Resampling

Zivid의 주요 장점 중 하나는 2D에서 3D로의 매핑이 가능하다는 것입니다. 포인트 클라우드는 organized array 형태로 반환되며, RGB 값은 해당 포인트와 일대일 대응됩니다.

However, the required resolution of the 3D data is not always the same as the required resolution of the 2D data. For example, one often requires higher resolution 2D than 3D. When you capture a separate 2D image and a 3D point cloud, the 2D image may have a higher resolution than the 3D point cloud. This can be done with the Settings::Sampling::Pixel settings, see Pixel and Pixel.

To maintain the 1-to-1 correspondence between the 2D and 3D data, the 3D data can be resampled to match the resolution of the 2D data. This can be done with Settings::Processing::Resampling.

리샘플링은 업샘플링과 다운샘플링을 모두 지칭할 수 있습니다. 두 가지 모두 다음 두 섹션에서 별도로 논의됩니다.

Upsampling

일반적으로 3D 해상도가 2D 해상도보다 낮은 경우 2D 해상도와 일치시키는 데 사용됩니다.

Examples

다음 예제에서는 2D 및 3D 데이터 간의 1:1 대응을 유지하면서 서로 다른 해상도로 2D 및 3D 데이터를 캡처하는 방법을 보여줍니다.

2x2 subsampled 3D and full resolution 2D

다음은 전체 해상도 2D 설정과 서브샘플링된 3D 설정으로 촬영되었습니다. 3D 데이터는 2D 이미지의 해상도에 맞춰 추가로 업샘플링됩니다.

auto settings2D = Zivid::Settings2D{
    Zivid::Settings2D::Acquisitions{ Zivid::Settings2D::Acquisition{} },
    Zivid::Settings2D::Sampling::Pixel::all,
};
auto settings = Zivid::Settings{
    Zivid::Settings::Engine::stripe,
    Zivid::Settings::Acquisitions{ Zivid::Settings::Acquisition{} },
    Zivid::Settings::Sampling::Pixel::blueSubsample2x2,
    Zivid::Settings::Sampling::Color::disabled,
    Zivid::Settings::Processing::Resampling::Mode::upsample2x2,
};

settings_2d = zivid.Settings2D()
settings_2d.acquisitions.append(zivid.Settings2D.Acquisition())
settings_2d.sampling.pixel = zivid.Settings2D.Sampling.Pixel.all
settings = zivid.Settings()
settings.engine = "stripe"
settings.acquisitions.append(zivid.Settings.Acquisition())
settings.sampling.pixel = zivid.Settings.Sampling.Pixel.blueSubsample2x2
settings.sampling.color = zivid.Settings.Sampling.Color.disabled
settings.processing.resampling.mode = zivid.Settings.Processing.Resampling.Mode.upsample2x2

Full Resolution 2D

Full resolution

3D Depth Image

Subsampled 2x2 then upsampled 2x2, depth image

3D Point Cloud

Subsampled 2x2 then upsampled 2x2, point cloud

(rotated viewpoint)

2D -> 3D 대응을 더 잘 나타내기 위해 해당 지역을 확대했습니다.
Full resolution
Subsampled 2x2 then upsampled 2x2, depth image
Subsampled 2x2 then upsampled 2x2, point cloud
Point cloud comparisons

Full resolution

Full resolution

Subsampled2x2

Subsampled 2x2

Subsampled2x2 + Upsampled2x2

Subsampled 2x2 + Upsampled 2x2
Full resolution
Subsampled 2x2
Subsampled 2x2 then upsampled 2x2

조심

Upsampling artifact

Full resolution

Full resolution where there is an artifact in the upsampled point cloud

Subsampled2x2

Subsampled 2x2 where there is an artifact in the upsampled point cloud

Subsampled2x2 + Upsampled2x2

업샘플링 아티팩트가 보입니다

상자 바닥과 체커보드 사이에 있는 점들을 주목하세요. 이는 2D 데이터의 해상도에 맞추기 위해 3D 데이터를 업샘플링하는 과정에서 발생한 아티팩트입니다.

4x4 subsampled 3D and 2x2 subsampled 2D

다음은 2x2 서브샘플링된 2D 설정으로 캡처한 이미지입니다. 3D 설정은 4x4 서브샘플링된 후, 2D 이미지의 해상도에 맞춰 2x2 업샘플링됩니다.

소스로 이동

source

auto settings = Zivid::Settings{ Zivid::Settings::Acquisitions{ Zivid::Settings::Acquisition{} },
                                 Zivid::Settings::Color{ Zivid::Settings2D{
                                     Zivid::Settings2D::Acquisitions{ Zivid::Settings2D::Acquisition{} } } } };

auto model = camera.info().model();
switch(model.value())
{
    case Zivid::CameraInfo::Model::ValueType::zividTwo:
    case Zivid::CameraInfo::Model::ValueType::zividTwoL100:
    {
        settings.set(Zivid::Settings::Sampling::Pixel::blueSubsample2x2);
        settings.set(Zivid::Settings::Processing::Resampling::Mode::upsample2x2);
        settings.color().value().set(Zivid::Settings2D::Sampling::Pixel::all);
        break;
    }
    case Zivid::CameraInfo::Model::ValueType::zivid2PlusM130:
    case Zivid::CameraInfo::Model::ValueType::zivid2PlusM60:
    case Zivid::CameraInfo::Model::ValueType::zivid2PlusL110:
    {
        settings.set(Zivid::Settings::Sampling::Pixel::blueSubsample4x4);
        settings.set(Zivid::Settings::Processing::Resampling::Mode::upsample2x2);
        settings.color().value().set(Zivid::Settings2D::Sampling::Pixel::blueSubsample2x2);
        break;
    }
    case Zivid::CameraInfo::Model::ValueType::zivid2PlusMR130:
    case Zivid::CameraInfo::Model::ValueType::zivid2PlusMR60:
    case Zivid::CameraInfo::Model::ValueType::zivid2PlusLR110:
    case Zivid::CameraInfo::Model::ValueType::zivid3XL250:
    {
        settings.set(Zivid::Settings::Sampling::Pixel::by4x4);
        settings.set(Zivid::Settings::Processing::Resampling::Mode::upsample2x2);
        settings.color().value().set(Zivid::Settings2D::Sampling::Pixel::by2x2);
        break;
    }
    case Zivid::CameraInfo::Model::ValueType::zividOnePlusSmall:
    case Zivid::CameraInfo::Model::ValueType::zividOnePlusMedium:
    case Zivid::CameraInfo::Model::ValueType::zividOnePlusLarge:
    {
        throw std::runtime_error("Unsupported camera model '" + model.toString() + "'");
    }
    default: throw std::runtime_error("Unhandled enum value '" + model.toString() + "'");
}
소스로 이동

source

settings = zivid.Settings(
    acquisitions=[zivid.Settings.Acquisition()],
    color=zivid.Settings2D(acquisitions=[zivid.Settings2D.Acquisition()]),
)

model = camera.info.model
if model in {zivid.CameraInfo.Model.zividTwo, zivid.CameraInfo.Model.zividTwoL100}:
    settings.sampling.pixel = zivid.Settings.Sampling.Pixel.blueSubsample2x2
    settings.processing.resampling.mode = zivid.Settings.Processing.Resampling.Mode.upsample2x2
    settings.color.sampling.pixel = zivid.Settings2D.Sampling.Pixel.all
elif model in {
    zivid.CameraInfo.Model.zivid2PlusM130,
    zivid.CameraInfo.Model.zivid2PlusM60,
    zivid.CameraInfo.Model.zivid2PlusL110,
}:
    settings.sampling.pixel = zivid.Settings.Sampling.Pixel.blueSubsample4x4
    settings.processing.resampling.mode = zivid.Settings.Processing.Resampling.Mode.upsample2x2
    settings.color.sampling.pixel = zivid.Settings2D.Sampling.Pixel.blueSubsample2x2
elif model in {
    zivid.CameraInfo.Model.zivid2PlusMR130,
    zivid.CameraInfo.Model.zivid2PlusMR60,
    zivid.CameraInfo.Model.zivid2PlusLR110,
    zivid.CameraInfo.Model.zivid3XL250,
}:
    settings.sampling.pixel = zivid.Settings.Sampling.Pixel.by4x4
    settings.processing.resampling.mode = zivid.Settings.Processing.Resampling.Mode.upsample2x2
    settings.color.sampling.pixel = zivid.Settings2D.Sampling.Pixel.by2x2
else:
    raise ValueError(f"Unsupported camera model '{model}'")

2x2 subsampled 2D

2x2 subsampled 2D

3D Depth Image

Subsampled 4x4 then upsampled 2x2, depth image

3D Point Cloud

Subsampled 4x4 then upsampled 2x2, point cloud

Downsampling

일반적으로 포인트 클라우드에서 노이즈가 적은 저해상도 포인트 클라우드를 얻는 데 사용됩니다. 자세한 내용은 Downsample 을 참조하세요.

Zivid SDK를 사용하여 포인트 클라우드를 다운샘플링하는 방법에는 두 가지가 있습니다.

  1. Via the setting Settings::Processing::Resampling, which means it’s controlled via the capture settings.

  2. Via the API PointCloud::downsample.

Zivid::Settings::Processing::Resampling 을 통해 다운샘플링합니다.

설정을 통해 다운샘플링이 호출되면 frame.pointCloud() 에서 반환된 포인트 클라우드는 이미 다운샘플링됩니다.

auto settings = Zivid::Settings{
    Zivid::Settings::Engine::stripe,
    Zivid::Settings::Acquisitions{ Zivid::Settings::Acquisition{} },
    Zivid::Settings::Sampling::Pixel::all,
    Zivid::Settings::Processing::Resampling::Mode::downsample2x2,
};

const auto cameraModel = camera.info().model();
if(cameraModel == Zivid::CameraInfo::Model::zivid2PlusM130
   || cameraModel == Zivid::CameraInfo::Model::zivid2PlusM60
   || cameraModel == Zivid::CameraInfo::Model::zivid2PlusL110)
{
    // For 2+, we must lower Brightness from the default 2.5 to 2.2, when using `all` mode.
    // This code can be removed by changing the Config.yml option 'Camera/Power/Limit'.
    for(auto &a : settings.acquisitions())
    {
        a.set(Zivid::Settings::Acquisition::Brightness{ 2.2 });
    }
}

std::cout << "Capturing frame" << std::endl;
const auto frame = camera.capture(settings);
const auto pointCloud = frame.pointCloud();
std::cout << "Getting BGRA image" << std::endl;
const auto image = pointCloud.copyColorsRGBA_SRGB();
const cv::Mat bgra(
    image.height(),
    image.width(),
    CV_8UC4, // NOLINTNEXTLINE(cppcoreguidelines-pro-type-const-cast)
    const_cast<void *>(static_cast<const void *>(image.data())));

std::cout << "Visualizing point cloud" << std::endl;
displayPointCloud(pointCloud, bgra);

settings = zivid.Settings()
settings.engine = "stripe"
settings.acquisitions.append(zivid.Settings.Acquisition())
settings.sampling.pixel = zivid.Settings.Sampling.Pixel.all
settings.processing.resampling.mode = zivid.Settings.Processing.Resampling.Mode.downsample2x2

model = camera.info.model
if model in (
    zivid.CameraInfo.Model.zivid2PlusM130,
    zivid.CameraInfo.Model.zivid2PlusM60,
    zivid.CameraInfo.Model.zivid2PlusL110,
):
    # For 2+, we must lower Brightness from the default 2.5 to 2.2, when using `all` mode.
    # This code can be removed by changing the Config.yml option 'Camera/Power/Limit'.
    for acquisition in settings.acquisitions:
        acquisition.brightness = 2.2

print("Capturing frame")
with camera.capture(settings) as frame:
    point_cloud = frame.point_cloud()
    xyz = point_cloud.copy_data("xyz")
    rgba = point_cloud.copy_data("rgba_srgb")

    print("Visualizing point cloud")
    display_pointcloud(xyz, rgba)

Zivid::PointCloud::downsample 을 통해 다운샘플링합니다.

The following is identical to the previous example, but the downsampling is invoked on the PointCloud instance returned by frame.pointCloud().

auto settings = Zivid::Settings{
    Zivid::Settings::Engine::stripe,
    Zivid::Settings::Acquisitions{ Zivid::Settings::Acquisition{} },
    Zivid::Settings::Sampling::Pixel{ Zivid::Settings::Sampling::Pixel::all },
};

const auto cameraModel = camera.info().model();
if(cameraModel == Zivid::CameraInfo::Model::zivid2PlusM130
   || cameraModel == Zivid::CameraInfo::Model::zivid2PlusM60
   || cameraModel == Zivid::CameraInfo::Model::zivid2PlusL110)
{
    // For 2+, we must lower Brightness from the default 2.5 to 2.2, when using `all` mode.
    // This code can be removed by changing the Config.yml option 'Camera/Power/Limit'.
    for(auto &a : settings.acquisitions())
    {
        a.set(Zivid::Settings::Acquisition::Brightness{ 2.2 });
    }
}

std::cout << "Capturing frame" << std::endl;
const auto frame = camera.capture(settings);
auto pointCloud = frame.pointCloud();
pointCloud.downsample(Zivid::PointCloud::Downsampling::by2x2);
std::cout << "Getting BGRA image" << std::endl;
const auto image = pointCloud.copyColorsRGBA_SRGB();
const cv::Mat bgra(
    image.height(),
    image.width(),
    CV_8UC4, // NOLINTNEXTLINE(cppcoreguidelines-pro-type-const-cast)
    const_cast<void *>(static_cast<const void *>(image.data())));

std::cout << "Visualizing point cloud" << std::endl;
displayPointCloud(pointCloud, bgra);

settings = zivid.Settings()
settings.engine = "stripe"
settings.acquisitions.append(zivid.Settings.Acquisition())
settings.sampling.pixel = zivid.Settings.Sampling.Pixel.all

model = camera.info.model
if model in (
    zivid.CameraInfo.Model.zivid2PlusM130,
    zivid.CameraInfo.Model.zivid2PlusM60,
    zivid.CameraInfo.Model.zivid2PlusL110,
):
    # For 2+, we must lower Brightness from the default 2.5 to 2.2, when using `all` mode.
    # This code can be removed by changing the Config.yml option 'Camera/Power/Limit'.
    for acquisition in settings.acquisitions:
        acquisition.brightness = 2.2

print("Capturing frame")
with camera.capture(settings) as frame:
    point_cloud = frame.point_cloud()
    point_cloud.downsample(zivid.PointCloud.Downsampling.by2x2)
    xyz = point_cloud.copy_data("xyz")
    rgba = point_cloud.copy_data("rgba_srgb")

    print("Visualizing point cloud")
    display_pointcloud(xyz, rgba)

다운샘플링은 현재 포인트 클라우드를 수정하는 방식으로 그 자리에서 수행할 수 있습니다.

소스로 이동

source

pointCloud.downsample(Zivid::PointCloud::Downsampling::by2x2);
소스로 이동

source

pointCloud.Downsample(Zivid.NET.PointCloud.Downsampling.By2x2);
소스로 이동

source

point_cloud.downsample(zivid.PointCloud.Downsampling.by2x2)

기존 포인트 클라우드를 변경하지 않고 다운샘플링된 포인트 클라우드를 새로운 포인트 클라우드 인스턴스로 가져오는 것도 가능합니다.

소스로 이동

source

auto downsampledPointCloud = pointCloud.downsampled(Zivid::PointCloud::Downsampling::by2x2);
소스로 이동

source

var downsampledPointCloud = pointCloud.Downsampled(Zivid.NET.PointCloud.Downsampling.By2x2);
소스로 이동

source

downsampled_point_cloud = point_cloud.downsampled(zivid.PointCloud.Downsampling.by2x2)

2D downsampling example

Full resolution

Full resolution

Subsampled

Quarter resolution subsampled, zoomed

Downsampled

Quarter resolution downsampled, zoomed
Full resolution
Quarter resolution subsampled, zoomed again
Quarter resolution downsampled, zoomed again