Resampling

One of the key advantages of Zivid is the inherent 2D to 3D mapping. The point clouds are returned in an organized array, and the RGB values correspond 1-to-1 with their respective points.

However, the required resolution of the 3D data is not always the same as the required resolution of the 2D data. For example, one often requires higher resolution 2D than 3D. When you capture a separate 2D image and a 3D point cloud, the 2D image may have a higher resolution than the 3D point cloud. This can be done with the Sampling::Pixel settings, see Pixel and Pixel.

To maintain the 1-to-1 correspondence between the 2D and 3D data, the 3D data can be resampled to match the resolution of the 2D data. This can be done with Settings::Processing::Resampling.

Resample can refer to either upsampling or downsampling. Both will be discussed separately in the following two sections.

Upsampling

Typically used to match 2D resolution, if 3D resolution is lower than 2D resolution.

Examples

The following examples show how to capture 2D and 3D data with different resolutions while maintaining the 1-to-1 correspondence between the 2D and 3D data.

2x2 subsampled 3D and full resolution 2D

The following is captured with full-resolution 2D settings, and subsampled 3D settings. The 3D data is further upsampled to match the resolution of the 2D image.

auto settings2D = Zivid::Settings2D{
    Zivid::Settings2D::Acquisitions{ Zivid::Settings2D::Acquisition{} },
    Zivid::Settings2D::Sampling::Pixel::all,
};
auto settings = Zivid::Settings{
    Zivid::Settings::Engine::phase,
    Zivid::Settings::Acquisitions{ Zivid::Settings::Acquisition{} },
    Zivid::Settings::Sampling::Pixel::blueSubsample2x2,
    Zivid::Settings::Sampling::Color::disabled,
    Zivid::Settings::Processing::Resampling::Mode::upsample2x2,
};

settings_2d = zivid.Settings2D()
settings_2d.acquisitions.append(zivid.Settings2D.Acquisition())
settings_2d.sampling.pixel = zivid.Settings2D.Sampling.Pixel.all
settings = zivid.Settings()
settings.engine = "phase"
settings.acquisitions.append(zivid.Settings.Acquisition())
settings.sampling.pixel = zivid.Settings.Sampling.Pixel.blueSubsample2x2
settings.sampling.color = zivid.Settings.Sampling.Color.disabled
settings.processing.resampling.mode = zivid.Settings.Processing.Resampling.Mode.upsample2x2

Full Resolution 2D

Full resolution

3D Depth Image

Subsampled 2x2 then upsampled 2x2, depth image

3D Point Cloud

Subsampled 2x2 then upsampled 2x2, point cloud

(rotated viewpoint)

Zoomed in on region to better indicate 2D -> 3D correspondence
Full resolution
Subsampled 2x2 then upsampled 2x2, depth image
Subsampled 2x2 then upsampled 2x2, point cloud
Point cloud comparisons

Full Resolution

Full resolution

Subsampled2x2

Subsampled 2x2

Subsampled2x2 + Upsampled2x2

Subsampled 2x2 + Upsampled 2x2
Full resolution
Subsampled 2x2
Subsampled 2x2 then upsampled 2x2

Caution

Upsampling artifact

Full Resolution

Full resolution where there is an artifact in the upsampled point cloud

Subsampled2x2

Subsampled 2x2 where there is an artifact in the upsampled point cloud

Subsampled2x2 + Upsampled2x2

Upsample artifact visible

Notice the string of points between the bin bottom and the checkerboard. This is an artifact from upsampling the 3D data to match the resolution of the 2D data.

4x4 subsampled 3D and 2x2 subsampled 2D

The following is captured with 2x2 subsampled 2D settings. The 3D settings are 4x4 subsampled and then 2x2 upsampled to match the resolution of the 2D image.

Go to source

source

auto settings2D = Zivid::Settings2D{ Zivid::Settings2D::Acquisitions{ Zivid::Settings2D::Acquisition{} } };
auto settings = Zivid::Settings{ Zivid::Settings::Acquisitions{ Zivid::Settings::Acquisition{} } };

auto model = camera.info().model();
switch(model.value())
{
    case Zivid::CameraInfo::Model::ValueType::zividTwo:
    case Zivid::CameraInfo::Model::ValueType::zividTwoL100:
    {
        settings2D.set(Zivid::Settings2D::Sampling::Pixel::all);
        settings.set(Zivid::Settings::Sampling::Pixel::blueSubsample2x2);
        settings.set(Zivid::Settings::Processing::Resampling::Mode::upsample2x2);
        break;
    }
    case Zivid::CameraInfo::Model::ValueType::zivid2PlusM130:
    case Zivid::CameraInfo::Model::ValueType::zivid2PlusM60:
    case Zivid::CameraInfo::Model::ValueType::zivid2PlusL110:
    {
        settings2D.set(Zivid::Settings2D::Sampling::Pixel::blueSubsample2x2);
        settings.set(Zivid::Settings::Sampling::Pixel::blueSubsample4x4);
        settings.set(Zivid::Settings::Processing::Resampling::Mode::upsample2x2);
        break;
    }
    case Zivid::CameraInfo::Model::ValueType::zivid2PlusMR130:
    case Zivid::CameraInfo::Model::ValueType::zivid2PlusMR60:
    case Zivid::CameraInfo::Model::ValueType::zivid2PlusLR110:
    {
        settings2D.set(Zivid::Settings2D::Sampling::Pixel::by2x2);
        settings.set(Zivid::Settings::Sampling::Pixel::by4x4);
        settings.set(Zivid::Settings::Processing::Resampling::Mode::upsample2x2);
        break;
    }
    case Zivid::CameraInfo::Model::ValueType::zividOnePlusSmall:
    case Zivid::CameraInfo::Model::ValueType::zividOnePlusMedium:
    case Zivid::CameraInfo::Model::ValueType::zividOnePlusLarge:
    {
        throw std::runtime_error("Unsupported camera model '" + model.toString() + "'");
    }
    default: throw std::runtime_error("Unhandled enum value '" + model.toString() + "'");
}
Go to source

source

settings_2d = zivid.Settings2D(acquisitions=[zivid.Settings2D.Acquisition()])
settings = zivid.Settings(acquisitions=[zivid.Settings.Acquisition()])

model = camera.info.model
if model in [zivid.CameraInfo.Model.zividTwo, zivid.CameraInfo.Model.zividTwoL100]:
    settings_2d.sampling.pixel = zivid.Settings2D.Sampling.Pixel.all
    settings.sampling.pixel = zivid.Settings.Sampling.Pixel.blueSubsample2x2
    settings.processing.resampling.mode = zivid.Settings.Processing.Resampling.Mode.upsample2x2
elif model in [
    zivid.CameraInfo.Model.zivid2PlusM130,
    zivid.CameraInfo.Model.zivid2PlusM60,
    zivid.CameraInfo.Model.zivid2PlusL110,
]:
    settings_2d.sampling.pixel = zivid.Settings2D.Sampling.Pixel.blueSubsample2x2
    settings.sampling.pixel = zivid.Settings.Sampling.Pixel.blueSubsample4x4
    settings.processing.resampling.mode = zivid.Settings.Processing.Resampling.Mode.upsample2x2
elif model in [
    zivid.CameraInfo.Model.zivid2PlusMR130,
    zivid.CameraInfo.Model.zivid2PlusMR60,
    zivid.CameraInfo.Model.zivid2PlusLR110,
]:
    settings_2d.sampling.pixel = zivid.Settings2D.Sampling.Pixel.by2x2
    settings.sampling.pixel = zivid.Settings.Sampling.Pixel.by4x4
    settings.processing.resampling.mode = zivid.Settings.Processing.Resampling.Mode.upsample2x2
else:
    raise ValueError(f"Unsupported camera model '{model}'")

2x2 subsampled 2D

2x2 subsampled 2D

3D Depth Image

Subsampled 4x4 then upsampled 2x2, depth image

3D Point Cloud

Subsampled 4x4 then upsampled 2x2, point cloud

Downsampling

Typically used on a point cloud to get a lower resolution point cloud, with less noise. For more information see Downsample.

There are two ways to downsample a point cloud with Zivid SDK:

  1. Via the setting Settings::Processing::Resampling, which means it’s controlled via the capture settings.

  2. Via the API Zivid::PointCloud::downsample.

Downsample via Zivid::Settings::Processing::Resampling

When downsampling is invoked via settings the point cloud returned by frame.pointCloud() is already downsampled.

auto settings = Zivid::Settings{
    Zivid::Settings::Engine::phase,
    Zivid::Settings::Acquisitions{ Zivid::Settings::Acquisition{} },
    Zivid::Settings::Sampling::Pixel::all,
    Zivid::Settings::Processing::Resampling::Mode::downsample2x2,
};

const auto cameraModel = camera.info().model();
if(cameraModel == Zivid::CameraInfo::Model::zivid2PlusM130
   || cameraModel == Zivid::CameraInfo::Model::zivid2PlusM60
   || cameraModel == Zivid::CameraInfo::Model::zivid2PlusL110)
{
    // For 2+, we must lower Brightness from the default 2.5 to 2.2, when using `all` mode.
    // This code can be removed by changing the Config.yml option 'Camera/Power/Limit'.
    for(auto &a : settings.acquisitions())
    {
        a.set(Zivid::Settings::Acquisition::Brightness{ 2.2 });
    }
}

std::cout << "Capturing frame" << std::endl;
const auto frame = camera.capture(settings);
const auto pointCloud = frame.pointCloud();
std::cout << "Getting BGRA image" << std::endl;
const auto image = pointCloud.copyColorsRGBA();
const cv::Mat bgra(
    image.height(),
    image.width(),
    CV_8UC4, // NOLINTNEXTLINE(cppcoreguidelines-pro-type-const-cast)
    const_cast<void *>(static_cast<const void *>(image.data())));

std::cout << "Visualizing point cloud" << std::endl;
displayPointCloud(pointCloud, bgra);

settings = zivid.Settings()
settings.engine = "phase"
settings.acquisitions.append(zivid.Settings.Acquisition())
settings.sampling.pixel = zivid.Settings.Sampling.Pixel.all
settings.processing.resampling.mode = zivid.Settings.Processing.Resampling.Mode.downsample2x2

model = camera.info.model
if model in (
    zivid.CameraInfo.Model.zivid2PlusM130,
    zivid.CameraInfo.Model.zivid2PlusM60,
    zivid.CameraInfo.Model.zivid2PlusL110,
):
    # For 2+, we must lower Brightness from the default 2.5 to 2.2, when using `all` mode.
    # This code can be removed by changing the Config.yml option 'Camera/Power/Limit'.
    for acquisition in settings.acquisitions:
        acquisition.brightness = 2.2

print("Capturing frame")
with camera.capture(settings) as frame:
    point_cloud = frame.point_cloud()
    xyz = point_cloud.copy_data("xyz")
    rgba = point_cloud.copy_data("rgba")

    print("Visualizing point cloud")
    display_pointcloud(xyz, rgba)

Downsample via Zivid::PointCloud::downsample

The following is identical to the previous example, but the downsampling is invoked on the Zivid::PointCloud instance returned by frame.pointCloud().

auto settings = Zivid::Settings{
    Zivid::Settings::Engine::phase,
    Zivid::Settings::Acquisitions{ Zivid::Settings::Acquisition{} },
    Zivid::Settings::Sampling::Pixel{ Zivid::Settings::Sampling::Pixel::all },
};

const auto cameraModel = camera.info().model();
if(cameraModel == Zivid::CameraInfo::Model::zivid2PlusM130
   || cameraModel == Zivid::CameraInfo::Model::zivid2PlusM60
   || cameraModel == Zivid::CameraInfo::Model::zivid2PlusL110)
{
    // For 2+, we must lower Brightness from the default 2.5 to 2.2, when using `all` mode.
    // This code can be removed by changing the Config.yml option 'Camera/Power/Limit'.
    for(auto &a : settings.acquisitions())
    {
        a.set(Zivid::Settings::Acquisition::Brightness{ 2.2 });
    }
}

std::cout << "Capturing frame" << std::endl;
const auto frame = camera.capture(settings);
auto pointCloud = frame.pointCloud();
pointCloud.downsample(Zivid::PointCloud::Downsampling::by2x2);
std::cout << "Getting BGRA image" << std::endl;
const auto image = pointCloud.copyColorsRGBA();
const cv::Mat bgra(
    image.height(),
    image.width(),
    CV_8UC4, // NOLINTNEXTLINE(cppcoreguidelines-pro-type-const-cast)
    const_cast<void *>(static_cast<const void *>(image.data())));

std::cout << "Visualizing point cloud" << std::endl;
displayPointCloud(pointCloud, bgra);

settings = zivid.Settings()
settings.engine = "phase"
settings.acquisitions.append(zivid.Settings.Acquisition())
settings.sampling.pixel = zivid.Settings.Sampling.Pixel.all

model = camera.info.model
if model in (
    zivid.CameraInfo.Model.zivid2PlusM130,
    zivid.CameraInfo.Model.zivid2PlusM60,
    zivid.CameraInfo.Model.zivid2PlusL110,
):
    # For 2+, we must lower Brightness from the default 2.5 to 2.2, when using `all` mode.
    # This code can be removed by changing the Config.yml option 'Camera/Power/Limit'.
    for acquisition in settings.acquisitions:
        acquisition.brightness = 2.2

print("Capturing frame")
with camera.capture(settings) as frame:
    point_cloud = frame.point_cloud()
    point_cloud.downsample(zivid.PointCloud.Downsampling.by2x2)
    xyz = point_cloud.copy_data("xyz")
    rgba = point_cloud.copy_data("rgba")

    print("Visualizing point cloud")
    display_pointcloud(xyz, rgba)

Downsampling can be done in-place, which modifies the current point cloud.

Go to source

source

pointCloud.downsample(Zivid::PointCloud::Downsampling::by2x2);
Go to source

source

pointCloud.Downsample(Zivid.NET.PointCloud.Downsampling.By2x2);
Go to source

source

point_cloud.downsample(zivid.PointCloud.Downsampling.by2x2)

It is also possible to get the downsampled point cloud as a new point cloud instance, which does not alter the existing point cloud.

Go to source

source

auto downsampledPointCloud = pointCloud.downsampled(Zivid::PointCloud::Downsampling::by2x2);
Go to source

source

var downsampledPointCloud = pointCloud.Downsampled(Zivid.NET.PointCloud.Downsampling.By2x2);
Go to source

source

downsampled_point_cloud = point_cloud.downsampled(zivid.PointCloud.Downsampling.by2x2)

2D downsampling example

Full Resolution

Full resolution

Subsampled

Quarter resolution subsampled, zoomed

Downsampled

Quarter resolution downsampled, zoomed
Full resolution
Quarter resolution subsampled, zoomed again
Quarter resolution downsampled, zoomed again