Point Cloud Tutorial

Introduction

This tutorial describes how to use Zivid SDK to work with Point Cloud data.

Tip

If you prefer watching a video, our webinar Getting your point cloud ready for your application covers the Point Cloud Tutorial.

Prerequisites

• Install Zivid Software.

• For Python: install zivid-python

Frame

C++

The Frame contains the point cloud and color image (stored on compute device memory) and the capture and camera information.

C#

The Frame contains the point cloud and color image (stored on compute device memory) and the capture and camera information.

Python

The zivid.Frame contains the point cloud and color image (stored on compute device memory) and the capture and camera information.

Capture

When you capture with Zivid, you get a frame in return. The point cloud is stored in the frame, and the frame is stored in the GPU memory. The capture can contain color or not, depending of the method that you call. For more information see this table with different capture modes.

Capture with color

If you want to capture a point cloud with color, you can use the Camera::capture2D3D() method.

C++

source

const auto frame = camera.capture2D3D(settings);

C#

source

using (var frame = camera.Capture2D3D(settings))

Python

source

frame = camera.capture_2d_3d(settings)

Capture without color

If you want to capture a point cloud without color, you can use the Camera::capture3D() method.

C++

source

const auto frame3D = camera.capture3D(settings);

C#

source

using (var frame3D = camera.Capture3D(settings))

Python

source

frame_3d = camera.capture_3d(settings)

Check Capture Tutorial for detailed instructions on how to capture.

Load

The frame can also be loaded from a ZDF file.

C++

source

const auto dataFile = std::string(ZIVID_SAMPLE_DATA_DIR) + "/Zivid3D.zdf";
std::cout << "Reading ZDF frame from file: " << dataFile << std::endl;
const auto frame = Zivid::Frame(dataFile);

C#

source

var dataFile =
    Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData) + "/Zivid/Zivid3D.zdf";
Console.WriteLine("Reading ZDF frame from file: " + dataFile);

using (var frame = new Zivid.NET.Frame(dataFile))
{

Python

source

data_file = get_sample_data_path() / "Zivid3D.zdf"
print(f"Reading point cloud from file: {data_file}")

frame = zivid.Frame(data_file)

Point Cloud

Get handle from Frame

You can now get a handle to the point cloud data on the GPU.

C++

source

const auto pointCloud = frame.pointCloud();

C#

source

var pointCloud = frame.PointCloud;

Python

source

point_cloud = frame.point_cloud()

Point cloud contains XYZ, RGB, and SNR, laid out on a 2D grid.

For more info check out Point Cloud Structure.

C++

The method Frame::pointCloud() does not perform any copying from GPU memory.

Note

Camera::capture2D3D() and Camera::capture3D() methods return at some moment in time after the camera completes capturing raw images. The handle from Frame::pointCloud() is available instantly. However, the actual point cloud data becomes available only after the processing on the GPU is finished. Any calls to data-copy functions (section below) will block and wait for processing to finish before proceeding with the requested copy operation.

For detailed explanation, see Point Cloud Capture Process.

C#

Getting the property Frame.PointCloud does not perform any copying from GPU memory.

Note

Camera.Capture2D3D() and Camera.Capture3D() methods return at some moment in time after the camera completes capturing raw images. The handle from Frame.PointCloud is available instantly. However, the actual point cloud data becomes available only after the processing on the GPU is finished. Any calls to data-copy methods (section below) will block and wait for processing to finish before proceeding with the requested copy operation.

For detailed explanation, see Point Cloud Capture Process.

Python

The function zivid.frame.point_cloud() does not perform any copying from GPU memory.

Note

zivid.camera.capture_2d_3d() and zivid.camera.capture_3d() methods return at some moment in time after the camera completes capturing raw images. The handle from zivid.frame.point_cloud() is available instantly. However, the actual point cloud data becomes available only after the processing on the GPU is finished. Any calls to data-copy functions (section below) will block and wait for processing to finish before proceeding with the requested copy operation.

For detailed explanation, see Point Cloud Capture Process.

Unorganized point cloud

It is possible to convert the organized point cloud to an unorganized point cloud. While doing so, all NaN values are removed, and the point cloud is flattened to a 1D array.

C++

source

const auto unorganizedPointCloud = frame.pointCloud().toUnorganizedPointCloud();

Python

source

unorganized_point_cloud = frame.point_cloud().to_unorganized_point_cloud()

Combining multiple unorganized point clouds

The unorganized point cloud can be extended with additional unorganized point clouds.

C++

source

stitchedPointCloud.extend(currentPointCloud.transform(transformationMatrixZivid));

Python

source

stitched_point_cloud.extend(current_point_cloud.transform(transformation_matrix))

Copy from GPU to CPU memory

You can now selectively copy data based on what is required. This is the complete list of output data formats and how to copy them from the GPU.

Tip

If you want to process the data directly on the GPU without copying to the CPU, see GPU Access Tutorial.

Most of these APIs also applies to the unorganized point cloud.

C++

Return type	Copy functions	Data per pixel	Total data
Zivid::Array2D<Zivid::PointXYZ>	PointCloud::copyPointsXYZ() or PointCloud::copyData<Zivid::PointXYZ>()	12 bytes	28 MB
Zivid::Array2D<Zivid::PointXYZW>	PointCloud::copyPointsXYZW() or PointCloud::copyData<Zivid::PointXYZW>()	16 bytes	37 MB
Zivid::Array2D<Zivid::PointZ>	PointCloud::copyPointsZ() or PointCloud::copyData<Zivid::PointZ>()	4 bytes	9 MB
Zivid::Array2D<Zivid::ColorRGBA>	PointCloud::copyColorsRGBA() or PointCloud::copyData<Zivid::ColorRGBA>()	4 bytes	9 MB
Zivid::Array2D<Zivid::SNR>	PointCloud::copySNRs() or PointCloud::copyData<Zivid::SNR>()	4 bytes	9 MB
Zivid::Array2D<Zivid::PointXYZColorRGBA>	PointCloud::copyData<PointXYZColorRGBA>()	16 bytes	37 MB
Zivid::Array2D<Zivid::PointXYZColorBGRA>	PointCloud::copyPointsXYZColorsBGRA() or PointCloud::copyData<PointXYZColorBGRA>()	16 bytes	37 MB
Zivid::Image<Zivid::ColorRGBA>	PointCloud::copyImageRGBA()	4 bytes	9 MB
Zivid::Image<Zivid::ColorBGRA>	PointCloud::copyImageBGRA()	4 bytes	9 MB
Zivid::Image<Zivid::ColorsRGB>	PointCloud::copyImagesRGB()	4 bytes	9 MB

C#

Return type	Copy methods	Data per pixel	Total data
float[height,width,3]	PointCloud.CopyPointsXYZ()	12 bytes	28 MB
float[height,width,4]	PointCloud.CopyPointsXYZW()	16 bytes	37 MB
float[height,width,1]	PointCloud.CopyPointsZ()	4 bytes	9 MB
byte[height,width,4]	PointCloud.CopyColorsRGBA()	4 bytes	9 MB
float[height,width]	PointCloud.CopySNRs()	4 bytes	9 MB
Zivid.NET.PointXYZColorRGBA[height, width]	PointCloud.CopyPointsXYZColorsRGBA()	16 bytes	37 MB
Zivid.NET.PointXYZColorBGRA[height, width]	PointCloud.CopyPointsXYZColorsBGRA()	16 bytes	37 MB
Zivid.NET.ImageRGBA	PointCloud.CopyImageRGBA()	4 bytes	9 MB
Zivid.NET.ImageBGRA	PointCloud.CopyImageBGRA()	4 bytes	9 MB
Zivid.NET.ImageSRGB	PointCloud.CopyImageSRGB()	4 bytes	9 MB

Python

Return type	Copy functions	Data per pixel	Total data
numpy.ndarray([height,width,3], dtype=float32)	PointCloud.copy_data("xyz")	12 bytes	28 MB
numpy.ndarray([height,width,3], dtype=float32)	PointCloud.copy_data("xyzw")	16 bytes	37 MB
numpy.ndarray([height,width], dtype=float32)	PointCloud.copy_data("z")	4 bytes	9 MB
numpy.ndarray([height,width,4], dtype=uint8)	PointCloud.copy_data("rgba")	4 bytes	9 MB
numpy.ndarray([height,width,4], dtype=uint8)	PointCloud.copy_data("bgra")	4 bytes	9 MB
numpy.ndarray([height,width,4], dtype=uint8)	PointCloud.copy_data("srgb")	4 bytes	9 MB
numpy.ndarray([height,width], dtype=float32)	PointCloud.copy_data("snr")	4 bytes	9 MB
numpy.ndarray([height,width], dtype=[('x', '<f4'), ('y', '<f4'), ('z', '<f4'), ('r', 'u1'), ('g', 'u1'), ('b', 'u1'), ('a', 'u1')])	PointCloud.copy_data("xyzrgba")	16 bytes	37 MB

Here is an example of how to copy data.

C++

source

const auto data = pointCloud.copyData<Zivid::PointXYZColorRGBA_SRGB>();

C#

source

var pointCloudData = pointCloud.CopyPointsXYZColorsRGBA_SRGB();

Python

source

xyz = point_cloud.copy_data("xyz")
rgba = point_cloud.copy_data("rgba_srgb")

Memory allocation options

In terms of memory allocation, there are two ways to copy data:

• The Zivid SDK can allocate a memory buffer and copy data to it.

• A user can pass a pointer to a pre-allocated memory buffer, and the Zivid SDK will copy the data to the pre-allocated memory buffer.

C++

We present examples for the two memory allocation options using OpenCV.

Copy selected data from GPU to CPU memory (Zivid-allocated)

If you are only concerned about e.g. RGB color data of the point cloud, you can copy only that data to the CPU memory.

source

std::cout << "Capturing frame" << std::endl;
frame = camera.capture2D3D(settings);

const auto frame2D = frame.frame2D();
if(!frame2D.has_value())
{
    throw std::runtime_error("Captured frame does not contain a 2D image.");
}
std::cout << "Copying colors with Zivid API from GPU to CPU" << std::endl;
auto colors = frame2D->imageBGRA_SRGB();

std::cout << "Casting the data pointer as a void*, since this is what the OpenCV matrix constructor requires."
          << std::endl;
auto *dataPtrZividAllocated = const_cast<void *>(static_cast<const void *>(colors.data()));

std::cout << "Wrapping this block of data in an OpenCV matrix. This is possible since the layout of \n"
          << "Zivid::ColorBGRA_SRGB exactly matches the layout of CV_8UC4. No copying occurs in this step."
          << std::endl;
const cv::Mat bgraZividAllocated(colors.height(), colors.width(), CV_8UC4, dataPtrZividAllocated);

std::cout << "Displaying image" << std::endl;
cv::imshow("BGRA image Zivid Allocated", bgraZividAllocated);
cv::waitKey(CI_WAITKEY_TIMEOUT_IN_MS);

Copy selected data from GPU to CPU memory (user-allocated)

In the above example, ownership of the data was held by the returned Zivid::Array2D<> objects. Alternatively, you may provide a pre-allocated memory buffer to Zivid::PointCloud::copyData(dataPtr). The type of dataPtr defines what shall be copied (PointXYZ, ColorRGBA, etc.).

Now let us look at the exact same use case as above. However, this time, we allow OpenCV to allocate the necessary storage. Then we ask the Zivid API to copy data directly from the GPU into this memory location.

source

std::cout << "Allocating the necessary storage with OpenCV API based on resolution info before any capturing"
          << std::endl;
auto bgraUserAllocated = cv::Mat(resolution.height(), resolution.width(), CV_8UC4);

std::cout << "Capturing frame" << std::endl;
auto frame = camera.capture2D3D(settings);
auto pointCloud = frame.pointCloud();

std::cout << "Copying data with Zivid API from the GPU into the memory location allocated by OpenCV"
          << std::endl;
pointCloud.copyData(&(*bgraUserAllocated.begin<Zivid::ColorBGRA_SRGB>()));

std::cout << "Displaying image" << std::endl;
cv::imshow("BGRA image User Allocated", bgraUserAllocated);
cv::waitKey(CI_WAITKEY_TIMEOUT_IN_MS);

See also

Copy Speed benchmarks for the time it takes to copy each data type from GPU to system memory.

Transform

You may want to transform the point cloud to change its origin from the camera to the robot base frame or, e.g., scale the point cloud by transforming it from mm to m.

C++

source

pointCloud.transform(baseToCameraTransform);

C#

source

pointCloud.Transform(transformBaseToCamera);

Python

source

point_cloud.transform(base_to_camera_transform)

Transformation can be done in-place:

• PointCloud::transform()

• UnorganizedPointCloud::transform()

or by creating a new instance:

• PointCloud::transformed()

• UnorganizedPointCloud::transformed()

The following example shows how create a new instance of UnorganizedPointCloud with a transformation applied to it. Note that in this sample is is not necessary to create a new instance, as the untransformed point cloud is not used after the transformation.

C++

source

const auto transformedUnorganizedPointCloud = unorganizedPointCloud.transformed(transformationMatrix);

Python

source

transformed_unorganized_point_cloud = unorganized_point_cloud.transformed(transformation_matrix)

Even the in-place API returns the transformed point cloud, so you can use it directly, as in the example below.

C++

source

stitchedPointCloud.extend(currentPointCloud.transform(transformationMatrixZivid));

Python

source

stitched_point_cloud.extend(current_point_cloud.transform(transformation_matrix))

If the object of interest is a subset of the ROI-cropped point cloud, you can further refine it by applying a binary mask. A typical workflow is to capture a 2D image, run a segmentation algorithm to identify the object, and apply the resulting mask to the 3D point cloud. This invalidates all points outside the segmented region.

See the Mask section of the Point Cloud Tutorial for API usage and code examples.

Mask

You can mask the point cloud to keep only a specific region of interest. A typical use case is to capture a 2D image, run it through a segmentation algorithm to produce a binary mask, and apply it to the 3D point cloud. Non-zero values in the mask invalidate the corresponding points (set to NaN), while zero values preserve them. The mask can be applied using mask() (in-place) or masked() (returns a new copy), on both PointCloud and Frame.

C++

source

// Create a ones-filled mask
Zivid::Mask mask(resolution);

// Calculate rectangle bounds
const int height = static_cast<int>(resolution.height());
const int width = static_cast<int>(resolution.width());
const int heightMin = (height - pixelsToDisplay) / 2;
const int heightMax = (height + pixelsToDisplay) / 2;
const int widthMin = (width - pixelsToDisplay) / 2;
const int widthMax = (width + pixelsToDisplay) / 2;

// Create OpenCV Mat wrapper for the mask data
cv::Mat maskMat(height, width, CV_8UC1, mask.data());

// Draw filled rectangle on the mask to unmask the central region
cv::rectangle(
    maskMat, cv::Point(widthMin, heightMin), cv::Point(widthMax, heightMax), cv::Scalar(0), cv::FILLED);

return mask;
auto maskedPointCloud = pointCloud.masked(mask);

C#

source

// Create a ones-filled mask
var mask = new Zivid.NET.Mask(resolution);

// Calculate rectangle bounds
int height = (int)resolution.Height;
int width = (int)resolution.Width;
int heightMin = (height - pixelsToDisplay) / 2;
int heightMax = (height + pixelsToDisplay) / 2;
int widthMin = (width - pixelsToDisplay) / 2;
int widthMax = (width + pixelsToDisplay) / 2;

// Set pixels inside the rectangle to zero
for (int y = heightMin; y < heightMax; ++y)
{
    for (int x = widthMin; x < widthMax; ++x)
    {
        mask[x, y] = 0;
    }
}

return mask;
    var maskedPointCloud = pointCloud.Masked(mask);

Python

source

pixels_to_display = 300
print(f"Generating binary mask of central {pixels_to_display} x {pixels_to_display} pixels")
height = frame.point_cloud().height
width = frame.point_cloud().width
mask = np.ones((height, width), bool)

h_min = (height - pixels_to_display) // 2
h_max = (height + pixels_to_display) // 2
w_min = (width - pixels_to_display) // 2
w_max = (width + pixels_to_display) // 2
mask[h_min:h_max, w_min:w_max] = 0
print("Masking point cloud")
point_cloud.mask(mask)

A mask can also be constructed from third-party data, such as an OpenCV image.

C++

source

// Create a circular mask in OpenCV
const int centerX = static_cast<int>(resolution.width()) / 2;
const int centerY = static_cast<int>(resolution.height()) / 2;
const int radius = pixelsToDisplay;
cv::circle(opencvMask, cv::Point(centerX, centerY), radius, cv::Scalar(0), cv::FILLED);

// Convert OpenCV mask to Zivid::Mask
const Zivid::Mask circularMask(resolution, opencvMask.datastart, opencvMask.dataend);

Python

source

opencv_mask = np.ones((height, width), dtype=np.uint8)
center_x = width // 2
center_y = height // 2
radius = pixels_to_display
cv2.circle(opencv_mask, (center_x, center_y), radius, 0, thickness=cv2.FILLED)

point_cloud.mask(opencv_mask.astype(bool))

Note

The mask supports implicit resampling: if the mask resolution differs from the point cloud resolution by an integer factor (2, 3, or 4), it is automatically resampled. This is useful when creating a mask from a 2D capture with a different resolution than the 3D capture.

Downsample

Sometimes you might not need a point cloud with as high spatial resolution as given from the camera. You may then downsample the point cloud.

Note

Sampling (3D) describes a hardware-based sub-/downsample method that reduces the resolution of the point cloud during capture while also reducing the acquisition and capture time.

Note

UnorganizedPointCloud does not support downsampling, but it does support voxel downsampling, see Voxel downsample.

Downsampling can be done in-place, which modifies the current point cloud.

C++

source

pointCloud.downsample(Zivid::PointCloud::Downsampling::by2x2);

C#

source

pointCloud.Downsample(Zivid.NET.PointCloud.Downsampling.By2x2);

Python

source

point_cloud.downsample(zivid.PointCloud.Downsampling.by2x2)

It is also possible to get the downsampled point cloud as a new point cloud instance, which does not alter the existing point cloud.

C++

source

auto downsampledPointCloud = pointCloud.downsampled(Zivid::PointCloud::Downsampling::by2x2);

C#

source

var downsampledPointCloud = pointCloud.Downsampled(Zivid.NET.PointCloud.Downsampling.By2x2);

Python

source

downsampled_point_cloud = point_cloud.downsampled(zivid.PointCloud.Downsampling.by2x2)

Zivid SDK supports the following downsampling rates: by2x2, by3x3, and by4x4, with the possibility to perform downsampling multiple times.

Voxel downsample

UnorganizedPointCloud supports voxel downsampling. The API takes two arguments:

1. voxelSize - the size of the voxel in millimeters.

2. minPointsPerVoxel - the minimum number of points per voxel to keep it.

Voxel downsampling subdivides 3D space into a grid of cubic voxels with a given size. If a given voxel contains a number of points at or above the given limit, all those source points are replaced with a single point with the following properties:

• Position (XYZ) is an SNR-weighted average of the source points’ positions, i.e. a high-confidence source point will have a greater influence on the resulting position than a low-confidence one.

• Color (RGBA) is the average of the source points’ colors.

• Signal-to-noise ratio (SNR) is sqrt(sum(SNR^2)) of the source points’ SNR values, i.e. the SNR of a new point will increase with both the number and the confidence of the source points that were used to compute its position.

Using minPointsPerVoxel > 1 is particularly useful for removing noise and artifacts from unorganized point clouds that are a combination of point clouds captured from different angles. This is because a given artifact is most likely only present in one of the captures, and minPointsPerVoxel can be used to only fill voxels that both captures “agree” on.

C++

source

const auto finalPointCloud = stitchedPointCloud.voxelDownsampled(0.5, 1);

Python

source

final_point_cloud = stitched_point_cloud.voxel_downsampled(0.5, 1)

Normals

Some applications require computing normals from the point cloud.

C++

source

std::cout << "Computing normals and copying them to CPU memory" << std::endl;
const auto normals = pointCloud.copyData<Zivid::NormalXYZ>();

C#

source

Console.WriteLine("Computing normals and copying them to CPU memory");
var normals = pointCloud.CopyNormalsXYZ();

Python

source

print("Computing normals and copying them to CPU memory")
normals = point_cloud.copy_data("normals")

The Normals API computes the normal at each point in the point cloud and copies normals from the GPU memory to the CPU memory. The result is a matrix of normal vectors, one for each point in the input point cloud. The size of normals is equal to the size of the input point cloud.

Visualize

Having the frame allows you to visualize the point cloud.

C++

source

std::cout << "Setting up visualization" << std::endl;
Zivid::Visualization::Visualizer visualizer;

std::cout << "Visualizing point cloud" << std::endl;
visualizer.showMaximized();
visualizer.show(frame);
visualizer.resetToFit();

std::cout << "Running visualizer. Blocking until window closes." << std::endl;
visualizer.run();

C#

source

Console.WriteLine("Setting up visualization");
using (var visualizer = new Zivid.NET.Visualization.Visualizer())
{
    Console.WriteLine("Visualizing point cloud");
    visualizer.Show(frame);
    visualizer.ShowMaximized();
    visualizer.ResetToFit();

    Console.WriteLine("Running visualizer. Blocking until window closes.");
    visualizer.Run();
}

You can visualize the point cloud from the point cloud object as well.

C++

source

std::cout << "Getting point cloud from frame" << std::endl;
auto pointCloud = frame.pointCloud();

std::cout << "Setting up visualization" << std::endl;
Zivid::Visualization::Visualizer visualizer;

std::cout << "Visualizing point cloud" << std::endl;
visualizer.showMaximized();
visualizer.show(pointCloud);
visualizer.resetToFit();

std::cout << "Running visualizer. Blocking until window closes." << std::endl;
visualizer.run();

C#

source

Console.WriteLine("Getting point cloud from frame");
var pointCloud = frame.PointCloud;

Console.WriteLine("Setting up visualization");
var visualizer = new Zivid.NET.Visualization.Visualizer();

Console.WriteLine("Visualizing point cloud");
visualizer.Show(pointCloud);
visualizer.ShowMaximized();
visualizer.ResetToFit();

Console.WriteLine("Running visualizer. Blocking until window closes.");
visualizer.Run();

For more information, check out Visualization Tutorial, where we cover point cloud, color image, depth map, and normals visualization, with implementations using third party libraries.

Conclusion

This tutorial shows how to use the Zivid SDK to extract the point cloud, manipulate it, transform it, and visualize it.

For the corresponding API, see Applications Basic.

Version History

SDK	Changes
2.18.0	Added Zivid::Mask class and mask() / masked() methods on PointCloud and Frame for post-capture binary masking.
2.16.0	Added support for UnorganizedPointCloud. transformed is added as a function to PointCloud (also available in UnorganizedPointCloud).
2.11.0	Added support for SRGB color space.
2.10.0	Monochrome Capture introduces a faster alternative to Downsample.