Point Cloud Tutorial
Introduction
This tutorial describes how to use Zivid SDK to work with Point Cloud data.
Tip
If you prefer watching a video, our webinar Getting your point cloud ready for your application covers the Point Cloud Tutorial.
Prerequisites
Install Zivid Software.
For Python: install zivid-python
Frame
The Zivid::Frame
contains the point cloud and color image (stored on compute device memory) and the capture and camera information.
The Zivid.NET.Frame
contains the point cloud and color image (stored on compute device memory) and the capture and camera information.
The zivid.Frame
contains the point cloud and color image (stored on compute device memory) and the capture and camera information.
Capture
When you capture with Zivid, you get a frame in return.
Check Capture Tutorial for detailed instructions on how to capture.
Load
The frame can also be loaded from a ZDF file.
const auto dataFile = std::string(ZIVID_SAMPLE_DATA_DIR) + "/Zivid3D.zdf";
std::cout << "Reading ZDF frame from file: " << dataFile << std::endl;
const auto frame = Zivid::Frame(dataFile);
Point Cloud
Get handle from Frame
You can now get a handle to the point cloud data on the GPU.
Point cloud contains XYZ, RGB, and SNR, laid out on a 2D grid.
For more info check out Point Cloud Structure.
The method Zivid::Frame::pointCloud()
does not perform any copying from GPU memory.
Note
Zivid::Camera::capture()
method returns at some moment in time after the camera completes capturing raw images.
The handle from Zivid::Frame::pointCloud()
is available instantly.
However, the actual point cloud data becomes available only after the processing on the GPU is finished.
Any calls to data-copy functions (section below) will block and wait for processing to finish before proceeding with the requested copy operation.
For detailed explanation, see Point Cloud Capture Process.
Getting the property Zivid.NET.Frame.PointCloud
does not perform any copying from GPU memory.
Note
Zivid.NET.Camera.Capture()
method returns at some moment in time after the camera completes capturing raw images.
The handle from Zivid.NET.Frame.PointCloud
is available instantly.
However, the actual point cloud data becomes available only after the processing on the GPU is finished.
Any calls to data-copy methods (section below) will block and wait for processing to finish before proceeding with the requested copy operation.
For detailed explanation, see Point Cloud Capture Process.
The function zivid.frame.point_cloud()
does not perform any copying from GPU memory.
Note
zivid.camera.capture()
method returns at some moment in time after the camera completes capturing raw images.
The handle from zivid.frame.point_cloud()
is available instantly.
However, the actual point cloud data becomes available only after the processing on the GPU is finished.
Any calls to data-copy functions (section below) will block and wait for processing to finish before proceeding with the requested copy operation.
For detailed explanation, see Point Cloud Capture Process.
Copy from GPU to CPU memory
You can now selectively copy data based on what is required. This is the complete list of output data formats and how to copy them from the GPU.
Return type |
Copy functions |
Data per pixel |
Total data |
---|---|---|---|
|
|
12 bytes |
28 MB |
|
|
16 bytes |
37 MB |
|
|
4 bytes |
9 MB |
|
|
4 bytes |
9 MB |
|
|
4 bytes |
9 MB |
|
|
16 bytes |
37 MB |
|
|
16 bytes |
37 MB |
|
|
4 bytes |
9 MB |
Return type |
Copy methods |
Data per pixel |
Total data |
---|---|---|---|
|
|
12 bytes |
28 MB |
|
|
16 bytes |
37 MB |
|
|
4 bytes |
9 MB |
|
|
4 bytes |
9 MB |
|
|
4 bytes |
9 MB |
|
|
16 bytes |
37 MB |
|
|
16 bytes |
37 MB |
|
|
4 bytes |
9 MB |
Return type |
Copy functions |
Data per pixel |
Total data |
---|---|---|---|
|
|
12 bytes |
28 MB |
|
|
16 bytes |
37 MB |
|
|
4 bytes |
9 MB |
|
|
4 bytes |
9 MB |
|
|
4 bytes |
9 MB |
|
|
16 bytes |
37 MB |
Here is an example of how to copy data.
Memory allocation options
In terms of memory allocation, there are two ways to copy data:
The Zivid SDK can allocate a memory buffer and copy data to it.
A user can pass a pointer to a pre-allocated memory buffer, and the Zivid SDK will copy the data to the pre-allocated memory buffer.
We present examples for the two memory allocation options using OpenCV.
Copy selected data from GPU to CPU memory (Zivid-allocated)
If you are only concerned about e.g. RGB color data of the point cloud, you can copy only that data to the CPU memory.
std::cout << "Capturing frame" << std::endl;
frame = camera.capture(settings);
pointCloud = frame.pointCloud();
std::cout << "Copying colors with Zivid API from GPU to CPU" << std::endl;
auto colors = pointCloud.copyColorsRGBA();
std::cout << "Casting the data pointer as a void*, since this is what the OpenCV matrix constructor requires."
<< std::endl;
auto *dataPtrZividAllocated = const_cast<void *>(static_cast<const void *>(colors.data()));
std::cout << "Wrapping this block of data in an OpenCV matrix. This is possible since the layout of \n"
<< "Zivid::ColorRGBA exactly matches the layout of CV_8UC4. No copying occurs in this step."
<< std::endl;
const cv::Mat rgbaZividAllocated(colors.height(), colors.width(), CV_8UC4, dataPtrZividAllocated);
Copy selected data from GPU to CPU memory (user-allocated)
In the above example, ownership of the data was held by the returned Zivid::Array2D<>
objects.
Alternatively, you may provide a pre-allocated memory buffer to Zivid::PointCloud::copyData(dataPtr)
.
The type of dataPtr
defines what shall be copied (PointXYZ
, ColorRGBA
, etc.).
Now let us look at the exact same use case as above. However, this time, we allow OpenCV to allocate the necessary storage. Then we ask the Zivid API to copy data directly from the GPU into this memory location.
std::cout << "Allocating the necessary storage with OpenCV API based on resolution info before any capturing"
<< std::endl;
auto rgbaUserAllocated = cv::Mat(resolution.height(), resolution.width(), CV_8UC4);
std::cout << "Capturing frame" << std::endl;
auto frame = camera.capture(settings);
auto pointCloud = frame.pointCloud();
std::cout << "Copying data with Zivid API from the GPU into the memory location allocated by OpenCV"
<< std::endl;
pointCloud.copyData(reinterpret_cast<Zivid::ColorRGBA *>(rgbaUserAllocated.data));
Transform
You may want to transform the point cloud to change its origin from the camera to the robot base frame or, e.g., scale the point cloud by transforming it from mm to m.
Downsample
Sometimes you might not need a point cloud with as high spatial resolution as given from the camera. You may then downsample the point cloud.
Downsampling can be done in-place, which modifies the current point cloud.
It is also possible to get the downsampled point cloud as a new point cloud instance, which does not alter the existing point cloud.
Zivid SDK supports the following downsampling rates: by2x2
, by3x3
, and by4x4
, with the possibility to perform downsampling multiple times.
Normals
Some applications require computing normals from the point cloud.
std::cout << "Computing normals and copying them to CPU memory" << std::endl;
const auto normals = pointCloud.copyData<Zivid::NormalXYZ>();
The Normals API computes the normal at each point in the point cloud and copies normals from the GPU memory to the CPU memory. The result is a matrix of normal vectors, one for each point in the input point cloud. The size of normals is equal to the size of the input point cloud.
Visualize
Having the frame allows you to visualize the point cloud.
std::cout << "Setting up visualization" << std::endl;
Zivid::Visualization::Visualizer visualizer;
std::cout << "Visualizing point cloud" << std::endl;
visualizer.showMaximized();
visualizer.show(frame);
visualizer.resetToFit();
std::cout << "Running visualizer. Blocking until window closes" << std::endl;
visualizer.run();
Console.WriteLine("Setting up visualization");
var visualizer = new Zivid.NET.Visualization.Visualizer();
Console.WriteLine("Visualizing point cloud");
visualizer.Show(frame);
visualizer.ShowMaximized();
visualizer.ResetToFit();
Console.WriteLine("Running visualizer. Blocking until window closes");
visualizer.Run();
You can visualize the point cloud from the point cloud object as well.
std::cout << "Getting point cloud from frame" << std::endl;
auto pointCloud = frame.pointCloud();
std::cout << "Setting up visualization" << std::endl;
Zivid::Visualization::Visualizer visualizer;
std::cout << "Visualizing point cloud" << std::endl;
visualizer.showMaximized();
visualizer.show(pointCloud);
visualizer.resetToFit();
std::cout << "Running visualizer. Blocking until window closes" << std::endl;
visualizer.run();
Console.WriteLine("Getting point cloud from frame");
var pointCloud = frame.PointCloud;
Console.WriteLine("Setting up visualization");
var visualizer = new Zivid.NET.Visualization.Visualizer();
Console.WriteLine("Visualizing point cloud");
visualizer.Show(pointCloud);
visualizer.ShowMaximized();
visualizer.ResetToFit();
Console.WriteLine("Running visualizer. Blocking until window closes");
visualizer.Run();
For more information, check out Visualization Tutorial, where we cover point cloud, color image, depth map, and normals visualization, with implementations using third party libraries.
Conclusion
This tutorial shows how to use the Zivid SDK to extract the point cloud, manipulate it, transform it, and visualize it.