点云教程

介绍

本教程介绍了如何使用Zivid SDK进行 点云 数据的相关操作。

小技巧

如果您更喜欢观看视频教程，我们的网络研讨会 Getting your point cloud ready for your application （为您的应用准备好点云）涵盖了点云教程。

先决条件

帧（Frame）

Zivid::Frame 包含了点云和彩色图像（存储在计算设备内存中）以及捕获设置和相机的信息。

Zivid.NET.Frame 包含了点云和彩色图像（存储在计算设备内存中）以及捕获设置和相机信息。

zivid.Frame 包含了点云和彩色图像（存储在计算设备内存中）以及捕获设置和相机信息。

捕获（Capture）

When you capture with Zivid, you get a frame in return. The point cloud is stored in the frame, and the frame is stored in the GPU memory. The capture can contain color or not, depending of the method that you call. For more information see this table with different capture modes.

Capture with color

If you want to capture a point cloud with color, you can use the Zivid::Camera::capture2D3D() method.

跳转到源码

source

const auto frame = camera.capture2D3D(settings);
跳转到源码

源码

using (var frame = camera.Capture2D3D(settings))
跳转到源码

source

with camera.capture_2d_3d(settings) as frame:

Capture without color

If you want to capture a point cloud without color, you can use the Zivid::Camera::capture3D() method.

跳转到源码

source

const auto frame3D = camera.capture3D(settings);
跳转到源码

source

using (var frame3D = camera.Capture3D(settings))
跳转到源码

source

with camera.capture_3d(settings) as frame_3d:

查看 捕获教程 了解有关如何捕获图像的详细说明。

加载（Load）

可以通过ZDF文件加载图像帧。

跳转到源码

源码

const auto dataFile = std::string(ZIVID_SAMPLE_DATA_DIR) + "/Zivid3D.zdf";
std::cout << "Reading ZDF frame from file: " << dataFile << std::endl;
const auto frame = Zivid::Frame(dataFile);
跳转到源码

source

var dataFile =
    Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData) + "/Zivid/Zivid3D.zdf";
Console.WriteLine("Reading ZDF frame from file: " + dataFile);

using (var frame = new Zivid.NET.Frame(dataFile))
{
跳转到源码

源码

data_file = get_sample_data_path() / "Zivid3D.zdf"
print(f"Reading point cloud from file: {data_file}")
frame = zivid.Frame(data_file)

点云

从Frame获取句柄

您现在可以从GPU上的点云数据获取句柄。

跳转到源码

源码

const auto pointCloud = frame.pointCloud();
跳转到源码

source

var pointCloud = frame.PointCloud;
跳转到源码

源码

point_cloud = frame.point_cloud()

点云包含了XYZ、RGB和SNR数据，分布在2D网格上。

如需了解更多信息，请查看 点云结构。

方法 Zivid::Frame::pointCloud() 不从GPU内存执行任何复制。

备注

Zivid::Camera::capture2D3D() and Zivid::Camera::capture3D() methods return at some moment in time after the camera completes capturing raw images. The handle from Zivid::Frame::pointCloud() is available instantly. However, the actual point cloud data becomes available only after the processing on the GPU is finished. Any calls to data-copy functions (section below) will block and wait for processing to finish before proceeding with the requested copy operation.

请查看 点云捕获过程 了解更多细节。

获取 Zivid.NET.Frame.PointCloud 不会从GPU内存执行任何复制。

备注

Zivid.NET.Camera.Capture2D3D() and Zivid.NET.Camera.Capture3D() methods return at some moment in time after the camera completes capturing raw images. The handle from Zivid.NET.Frame.PointCloud is available instantly. However, the actual point cloud data becomes available only after the processing on the GPU is finished. Any calls to data-copy methods (section below) will block and wait for processing to finish before proceeding with the requested copy operation.

请查看 点云捕获过程 了解更多细节。

功能 zivid.frame.point_cloud() 不从GPU内存执行任何复制。

备注

zivid.camera.capture_2d_3d() and zivid.camera.capture_3d() methods return at some moment in time after the camera completes capturing raw images. The handle from zivid.frame.point_cloud() is available instantly. However, the actual point cloud data becomes available only after the processing on the GPU is finished. Any calls to data-copy functions (section below) will block and wait for processing to finish before proceeding with the requested copy operation.

请查看 点云捕获过程 了解更多细节。

将数据从GPU复制到CPU内存

您现在可以根据需要有选择地复制数据。下面的列表列出了输出数据格式以及如何从GPU复制它们。

返回类型

复制功能

单像素数据量

总数据量

Zivid::Array2D<Zivid::PointXYZ>

PointCloud::copyPointsXYZ() 或 PointCloud::copyData<Zivid::PointXYZ>()

12 bytes

28 MB

Zivid::Array2D<Zivid::PointXYZW>

PointCloud::copyPointsXYZW() 或 PointCloud::copyData<Zivid::PointXYZW>()

16 bytes

37 MB

Zivid::Array2D<Zivid::PointZ>

PointCloud::copyPointsZ() 或 PointCloud::copyData<Zivid::PointZ>()

4 bytes

9 MB

Zivid::Array2D<Zivid::ColorRGBA>

PointCloud::copyColorsRGBA() 或 PointCloud::copyData<Zivid::ColorRGBA>()

4 bytes

9 MB

Zivid::Array2D<Zivid::SNR>

PointCloud::copySNRs() 或 PointCloud::copyData<Zivid::SNR>()

4 bytes

9 MB

Zivid::Array2D<Zivid::PointXYZColorRGBA>

PointCloud::copyData<PointXYZColorRGBA>()

16 bytes

37 MB

Zivid::Array2D<Zivid::PointXYZColorBGRA>

PointCloud::copyPointsXYZColorsBGRA() 或 PointCloud::copyData<PointXYZColorBGRA>()

16 bytes

37 MB

Zivid::Image<Zivid::ColorRGBA>

PointCloud::copyImageRGBA()

4 bytes

9 MB

Zivid::Image<Zivid::ColorBGRA>

PointCloud::copyImageBGRA()

4 bytes

9 MB

Zivid::Image<Zivid::ColorsRGB>

PointCloud::copyImagesRGB()

4 bytes

9 MB

返回类型

复制方法

单像素数据量

总数据量

float[height,width,3]

PointCloud.CopyPointsXYZ()

12 bytes

28 MB

float[height,width,4]

PointCloud.CopyPointsXYZW()

16 bytes

37 MB

float[height,width,1]

PointCloud.CopyPointsZ()

4 bytes

9 MB

byte[height,width,4]

PointCloud.CopyColorsRGBA()

4 bytes

9 MB

float[height,width]

PointCloud.CopySNRs()

4 bytes

9 MB

Zivid.NET.PointXYZColorRGBA[height, width]

PointCloud.CopyPointsXYZColorsRGBA()

16 bytes

37 MB

Zivid.NET.PointXYZColorBGRA[height, width]

PointCloud.CopyPointsXYZColorsBGRA()

16 bytes

37 MB

Zivid.NET.ImageRGBA

PointCloud.CopyImageRGBA()

4 bytes

9 MB

Zivid.NET.ImageBGRA

PointCloud.CopyImageBGRA()

4 bytes

9 MB

Zivid.NET.ImageSRGB

PointCloud.CopyImageSRGB()

4 bytes

9 MB

返回类型

复制功能

单像素数据量

总数据量

numpy.ndarray([height,width,3], dtype=float32)

PointCloud.copy_data("xyz")

12 bytes

28 MB

numpy.ndarray([height,width,3], dtype=float32)

PointCloud.copy_data("xyzw")

16 bytes

37 MB

numpy.ndarray([height,width], dtype=float32)

PointCloud.copy_data("z")

4 bytes

9 MB

numpy.ndarray([height,width,4], dtype=uint8)

PointCloud.copy_data("rgba")

4 bytes

9 MB

numpy.ndarray([height,width,4], dtype=uint8)

PointCloud.copy_data("bgra")

4 bytes

9 MB

numpy.ndarray([height,width,4], dtype=uint8)

PointCloud.copy_data("srgb")

4 bytes

9 MB

numpy.ndarray([height,width], dtype=float32)

PointCloud.copy_data("snr")

4 bytes

9 MB

numpy.ndarray([height,width], dtype=[('x', '<f4'), ('y', '<f4'), ('z', '<f4'), (' r', 'u1'), ('g', 'u1'), ('b', 'u1'), ('a', 'u1')])

PointCloud.copy_data("xyzrgba")

16 bytes

37 MB

以下是如何复制数据的示例。

跳转到源码

源码

const auto data = pointCloud.copyData<Zivid::PointXYZColorRGBA_SRGB>();
跳转到源码

source

var pointCloudData = pointCloud.CopyPointsXYZColorsRGBA();
跳转到源码

源码

xyz = point_cloud.copy_data("xyz")
rgba = point_cloud.copy_data("rgba_srgb")

内存分配选项

在内存分配方面，复制数据有两种方式：

  • Zivid SDK可以分配内存缓冲区并将数据复制到其中。

  • 用户可以将指针传递给预分配的内存缓冲区，Zivid SDK会将数据复制到预分配的内存缓冲区。

我们将展示两个使用了OpenCV的内存分配的示例。

将选定数据从GPU复制到 CPU内存（Zivid 分配）

如果您只关心例如点云的RGB颜色数据，您可以仅将该数据复制到CPU内存。

跳转到源码

source

std::cout << "Capturing frame" << std::endl;
frame = camera.capture2D3D(settings);

std::cout << "Copying colors with Zivid API from GPU to CPU" << std::endl;
auto colors = frame.frame2D().value().imageBGRA_SRGB();

std::cout << "Casting the data pointer as a void*, since this is what the OpenCV matrix constructor requires."
          << std::endl;
auto *dataPtrZividAllocated = const_cast<void *>(static_cast<const void *>(colors.data()));

std::cout << "Wrapping this block of data in an OpenCV matrix. This is possible since the layout of \n"
          << "Zivid::ColorBGRA_SRGB exactly matches the layout of CV_8UC4. No copying occurs in this step."
          << std::endl;
const cv::Mat bgraZividAllocated(colors.height(), colors.width(), CV_8UC4, dataPtrZividAllocated);

std::cout << "Displaying image" << std::endl;
cv::imshow("BGRA image Zivid Allocated", bgraZividAllocated);
cv::waitKey(0);

将选定数据从GPU复制到CPU内存（用户分配）

在上面的示例中，数据的所有权由返回的 Zivid::Array2D<> 对象拥有。或者您可以提供预先分配的内存缓冲区到 Zivid::PointCloud::copyData(dataPtr)。 dataPtr 的类型定义了将被复制的数据（PointXYZ, ColorRGBA 等。）。

现在让我们看一下与上面完全相同的用例。但是，这一次我们允许OpenCV来分配必要的存储空间。然后我们让Zivid API将数据直接从GPU复制到这个内存位置。

跳转到源码

source

std::cout << "Allocating the necessary storage with OpenCV API based on resolution info before any capturing"
          << std::endl;
auto bgraUserAllocated = cv::Mat(resolution.height(), resolution.width(), CV_8UC4);

std::cout << "Capturing frame" << std::endl;
auto frame = camera.capture2D3D(settings);
auto pointCloud = frame.pointCloud();

std::cout << "Copying data with Zivid API from the GPU into the memory location allocated by OpenCV"
          << std::endl;
pointCloud.copyData(&(*bgraUserAllocated.begin<Zivid::ColorBGRA_SRGB>()));

std::cout << "Displaying image" << std::endl;
cv::imshow("BGRA image User Allocated", bgraUserAllocated);
cv::waitKey(0);

转换

你可能想要 转换 点云，将其原点从相机坐标系转换到机器人基坐标系，或者比如， 通过将点云从mm转换为m来缩放点云。

跳转到源码

源码

pointCloud.transform(baseToCameraTransform);
跳转到源码

source

pointCloud.Transform(transformBaseToCamera);
跳转到源码

源码

point_cloud.transform(base_to_camera_transform)

降采样

有时您可能不需要相机输出的高空间分辨率（高空间分辨率意味着更多的细节和更短的点之间的距离）点云。那么您对点云进行 降采样 。

备注

Sampling（采样） - 3D 描述了一种基于硬件的子/降采样方法，它可以降低捕获过程中点云的分辨率，同时也减少采集和捕获时间。

可以就地进行降采样，从而修改当前的点云。

跳转到源码

源码

pointCloud.downsample(Zivid::PointCloud::Downsampling::by2x2);
跳转到源码

source

pointCloud.Downsample(Zivid.NET.PointCloud.Downsampling.By2x2);
跳转到源码

源码

point_cloud.downsample(zivid.PointCloud.Downsampling.by2x2)

也可以将降采样后的点云作为一个新的点云实例，它不会改变现有的点云。

跳转到源码

源码

auto downsampledPointCloud = pointCloud.downsampled(Zivid::PointCloud::Downsampling::by2x2);
跳转到源码

source

var downsampledPointCloud = pointCloud.Downsampled(Zivid.NET.PointCloud.Downsampling.By2x2);
跳转到源码

源码

downsampled_point_cloud = point_cloud.downsampled(zivid.PointCloud.Downsampling.by2x2)

Zivid SDK支持以下降采样率： by2x2, by3x3， 和 by4x4, 可以进行多次降采样。

法线

一些应用需要计算点云的 法线 数据。

跳转到源码

源码

std::cout << "Computing normals and copying them to CPU memory" << std::endl;
const auto normals = pointCloud.copyData<Zivid::NormalXYZ>();
跳转到源码

source

Console.WriteLine("Computing normals and copying them to CPU memory");
var normals = pointCloud.CopyNormalsXYZ();
跳转到源码

源码

print("Computing normals and copying them to CPU memory")
normals = point_cloud.copy_data("normals")

法线API将计算点云中每个点的法线，并将法线从GPU内存复制到CPU内存。其结果是一个法向量矩阵，每条法线对应输入点云的每个点。法线的大小和输入点云的大小相等。

可视化

您可以通过帧（frame）可视化点云。

跳转到源码

source

std::cout << "Setting up visualization" << std::endl;
Zivid::Visualization::Visualizer visualizer;

std::cout << "Visualizing point cloud" << std::endl;
visualizer.showMaximized();
visualizer.show(frame);
visualizer.resetToFit();

std::cout << "Running visualizer. Blocking until window closes." << std::endl;
visualizer.run();
跳转到源码

source

Console.WriteLine("Setting up visualization");
using (var visualizer = new Zivid.NET.Visualization.Visualizer())
{
    Console.WriteLine("Visualizing point cloud");
    visualizer.Show(frame);
    visualizer.ShowMaximized();
    visualizer.ResetToFit();

    Console.WriteLine("Running visualizer. Blocking until window closes.");
    visualizer.Run();
}

您也可以从点云对象来可视化点云。

跳转到源码

源码

std::cout << "Getting point cloud from frame" << std::endl;
auto pointCloud = frame.pointCloud();

std::cout << "Setting up visualization" << std::endl;
Zivid::Visualization::Visualizer visualizer;

std::cout << "Visualizing point cloud" << std::endl;
visualizer.showMaximized();
visualizer.show(pointCloud);
visualizer.resetToFit();

std::cout << "Running visualizer. Blocking until window closes." << std::endl;
visualizer.run();
跳转到源码

source

Console.WriteLine("Getting point cloud from frame");
var pointCloud = frame.PointCloud;

Console.WriteLine("Setting up visualization");
var visualizer = new Zivid.NET.Visualization.Visualizer();

Console.WriteLine("Visualizing point cloud");
visualizer.Show(pointCloud);
visualizer.ShowMaximized();
visualizer.ResetToFit();

Console.WriteLine("Running visualizer. Blocking until window closes.");
visualizer.Run();

如需了解更多信息，请查看 可视化教程，里面包含了如何使用第三方库实现点云、彩色图像、深度图和法线的可视化。

结论

本教程展示了如何使用Zivid SDK来提取、操作、转换和可视化点云。

版本历史

SDK

变更

2.11.0

添加了对 SRGB 色彩空间（SRGB color space）的支持。

2.10.0

Monochrome Capture（单色捕获） 引入了比 降采样 更快的替代方案。