点云教程

介绍

本教程介绍了如何使用Zivid SDK进行 点云 数据的相关操作。

小技巧

如果您更喜欢观看视频教程，我们的网络研讨会 Getting your point cloud ready for your application （为您的应用准备好点云）涵盖了点云教程。

先决条件

• 安装 Zivid软件.

• 对于Python：安装 Zivid-python

帧（Frame）

C++

Zivid::Frame 包含了点云和彩色图像（存储在计算设备内存中）以及捕获设置和相机的信息。

C＃

Zivid.NET.Frame 包含了点云和彩色图像（存储在计算设备内存中）以及捕获设置和相机信息。

Python

zivid.Frame 包含了点云和彩色图像（存储在计算设备内存中）以及捕获设置和相机信息。

捕获（Capture）

使用 Zivid 进行捕获时，您会得到一个 frame 作为返回数据。点云存储在该 frame 中，而该 frame 又存储在 GPU 内存中。捕获的图像可以选择包含或者不包含颜色信息，具体取决于您调用的方法。更多信息请参阅 包含不同捕获模式的表格 。

带颜色数据的捕获

如果要捕获带有颜色的点云，可以使用 Zivid::Camera::capture2D3D() 方法。

C++

源码

const auto frame = camera.capture2D3D(settings);

C＃

源码

using (var frame = camera.Capture2D3D(settings))

Python

source

frame = camera.capture_2d_3d(settings)

不带颜色数据的捕获

如果要捕获没有颜色信息的点云，可以使用 Zivid::Camera::capture3D() 方法。

C++

源码

const auto frame3D = camera.capture3D(settings);

C＃

源码

using (var frame3D = camera.Capture3D(settings))

Python

源码

frame_3d = camera.capture_3d(settings)

查看 捕获教程 了解有关如何捕获图像的详细说明。

加载（Load）

可以通过ZDF文件加载图像帧。

C++

源码

const auto dataFile = std::string(ZIVID_SAMPLE_DATA_DIR) + "/Zivid3D.zdf";
std::cout << "Reading ZDF frame from file: " << dataFile << std::endl;
const auto frame = Zivid::Frame(dataFile);

C＃

源码

var dataFile =
    Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData) + "/Zivid/Zivid3D.zdf";
Console.WriteLine("Reading ZDF frame from file: " + dataFile);

using (var frame = new Zivid.NET.Frame(dataFile))
{

Python

source

data_file = get_sample_data_path() / "Zivid3D.zdf"
print(f"Reading point cloud from file: {data_file}")

frame = zivid.Frame(data_file)

点云

从Frame获取句柄

您现在可以从GPU上的点云数据获取句柄。

C++

源码

const auto pointCloud = frame.pointCloud();

C＃

源码

var pointCloud = frame.PointCloud;

Python

source

point_cloud = frame.point_cloud()

点云包含了XYZ、RGB和SNR数据，分布在2D网格上。

如需了解更多信息，请查看 点云结构。

C++

方法 Zivid::Frame::pointCloud() 不从GPU内存执行任何复制。

备注

Zivid::Camera::capture2D3D() ` 和 :code:`Zivid::Camera::capture3D() 方法会在相机完成原始图像采集后的某个时刻返回。 Zivid::Frame::pointCloud() 句柄可立即使用。然而，实际的点云数据只有在 GPU 处理完成后才可用。任何对数据复制函数（见下文）的调用都会被阻塞，并等待处理完成后再继续执行请求的复制操作。

请查看 点云捕获过程 了解更多细节。

C＃

获取 Zivid.NET.Frame.PointCloud 不会从GPU内存执行任何复制。

备注

Zivid.NET.Camera.Capture2D3D() 和 Zivid.NET.Camera.Capture3D() 方法会在相机完成原始图像采集后的某个时刻返回。 Zivid.NET.Frame.PointCloud 句柄可立即使用。然而，实际的点云数据只有在 GPU 处理完成后才可用。任何对数据复制方法（见下文）的调用都会被阻塞，并等待处理完成后再继续执行请求的复制操作。

请查看 点云捕获过程 了解更多细节。

Python

功能 zivid.frame.point_cloud() 不从GPU内存执行任何复制。

备注

zivid.camera.capture_2d_3d() 和 zivid.camera.capture_3d() 方法会在相机完成原始图像采集后的某个时刻返回。 zivid.frame.point_cloud() 句柄可立即使用。但是，实际的点云数据只有在 GPU 处理完成后才可用。任何对数据复制函数（见下文）的调用都会被阻塞，并等待处理完成后再继续执行请求的复制操作。

请查看 点云捕获过程 了解更多细节。

Unorganized point cloud

It is possible to convert the organized point cloud to an unorganized point cloud. While doing so, all NaN values are removed, and the point cloud is flattened to a 1D array.

C++

source

const auto unorganizedPointCloud = frame.pointCloud().toUnorganizedPointCloud();

Python

source

unorganized_point_cloud = frame.point_cloud().to_unorganized_point_cloud()

Combining multiple unorganized point clouds

The unorganized point cloud can be extended with additional unorganized point clouds.

C++

source

stitchedPointCloud.extend(currentPointCloud.transform(transformationMatrixZivid));

Python

source

stitched_point_cloud.extend(current_point_cloud.transform(transformation_matrix))

将数据从GPU复制到CPU内存

You can now selectively copy data based on what is required. This is the complete list of output data formats and how to copy them from the GPU. Most of these APIs also applies to the unorganized point cloud.

C++

返回类型	复制功能	单像素数据量	总数据量
Zivid::Array2D<Zivid::PointXYZ>	PointCloud::copyPointsXYZ() 或 PointCloud::copyData<Zivid::PointXYZ>()	12 bytes	28 MB
Zivid::Array2D<Zivid::PointXYZW>	PointCloud::copyPointsXYZW() 或 PointCloud::copyData<Zivid::PointXYZW>()	16 bytes	37 MB
Zivid::Array2D<Zivid::PointZ>	PointCloud::copyPointsZ() 或 PointCloud::copyData<Zivid::PointZ>()	4 bytes	9 MB
Zivid::Array2D<Zivid::ColorRGBA>	PointCloud::copyColorsRGBA() 或 PointCloud::copyData<Zivid::ColorRGBA>()	4 bytes	9 MB
Zivid::Array2D<Zivid::SNR>	PointCloud::copySNRs() 或 PointCloud::copyData<Zivid::SNR>()	4 bytes	9 MB
Zivid::Array2D<Zivid::PointXYZColorRGBA>	PointCloud::copyData<PointXYZColorRGBA>()	16 bytes	37 MB
Zivid::Array2D<Zivid::PointXYZColorBGRA>	PointCloud::copyPointsXYZColorsBGRA() 或 PointCloud::copyData<PointXYZColorBGRA>()	16 bytes	37 MB
Zivid::Image<Zivid::ColorRGBA>	PointCloud::copyImageRGBA()	4 bytes	9 MB
Zivid::Image<Zivid::ColorBGRA>	PointCloud::copyImageBGRA()	4 bytes	9 MB
Zivid::Image<Zivid::ColorsRGB>	PointCloud::copyImagesRGB()	4 bytes	9 MB

C＃

返回类型	复制方法	单像素数据量	总数据量
float[height,width,3]	PointCloud.CopyPointsXYZ()	12 bytes	28 MB
float[height,width,4]	PointCloud.CopyPointsXYZW()	16 bytes	37 MB
float[height,width,1]	PointCloud.CopyPointsZ()	4 bytes	9 MB
byte[height,width,4]	PointCloud.CopyColorsRGBA()	4 bytes	9 MB
float[height,width]	PointCloud.CopySNRs()	4 bytes	9 MB
Zivid.NET.PointXYZColorRGBA[height, width]	PointCloud.CopyPointsXYZColorsRGBA()	16 bytes	37 MB
Zivid.NET.PointXYZColorBGRA[height, width]	PointCloud.CopyPointsXYZColorsBGRA()	16 bytes	37 MB
Zivid.NET.ImageRGBA	PointCloud.CopyImageRGBA()	4 bytes	9 MB
Zivid.NET.ImageBGRA	PointCloud.CopyImageBGRA()	4 bytes	9 MB
Zivid.NET.ImageSRGB	PointCloud.CopyImageSRGB()	4 bytes	9 MB

Python

返回类型	复制功能	单像素数据量	总数据量
numpy.ndarray([height,width,3], dtype=float32)	PointCloud.copy_data("xyz")	12 bytes	28 MB
numpy.ndarray([height,width,3], dtype=float32)	PointCloud.copy_data("xyzw")	16 bytes	37 MB
numpy.ndarray([height,width], dtype=float32)	PointCloud.copy_data("z")	4 bytes	9 MB
numpy.ndarray([height,width,4], dtype=uint8)	PointCloud.copy_data("rgba")	4 bytes	9 MB
numpy.ndarray([height,width,4], dtype=uint8)	PointCloud.copy_data("bgra")	4 bytes	9 MB
numpy.ndarray([height,width,4], dtype=uint8)	PointCloud.copy_data("srgb")	4 bytes	9 MB
numpy.ndarray([height,width], dtype=float32)	PointCloud.copy_data("snr")	4 bytes	9 MB
numpy.ndarray([height,width], dtype=[('x', '<f4'), ('y', '<f4'), ('z', '<f4'), (' r', 'u1'), ('g', 'u1'), ('b', 'u1'), ('a', 'u1')])	PointCloud.copy_data("xyzrgba")	16 bytes	37 MB

以下是如何复制数据的示例。

C++

源码

const auto data = pointCloud.copyData<Zivid::PointXYZColorRGBA_SRGB>();

C＃

源码

var pointCloudData = pointCloud.CopyPointsXYZColorsRGBA();

Python

source

xyz = point_cloud.copy_data("xyz")
rgba = point_cloud.copy_data("rgba_srgb")

内存分配选项

在内存分配方面，复制数据有两种方式：

• Zivid SDK可以分配内存缓冲区并将数据复制到其中。

• 用户可以将指针传递给预分配的内存缓冲区，Zivid SDK会将数据复制到预分配的内存缓冲区。

C++

我们将展示两个使用了OpenCV的内存分配的示例。

将选定数据从GPU复制到 CPU内存（Zivid 分配）

如果您只关心例如点云的RGB颜色数据，您可以仅将该数据复制到CPU内存。

源码

std::cout << "Capturing frame" << std::endl;
frame = camera.capture2D3D(settings);

std::cout << "Copying colors with Zivid API from GPU to CPU" << std::endl;
auto colors = frame.frame2D().value().imageBGRA_SRGB();

std::cout << "Casting the data pointer as a void*, since this is what the OpenCV matrix constructor requires."
          << std::endl;
auto *dataPtrZividAllocated = const_cast<void *>(static_cast<const void *>(colors.data()));

std::cout << "Wrapping this block of data in an OpenCV matrix. This is possible since the layout of \n"
          << "Zivid::ColorBGRA_SRGB exactly matches the layout of CV_8UC4. No copying occurs in this step."
          << std::endl;
const cv::Mat bgraZividAllocated(colors.height(), colors.width(), CV_8UC4, dataPtrZividAllocated);

std::cout << "Displaying image" << std::endl;
cv::imshow("BGRA image Zivid Allocated", bgraZividAllocated);
cv::waitKey(CI_WAITKEY_TIMEOUT_IN_MS);

将选定数据从GPU复制到CPU内存（用户分配）

在上面的示例中，数据的所有权由返回的 Zivid::Array2D<> 对象拥有。或者您可以提供预先分配的内存缓冲区到 Zivid::PointCloud::copyData(dataPtr)。 dataPtr 的类型定义了将被复制的数据（PointXYZ, ColorRGBA 等。）。

现在让我们看一下与上面完全相同的用例。但是，这一次我们允许OpenCV来分配必要的存储空间。然后我们让Zivid API将数据直接从GPU复制到这个内存位置。

源码

std::cout << "Allocating the necessary storage with OpenCV API based on resolution info before any capturing"
          << std::endl;
auto bgraUserAllocated = cv::Mat(resolution.height(), resolution.width(), CV_8UC4);

std::cout << "Capturing frame" << std::endl;
auto frame = camera.capture2D3D(settings);
auto pointCloud = frame.pointCloud();

std::cout << "Copying data with Zivid API from the GPU into the memory location allocated by OpenCV"
          << std::endl;
pointCloud.copyData(&(*bgraUserAllocated.begin<Zivid::ColorBGRA_SRGB>()));

std::cout << "Displaying image" << std::endl;
cv::imshow("BGRA image User Allocated", bgraUserAllocated);
cv::waitKey(CI_WAITKEY_TIMEOUT_IN_MS);

Copy unorganized point cloud data from GPU to CPU memory (Open3D-tensor)

source

open3d::t::geometry::PointCloud copyToOpen3D(const Zivid::UnorganizedPointCloud &pointCloud)
{
    using namespace open3d::core;
    auto device = Device("CPU:0");
    auto xyzTensor = Tensor({ static_cast<int64_t>(pointCloud.size()), 3 }, Dtype::Float32, device);
    auto rgbTensor = Tensor({ static_cast<int64_t>(pointCloud.size()), 3 }, Dtype::Float32, device);

    pointCloud.copyData(reinterpret_cast<Zivid::PointXYZ *>(xyzTensor.GetDataPtr<float>()));

    // Open3D does not store colors in 8-bit
    const auto rgbaColors = pointCloud.copyColorsRGBA_SRGB();
    for(size_t i = 0; i < pointCloud.size(); ++i)
    {
        const auto r = static_cast<float>(rgbaColors(i).r) / 255.0f;
        const auto g = static_cast<float>(rgbaColors(i).g) / 255.0f;
        const auto b = static_cast<float>(rgbaColors(i).b) / 255.0f;
        rgbTensor.SetItem(TensorKey::Index(i), Tensor::Init({ r, g, b }));
    }

    open3d::t::geometry::PointCloud cloud(device);
    cloud.SetPointPositions(xyzTensor);
    cloud.SetPointColors(rgbTensor);
    return cloud;
}

转换

你可能想要 转换 点云，将其原点从相机坐标系转换到机器人基坐标系，或者比如， 通过将点云从mm转换为m来缩放点云。

C++

源码

pointCloud.transform(baseToCameraTransform);

C＃

源码

pointCloud.Transform(transformBaseToCamera);

Python

source

point_cloud.transform(base_to_camera_transform)

Transformation can be done in-place:

• Zivid::PointCloud::transform()

• Zivid::UnorganizedPointCloud::transform()

or by creating a new instance:

• Zivid::PointCloud::transformed()

• Zivid::UnorganizedPointCloud::transformed()

The following example shows how create a new instance of Zivid::UnorganizedPointCloud with a transformation applied to it. Note that in this sample is is not necessary to create a new instance, as the untransformed point cloud is not used after the transformation.

C++

source

const auto transformedUnorganizedPointCloud = unorganizedPointCloud.transformed(transformationMatrix);

Python

source

transformed_unorganized_point_cloud = unorganized_point_cloud.transformed(transformation_matrix)

Even the in-place API returns the transformed point cloud, so you can use it directly, as in the example below.

C++

source

stitchedPointCloud.extend(currentPointCloud.transform(transformationMatrixZivid));

Python

source

stitched_point_cloud.extend(current_point_cloud.transform(transformation_matrix))

降采样

有时您可能不需要相机输出的高空间分辨率（高空间分辨率意味着更多的细节和更短的点之间的距离）点云。那么您对点云进行 降采样 。

备注

Sampling（采样） - 3D 描述了一种基于硬件的子/降采样方法，它可以降低捕获过程中点云的分辨率，同时也减少采集和捕获时间。

备注

Zivid::UnorganizedPointCloud does not support downsampling, but it does support voxel downsampling, see Voxel downsample.

可以就地进行降采样，从而修改当前的点云。

C++

源码

pointCloud.downsample(Zivid::PointCloud::Downsampling::by2x2);

C＃

源码

pointCloud.Downsample(Zivid.NET.PointCloud.Downsampling.By2x2);

Python

source

point_cloud.downsample(zivid.PointCloud.Downsampling.by2x2)

也可以将降采样后的点云作为一个新的点云实例，它不会改变现有的点云。

C++

源码

auto downsampledPointCloud = pointCloud.downsampled(Zivid::PointCloud::Downsampling::by2x2);

C＃

源码

var downsampledPointCloud = pointCloud.Downsampled(Zivid.NET.PointCloud.Downsampling.By2x2);

Python

source

downsampled_point_cloud = point_cloud.downsampled(zivid.PointCloud.Downsampling.by2x2)

Zivid SDK支持以下降采样率： by2x2, by3x3， 和 by4x4, 可以进行多次降采样。

Voxel downsample

Zivid::UnorganizedPointCloud supports voxel downsampling. The API takes two arguments:

1. voxelSize - the size of the voxel in millimeters.

2. minPointsPerVoxel - the minimum number of points per voxel to keep it.

Voxel downsampling subdivides 3D space into a grid of cubic voxels with a given size. If a given voxel contains a number of points at or above the given limit, all those source points are replaced with a single point with the following properties:

• Position (XYZ) is an SNR-weighted average of the source points' positions, i.e. a high-confidence source point will have a greater influence on the resulting position than a low-confidence one.

• Color (RGBA) is the average of the source points' colors.

• Signal-to-noise ratio (SNR) is sqrt(sum(SNR^2)) of the source points' SNR values, i.e. the SNR of a new point will increase with both the number and the confidence of the source points that were used to compute its position.

Using minPointsPerVoxel > 1 is particularly useful for removing noise and artifacts from unorganized point clouds that are a combination of point clouds captured from different angles. This is because a given artifact is most likely only present in one of the captures, and minPointsPerVoxel can be used to only fill voxels that both captures "agree" on.

C++

source

const auto finalPointCloud = stitchedPointCloud.voxelDownsampled(0.5, 1);

Python

source

final_point_cloud = stitched_point_cloud.voxel_downsampled(0.5, 1)

法线

一些应用需要计算点云的 法线 数据。

C++

源码

std::cout << "Computing normals and copying them to CPU memory" << std::endl;
const auto normals = pointCloud.copyData<Zivid::NormalXYZ>();

C＃

源码

Console.WriteLine("Computing normals and copying them to CPU memory");
var normals = pointCloud.CopyNormalsXYZ();

Python

源码

print("Computing normals and copying them to CPU memory")
normals = point_cloud.copy_data("normals")

法线API将计算点云中每个点的法线，并将法线从GPU内存复制到CPU内存。其结果是一个法向量矩阵，每条法线对应输入点云的每个点。法线的大小和输入点云的大小相等。

可视化

您可以通过帧（frame）可视化点云。

C++

源码

std::cout << "Setting up visualization" << std::endl;
Zivid::Visualization::Visualizer visualizer;

std::cout << "Visualizing point cloud" << std::endl;
visualizer.showMaximized();
visualizer.show(frame);
visualizer.resetToFit();

std::cout << "Running visualizer. Blocking until window closes." << std::endl;
visualizer.run();

C＃

源码

Console.WriteLine("Setting up visualization");
using (var visualizer = new Zivid.NET.Visualization.Visualizer())
{
    Console.WriteLine("Visualizing point cloud");
    visualizer.Show(frame);
    visualizer.ShowMaximized();
    visualizer.ResetToFit();

    Console.WriteLine("Running visualizer. Blocking until window closes.");
    visualizer.Run();
}

您也可以从点云对象来可视化点云。

C++

源码

std::cout << "Getting point cloud from frame" << std::endl;
auto pointCloud = frame.pointCloud();

std::cout << "Setting up visualization" << std::endl;
Zivid::Visualization::Visualizer visualizer;

std::cout << "Visualizing point cloud" << std::endl;
visualizer.showMaximized();
visualizer.show(pointCloud);
visualizer.resetToFit();

std::cout << "Running visualizer. Blocking until window closes." << std::endl;
visualizer.run();

C＃

源码

Console.WriteLine("Getting point cloud from frame");
var pointCloud = frame.PointCloud;

Console.WriteLine("Setting up visualization");
var visualizer = new Zivid.NET.Visualization.Visualizer();

Console.WriteLine("Visualizing point cloud");
visualizer.Show(pointCloud);
visualizer.ShowMaximized();
visualizer.ResetToFit();

Console.WriteLine("Running visualizer. Blocking until window closes.");
visualizer.Run();

如需了解更多信息，请查看 可视化教程，里面包含了如何使用第三方库实现点云、彩色图像、深度图和法线的可视化。

结论

本教程展示了如何使用Zivid SDK来提取、操作、转换和可视化点云。

版本历史

SDK	变更
2.16.0	Added support for Zivid::UnorganizedPointCloud. transformed is added as a function to Zivid::PointCloud (also available in Zivid::UnorganizedPointCloud).
2.11.0	添加了对 SRGB 色彩空间（SRGB color space）的支持。
2.10.0	Monochrome Capture（单色捕获） 引入了比 降采样 更快的替代方案。