2D+3D捕获策略

请注意，如果您不关心颜色信息，请直接跳过本章节，查看下一章节： 根据捕获速度选择 3D 和 2D 的设置 。

单品拣选（piece-picking）应用中常用的许多检测算法依赖于 2D 数据来识别要拾取的对象。在本文中，我们将深入探讨获取 2D 信息的不同方法、它们的优缺点以及外部照明条件。我们还会讨论各种 2D-3D 方法、它们的数据质量以及它们如何影响周期时间。

获取2D数据有两种方法：

1. 通过 camera.capture(Zivid::Settings2D).imageRGBA() 进行单独的 2D 捕获，请参阅 2D图像捕获流程 。

2. 3D 捕获 camera.capture(Zivid::Settings).pointCloud.copyImageRGBA() 中的一部分，请参阅 点云捕获过程 。

Which one to use depends on your requirements and the machine vision pipeline. We advocate for a dedicated 2D capture as it can leverage multi-threading and optimized scheduling. Utilizing 2D data from the 3D capture is simpler, but you may have to compromise speed to get desired 2D quality.

小技巧

当您单独捕获 2D 时，您应该在 3D 捕获中禁用 RGB。这可以节省采集和处理时间。通过将 Sampling::Color 设置为 disabled 来禁用 3D 捕获中的 RGB。

我们的建议：

• 单独的全分辨率 2D 捕获并打开投影仪。

• 不包含颜色数据的子采样 3D 捕获。

相机分辨率和一对一映射

对于精确的 2D 分割和检测，高分辨率彩色图像很有帮助。 Zivid 2+ 具有 5 MPx 成像传感器，而 Zivid 2 具有 2.3 MPx 传感器。下表显示了不同相机的 2D 和 3D 捕获的分辨率输出。

2D 捕获辨率

2D 捕获	Zivid 2+	Zivid 2
全分辨率	2448×2048	1944×1200
2x2 子采样	1224×1024	972×600
4x4 子采样	612×512	不支持

3D 捕获分辨率

3D 捕获	Zivid 2+	Zivid 2
全分辨率 [1]	2448×2048	1944×1200
2x2 子采样 [1]	1224×1024	972×600
4x4 子采样 [1]	612×512	不支持

[1] (1,2,3)

The resolution of the 2D image obtained from the 3D capture can be different from the 3D data resolution.

Output resolution of 2D captures is controlled via the Sampling::Pixel and the output resolution of 3D captures via the combination of the Sampling::Pixel and the Processing::Resampling settings. See Pixel Sampling and Resampling（重采样）. This means that it is possible to no longer have a 1-to-1 correlation between a 2D pixel and a 3D point. Consequently, it is more challenging to extract the 3D data from a segmented mask in the 2D image.

As mentioned, it is common to require high-resolution 2D data for segmentation and detection. For example, our recommended preset for Consumer Goods（消费品） Z2+ MR130 Quality preset uses Sampling::Pixel set to by2x2. In this case we should either:

• 对 3D 数据进行上采样，以恢复 1:1 对应关系，或

• 将二维索引映射到子采样三维数据中的索引。

重采样（Resampling）

为了匹配 2D 捕获的分辨率，只需应用取消子采样的上采样即可。这样操作仍保留了子采样捕获的速度优势。例如：

C++

auto settings2D = Zivid::Settings2D{
    Zivid::Settings2D::Acquisitions{ Zivid::Settings2D::Acquisition{} },
    Zivid::Settings2D::Sampling::Pixel::all,
};
auto settings = Zivid::Settings{
    Zivid::Settings::Engine::phase,
    Zivid::Settings::Acquisitions{ Zivid::Settings::Acquisition{} },
    Zivid::Settings::Sampling::Pixel::blueSubsample2x2,
    Zivid::Settings::Sampling::Color::disabled,
    Zivid::Settings::Processing::Resampling::Mode::upsample2x2,
};

Python

settings_2d = zivid.Settings2D()
settings_2d.acquisitions.append(zivid.Settings2D.Acquisition())
settings_2d.sampling.pixel = zivid.Settings2D.Sampling.Pixel.all
settings = zivid.Settings()
settings.engine = "phase"
settings.acquisitions.append(zivid.Settings.Acquisition())
settings.sampling.pixel = zivid.Settings.Sampling.Pixel.blueSubsample2x2
settings.sampling.color = zivid.Settings.Sampling.Color.disabled
settings.processing.resampling.mode = zivid.Settings.Processing.Resampling.Mode.upsample2x2

更多相关的详细信息，请参阅 Resampling（重采样） 。

另一种选择是将 2D 索引映射到子采样 3D 数据中的索引。此方法稍微复杂一些，但可能更高效。点云可以保持子采样，因此消耗更少的内存和处理能力。

为了建立全分辨率 2D 图像与子采样点云之间的关联，需要一种特定的映射技术。此过程涉及从与拜耳网格中的蓝色或红色像素相对应的像素中提取 RGB 值。

Zivid::Experimental::Calibration::pixelMapping(camera, settings); 可用于获取执行此映射所需的参数。以下是使用此函数的示例。

C++

const auto pixelMapping = Zivid::Experimental::Calibration::pixelMapping(camera, settings);
std::cout << "Pixel mapping: " << pixelMapping << std::endl;
cv::Mat mappedBGR(
    fullResolutionBGR.rows / pixelMapping.rowStride(),
    fullResolutionBGR.cols / pixelMapping.colStride(),
    CV_8UC3);
std::cout << "Mapped width: " << mappedBGR.cols << ", height: " << mappedBGR.rows << std::endl;
for(size_t row = 0; row < static_cast<size_t>(fullResolutionBGR.rows - pixelMapping.rowOffset());
    row += pixelMapping.rowStride())
{
    for(size_t col = 0; col < static_cast<size_t>(fullResolutionBGR.cols - pixelMapping.colOffset());
        col += pixelMapping.colStride())
    {
        mappedBGR.at<cv::Vec3b>(row / pixelMapping.rowStride(), col / pixelMapping.colStride()) =
            fullResolutionBGR.at<cv::Vec3b>(row + pixelMapping.rowOffset(), col + pixelMapping.colOffset());
    }
}
return mappedBGR;

Python

pixel_mapping = calibration.pixel_mapping(camera, settings)
return rgba[
    int(pixel_mapping.row_offset) :: pixel_mapping.row_stride,
    int(pixel_mapping.col_offset) :: pixel_mapping.col_stride,
    0:3,
]

备注

如果您使用内部函数并且 2D 和 3D 捕获具有不同的分辨率，请确保正确地使用它们。请参阅 相机内参 了解更多信息。

关于外部光源的考量

The ideal light source for a 2D capture is strong, because it reduces the influence of varying ambient light, and diffuse, because this limits the blooming effects. For Zivid cameras, this light source comes from the internal projector. Therefore, you do not need any additional lighting in your robot cell.

You may encounter blooming when utilizing the internal projector as light source. Tilting the camera, changing the background, or tuning the 2D acquisition settings can mitigate the blooming effect.

Exposure variations caused by changes in ambient light, such as transitions from day to night, doors opening and closing, or changes in ceiling lighting, affects 2D and 3D data differently. For 2D data, they can impact segmentation performance, especially when it is trained on specific datasets. For 3D data, exposure variations may affect point cloud completeness due to varying noise levels. Zivid cameras are very robust to such exposure variations.

The below table summarizes the pros and cons of using Zivid camera with respect to 2D quality.

	内置投影仪
机器人单元设置	简单的
对环境光变化的适应能力	Strong
2D图像出现光晕现象	可能
需要设置 2D 白平衡	否

捕获策略

There are three capture strategies depending on which data (2D or 3D) you need first.

• 2D数据先于3D数据

• 2D数据作为如何使用3D数据的一部分

• 2D数据在使用3D数据之后

Which strategy you should go for depends on your machine vision algorithms and pipeline. We recommend 2D data before 3D data (taking a 2D capture first, followed by a 3D capture with color disabled). This approach allows you to process the color image (e.g., segmentation) in parallel with capturing 3D data, thus achieving the best pick rates on the system level.

Below we summarize the performance of the different strategies. For a more in-depth understanding and comprehensive ZividBenchmarks, please see 2D+3D捕获策略.

The following tables list the different 2D+3D capture configurations. It shows how they are expected to perform relative to each other with respect to speed.

捕获周期	速度
	Zivid 2	Zivid 2+	Zivid 2+ R
	较快	快速	最快
3D ➞ 2D / 2D ➞ 3D	快速	快速	较快
3D（启用 w/RGB）	较快	较快	最快

下表列出了不同硬件上的实际测量结果。这里我们使用了Fast Consumer Goods的预设设置进行3D捕获。

Zivid 2+

(Z2+ MR130 Fast)

( Z2+ M130 Fast )

Zivid 2

( Z2 M70 Fast )

小技巧

如需在 PC 上测试不同的 2D-3D 策略，您可以运行 ZividBenchmark.cpp 示例，并使用从 YML 文件加载的设置。您可以在 示例 页面选择 C++ 选项查看说明。

在下一章节中，我们将介绍如何 基于捕获速度选择3D和2D设置 。

版本历史

SDK	变更
2.12.0	对于 Zivid 2+，2D 捕获的采集时间最多可减少 50%，3D 捕获的采集时间最多可减少 5%。请注意，Zivid One+已达到其生命周期终点，不再提供支持。
2.11.0	添加了对 redSubsample4x4 和 blueSubsample4x4 的支持。