2D+3D 捕获策略

Many detection algorithms commonly used in piece-picking applications rely on 2D data to identify which object to pick. This article explains the different ways to acquire that 2D data, their tradeoffs in data quality and cycle time, and how external lighting affects the result.

If you do not care about color information, skip ahead to selecting 3D and 2D settings based on capture speed.

获取2D数据有两种方法：

1. 通过 camera.capture2D(Zivid::Settings).imageRGBA() 进行单独的 2D 捕获，请参阅 2D图像捕获流程 。

2. 3D 捕获 camera.capture2D3D(Zivid::Settings).pointCloud().copyImageRGBA() 中的一部分，请参阅 点云捕获过程 。

Which one to use depends on your requirements and the machine vision pipeline. We advocate for a dedicated 2D capture as it can use multi-threading and optimized scheduling. Utilizing 2D data from the 3D capture is simpler, but you may have to compromise speed to get desired 2D quality.

小技巧

When you capture 2D separately, disable RGB in the 3D capture. This saves both on acquisition and processing time. Disable RGB in 3D capture by setting Sampling::Color to disabled.

我们的建议：

• 单独的全分辨率 2D 捕获并打开投影仪。

• 不包含颜色数据的子采样 3D 捕获。

相机分辨率和一对一映射

For accurate 2D segmentation and detection, it is beneficial with a high-resolution color image. Zivid 3 has a 8 MPx imaging sensor, Zivid 2+ a 5 MPx sensor, while Zivid 2 has a 2.3 MPx sensor. The following table shows the resolution outputs of the different cameras for both 2D and 3D captures.

2D 捕获辨率

2D 捕获	Zivid 3	Zivid 2+	Zivid 2
全分辨率	2816 x 2816	2448×2048	1944×1200
2x2 子采样	1408 x 1408	1224×1024	972×600
4x4 子采样	704 x 704	612×512	不支持

3D 捕获分辨率

3D 捕获	Zivid 3	Zivid 2+	Zivid 2
全分辨率	2816 x 2816	2448×2048	1944×1200
2x2 子采样	1408 x 1420	1224×1024	972×600
4x4 子采样	704 x 704	612×512	不支持

执行 capture2D3D() 捕获时，结果是包含 2D 和 3D 数据的 frame 。

2D 数据可以通过两种方式提取：

frame.frame2D().imageRGBA_SRGB()

这与您单独捕获 2D 是一样的。

frame.pointCloud().copyImageRGBA_SRGB()

这可以确保即使将 2D 和 3D 设置为具有不同分辨率的情况下，它们也是 1:1 映射的。

Output resolution of 2D captures is controlled via the Settings2D::Sampling::Pixel setting and the output resolution of 3D captures via the combination of the Settings::Sampling::Pixel and the Settings::Processing::Resampling settings. See Pixel Sampling (2D), Pixel Sampling (3D) and Resampling（重采样）.

For accurate 2D segmentation and detection, it is common to require high-resolution 2D data. For example, our recommended preset for Consumer Goods（消费品） Z2+ MR130 Quality preset uses Settings::Sampling::Pixel set to by2x2. In this case, either:

• 对 3D 数据进行上采样，以恢复 1:1 对应关系，或

• 将二维索引映射到子采样三维数据中的索引，或者

• 通过 frame.pointCloud().copyImageRGBA_SRGB() 从点云获取 2D 数据

重采样（Resampling）

To match the resolution of the 2D capture, apply an upsampling which undoes the subsampling. This retains the speed advantages of the subsampled capture. For example:

C++

auto settings2D = Zivid::Settings2D{
    Zivid::Settings2D::Acquisitions{ Zivid::Settings2D::Acquisition{} },
    Zivid::Settings2D::Sampling::Pixel::all,
};
auto settings = Zivid::Settings{
    Zivid::Settings::Engine::stripe,
    Zivid::Settings::Acquisitions{ Zivid::Settings::Acquisition{} },
    Zivid::Settings::Sampling::Pixel::blueSubsample2x2,
    Zivid::Settings::Sampling::Color::disabled,
    Zivid::Settings::Processing::Resampling::Mode::upsample2x2,
};

Python

settings_2d = zivid.Settings2D()
settings_2d.acquisitions.append(zivid.Settings2D.Acquisition())
settings_2d.sampling.pixel = zivid.Settings2D.Sampling.Pixel.all
settings = zivid.Settings()
settings.engine = "stripe"
settings.acquisitions.append(zivid.Settings.Acquisition())
settings.sampling.pixel = zivid.Settings.Sampling.Pixel.blueSubsample2x2
settings.sampling.color = zivid.Settings.Sampling.Color.disabled
settings.processing.resampling.mode = zivid.Settings.Processing.Resampling.Mode.upsample2x2

更多相关的详细信息，请参阅 Resampling（重采样） 。

另一种选择是将 2D 索引映射到子采样 3D 数据中的索引。此方法稍微复杂一些，但可能更高效。点云可以保持子采样，因此消耗更少的内存和处理能力。

为了建立全分辨率 2D 数据与子采样点云之间的关联，需要一种特定的映射技术。此过程涉及从与拜耳网格中的蓝色或红色像素相对应的像素中提取 RGB 值。

Zivid::Experimental::Calibration::pixelMapping(camera, settings); 可用于获取执行此映射所需的参数。以下是使用此函数的示例。

C++

const auto pixelMapping = Zivid::Experimental::Calibration::pixelMapping(camera, settings);
std::cout << "Pixel mapping: " << pixelMapping << std::endl;
cv::Mat mappedBGR(
    fullResolutionBGR.rows / pixelMapping.rowStride(),
    fullResolutionBGR.cols / pixelMapping.colStride(),
    CV_8UC3);
std::cout << "Mapped width: " << mappedBGR.cols << ", height: " << mappedBGR.rows << std::endl;
for(size_t row = 0; row < static_cast<size_t>(fullResolutionBGR.rows - pixelMapping.rowOffset());
    row += pixelMapping.rowStride())
{
    for(size_t col = 0; col < static_cast<size_t>(fullResolutionBGR.cols - pixelMapping.colOffset());
        col += pixelMapping.colStride())
    {
        mappedBGR.at<cv::Vec3b>(row / pixelMapping.rowStride(), col / pixelMapping.colStride()) =
            fullResolutionBGR.at<cv::Vec3b>(row + pixelMapping.rowOffset(), col + pixelMapping.colOffset());
    }
}
return mappedBGR;

Python

pixel_mapping = calibration.pixel_mapping(camera, settings)
return rgba[
    int(pixel_mapping.row_offset) :: pixel_mapping.row_stride,
    int(pixel_mapping.col_offset) :: pixel_mapping.col_stride,
    0:3,
]

备注

如果您会使用到相机内参，并且 2D 和 3D 捕获具有不同的分辨率，请确保正确地使用它们。请参阅 相机内参 了解更多信息。

关于外部光源的考量

对于 2D 捕获而言，理想的光源应当是强光，因为它能降低环境光变化带来的影响；同时也应当是漫射光，因为这样可以限制 泛光效应（blooming effects）。

You can encounter blooming when using the internal projector as light source. Tilting the camera, changing the background, or tuning the 2D acquisition settings can mitigate the blooming effect.

Exposure variations caused by changes in ambient light, such as transitions from day to night, doors opening and closing, or changes in ceiling lighting, affects 2D and 3D data differently. For 2D data, they can impact segmentation performance, especially when it is trained on specific datasets. For 3D data, exposure variations may affect point cloud completeness due to varying noise levels. Zivid cameras are robust to such exposure variations.

下表总结了使用 Zivid 相机在 2D 质量方面的优缺点。

	内置投影仪
机器人单元设置	简单
对环境光变化的适应能力	强
2D图像出现光晕现象	可能
需要设置 2D 白平衡	否

捕获策略

根据您首先需要哪种数据（2D 或 3D），有三种捕获策略。

• 2D数据先于3D数据

• 2D数据作为如何使用3D数据的一部分

• 2D数据在使用3D数据之后

Which strategy fits best depends on your machine vision algorithms and pipeline. We recommend 2D data before 3D data (taking a 2D capture first, followed by a 3D capture with color disabled). This approach allows you to process the color image (e.g., segmentation) in parallel with capturing 3D data, thus achieving the best pick rates on the system level.

Below we summarize the performance of the different strategies. For a more in-depth understanding and ZividBenchmarks, please see 2D+3D捕获策略.

下表显示了不同硬件上的实际测量结果。

Zivid 3

（ Z3 XL250 Fast ）

Zivid 2+

（ Z2+ LR110 Fast ）

（ Z2+ L110 Fast ）

Zivid 2

（ Z2 M70 Matte ）

小技巧

如需在 PC 上测试不同的 2D-3D 策略，您可以运行 ZividBenchmark.cpp 示例，并使用从 YML 文件加载的设置。您可以在 示例 页面选择 C++ 选项查看说明。

Next, see selecting 3D and 2D settings based on capture speed.

Version History

SDK	变更
2.12.0	对于 Zivid 2+，2D 捕获的采集时间最多可减少 50%，3D 捕获的采集时间最多可减少 5%。请注意，Zivid One+已达到其生命周期终点，不再提供支持。
2.11.0	添加了对 redSubsample4x4 和 blueSubsample4x4 的支持。