2D+3D捕获策略

如果您的应用同时需要2D和3D数据，那么您可以参考本教程。

我们解释并强调了不同的2D-3D捕获方法的优缺点，阐明了一些限制，并解释了它们如何影响循环时间。我们也谈到了使用外部光源和使用内部投影仪来拍摄2D图像的区别。

2D数据

RGB图像

3D数据

点云

获取2D数据有两种不同的方式：

1. Independently via camera.capture2D(Zivid::Settings2D or Zivid::Settings ).imageSRGB(), see 2D图像捕获流程.

2. As part of 2D3D capture frame = camera.capture2D3D(Zivid::Settings):

   1. frame.pointCloud().copyImageSRGB(). This ensures 1-to-1 mapping between 2D and 3D data.

   2. frame.frame2D().imageSRGB(). This is the same as if you would have captured 2D independently.

使用哪一种方式来获取2D数据取决于您的要求。

不同的场景会导致不同的权衡。我们首先根据您需要的数据对其进行细分。然后我们将讨论不同场景下捕获速度与图像质量的权衡。

• 我需要 2D data before 3D data

• 我需要 2D data as part of how I use 3D data

• 我只需要 2D data after I have used the 3D data

关于外部光源

在我们讨论不同的策略之前，我们需要讨论外部光源。2D捕获的理想光源是强漫射光，因为这会限制 光晕效果 。使用内部投影仪作为光源时，光晕效果几乎是不可避免的。 以一定角度安装相机 可显着降低这种影响，但外部漫射光源仍然会提供更好的效果。外部光源会导致3D数据中出现额外噪声，因此最好在3D采集期间关闭外部光源。

除了减少光晕效应外，高照度的外部光源还可以消除由环境光变化引起的曝光变化。环境光变化的典型来源有：

• 日光变化（白天/黑夜、云等）

• 开门和关门

• 顶灯打开和关闭

备注

Zivid 2+R is largely resilient to changes in ambient light.

Considerations for Zivid 2+ M60/M130/L110 and Zivid 2

这些曝光变化对3D和2D数据的影响不同。2D数据中曝光变化的影响取决于所使用的检测算法。如果在2D中执行分割，这些变化可能会，也可能不会影响分割性能。对于点云，您可能会发现由于噪声的变化而导致点云完整性的变化。

这就引出了一个问题：我们是否应该使用相机的投影仪来捕获2D图像？

Check out 优化彩色图像 for more information on this topic.

2D数据先于3D数据

如果您在2D中执行分割，然后确定您的抓取位姿，那么您需要在获取3D之前得到2D数据。获取2D数据的最快方法是使用单独的2D捕获。因此，如果您在使用3D数据之前就需要使用2D数据，那么您应该先执行单独的2D捕获。

警告

Some camera reconfiguration may be needed when switching between 2D and 3D captures. This can lead to an increase in capture time for 2D+3D, where captures happen right after each other. In particular:

• If possible, keep the aperture of the 2D acquisition equal to the 3D acquisition(s).

• Consider using the same sampling setting in 2D and 3D, see Sampling（采样）- 2D and Sampling（采样） - 3D.

  • The penalty is 35ms for switching sampling setting.

  • The gain from using low resolution in 3D is often larger than the penalty, in particular for challenging scenes.

以下代码示例展示了您可以如何：

1. 捕获2D图像

2. 使用2D数据和并行执行3D捕获

C++

源码

const auto frame2dAndCaptureTime = captureAndMeasure2D(camera, settings2D);
std::future<Duration> userThread =
    std::async(std::launch::async, useFrame<Zivid::Frame2D>, std::ref(frame2dAndCaptureTime.frame));
const auto frameAndCaptureTime = captureAndMeasure3D(camera, settings2D3D);
const auto processTime = useFrame(frameAndCaptureTime.frame);
const auto processTime2D = userThread.get();

下面显示了实际的基准数字。您会在页面底部找到更详细的表格。

Details about configurations

Display Options

Standard Deviation

Capture 2D + 3D (hide)

hidden

hidden

	2+R		2+		2
	High-end NVIDIA		High-end NVIDIA		High-end NVIDIA
	2D	3D	2D	3D	2D	3D
Consumer Goods Settings
Consumer Goods Fast	22 ms (±0)120 ms (±1)135 ms (±1)		23 ms (±0)470 ms (±1)480 ms (±1)
Consumer Goods Quality	22 ms (±0)250 ms (±2)265 ms (±2)		50 ms (±3)885 ms (±2)910 ms (±4)

2D数据作为如何使用3D数据的一部分

In this case, we don’t have to get access to the 2D data before the 3D data. Thus we only have to care about overall speed and quality.

速度

For optimal speed, we should avoid switching penalties.

• Keep the aperture of the 2D acquisition equal to the 3D acquisition(s).

• Use the same sampling setting in 2D and 3D, see Sampling（采样）- 2D and Sampling（采样） - 3D.

  • The penalty is 35ms for switching sampling setting.

  • The gain from using low resolution in 3D is often larger than the penalty, in particular for challenging scenes.

2D图像质量

For optimal 2D quality tune 2D settings according to your requirements. See 优化彩色图像 for more information on this topic.

下面显示了实际的基准数字。您会在页面底部找到更详细的表格。

Details about configurations

hidden

	2+R	2+	2
	High-end NVIDIA	High-end NVIDIA	High-end NVIDIA
Consumer Goods Settings
Consumer Goods Fast	130 ms (±2)	475 ms (±2)	310 ms (±18)
Consumer Goods Quality	260 ms (±1)	905 ms (±6)	455 ms (±19)

2D数据在使用3D数据之后

The table below shows 3D+2D capture time examples.

Details about configurations

hidden

hidden

	2+R				2+				2
	High-end NVIDIA		High-end Intel		High-end NVIDIA		High-end Intel		High-end NVIDIA		High-end Intel
	2D	3D	2D	3D	2D	3D	2D	3D	2D	3D	2D	3D
Capture 3D and then 2D
Projector On for 2D					44 ms155 ms190 ms		60 ms200 ms205 ms		55 ms315 ms315 ms		95 ms800 ms800 ms
Projector Off for 2D					42 ms155 ms185 ms		60 ms200 ms205 ms		55 ms315 ms315 ms		95 ms800 ms800 ms

相机分辨率和 1:1 映射

为了实现准确的 2D 分割和检测，高分辨率彩色图像非常有用。Zivid 2+ 配备了 5 MPx 成像传感器，而 Zivid 2 配备了 2.3 MPx 传感器。下表显示了不同相机在 2D 和 3D 捕获时的分辨率输出。

2D 捕获分辨率

2D 捕获	Zivid 2+	Zivid 2
全分辨率	2448 x 2048	1944 x 1200
2x2 子采样	1224 x 1024	972 x 600
4x4 子采样	612 x 512	不支持

3D 捕获分辨率

3D 捕获	Zivid 2+	Zivid 2
全分辨率	2448 x 2048	1944 x 1200
2x2 子采样	1224 x 1024	972 x 600
4x4 子采样	612 x 512	不支持

When performing a capture2D3D() capture the result is a frame that contains both 2D and 3D data.

2D data can be extracted in two ways:

frame.frame2D().imageSRGB()

This is the same as if you would have captured 2D independently.

frame.pointCloud().copyImageSRGB()

This will ensure 1-to-1 mapping, even in cases where the settings define that 2D and 3D should have different resolutions.

Output resolution of 2D captures is controlled via the Settings2D::Sampling::Pixel setting and the output resolution of 3D captures via the combination of the Settings::Sampling::Pixel and the Settings::Processing::Resampling settings. See Pixel Sampling (2D), Pixel Sampling (3D) and Resampling（重采样）.

如上所述，分割和检测通常需要高分辨率 2D 数据。例如，我们推荐的 Consumer Goods（消费品） Z2+ MR130 Quality 预设使用的 Sampling::Pixel 设置为 by2x2 。在这种情况下，我们应该：

• 对 3D 数据进行上采样，以恢复 1:1 对应关系，或

• Map 2D indices to the indices in the subsampled 3D data, or

• Get 2D data from the pointcloud via frame.pointCloud().copyImageSRGB()

重采样

为了匹配 2D 捕获的分辨率，只需应用取消子采样的上采样即可。这样操作仍保留了子采样捕获的速度优势。例如：

C++

auto settings2D = Zivid::Settings2D{
    Zivid::Settings2D::Acquisitions{ Zivid::Settings2D::Acquisition{} },
    Zivid::Settings2D::Sampling::Pixel::all,
};
auto settings = Zivid::Settings{
    Zivid::Settings::Engine::phase,
    Zivid::Settings::Acquisitions{ Zivid::Settings::Acquisition{} },
    Zivid::Settings::Sampling::Pixel::blueSubsample2x2,
    Zivid::Settings::Sampling::Color::disabled,
    Zivid::Settings::Processing::Resampling::Mode::upsample2x2,
};

Python

settings_2d = zivid.Settings2D()
settings_2d.acquisitions.append(zivid.Settings2D.Acquisition())
settings_2d.sampling.pixel = zivid.Settings2D.Sampling.Pixel.all
settings = zivid.Settings()
settings.engine = "phase"
settings.acquisitions.append(zivid.Settings.Acquisition())
settings.sampling.pixel = zivid.Settings.Sampling.Pixel.blueSubsample2x2
settings.sampling.color = zivid.Settings.Sampling.Color.disabled
settings.processing.resampling.mode = zivid.Settings.Processing.Resampling.Mode.upsample2x2

更多相关的详细信息，请参阅 Resampling（重采样） 。

在不同分辨率之间映射像素索引

另一种方法是将 2D 索引映射到子采样 3D 数据中的索引。此选项稍微复杂一些，但可能更高效。点云可以保持子采样，因此消耗更少的内存和处理能力。

To establish a correlation between the full-resolution 2D data and the subsampled point cloud, a specific mapping technique is required. This process involves extracting RGB values from the pixels that correspond to the Blue or Red pixels from the Bayer grid.

Zivid::Experimental::Calibration::pixelMapping(camera, settings); 可用于获取执行此映射所需的参数。以下是使用此函数的示例。

C++

const auto pixelMapping = Zivid::Experimental::Calibration::pixelMapping(camera, settings);
std::cout << "Pixel mapping: " << pixelMapping << std::endl;
cv::Mat mappedBGR(
    fullResolutionBGR.rows / pixelMapping.rowStride(),
    fullResolutionBGR.cols / pixelMapping.colStride(),
    CV_8UC3);
std::cout << "Mapped width: " << mappedBGR.cols << ", height: " << mappedBGR.rows << std::endl;
for(size_t row = 0; row < static_cast<size_t>(fullResolutionBGR.rows - pixelMapping.rowOffset());
    row += pixelMapping.rowStride())
{
    for(size_t col = 0; col < static_cast<size_t>(fullResolutionBGR.cols - pixelMapping.colOffset());
        col += pixelMapping.colStride())
    {
        mappedBGR.at<cv::Vec3b>(row / pixelMapping.rowStride(), col / pixelMapping.colStride()) =
            fullResolutionBGR.at<cv::Vec3b>(row + pixelMapping.rowOffset(), col + pixelMapping.colOffset());
    }
}
return mappedBGR;

Python

pixel_mapping = calibration.pixel_mapping(camera, settings)
return rgba[
    int(pixel_mapping.row_offset) :: pixel_mapping.row_stride,
    int(pixel_mapping.col_offset) :: pixel_mapping.col_stride,
    0:3,
]

有关映射的更多详细信息（例如 blueSubsample2x2 ）

为了提取与蓝色像素相对应的所有 RGB 值，我们使用以下索引：

$$\begin{array}{r}\left[\begin{array}{ccccc}(0,0)& (0,2)& (0,4)& (0,...)& (0,W)\\ (2,0)& (2,2)& (2,4)& (2,...)& (2,W)\\ (4,0)& (4,2)& (4,4)& (4,...)& (4,W)\\ (...,0)& (...,2)& (...,4)& (...,...)& (...,...)\\ (H,0)& (H,2)& (H,4)& (H,...)& (H,W)\end{array}\right]\text{, where }\begin{array}{c}W=width-2\\ H=height-2\end{array}\end{array}$$

为了提取与红色像素相对应的所有 RGB 值，我们使用以下索引：

$$\begin{array}{r}\left[\begin{array}{ccccc}(1,1)& (1,3)& (1,5)& (1,...)& (1,W)\\ (3,1)& (3,3)& (3,5)& (3,...)& (3,W)\\ (5,1)& (5,3)& (5,5)& (5,...)& (5,W)\\ (...,1)& (...,3)& (...,5)& (...,...)& (...,...)\\ (H,1)& (H,3)& (H,5)& (H,...)& (H,W)\end{array}\right]\text{, where }\begin{array}{c}W=width-1\\ H=height-1\end{array}\end{array}$$

备注

如果您需要使用内参，并且 2D 和 3D 捕获具有不同的分辨率，请确保正确使用它们。有关更多信息，请参阅 相机内参 。

总结

我们的建议：

• 使用全分辨率进行 2D 捕获

• 3D capture with subsampled resolution

备注

仅当您在捕获和复制具有不同分辨率的 2D 和 3D 时想要实现 1 对 1 像素对应关系时，才需要在用户代码中进行子采样或降采样。

Following is a table showing actual measurements on different hardware.

Zivid 2+

(Z2+ LR110 Fast)

(Z2+ L110 Fast)

Zivid 2

( Z2 M70 Fast )

小技巧

如需在 PC 上测试不同的 2D-3D 策略，您可以运行 ZividBenchmark.cpp 示例，并使用从 YML 文件加载的设置。您可以在 示例 页面选择 C++ 选项查看说明。

Cameras

Configurations

Low-end Intel High-end NVIDIA High-end Intel Low-end Jetson NVIDIA High-end Jetson NVIDIA

Details about configurations

Display Options

Standard Deviation

Capture 2D + 3D (hide)

Capture 2D and then 3D

Capture 3D and then 2D

Capture 3D including 2D

hidden

hidden

	2+R		2+
	High-end NVIDIA		High-end NVIDIA
	2D	3D	2D	3D
Capture 2D and then 3D
Projector On for 2D	47 ms290 ms335 ms		23 ms180 ms190 ms
Projector Off for 2D			21 ms175 ms190 ms
Capture 3D and then 2D
Projector On for 2D			44 ms155 ms190 ms
Projector Off for 2D			42 ms155 ms185 ms
Capture 3D including 2D
Projector Off for 2D			0 ms190 ms0 ms

版本历史

SDK	变更
2.14.0	增加了对 Zivid 2+ MR130、LR110 和 MR60 的支持，与旧款 Zivid 2+ 型号相比，它们速度更快，更适合拣选物品。
2.12.0	对于 Zivid 2+，2D 捕获的采集时间最多可减少 50%，3D 捕获的采集时间最多可减少 5%。请注意，Zivid One+已达到其生命周期终点，不再提供支持。因此，与 2D+3D 捕获相关的大多数复杂性不再适用。
2.11.0	Zivid 2 和 Zivid 2+ 现在支持 3D ➞ 2D 和 3D ➞ 2D 的并发处理和采集，并且优化了捕获模式之间的切换。