2D + 3D Capture Strategy

애플리케이션에 2D 및 3D 데이터가 모두 필요한 경우 이 튜토리얼이 도움이 됩니다.

다양한 2D-3D 캡처 접근 방식의 장단점을 설명 및 강조하고, 몇 가지 제한 사항을 명확히 하고, 사이클 시간에 미치는 영향을 설명합니다. 2D용 외부 조명과 내부 프로젝터 사용의 차이점에 대해 다룹니다.

2D data

RGB image

3D data

Point Cloud

2D 데이터를 얻는 방법에는 크게 두 가지가 있습니다.

1. Independently via camera.capture2D(Zivid::Settings2D or Zivid::Settings ).imageSRGB(), see 2D Image Capture Process.

2. As part of 2D3D capture frame = camera.capture2D3D(Zivid::Settings):

   1. frame.pointCloud().copyImageSRGB(). This ensures 1-to-1 mapping between 2D and 3D data.

   2. frame.frame2D().imageSRGB(). This is the same as if you would have captured 2D independently.

그러나 어느 것을 사용할지는 요구 사항에 따라 다릅니다.

다른 시나리오는 다른 장단점으로 이어질 것입니다. 먼저 필요한 데이터를 기준으로 분류합니다. 그런 다음 다양한 시나리오에 대한 속도와 품질의 장단점에 대해 논의합니다.

• 2D data before 3D data 의 경우

• 2D data as part of how I use 3D data 의 경우

• 2D data after I have used the 3D data 의 경우

About external light

다양한 전략을 살펴보기 전에 외부 광원에 대해 먼저 설명해야 합니다. 2D 캡처에 이상적인 광원은 blooming effects 를 제한하기 때문에 강하고 확산되는 광원입니다. 내부 프로젝터를 광원으로 사용하면 블루밍 효과는 거의 피할 수 없습니다. Mounting the camera at an angle 하면 이 효과가 크게 줄어들지만 여전히 외부 확산 광원이 더 좋습니다. 외부 조명은 3D 데이터에 노이즈를 유발하므로 3D 캡처 중에는 외부 조명을 끄는 것이 이상적입니다.

블루밍 효과의 감소 외에도 강한 외부 조명은 주변 조명의 변화로 인한 노출 변화를 완화할 수 있습니다. 주변광 변화의 일반적인 예시:

• 일광의 변화(낮/밤, 구름 등)

• 도어 개폐

• 천장 조명의 켜짐 및 꺼짐

참고

Zivid 2+R is largely resilient to changes in ambient light.

Considerations for Zivid 2+ M60/M130/L110 and Zivid 2

노출의 이러한 변화는 3D 및 2D 데이터에 다르게 영향을 미칩니다. 2D 데이터에서 노출 변화의 영향은 사용된 감지 알고리즘에 따라 다릅니다. 분할이 2D에서 수행되는 경우 이러한 변형은 분할 성능에 영향을 미칠 수도 있고 그렇지 않을 수도 있습니다. 포인트 클라우드의 경우 노이즈 변동으로 인해 포인트 클라우드 완성도에 변동이 있을 수 있습니다.

이것은 다음과 같은 질문으로 이어집니다. 2D에 프로젝터를 사용해야 합니까?

Check out Optimizing Color Image for more information on this topic.

2D data before 3D data

예를 들어 2D에서 분할을 수행한 다음 나중에 피킹 포즈를 결정하는 경우 3D보다 빠른 2D가 필요합니다. 2D 데이터를 얻는 가장 빠른 방법은 별도의 2D 캡처를 이용하는 것입니다. 따라서 3D 데이터보다 먼저 2D 데이터가 필요한 경우에는 별도의 2D 캡처를 수행해야 합니다.

경고

Some camera reconfiguration may be needed when switching between 2D and 3D captures. This can lead to an increase in capture time for 2D+3D, where captures happen right after each other. In particular:

• If possible, keep the aperture of the 2D acquisition equal to the 3D acquisition(s).

• Consider using the same sampling setting in 2D and 3D, see Sampling (2D) and Sampling (3D).

  • The penalty is 35ms for switching sampling setting.

  • The gain from using low resolution in 3D is often larger than the penalty, in particular for challenging scenes.

다음 코드 샘플은 아래의 내용을 수행하는 방법을 보여줍니다.

1. 2D 캡처

2. 2D 데이터를 사용하고 3D를 동시에 캡처

C++

source

const auto frame2dAndCaptureTime = captureAndMeasure2D(camera, settings2D);
std::future<Duration> userThread =
    std::async(std::launch::async, useFrame<Zivid::Frame2D>, std::ref(frame2dAndCaptureTime.frame));
const auto frameAndCaptureTime = captureAndMeasure3D(camera, settings2D3D);
const auto processTime = useFrame(frameAndCaptureTime.frame);
const auto processTime2D = userThread.get();

The following shows actual benchmark numbers. You will find a more extensive table at the bottom of the page.

Details about configurations

Display Options

Standard Deviation

Capture 2D + 3D (hide)

hidden

hidden

	2+R		2+		2
	High-end NVIDIA		High-end NVIDIA		High-end NVIDIA
	2D	3D	2D	3D	2D	3D
Consumer Goods Settings
Consumer Goods Fast	22 ms (±0)120 ms (±1)135 ms (±1)		23 ms (±0)470 ms (±1)480 ms (±1)
Consumer Goods Quality	22 ms (±0)250 ms (±2)265 ms (±2)		50 ms (±3)885 ms (±2)910 ms (±4)

3D 데이터를 사용해서 얻은 2D 데이터

In this case, we don’t have to get access to the 2D data before the 3D data. Thus we only have to care about overall speed and quality.

Speed

For optimal speed, we should avoid switching penalties.

• Keep the aperture of the 2D acquisition equal to the 3D acquisition(s).

• Use the same sampling setting in 2D and 3D, see Sampling (2D) and Sampling (3D).

  • The penalty is 35ms for switching sampling setting.

  • The gain from using low resolution in 3D is often larger than the penalty, in particular for challenging scenes.

2D Quality

For optimal 2D quality tune 2D settings according to your requirements. See Optimizing Color Image for more information on this topic.

The following shows actual benchmark numbers. You will find a more extensive table at the bottom of the page.

Details about configurations

hidden

	2+R	2+	2
	High-end NVIDIA	High-end NVIDIA	High-end NVIDIA
Consumer Goods Settings
Consumer Goods Fast	130 ms (±2)	475 ms (±2)	310 ms (±18)
Consumer Goods Quality	260 ms (±1)	905 ms (±6)	455 ms (±19)

2D data after I have used the 3D data

The table below shows 3D+2D capture time examples.

Details about configurations

hidden

hidden

	2+R				2+				2
	High-end NVIDIA		High-end Intel		High-end NVIDIA		High-end Intel		High-end NVIDIA		High-end Intel
	2D	3D	2D	3D	2D	3D	2D	3D	2D	3D	2D	3D
Capture 3D and then 2D
Projector On for 2D					44 ms155 ms190 ms		60 ms200 ms205 ms		55 ms315 ms315 ms		95 ms800 ms800 ms
Projector Off for 2D					42 ms155 ms185 ms		60 ms200 ms205 ms		55 ms315 ms315 ms		95 ms800 ms800 ms

Camera resolution and 1-to-1 mapping

For accurate 2D segmentation and detection, it is beneficial with a high-resolution color image. Zivid 2+ has a 5 MPx imaging sensor, while Zivid 2 has a 2.3 MPx sensors. The following table shows the resolution outputs of the different cameras for both 2D and 3D captures.

2D capture resolutions

2D capture	Zivid 2+	Zivid 2
Full resolution	2448 x 2048	1944 x 1200
2x2 subsampled	1224 x 1024	972 x 600
4x4 subsampled	612 x 512	Not available

3D capture resolutions

3D capture	Zivid 2+	Zivid 2
Full resolution	2448 x 2048	1944 x 1200
2x2 subsampled	1224 x 1024	972 x 600
4x4 subsampled	612 x 512	Not available

When performing a capture2D3D() capture the result is a frame that contains both 2D and 3D data.

2D data can be extracted in two ways:

frame.frame2D().imageSRGB()

This is the same as if you would have captured 2D independently.

frame.pointCloud().copyImageSRGB()

This will ensure 1-to-1 mapping, even in cases where the settings define that 2D and 3D should have different resolutions.

Output resolution of 2D captures is controlled via the Settings2D::Sampling::Pixel setting and the output resolution of 3D captures via the combination of the Settings::Sampling::Pixel and the Settings::Processing::Resampling settings. See Pixel Sampling (2D), Pixel Sampling (3D) and Resampling.

As mentioned, it is common to require high-resolution 2D data for segmentation and detection. For example, our recommended preset for Consumer Goods Z2+ MR130 Quality preset uses Sampling::Pixel set to by2x2. In this case we should either:

• Upsample the 3D data to restore 1-to-1 correspondence, or

• Map 2D indices to the indices in the subsampled 3D data, or

• Get 2D data from the pointcloud via frame.pointCloud().copyImageSRGB()

Resampling

In order to match the resolution of the 2D capture, simply apply an upsampling which undoes the subsampling. This retains the speed advantages of the subsampled capture. For example:

C++

auto settings2D = Zivid::Settings2D{
    Zivid::Settings2D::Acquisitions{ Zivid::Settings2D::Acquisition{} },
    Zivid::Settings2D::Sampling::Pixel::all,
};
auto settings = Zivid::Settings{
    Zivid::Settings::Engine::phase,
    Zivid::Settings::Acquisitions{ Zivid::Settings::Acquisition{} },
    Zivid::Settings::Sampling::Pixel::blueSubsample2x2,
    Zivid::Settings::Sampling::Color::disabled,
    Zivid::Settings::Processing::Resampling::Mode::upsample2x2,
};

Python

settings_2d = zivid.Settings2D()
settings_2d.acquisitions.append(zivid.Settings2D.Acquisition())
settings_2d.sampling.pixel = zivid.Settings2D.Sampling.Pixel.all
settings = zivid.Settings()
settings.engine = "phase"
settings.acquisitions.append(zivid.Settings.Acquisition())
settings.sampling.pixel = zivid.Settings.Sampling.Pixel.blueSubsample2x2
settings.sampling.color = zivid.Settings.Sampling.Color.disabled
settings.processing.resampling.mode = zivid.Settings.Processing.Resampling.Mode.upsample2x2

For more details see Resampling.

Mapping pixel indices between different resolutions

The other option is to map the 2D indices to the indices in the subsampled 3D data. This option is a bit more complicated, but it is potentially more efficient. The point cloud can remain subsampled, and thus consume less memory and processing power.

To establish a correlation between the full-resolution 2D data and the subsampled point cloud, a specific mapping technique is required. This process involves extracting RGB values from the pixels that correspond to the Blue or Red pixels from the Bayer grid.

Zivid::Experimental::Calibration::pixelMapping(camera, settings); can be used to get parameters required to perform this mapping. Following is an example which uses this function.

C++

const auto pixelMapping = Zivid::Experimental::Calibration::pixelMapping(camera, settings);
std::cout << "Pixel mapping: " << pixelMapping << std::endl;
cv::Mat mappedBGR(
    fullResolutionBGR.rows / pixelMapping.rowStride(),
    fullResolutionBGR.cols / pixelMapping.colStride(),
    CV_8UC3);
std::cout << "Mapped width: " << mappedBGR.cols << ", height: " << mappedBGR.rows << std::endl;
for(size_t row = 0; row < static_cast<size_t>(fullResolutionBGR.rows - pixelMapping.rowOffset());
    row += pixelMapping.rowStride())
{
    for(size_t col = 0; col < static_cast<size_t>(fullResolutionBGR.cols - pixelMapping.colOffset());
        col += pixelMapping.colStride())
    {
        mappedBGR.at<cv::Vec3b>(row / pixelMapping.rowStride(), col / pixelMapping.colStride()) =
            fullResolutionBGR.at<cv::Vec3b>(row + pixelMapping.rowOffset(), col + pixelMapping.colOffset());
    }
}
return mappedBGR;

Python

pixel_mapping = calibration.pixel_mapping(camera, settings)
return rgba[
    int(pixel_mapping.row_offset) :: pixel_mapping.row_stride,
    int(pixel_mapping.col_offset) :: pixel_mapping.col_stride,
    0:3,
]

For more details about mapping (example for blueSubsample2x2)

In order to extract all the RGB values which correspond to the blue pixels we use the indices:

$$\begin{array}{r}\left[\begin{array}{ccccc}(0,0)& (0,2)& (0,4)& (0,...)& (0,W)\\ (2,0)& (2,2)& (2,4)& (2,...)& (2,W)\\ (4,0)& (4,2)& (4,4)& (4,...)& (4,W)\\ (...,0)& (...,2)& (...,4)& (...,...)& (...,...)\\ (H,0)& (H,2)& (H,4)& (H,...)& (H,W)\end{array}\right]\text{, where }\begin{array}{c}W=width-2\\ H=height-2\end{array}\end{array}$$

In order to extract all the RGB values which correspond to the red pixels we use the indices:

$$\begin{array}{r}\left[\begin{array}{ccccc}(1,1)& (1,3)& (1,5)& (1,...)& (1,W)\\ (3,1)& (3,3)& (3,5)& (3,...)& (3,W)\\ (5,1)& (5,3)& (5,5)& (5,...)& (5,W)\\ (...,1)& (...,3)& (...,5)& (...,...)& (...,...)\\ (H,1)& (H,3)& (H,5)& (H,...)& (H,W)\end{array}\right]\text{, where }\begin{array}{c}W=width-1\\ H=height-1\end{array}\end{array}$$

참고

If you use intrinsics and 2D and 3D capture have different resolutions, ensure you use them correctly. See Camera Intrinsics for more information.

Summary

Our recommendation:

• 2D capture with full resolution

• 3D capture with subsampled resolution

참고

Subsampling or downsampling in user code is only necessary if you want to have 1-to-1 pixel correspondence when you capture and copy 2D and 3D with different resolutions.

Following is a table showing actual measurements on different hardware.

Zivid 2+

(Z2+ LR110 Fast)

(Z2+ L110 Fast)

Zivid 2

(Z2 M70 Fast)

팁

PC에서 다양한 2D-3D 전략을 테스트하려면 YML 파일에서 로드된 설정으로 ZividBenchmark.cpp 샘플을 실행할 수 있습니다. Samples 로 이동하고 튜토리얼을 보려면 C++를 선택합니다.

Cameras

Configurations

Low-end Intel High-end NVIDIA High-end Intel Low-end Jetson NVIDIA High-end Jetson NVIDIA

Details about configurations

Display Options

Standard Deviation

Capture 2D + 3D (hide)

Capture 2D and then 3D

Capture 3D and then 2D

Capture 3D including 2D

hidden

hidden

	2+R		2+
	High-end NVIDIA		High-end NVIDIA
	2D	3D	2D	3D
Capture 2D and then 3D
Projector On for 2D	47 ms290 ms335 ms		23 ms180 ms190 ms
Projector Off for 2D			21 ms175 ms190 ms
Capture 3D and then 2D
Projector On for 2D			44 ms155 ms190 ms
Projector Off for 2D			42 ms155 ms185 ms
Capture 3D including 2D
Projector Off for 2D			0 ms190 ms0 ms

Version History

SDK	Changes
2.14.0	Added support for Zivid 2+ MR130, LR110, and MR60, which are faster and better fit for piece picking than older Zivid 2+ models.
2.12.0	Acquisition time is reduced by up to 50% for 2D captures and up to 5% for 3D captures for Zivid 2+. Zivid One+ has reached its End-of-Life and is no longer supported; thus, most of the complexities related to 2D+3D captures are no longer applicable.
2.11.0	Zivid 2 and Zivid 2+ now support concurrent processing and acquisition for 3D ➞ 2D and 3D ➞ 2D, and switching between capture modes have been optimized.