Point Cloud Tutorial

Introduction

이 튜토리얼은 Zivid SDK를 사용하여 Point Cloud 데이터를 처리하는 방법을 설명합니다.

팁

비디오 시청을 선호하는 경우 웨비나 Getting your point cloud ready for your application 에서 포인트 클라우드에 대해 다룹니다.

Prerequisites

• Zivid Software 를 설치합니다.

• Python의 경우: zivid-python 를 설치합니다.

Frame

C++

Zivid::Frame 은 포인트 클라우드 및 컬러 이미지(컴퓨팅 장치 메모리에 저장된)와 캡처 및 카메라 정보가 포함됩니다.

C#

Zivid.NET.Frame 은 포인트 클라우드 및 컬러 이미지(컴퓨팅 장치 메모리에 저장됨)와 캡처 및 카메라 정보가 포함됩니다.

Python

zivid.Frame 은 포인트 클라우드 및 컬러 이미지(컴퓨팅 장치 메모리에 저장됨)와 캡처 및 카메라 정보가 포함됩니다.

Capture

When you capture with Zivid, you get a frame in return. The point cloud is stored in the frame, and the frame is stored in the GPU memory. The capture can contain color or not, depending of the method that you call. For more information see this table with different capture modes.

Capture with color

If you want to capture a point cloud with color, you can use the Zivid::Camera::capture2D3D() method.

C++

source

const auto frame = camera.capture2D3D(settings);

C#

source

using (var frame = camera.Capture2D3D(settings))

Python

source

frame = camera.capture_2d_3d(settings)

Capture without color

If you want to capture a point cloud without color, you can use the Zivid::Camera::capture3D() method.

C++

source

const auto frame3D = camera.capture3D(settings);

C#

source

using (var frame3D = camera.Capture3D(settings))

Python

source

frame_3d = camera.capture_3d(settings)

캡처 방법에 대한 자세한 지침은 Capture Tutorial 에서 확인하십시오.

Load

Frame은 ZDF 파일에서 로드할 수도 있습니다.

C++

소스

const auto dataFile = std::string(ZIVID_SAMPLE_DATA_DIR) + "/Zivid3D.zdf";
std::cout << "Reading ZDF frame from file: " << dataFile << std::endl;
const auto frame = Zivid::Frame(dataFile);

C#

source

var dataFile =
    Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData) + "/Zivid/Zivid3D.zdf";
Console.WriteLine("Reading ZDF frame from file: " + dataFile);

using (var frame = new Zivid.NET.Frame(dataFile))
{

Python

source

data_file = get_sample_data_path() / "Zivid3D.zdf"
print(f"Reading point cloud from file: {data_file}")

frame = zivid.Frame(data_file)

Point Cloud

Get handle from Frame

이제 GPU에서 포인트 클라우드 데이터를 얻을 수 있습니다.

C++

소스

const auto pointCloud = frame.pointCloud();

C#

source

var pointCloud = frame.PointCloud;

Python

source

point_cloud = frame.point_cloud()

포인트 클라우드는 2D 그리드에 배치된 XYZ, RGB 및 SNR을 포함합니다.

더 많은 정보를 확인하려면 Point Cloud Structure 를 참고하세요.

C++

이 메서드 Zivid::Frame::pointCloud() 는 GPU 메모리에서 복사를 수행하지 않습니다.

참고

Zivid::Camera::capture2D3D() and Zivid::Camera::capture3D() methods return at some moment in time after the camera completes capturing raw images. The handle from Zivid::Frame::pointCloud() is available instantly. However, the actual point cloud data becomes available only after the processing on the GPU is finished. Any calls to data-copy functions (section below) will block and wait for processing to finish before proceeding with the requested copy operation.

자세한 설명은 다음을 참조하십시오. Point Cloud Capture Process.

C#

Zivid.NET.Frame.PointCloud 특성을 얻을 때는 GPU 메모리에서 복사를 수행하지 않습니다.

참고

Zivid.NET.Camera.Capture2D3D() and Zivid.NET.Camera.Capture3D() methods return at some moment in time after the camera completes capturing raw images. The handle from Zivid.NET.Frame.PointCloud is available instantly. However, the actual point cloud data becomes available only after the processing on the GPU is finished. Any calls to data-copy methods (section below) will block and wait for processing to finish before proceeding with the requested copy operation.

자세한 설명은 다음을 참조하십시오. Point Cloud Capture Process.

Python

zivid.frame.point_cloud() 함수는 GPU 메모리에서 복사를 수행하지 않습니다.

참고

zivid.camera.capture_2d_3d() and zivid.camera.capture_3d() methods return at some moment in time after the camera completes capturing raw images. The handle from zivid.frame.point_cloud() is available instantly. However, the actual point cloud data becomes available only after the processing on the GPU is finished. Any calls to data-copy functions (section below) will block and wait for processing to finish before proceeding with the requested copy operation.

자세한 설명은 다음을 참조하십시오. Point Cloud Capture Process.

Unorganized point cloud

It is possible to convert the organized point cloud to an unorganized point cloud. While doing so, all NaN values are removed, and the point cloud is flattened to a 1D array.

C++

source

const auto unorganizedPointCloud = frame.pointCloud().toUnorganizedPointCloud();

Python

source

unorganized_point_cloud = frame.point_cloud().to_unorganized_point_cloud()

Combining multiple unorganized point clouds

The unorganized point cloud can be extended with additional unorganized point clouds.

C++

source

stitchedPointCloud.extend(currentPointCloud.transform(transformationMatrixZivid));

Python

source

stitched_point_cloud.extend(current_point_cloud.transform(transformation_matrix))

Copy from GPU to CPU memory

You can now selectively copy data based on what is required. This is the complete list of output data formats and how to copy them from the GPU. Most of these APIs also applies to the unorganized point cloud.

C++

Return type	Copy functions	Data per pixel	Total data
Zivid::Array2D<Zivid::PointXYZ>	PointCloud::copyPointsXYZ() 또는 PointCloud::copyData<Zivid::PointXYZ>()	12 bytes	28 MB
Zivid::Array2D<Zivid::PointXYZW>	PointCloud::copyPointsXYZW() 또는 PointCloud::copyData<Zivid::PointXYZW>()	16 bytes	37 MB
Zivid::Array2D<Zivid::PointZ>	PointCloud::copyPointsZ() 또는 PointCloud::copyData<Zivid::PointZ>()	4 bytes	9 MB
Zivid::Array2D<Zivid::ColorRGBA>	PointCloud::copyColorsRGBA() 또는 PointCloud::copyData<Zivid::ColorRGBA>()	4 bytes	9 MB
Zivid::Array2D<Zivid::SNR>	PointCloud::copySNRs() 또는 PointCloud::copyData<Zivid::SNR>()	4 bytes	9 MB
Zivid::Array2D<Zivid::PointXYZColorRGBA>	PointCloud::copyData<PointXYZColorRGBA>()	16 bytes	37 MB
Zivid::Array2D<Zivid::PointXYZColorBGRA>	PointCloud::copyPointsXYZColorsBGRA() 또는 PointCloud::copyData<PointXYZColorBGRA>()	16 bytes	37 MB
Zivid::Image<Zivid::ColorRGBA>	PointCloud::copyImageRGBA()	4 bytes	9 MB
Zivid::Image<Zivid::ColorBGRA>	PointCloud::copyImageBGRA()	4 bytes	9 MB
Zivid::Image<Zivid::ColorsRGB>	PointCloud::copyImagesRGB()	4 bytes	9 MB

C#

Return type	Copy methods	Data per pixel	Total data
float[height,width, 3]	PointCloud.CopyPointsXYZ()	12 bytes	28 MB
float[height,width, 4]	PointCloud.CopyPointsXYZW()	16 bytes	37 MB
float[height,width, 1]	PointCloud.CopyPointsZ()	4 bytes	9 MB
byte[height,width, 4]	PointCloud.CopyColorsRGBA()	4 bytes	9 MB
float[height,width]	PointCloud.CopySNR()	4 bytes	9 MB
Zivid.NET.PointXYZColorRGBA[height,width]	PointCloud.CopyPointsXYZColorsRGBA()	16 bytes	37 MB
Zivid.NET.PointXYZColorBGRA[height,width]	PointCloud.CopyPointsXYZColorsBGRA()	16 bytes	37 MB
Zivid.NET.ImageRGBA	PointCloud.CopyImageRGBA()	4 bytes	9 MB
Zivid.NET.ImageBGRA	PointCloud.CopyImageBGRA()	4 bytes	9 MB
Zivid.NET.ImageSRGB	PointCloud.CopyImageSRGB()	4 bytes	9 MB

Python

Return type	Copy functions	Data per pixel	Total data
numpy.ndarray([height,width, 3], dtype=float32)	PointCloud.copy_data("xyz")	12 bytes	28 MB
numpy.ndarray([height,width, 3], dtype=float32)	PointCloud.copy_data("xyzw")	16 bytes	37 MB
numpy.ndarray([height,width], dtype=float32)	PointCloud.copy_data("z")	4 bytes	9 MB
numpy.ndarray([height,width, 4], dtype=uint8)	PointCloud.copy_data("rgba")	4 bytes	9 MB
numpy.ndarray([height,width, 4], dtype=uint8)	PointCloud.copy_data("bgra")	4 bytes	9 MB
numpy.ndarray([height,width, 4], dtype=uint8)	PointCloud.copy_data("srgb")	4 bytes	9 MB
numpy.ndarray([height,width], dtype=float32)	PointCloud.copy_data("snr")	4 bytes	9 MB
numpy.ndarray([height,width], dtype=[('x', '<f4'), ('y', '<f4'), ('z', '<f4'), (' r', 'u1'), ('g', 'u1'), ('b', 'u1'), ('a', 'u1')])	PointCloud.copy_data("xyzrgba")	16 bytes	37 MB

다음은 데이터를 복사하는 방법의 예입니다.

C++

소스

const auto data = pointCloud.copyData<Zivid::PointXYZColorRGBA_SRGB>();

C#

source

var pointCloudData = pointCloud.CopyPointsXYZColorsRGBA();

Python

source

xyz = point_cloud.copy_data("xyz")
rgba = point_cloud.copy_data("rgba_srgb")

Memory allocation options

메모리 할당 측면에서 데이터를 복사하는 두 가지 방법이 있습니다.

• Zivid SDK는 메모리 버퍼를 할당하고 데이터를 복사할 수 있습니다.

• 사용자는 사전 할당된 메모리 버퍼에 대한 포인터를 전달할 수 있으며 Zivid SDK는 데이터를 사전 할당된 메모리 버퍼에 복사합니다.

C++

OpenCV를 사용하는 두 가지 메모리 할당 옵션에 대한 예를 제시합니다.

Copy selected data from GPU to CPU memory (Zivid-allocated)

예를 들어 포인트 클라우드의 RGB 색상 데이터는 해당 데이터만 CPU 메모리에 복사할 수 있습니다.

source

std::cout << "Capturing frame" << std::endl;
frame = camera.capture2D3D(settings);

std::cout << "Copying colors with Zivid API from GPU to CPU" << std::endl;
auto colors = frame.frame2D().value().imageBGRA_SRGB();

std::cout << "Casting the data pointer as a void*, since this is what the OpenCV matrix constructor requires."
          << std::endl;
auto *dataPtrZividAllocated = const_cast<void *>(static_cast<const void *>(colors.data()));

std::cout << "Wrapping this block of data in an OpenCV matrix. This is possible since the layout of \n"
          << "Zivid::ColorBGRA_SRGB exactly matches the layout of CV_8UC4. No copying occurs in this step."
          << std::endl;
const cv::Mat bgraZividAllocated(colors.height(), colors.width(), CV_8UC4, dataPtrZividAllocated);

std::cout << "Displaying image" << std::endl;
cv::imshow("BGRA image Zivid Allocated", bgraZividAllocated);
cv::waitKey(CI_WAITKEY_TIMEOUT_IN_MS);

Copy selected data from GPU to CPU memory (user-allocated)

위 예에서 데이터의 소유권은 반환된 Zivid::Array2D<> 개체에 의해 유지됩니다. 또는 사전 Zivid::PointCloud::copyData(dataPtr) 에 할당된 메모리 버퍼를 제공할 수 있습니다. dataPtr 의 유형은 복사할 대상을 정의합니다(PointXYZ, ColorRGBA, 등.).

이제 위와 똑같은 사용 사례를 살펴보겠습니다. 그러나 이번에는 OpenCV가 필요한 스토리지를 할당하도록 허용합니다. 그런 다음 GPU에서 이 메모리 위치로 데이터를 직접 복사하도록 Zivid API에 요청합니다.

source

std::cout << "Allocating the necessary storage with OpenCV API based on resolution info before any capturing"
          << std::endl;
auto bgraUserAllocated = cv::Mat(resolution.height(), resolution.width(), CV_8UC4);

std::cout << "Capturing frame" << std::endl;
auto frame = camera.capture2D3D(settings);
auto pointCloud = frame.pointCloud();

std::cout << "Copying data with Zivid API from the GPU into the memory location allocated by OpenCV"
          << std::endl;
pointCloud.copyData(&(*bgraUserAllocated.begin<Zivid::ColorBGRA_SRGB>()));

std::cout << "Displaying image" << std::endl;
cv::imshow("BGRA image User Allocated", bgraUserAllocated);
cv::waitKey(CI_WAITKEY_TIMEOUT_IN_MS);

Copy unorganized point cloud data from GPU to CPU memory (Open3D-tensor)

source

open3d::t::geometry::PointCloud copyToOpen3D(const Zivid::UnorganizedPointCloud &pointCloud)
{
    using namespace open3d::core;
    auto device = Device("CPU:0");
    auto xyzTensor = Tensor({ static_cast<int64_t>(pointCloud.size()), 3 }, Dtype::Float32, device);
    auto rgbTensor = Tensor({ static_cast<int64_t>(pointCloud.size()), 3 }, Dtype::Float32, device);

    pointCloud.copyData(reinterpret_cast<Zivid::PointXYZ *>(xyzTensor.GetDataPtr<float>()));

    // Open3D does not store colors in 8-bit
    const auto rgbaColors = pointCloud.copyColorsRGBA_SRGB();
    for(size_t i = 0; i < pointCloud.size(); ++i)
    {
        const auto r = static_cast<float>(rgbaColors(i).r) / 255.0f;
        const auto g = static_cast<float>(rgbaColors(i).g) / 255.0f;
        const auto b = static_cast<float>(rgbaColors(i).b) / 255.0f;
        rgbTensor.SetItem(TensorKey::Index(i), Tensor::Init({ r, g, b }));
    }

    open3d::t::geometry::PointCloud cloud(device);
    cloud.SetPointPositions(xyzTensor);
    cloud.SetPointColors(rgbTensor);
    return cloud;
}

Transform

포인트 클라우드의 원점을 카메라에서 로봇 기본 프레임 또는 예: scale the point cloud by transforming it from mm to m 으로 변경하기 위해 transform 를 원할 수 있습니다.

C++

source

pointCloud.transform(baseToCameraTransform);

C#

source

pointCloud.Transform(transformBaseToCamera);

Python

source

point_cloud.transform(base_to_camera_transform)

Transformation can be done in-place:

• Zivid::PointCloud::transform()

• Zivid::UnorganizedPointCloud::transform()

or by creating a new instance:

• Zivid::PointCloud::transformed()

• Zivid::UnorganizedPointCloud::transformed()

The following example shows how create a new instance of Zivid::UnorganizedPointCloud with a transformation applied to it. Note that in this sample is is not necessary to create a new instance, as the untransformed point cloud is not used after the transformation.

C++

source

const auto transformedUnorganizedPointCloud = unorganizedPointCloud.transformed(transformationMatrix);

Python

source

transformed_unorganized_point_cloud = unorganized_point_cloud.transformed(transformation_matrix)

Even the in-place API returns the transformed point cloud, so you can use it directly, as in the example below.

C++

source

stitchedPointCloud.extend(currentPointCloud.transform(transformationMatrixZivid));

Python

source

stitched_point_cloud.extend(current_point_cloud.transform(transformation_matrix))

Downsample

때로는 카메라에서 촬영되는 high spatial resolution 와 같은 포인트 클라우드가 필요하지 않을 수 있습니다. 직접 포인트 클라우드를 downsample 할 수 있습니다.

참고

Sampling (3D) describes a hardware-based sub-/downsample method that reduces the resolution of the point cloud during capture while also reducing the acquisition and capture time.

참고

Zivid::UnorganizedPointCloud does not support downsampling, but it does support voxel downsampling, see Voxel downsample.

다운샘플링은 현재 포인트 클라우드를 수정하면서 동시에 수행할 수 있습니다.

C++

소스

pointCloud.downsample(Zivid::PointCloud::Downsampling::by2x2);

C#

source

pointCloud.Downsample(Zivid.NET.PointCloud.Downsampling.By2x2);

Python

source

point_cloud.downsample(zivid.PointCloud.Downsampling.by2x2)

다운샘플링된 포인트 클라우드를 기존 포인트 클라우드를 변경하지 않는 새로운 포인트 클라우드 인스턴스로 얻을 수도 있습니다.

C++

소스

auto downsampledPointCloud = pointCloud.downsampled(Zivid::PointCloud::Downsampling::by2x2);

C#

source

var downsampledPointCloud = pointCloud.Downsampled(Zivid.NET.PointCloud.Downsampling.By2x2);

Python

source

downsampled_point_cloud = point_cloud.downsampled(zivid.PointCloud.Downsampling.by2x2)

Zivid SDK는 다음과 같은 다운샘플링 속도를 지원합니다. by2x2, by3x3, 그리고 by4x4, 다운샘플링을 여러 번 수행할 수 있습니다.

Voxel downsample

Zivid::UnorganizedPointCloud supports voxel downsampling. The API takes two arguments:

1. voxelSize - the size of the voxel in millimeters.

2. minPointsPerVoxel - the minimum number of points per voxel to keep it.

Voxel downsampling subdivides 3D space into a grid of cubic voxels with a given size. If a given voxel contains a number of points at or above the given limit, all those source points are replaced with a single point with the following properties:

• Position (XYZ) is an SNR-weighted average of the source points’ positions, i.e. a high-confidence source point will have a greater influence on the resulting position than a low-confidence one.

• Color (RGBA) is the average of the source points’ colors.

• Signal-to-noise ratio (SNR) is sqrt(sum(SNR^2)) of the source points’ SNR values, i.e. the SNR of a new point will increase with both the number and the confidence of the source points that were used to compute its position.

Using minPointsPerVoxel > 1 is particularly useful for removing noise and artifacts from unorganized point clouds that are a combination of point clouds captured from different angles. This is because a given artifact is most likely only present in one of the captures, and minPointsPerVoxel can be used to only fill voxels that both captures “agree” on.

C++

source

const auto finalPointCloud = stitchedPointCloud.voxelDownsampled(0.5, 1);

Python

source

final_point_cloud = stitched_point_cloud.voxel_downsampled(0.5, 1)

Normals

일부 어플리케이션에는 포인트 클라우드에서 normals 얻기 위해 컴퓨팅이 필요합니다.

C++

소스

std::cout << "Computing normals and copying them to CPU memory" << std::endl;
const auto normals = pointCloud.copyData<Zivid::NormalXYZ>();

C#

source

Console.WriteLine("Computing normals and copying them to CPU memory");
var normals = pointCloud.CopyNormalsXYZ();

Python

source

print("Computing normals and copying them to CPU memory")
normals = point_cloud.copy_data("normals")

Normals API는 포인트 클라우드의 각 지점에서 법선을 계산하고 GPU 메모리에서 CPU 메모리로 법선을 복사합니다. 결과는 입력 포인트 클라우드의 각 포인트에 대해 하나씩, 법선 벡터의 행렬입니다. 법선의 크기는 입력 포인트 클라우드의 크기와 같습니다.

Visualize

프레임이 있으면 포인트 클라우드를 시각화할 수 있습니다.

C++

source

std::cout << "Setting up visualization" << std::endl;
Zivid::Visualization::Visualizer visualizer;

std::cout << "Visualizing point cloud" << std::endl;
visualizer.showMaximized();
visualizer.show(frame);
visualizer.resetToFit();

std::cout << "Running visualizer. Blocking until window closes." << std::endl;
visualizer.run();

C#

source

Console.WriteLine("Setting up visualization");
using (var visualizer = new Zivid.NET.Visualization.Visualizer())
{
    Console.WriteLine("Visualizing point cloud");
    visualizer.Show(frame);
    visualizer.ShowMaximized();
    visualizer.ResetToFit();

    Console.WriteLine("Running visualizer. Blocking until window closes.");
    visualizer.Run();
}

포인트 클라우드 객체에서도 포인트 클라우드를 시각화할 수 있습니다.

C++

소스

std::cout << "Getting point cloud from frame" << std::endl;
auto pointCloud = frame.pointCloud();

std::cout << "Setting up visualization" << std::endl;
Zivid::Visualization::Visualizer visualizer;

std::cout << "Visualizing point cloud" << std::endl;
visualizer.showMaximized();
visualizer.show(pointCloud);
visualizer.resetToFit();

std::cout << "Running visualizer. Blocking until window closes." << std::endl;
visualizer.run();

C#

source

Console.WriteLine("Getting point cloud from frame");
var pointCloud = frame.PointCloud;

Console.WriteLine("Setting up visualization");
var visualizer = new Zivid.NET.Visualization.Visualizer();

Console.WriteLine("Visualizing point cloud");
visualizer.Show(pointCloud);
visualizer.ShowMaximized();
visualizer.ResetToFit();

Console.WriteLine("Running visualizer. Blocking until window closes.");
visualizer.Run();

자세한 내용은 다음을 확인하세요. Visualization Tutorial, 여기서 우리는 타사 라이브러리를 사용하여 포인트 클라우드, 컬러 이미지, 깊이 맵 및 법선 시각화를 다룹니다.

Conclusion

이 튜토리얼은 Zivid SDK를 사용하여 포인트 클라우드를 추출하고, 조작하고, 변환하고, 시각화하는 방법을 보여줍니다.

Version History

SDK	Changes
2.16.0	Added support for Zivid::UnorganizedPointCloud. transformed is added as a function to Zivid::PointCloud (also available in Zivid::UnorganizedPointCloud).
2.11.0	Added support for SRGB color space.
2.10.0	Monochrome Capture 는 Downsample 을 대체할 수 있는 보다 빠른 대안을 제공합니다.