通过ArUco标记定义的ROI盒

This tutorial demonstrates how to find the ROI box parameters using an ArUco marker and how to filter the contents of a bin using it. We are here using the ArUco marker on the Zivid calibration board (7x8 30mm, 300x300mm), and assume the calibration board is placed in the bottom right corner of the bin. The bin size is also assumed known and is used to set the dimensions of the ROI box. This way you can automatically find the ROI parameters in the camera frame.

备注

本教程使用了一个档案相机（file camera）对下图中的场景进行演示。

小技巧

ROI 框过滤可以减少捕获时间。

档案相机可以从 Sample Data（示例数据） 下载。

首先，我们需要捕获ArUco标记的点云。

C++

source

const auto fileCamera = std::string(ZIVID_SAMPLE_DATA_DIR) + "/BinWithCalibrationBoard.zdf";
const auto loadedFrameWithDiagnostics = Zivid::Frame(fileCamera);

std::cout << "Creating virtual camera using file: " << fileCamera << std::endl;
auto camera = zivid.createFileCamera(loadedFrameWithDiagnostics);

auto settings = loadedFrameWithDiagnostics.settings();

const auto originalFrame = camera.capture2D3D(settings);
auto pointCloud = originalFrame.pointCloud();

C＃

source

string fileCamera = Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData) + "/Zivid/BinWithCalibrationBoard.zdf";
var loadedFrameWithDiagnostics = new Zivid.NET.Frame(fileCamera);

Console.WriteLine("Creating virtual camera using file: " + fileCamera);
var camera = zivid.CreateFileCamera(loadedFrameWithDiagnostics);

var settings = loadedFrameWithDiagnostics.Settings;

using (var originalFrame = camera.Capture2D3D(settings))
{
    var pointCloud = originalFrame.PointCloud;

Python

source

file_camera = get_sample_data_path() / "BinWithCalibrationBoard.zdf"
loaded_frame_with_diagnostics = zivid.Frame(file_camera)

print(f"Creating virtual camera using file: {file_camera}")
camera = app.create_file_camera(loaded_frame_with_diagnostics)

settings = loaded_frame_with_diagnostics.settings

original_frame = camera.capture_2d_3d(settings)
point_cloud = original_frame.point_cloud()

ArUco标记坐标系的原点位于ArUco标记的中心。

我们定义了ROI盒右下角相对于ArUco标记坐标系的位置，以及ROI盒尺寸。然后我们从长度和宽度中减去箱边缘的宽度以移除箱壁。

C++

source

const float roiBoxLength = 545.F;
const float roiBoxWidth = 345.F;
const float roiBoxHeight = 150.F;
// Coordinates are relative to the ArUco marker origin which lies in the center of the ArUco marker.
// Positive x-axis is "East", y-axis is "South" and z-axis is "Down".
const Zivid::PointXYZ roiBoxLowerRightCornerInArUcoFrame{ 240.F, 30.F, 5.F };
const Zivid::PointXYZ roiBoxUpperRightCornerInArUcoFrame{ roiBoxLowerRightCornerInArUcoFrame.x,
                                                          roiBoxLowerRightCornerInArUcoFrame.y - roiBoxWidth,
                                                          roiBoxLowerRightCornerInArUcoFrame.z };
const Zivid::PointXYZ roiBoxLowerLeftCornerInArUcoFrame{ roiBoxLowerRightCornerInArUcoFrame.x - roiBoxLength,
                                                         roiBoxLowerRightCornerInArUcoFrame.y,
                                                         roiBoxLowerRightCornerInArUcoFrame.z };

C＃

source

float roiBoxLength = 545F;
float roiBoxWidth = 345F;
float roiBoxHeight = 150F;
// Coordinates are relative to the checkerboard origin which lies in the intersection between the four checkers
// in the top-left corner of the checkerboard: Positive x-axis is "East", y-axis is "South" and z-axis is "Down"
var roiBoxLowerRightCornerInCheckerboardFrame = new Zivid.NET.PointXYZ
{
    x = 240F,
    y = 30F,
    z = 5F
};
var roiBoxUpperRightCornerInCheckerboardFrame = new Zivid.NET.PointXYZ
{
    x = roiBoxLowerRightCornerInCheckerboardFrame.x,
    y = roiBoxLowerRightCornerInCheckerboardFrame.y - roiBoxWidth,
    z = roiBoxLowerRightCornerInCheckerboardFrame.z
};
var roiBoxLowerLeftCornerInCheckerboardFrame = new Zivid.NET.PointXYZ
{
    x = roiBoxLowerRightCornerInCheckerboardFrame.x - roiBoxLength,
    y = roiBoxLowerRightCornerInCheckerboardFrame.y,
    z = roiBoxLowerRightCornerInCheckerboardFrame.z

};

Python

source

roi_box_length = 545
roi_box_width = 345
roi_box_height = 150
roi_box_lower_right_corner_in_aruco_frame = np.array([240, 30, 5])
roi_box_upper_right_corner_in_aruco_frame = np.array(
    [
        roi_box_lower_right_corner_in_aruco_frame[0],
        roi_box_lower_right_corner_in_aruco_frame[1] - roi_box_width,
        roi_box_lower_right_corner_in_aruco_frame[2],
    ]
)
roi_box_lower_left_corner_in_aruco_frame = np.array(
    [
        roi_box_lower_right_corner_in_aruco_frame[0] - roi_box_length,
        roi_box_lower_right_corner_in_aruco_frame[1],
        roi_box_lower_right_corner_in_aruco_frame[2],
    ]
)

然后可以设置定义ROI盒基坐标系的三个点。

C++

source

const Zivid::PointXYZ pointOInArUcoFrame = roiBoxLowerRightCornerInArUcoFrame;
const Zivid::PointXYZ pointAInArUcoFrame = roiBoxUpperRightCornerInArUcoFrame;
const Zivid::PointXYZ pointBInArUcoFrame = roiBoxLowerLeftCornerInArUcoFrame;

C＃

source

var pointOInCheckerboardFrame = roiBoxLowerRightCornerInCheckerboardFrame;
var pointAInCheckerboardFrame = roiBoxUpperRightCornerInCheckerboardFrame;
var pointBInCheckerboardFrame = roiBoxLowerLeftCornerInCheckerboardFrame;

Python

source

point_o_in_aruco_frame = roi_box_lower_right_corner_in_aruco_frame
point_a_in_aruco_frame = roi_box_upper_right_corner_in_aruco_frame
point_b_in_aruco_frame = roi_box_lower_left_corner_in_aruco_frame

然后我们将配置ArUco标记。

C++

source

std::cout << "Configuring ArUco marker" << std::endl;
const auto markerDictionary = Zivid::Calibration::MarkerDictionary::aruco4x4_50;
std::vector<int> markerId = { 1 };

C＃

source

Console.WriteLine("Configuring ArUco marker");
var markerDictionary = Zivid.NET.MarkerDictionary.Aruco4x4_50;
var markerId = new List<int> { 1 };

Python

source

print("Configuring ArUco marker")
marker_dictionary = zivid.calibration.MarkerDictionary.aruco4x4_50
marker_id = [1]

然后我们检测 ArUco 标记。

C++

source

std::cout << "Detecting ArUco marker" << std::endl;
const auto detectionResult = Zivid::Calibration::detectMarkers(originalFrame, markerId, markerDictionary);

C＃

source

Console.WriteLine("Detecting ArUco marker");
var detectionResult = Detector.DetectMarkers(originalFrame, markerId, markerDictionary);

Python

source

print("Detecting ArUco marker")
detection_result = zivid.calibration.detect_markers(original_frame, marker_id, marker_dictionary)

然后我们需要估计ArUco标记的位姿，将这三个点转换到相机参考系。

C++

source

std::cout << "Estimating pose of detected ArUco marker" << std::endl;
const auto cameraToMarkerTransform = detectionResult.detectedMarkers()[0].pose().toMatrix();
std::cout << "Transforming the ROI base frame points to the camera frame" << std::endl;
const auto roiPointsInCameraFrame = transformPoints(
    std::vector<Zivid::PointXYZ>{ pointOInArUcoFrame, pointAInArUcoFrame, pointBInArUcoFrame },
    cameraToMarkerTransform);

C＃

source

Console.WriteLine("Estimating pose of detected ArUco marker");
var cameraToMarkerTransform = new Zivid.NET.Matrix4x4(detectionResult.DetectedMarkers()[0].Pose().ToMatrix());
Console.WriteLine("Transforming the ROI base frame points to the camera frame");
var roiPointsInCameraFrame = TransformPoints(
    new List<Zivid.NET.PointXYZ> { pointOInCheckerboardFrame, pointAInCheckerboardFrame, pointBInCheckerboardFrame },
    cameraToMarkerTransform);

Python

source

print("Estimating pose of detected ArUco marker")
camera_to_marker_transform = detection_result.detected_markers()[0].pose.to_matrix()
print("Transforming the ROI base frame points to the camera frame")
roi_points_in_camera_frame = _transform_points(
    [point_o_in_aruco_frame, point_a_in_aruco_frame, point_b_in_aruco_frame],
    camera_to_marker_transform,
)

提示

详细了解 位置、方向和坐标变换 来了解其工作原理。

现在我们可以根据ROI盒的大小和位置过滤点云。将第一个范围（extent）设置为一个较小的负数值来避免过滤掉底板，第二个范围（extent）设置为所需的盒子高度。

C++

source

const auto roiSettings = Zivid::Settings::RegionOfInterest::Box{
    Zivid::Settings::RegionOfInterest::Box::Enabled::yes,
    Zivid::Settings::RegionOfInterest::Box::PointO{ roiPointsInCameraFrame[0] },
    Zivid::Settings::RegionOfInterest::Box::PointA{ roiPointsInCameraFrame[1] },
    Zivid::Settings::RegionOfInterest::Box::PointB{ roiPointsInCameraFrame[2] },
    Zivid::Settings::RegionOfInterest::Box::Extents{ -10, roiBoxHeight }
};

C＃

source

var roiSettings = new Zivid.NET.Settings.RegionOfInterestGroup.BoxGroup
{
    Enabled = true,
    PointO = roiPointsInCameraFrame[0],
    PointA = roiPointsInCameraFrame[1],
    PointB = roiPointsInCameraFrame[2]
};
roiSettings.Extents = new Zivid.NET.Range<double>(-10, roiBoxHeight);

Python

source

roi_settings = zivid.Settings.RegionOfInterest.Box(
    enabled=True,
    point_o=roi_points_in_camera_frame[0],
    point_a=roi_points_in_camera_frame[1],
    point_b=roi_points_in_camera_frame[2],
    extents=(-10, roi_box_height),
)

现在，我们可以利用感兴趣区域（ROI）对捕获的点云进行掩膜处理，并可视化最终结果。

C++

source

const auto roiPointCloud = pointCloud.maskedByRegionOfInterest(roiSettings);

std::cout << "Displaying the ROI-filtered point cloud" << std::endl;
visualizeZividPointCloud(roiPointCloud);

C＃

source

using (var roiPointCloud = pointCloud.MaskedByRegionOfInterest(roiSettings))
{
    Console.WriteLine("Displaying the ROI-filtered point cloud");
    VisualizeZividPointCloud(roiPointCloud);
}

Python

source

roi_point_cloud = point_cloud.masked_by_region_of_interest(roi_settings)
print("Displaying the ROI-filtered point cloud")
display_pointcloud(roi_point_cloud)

最后，我们将 ROI 框添加到捕获设置中，直接应用 ROI 过滤器来捕获点云。这种方法比捕获后再进行掩膜处理速度更快，并且允许您保存包含 ROI 参数的捕获设置，以便在移除标定板后重复使用。

C++

source

std::cout << "Adding the ROI box to the capture settings and capturing again" << std::endl;
settings.set(Zivid::Settings::RegionOfInterest{ roiSettings });

const auto roiFrame = camera.capture2D3D(settings);

std::cout << "Displaying the ROI-filtered point cloud from the new capture" << std::endl;
visualizeZividPointCloud(roiFrame.pointCloud());

C＃

source

Console.WriteLine("Adding the ROI box to the capture settings and capturing again");
settings.RegionOfInterest.Box = roiSettings;

using (var roiFrame = camera.Capture2D3D(settings))
{
    Console.WriteLine("Displaying the ROI-filtered point cloud from the new capture");
    VisualizeZividPointCloud(roiFrame);
}

Python

source

print("Adding the ROI box to the capture settings and capturing again")
settings.region_of_interest.box = roi_settings

roi_frame_point_cloud = camera.capture_2d_3d(settings).point_cloud()
print("Displaying the ROI-filtered point cloud from the new capture")
display_pointcloud(roi_frame_point_cloud)

如需根据ROI盒过滤点云，您可以运行我们的代码示例。

C++

示例： ROIBoxViaArucoMarker.cpp

./ROIBoxViaArucoMarker

C＃

示例： ROIBoxViaArucoMarker.cs

./ROIBoxViaArucoMarker

Python

示例： roi_box_via_aruco_marker.py

python roi_box_via_aruco_marker.py

小技巧

如果您希望在自己的设置中使用它，您可以自行修改代码示例：

1. 用您的实际相机和设置替换档案相机。

2. 将标定板放在料箱的右下角。

3. 将ROI盒尺寸修改为您的料箱的尺寸。

4. 运行示例！

您现在可以保存包含了ROI参数的捕获设置了，然后拿走标定板并在整个料箱上使用该设置。

Version History

SDK	Changes
2.18.0	Mask the captured point cloud by the ROI box instead of re-capturing.