ROI Box via Checkerboard

This tutorial demonstrates how to find the ROI box parameters using the Zivid calibration board and how to filter the contents of a bin using it. We assume the calibration board is placed in the bottom right corner of the bin. The bin size is also assumed known and is used to set the dimensions of the ROI box. This way you can automatically find the ROI parameters in the camera frame.

Note

This tutorial uses a file camera for the scene in the image below for demonstration.

Tip

ROI box filtering reduces capture time.

The file camera can be downloaded from Sample Data.

First, we capture a point cloud of the checkerboard.

C++

const auto fileCamera = std::string(ZIVID_SAMPLE_DATA_DIR) + "/BinWithCalibrationBoard.zfc";

std::cout << "Creating virtual camera using file: " << fileCamera << std::endl;
auto camera = zivid.createFileCamera(fileCamera);

auto settings = Zivid::Settings{ Zivid::Settings::Acquisitions{ Zivid::Settings::Acquisition{} } };

const auto originalFrame = camera.capture(settings);
auto pointCloud = originalFrame.pointCloud();

Python

file_camera = get_sample_data_path() / "BinWithCalibrationBoard.zfc"

print(f"Creating virtual camera using file: {file_camera}")
camera = app.create_file_camera(file_camera)

settings = zivid.Settings([zivid.Settings.Acquisition()])

original_frame = camera.capture(settings)
original_point_cloud = original_frame.point_cloud()

The checkerboard frame has its origin in the intersection between the four checkers in the top-left corner of the board.

ROI box size and bottom-right corner relative to the checkerboard frame.

We define the position of the bottom-right corner of the ROI box relative to the checkerboard frame, and the ROI box size. Then we subtract the width of the bin edge from the length and width to remove the bin walls.

C++

// Coordinates are relative to the checkerboard origin which lies in the intersection between the four checkers
// in the top-left corner of the checkerboard: Positive x-axis is "East", y-axis is "South" and z-axis is "Down"
const Zivid::PointXYZ roiBoxLowerRightCornerInCheckerboardFrame{ 240.F, 260.F, 5.F };
const Zivid::PointXYZ roiBoxUpperRightCornerInCheckerboardFrame{ roiBoxLowerRightCornerInCheckerboardFrame.x,
                                                                 roiBoxLowerRightCornerInCheckerboardFrame.y
                                                                     - roiBoxWidth,
                                                                 roiBoxLowerRightCornerInCheckerboardFrame.z };
const Zivid::PointXYZ roiBoxLowerLeftCornerInCheckerboardFrame{ roiBoxLowerRightCornerInCheckerboardFrame.x
                                                                    - roiBoxLength,
                                                                roiBoxLowerRightCornerInCheckerboardFrame.y,
                                                                roiBoxLowerRightCornerInCheckerboardFrame.z };
const float roiBoxLength = 545.F;
const float roiBoxWidth = 345.F;
const float roiBoxHeight = 150.F;

Python

roi_box_lower_right_corner = np.array([240, 260, 0.5])
roi_box_upper_right_corner = np.array(
    [
        roi_box_lower_right_corner[0],
        roi_box_lower_right_corner[1] - roi_box_width,
        roi_box_lower_right_corner[2],
    ]
)
roi_box_lower_left_corner = np.array(
    [roi_box_lower_right_corner[0] - roi_box_length, roi_box_lower_right_corner[1], roi_box_lower_right_corner[2]]
)
roi_box_length = 545
roi_box_width = 345
roi_box_height = 150

The three points that define the base frame of the ROI box can then be set.

C++

const Zivid::PointXYZ pointOInCheckerboardFrame = roiBoxLowerRightCornerInCheckerboardFrame;
const Zivid::PointXYZ pointAInCheckerboardFrame = roiBoxUpperRightCornerInCheckerboardFrame;
const Zivid::PointXYZ pointBInCheckerboardFrame = roiBoxLowerLeftCornerInCheckerboardFrame;

Python

point_o_in_checkerboard_frame = roi_box_lower_right_corner
point_a_in_checkerboard_frame = roi_box_upper_right_corner
point_b_in_checkerboard_frame = roi_box_lower_left_corner

Then we estimate the pose of the checkerboard to transform the three points into the camera frame of reference.

C++

std::cout << "Detecting and estimating pose of the Zivid checkerboard in the camera frame" << std::endl;
const auto detectionResult = Zivid::Calibration::detectFeaturePoints(originalFrame.pointCloud());
const auto transformCheckerboardToCamera = detectionResult.pose().toMatrix();

std::cout << "Transforming the ROI base frame points to the camera frame" << std::endl;
const auto roiPointsInCameraFrame = transformPoints(
    std::vector<Zivid::PointXYZ>{
        pointOInCheckerboardFrame, pointAInCheckerboardFrame, pointBInCheckerboardFrame },
    transformCheckerboardToCamera);

Python

print("Detecting and estimating pose of the Zivid checkerboard in the camera frame")
detection_result = zivid.experimental.calibration.detect_feature_points(original_frame)
transform_checkerboard_to_camera = detection_result.pose().to_matrix()

print("Transforming the ROI base frame points to the camera frame")
roi_points_in_camera_frame = _transform_points(
    [point_o_in_checkerboard_frame, point_a_in_checkerboard_frame, point_b_in_checkerboard_frame],
    transform_checkerboard_to_camera,
)

Hint

Learn more about Position, Orientation and Coordinate Transformations to understand how this works.

Now we can filter the point cloud based on the size and position of the ROI box. The first extent is set to a small negative value to include the floor, and the second extent is set to the desired height of the box.

C++

settings.set(Zivid::Settings::RegionOfInterest{
    Zivid::Settings::RegionOfInterest::Box::Enabled::yes,
    Zivid::Settings::RegionOfInterest::Box::PointO{ roiPointsInCameraFrame[0] },
    Zivid::Settings::RegionOfInterest::Box::PointA{ roiPointsInCameraFrame[1] },
    Zivid::Settings::RegionOfInterest::Box::PointB{ roiPointsInCameraFrame[2] },
    Zivid::Settings::RegionOfInterest::Box::Extents{ -10, roiBoxHeight } });

Python

settings.region_of_interest.box.enabled = True
settings.region_of_interest.box.point_o = roi_points_in_camera_frame[0]
settings.region_of_interest.box.point_a = roi_points_in_camera_frame[1]
settings.region_of_interest.box.point_b = roi_points_in_camera_frame[2]
settings.region_of_interest.box.extents = (-10, roi_box_height)

Finally, we take a new capture and visualize the filtered point cloud.

C++

const auto roiFrame = camera.capture(settings);

std::cout << "Displaying the ROI-filtered point cloud" << std::endl;
visualizeZividPointCloud(roiFrame);

Python

roi_point_cloud = camera.capture(settings).point_cloud()

print("Displaying the ROI-filtered point cloud")
display_pointcloud(roi_point_cloud.copy_data("xyz"), roi_point_cloud.copy_data("rgba")[:, :, :3])

ROI filtered and transformed point cloud of a bin with checkerboard

To filter the point cloud based on a ROI box, you can run our code sample.

C++

Sample: ROIBoxViaCheckerboard.cpp

./ROIBoxViaCheckerboard

Python

Sample: roi_box_via_checkerboard.py

python roi_box_via_checkerboard.py

Tip

Modify the code sample if you wish to use this in your own setup:

Replace the file camera with your actual camera and settings.
Place the checkerboard in the bottom right corner of your bin.
Modify the ROI box size to your bin size.
Run sample!

You can now save the capture settings with the ROI parameters, remove the checkerboard and use the settings on the whole bin.