Stitch by Transform Tutorial

This tutorial describes how to use the results of calibrating multiple cameras to each other. The results of multi-camera transformation are transformation matrices that can be used to transform one point cloud into the coordinate frame of another. This is a very good first step in stitching point clouds together.

Prerequisites

You should have completed Multi-Camera Calibration Tutorial before going through this tutorial.

Load associated transformation matrices and map to point cloud

To apply the correct transformation matrix, we must map it to its corresponding frame. We use the camera or ZDF file's serial number to find the YAML file with a matching name.

When capturing directly from the camera, we read its serial number and search for a YAML file with that name, then map the transformation matrix to the camera.

C++

source

const auto transformsMappedToCameras =
    getTransformationMatricesFromYAML(transformationMatricesfileList, connectedCameras);

Python

source

transforms_mapped_to_cameras = get_transformation_matrices_from_yaml(args.yaml_files, connected_cameras)

On the other hand, when loading from ZDF files, we need to find a ZDF file from the list of files and extract its serial number from the frame. And then, we search for a YAML file on the same file list that uses that serial number as its name. Then we map a transformation matrix with a Frame. In getTransformedPointClouds we apply the transformation and extend the unorganized point cloud, right away.

C++

source

Zivid::UnorganizedPointCloud getTransformedPointClouds(
    Zivid::UnorganizedPointCloud stitchedPointCloud;
                Zivid::Matrix4x4 transformationMatrixZivid(yamlFileName);
                Zivid::UnorganizedPointCloud currentPointCloud = frame.pointCloud().toUnorganizedPointCloud();
                stitchedPointCloud.extend(currentPointCloud.transform(transformationMatrixZivid));

Python

source

def get_transformed_point_clouds(
    stitched_point_cloud = zivid.UnorganizedPointCloud()
        transformation_matrix = zivid.Matrix4x4(yaml_file)
        current_point_cloud = frame.point_cloud().to_unorganized_point_cloud()
        stitched_point_cloud.extend(current_point_cloud.transform(transformation_matrix))

Apply transformation matrix and stitch transformed point cloud with previous point clouds

We have mapped capture with a transformation matrix. Thus, we are ready to transform and stitch.

We can capture and extract the correction transform from transformsMappedToCameras. For each capture we transform and extend the unorganized point cloud.

C++

source

for(auto &camera : connectedCameras)
{
    if(pathNotProvided)
    {
        settingsPath = std::string(ZIVID_SAMPLE_DATA_DIR) + "/Settings/" + sanitizedModelName(camera)
                       + "_ManufacturingSpecular.yml";
    }

    std::cout << "Imaging from camera: " << camera.info().serialNumber() << std::endl;
    const auto frame = camera.capture2D3D(Zivid::Settings(settingsPath));
    const auto unorganizedPointCloud = frame.pointCloud().toUnorganizedPointCloud();
    const auto transformationMatrix = transformsMappedToCameras.at(camera.info().serialNumber().toString());
    const auto transformedUnorganizedPointCloud = unorganizedPointCloud.transformed(transformationMatrix);
    stitchedPointCloud.extend(transformedUnorganizedPointCloud);
}

Python

source

for camera in connected_cameras:
    if args.settings_path is not None:
        settings_path = args.settings_path
    else:
        settings_path = (
            get_sample_data_path() / "Settings" / f"{sanitized_model_name(camera)}_ManufacturingSpecular.yml"
        )
    print(f"Imaging from camera: {camera.info.serial_number}")
    frame = camera.capture(zivid.Settings.load(settings_path))
    unorganized_point_cloud = frame.point_cloud().to_unorganized_point_cloud()
    transformation_matrix = transforms_mapped_to_cameras[camera.info.serial_number]
    transformed_unorganized_point_cloud = unorganized_point_cloud.transformed(transformation_matrix)
    stitched_point_cloud.extend(transformed_unorganized_point_cloud)

Optional: refine alignment with local point cloud registration

The transformation matrices from multi-camera calibration are rarely perfect and usually leave small residual alignment errors between the stitched point clouds. These residuals can be reduced by applying Experimental::Toolbox::localPointCloudRegistration() to refine the alignment between overlapping point clouds after the transform-based pre-alignment.

For an explanation of this refinement step applied on top of the pre-alignment based on multi-camera calibration, see Stitching Workflow.

Voxel downsample

Before visualization we clean up the stitched point cloud:

C++

source

const auto finalPointCloud = stitchedPointCloud.voxelDownsampled(0.5, 1);

Python

source

final_point_cloud = stitched_point_cloud.voxel_downsampled(0.5, 1)

Here we're asking for voxel size of 0.5 mm. The second parameter is useful if more than 2 cameras are involved. Then it can be used to have a majority vote within a pixel. For more information see Voxel downsample.

Visualize

We use the Visualization::Visualizer to visualize the stitched point cloud.

C++

source

void visualizePointCloud(const Zivid::UnorganizedPointCloud &unorganizedPointCloud)
{
    Zivid::Visualization::Visualizer visualizer;

    visualizer.showMaximized();
    visualizer.show(unorganizedPointCloud);
    visualizer.resetToFit();

    std::cout << "Running visualizer. Blocking until window closes." << std::endl;
    visualizer.run();
}

Save stitched point cloud

Finally we can optionally save the point cloud:

C++

source

const std::string &fileName = stitchedPointCloudFileName;
std::cout << "Saving " << finalPointCloud.size() << " data points to " << fileName << std::endl;

using PLY = Zivid::Experimental::PointCloudExport::FileFormat::PLY;
const auto colorSpace = Zivid::Experimental::PointCloudExport::ColorSpace::sRGB;
Zivid::Experimental::PointCloudExport::exportUnorganizedPointCloud(
    finalPointCloud, PLY{ fileName, PLY::Layout::unordered, colorSpace });

Python

source

print(f"Saving {final_point_cloud.size} data points to {args.output_file}")
export_unorganized_point_cloud(
    final_point_cloud, PLY(str(args.output_file), layout=PLY.Layout.unordered, color_space=ColorSpace.srgb)
)