可视化教程

介绍

本教程介绍如何使用Zivid SDK和第三方库来可视化Zivid相机捕获的3D和2D数据。

先决条件

本教程从Zivid帧（frame）开始。查看 捕获教程 了解更多有关如何捕获帧的信息。

点云

在C++和C#中使用Zivid SDK

您可以通过帧（frame）可视化点云。

跳转到源码

源码

std::cout << "Setting up visualization" << std::endl;
Zivid::Visualization::Visualizer visualizer;

std::cout << "Visualizing point cloud" << std::endl;
visualizer.showMaximized();
visualizer.show(frame);
visualizer.resetToFit();

std::cout << "Running visualizer. Blocking until window closes." << std::endl;
visualizer.run();
跳转到源码

源码

Console.WriteLine("Setting up visualization");
var visualizer = new Zivid.NET.Visualization.Visualizer();

Console.WriteLine("Visualizing point cloud");
visualizer.Show(frame);
visualizer.ShowMaximized();
visualizer.ResetToFit();

Console.WriteLine("Running visualizer. Blocking until window closes.");
visualizer.Run();

您也可以从点云对象来可视化点云。

跳转到源码

源码

std::cout << "Getting point cloud from frame" << std::endl;
auto pointCloud = frame.pointCloud();

std::cout << "Setting up visualization" << std::endl;
Zivid::Visualization::Visualizer visualizer;

std::cout << "Visualizing point cloud" << std::endl;
visualizer.showMaximized();
visualizer.show(pointCloud);
visualizer.resetToFit();

std::cout << "Running visualizer. Blocking until window closes." << std::endl;
visualizer.run();
跳转到源码

源码

Console.WriteLine("Getting point cloud from frame");
var pointCloud = frame.PointCloud;

Console.WriteLine("Setting up visualization");
var visualizer = new Zivid.NET.Visualization.Visualizer();

Console.WriteLine("Visualizing point cloud");
visualizer.Show(pointCloud);
visualizer.ShowMaximized();
visualizer.ResetToFit();

Console.WriteLine("Running visualizer. Blocking until window closes.");
visualizer.Run();

在Python中使用Open3D

Zivid-Python 不支持点云可视化，这里我们使用 Open3D 来实现该功能。

跳转到源码

source

point_cloud = frame.point_cloud()
xyz = point_cloud.copy_data("xyz")
rgba = point_cloud.copy_data("rgba")

print("Visualizing point cloud")
display_pointcloud(xyz, rgba[:, :, 0:3])

下面展示了如何实现点云可视化的功能。

跳转到源码

source

def display_pointcloud(xyz: np.ndarray, rgb: np.ndarray, normals: Optional[np.ndarray] = None) -> None:
    """Display point cloud provided from 'xyz' with colors from 'rgb'.

    Args:
        rgb: RGB image
        xyz: A numpy array of X, Y and Z point cloud coordinates
        normals: Ordered array of normal vectors, mapped to xyz

    """
    xyz = np.nan_to_num(xyz).reshape(-1, 3)
    if normals is not None:
        normals = np.nan_to_num(normals).reshape(-1, 3)
    rgb = rgb.reshape(-1, 3)

    point_cloud_open3d = o3d.geometry.PointCloud(o3d.utility.Vector3dVector(xyz))
    point_cloud_open3d.colors = o3d.utility.Vector3dVector(rgb / 255)
    if normals is not None:
        point_cloud_open3d.normals = o3d.utility.Vector3dVector(normals)
        print("Open 3D controls:")
        print("  n: for normals")
        print("  9: for point cloud colored by normals")
        print("  h: for all controls")

    visualizer = o3d.visualization.Visualizer()  # pylint: disable=no-member
    visualizer.create_window()
    visualizer.add_geometry(point_cloud_open3d)

    if normals is None:
        visualizer.get_render_option().background_color = (0, 0, 0)
    visualizer.get_render_option().point_size = 1
    visualizer.get_render_option().show_coordinate_frame = True
    visualizer.get_view_control().set_front([0, 0, -1])
    visualizer.get_view_control().set_up([0, -1, 0])

    visualizer.run()
    visualizer.destroy_window()

彩色图像

由于Zivid SDK 和 Zivid-Python 不支持2D彩色图像可视化，我们使用第三方库实现该功能： 使 用 OpenCV 在C++和Python中实现，以及使 用 Matplotlib 在Python中实现。

C++中使用OpenCV实现

首先，我们将点云转换为OpenCV彩色图像。

跳转到源码

source

std::cout << "Converting point cloud to BGRA image in OpenCV format" << std::endl;
cv::Mat bgra = pointCloudToCvBGRA(pointCloud);

下面展示了如何实现将点云转换为彩色图像的功能。

跳转到源码

source

cv::Mat pointCloudToCvBGRA(const Zivid::PointCloud &pointCloud)

{
    auto bgra = cv::Mat(pointCloud.height(), pointCloud.width(), CV_8UC4);
    pointCloud.copyData(&(*bgra.begin<Zivid::ColorBGRA>()));

    return bgra;
}

小技巧

也可以直接从Zivid 2D彩色图像中获取OpenCV彩色图像。

我们现在可以可视化彩色图像了。

跳转到源码

source

cv::namedWindow("BGR image", cv::WINDOW_AUTOSIZE);
cv::imshow("BGR image", bgra);
cv::waitKey(0);

Python中使用OpenCV实现

首先，我们将点云转换为OpenCV彩色图像。

跳转到源码

源码

print("Converting to BGR image in OpenCV format")
bgr = _point_cloud_to_cv_bgr(point_cloud)

下面展示了如何实现将点云转换为彩色图像的功能。

跳转到源码

源码

def _point_cloud_to_cv_bgr(point_cloud: zivid.PointCloud) -> np.ndarray:
    """Get bgr image from frame.

    Args:
        point_cloud: Zivid point cloud

    Returns:
        bgr: BGR image (HxWx3 ndarray)

    """
    bgra = point_cloud.copy_data("bgra")

    return bgra[:, :, :3]

小技巧

也可以直接从Zivid 2D彩色图像中获取OpenCV彩色图像。

我们现在可以可视化彩色图像了。

跳转到源码

源码

_visualize_and_save_image(bgr, bgr_image_file, "BGR image")

下面展示了如何实现可视化功能。

跳转到源码

源码

def _visualize_and_save_image(image: np.ndarray, image_file: str, title: str) -> None:
    """Visualize and save image to file.

    Args:
        image: BGR image (HxWx3 ndarray)
        image_file: File name
        title: OpenCV Window name

    """
    display_bgr(image, title)
    cv2.imwrite(image_file, image)

Python中通过Matplotlib实现

Matplotlib提供了一种更简单的在Python中可视化彩色图像的方法。

跳转到源码

源码

display_rgb(rgba[:, :, 0:3], block=False)

函数实现如下所示。

跳转到源码

源码

def display_rgb(rgb: np.ndarray, title: str = "RGB image", block: bool = True) -> None:
    """Display RGB image.

    Args:
        rgb: RGB image (HxWx3 ndarray)
        title: Image title
        block: Stops the running program until the windows is closed

    """
    plt.figure()
    plt.imshow(rgb)
    plt.title(title)
    plt.show(block=block)

深度图

由于Zivid SDK 和 Zivid-Python 不支持深度图的可视化功能，我们使用第三方库实现该功能： 使 用 OpenCV 在C++和Python中实现，以及使 用 Matplotlib 在Python中实现。

C++中使用OpenCV实现

首先，我们将点云转换为OpenCV深度图。

跳转到源码

source

std::cout << "Converting to Depth map in OpenCV format" << std::endl;
cv::Mat zColorMap = pointCloudToCvZ(pointCloud);

下面展示了将点云转换为深度图的函数。

跳转到源码

source

cv::Mat pointCloudToCvZ(const Zivid::PointCloud &pointCloud)
{
    cv::Mat z(pointCloud.height(), pointCloud.width(), CV_8UC1, cv::Scalar(0)); // NOLINT(hicpp-signed-bitwise)
    const auto points = pointCloud.copyPointsZ();

    // Getting min and max values for X, Y, Z images
    const auto *maxZ = std::max_element(points.data(), points.data() + pointCloud.size(), isLesserOrNan);
    const auto *minZ = std::max_element(points.data(), points.data() + pointCloud.size(), isGreaterOrNaN);

    // Filling in OpenCV matrix with the cloud data
    for(size_t i = 0; i < pointCloud.height(); i++)
    {
        for(size_t j = 0; j < pointCloud.width(); j++)
        {
            if(std::isnan(points(i, j).z))
            {
                z.at<uchar>(i, j) = 0;
            }
            else
            {
                z.at<uchar>(i, j) =
                    static_cast<unsigned char>((255.0F * (points(i, j).z - minZ->z) / (maxZ->z - minZ->z)));
            }
        }
    }

    // Applying color map
    cv::Mat zColorMap;
    cv::applyColorMap(z, zColorMap, cv::COLORMAP_VIRIDIS);

    // Setting invalid points (nan) to black
    for(size_t i = 0; i < pointCloud.height(); i++)
    {
        for(size_t j = 0; j < pointCloud.width(); j++)
        {
            if(std::isnan(points(i, j).z))
            {
                auto &zRGB = zColorMap.at<cv::Vec3b>(i, j);
                zRGB[0] = 0;
                zRGB[1] = 0;
                zRGB[2] = 0;
            }
        }
    }

    return zColorMap;
}

我们现在可以可视化深度图了。

跳转到源码

source

cv::namedWindow("Depth map", cv::WINDOW_AUTOSIZE);
cv::imshow("Depth map", zColorMap);
cv::waitKey(0);

Python中使用OpenCV实现

首先，我们将点云转换为OpenCV深度图。

跳转到源码

源码

print("Converting to Depth map in OpenCV format")
z_color_map = _point_cloud_to_cv_z(point_cloud)

下面展示了将点云转换为深度图的函数。

跳转到源码

源码

def _point_cloud_to_cv_z(point_cloud: zivid.PointCloud) -> np.ndarray:
    """Get depth map from frame.

    Args:
        point_cloud: Zivid point cloud

    Returns:
        depth_map_color_map: Depth map (HxWx1 ndarray)

    """
    depth_map = point_cloud.copy_data("z")
    depth_map_uint8 = ((depth_map - np.nanmin(depth_map)) / (np.nanmax(depth_map) - np.nanmin(depth_map)) * 255).astype(
        np.uint8
    )

    depth_map_color_map = cv2.applyColorMap(depth_map_uint8, cv2.COLORMAP_VIRIDIS)

    # Setting nans to black
    depth_map_color_map[np.isnan(depth_map)[:, :]] = 0

    return depth_map_color_map

我们现在可以可视化深度图了。

跳转到源码

源码

_visualize_and_save_image(z_color_map, depth_map_file, "Depth map")

函数实现如下所示。

跳转到源码

源码

def _visualize_and_save_image(image: np.ndarray, image_file: str, title: str) -> None:
    """Visualize and save image to file.

    Args:
        image: BGR image (HxWx3 ndarray)
        image_file: File name
        title: OpenCV Window name

    """
    display_bgr(image, title)
    cv2.imwrite(image_file, image)

Python中通过Matplotlib实现

Matplotlib提供了一种更简单的在Python中可视化深度图的方法。

跳转到源码

源码

display_depthmap(xyz, block=True)

函数实现如下所示。

跳转到源码

source

def display_depthmap(xyz: np.ndarray, block: bool = True) -> None:
    """Create and display depthmap.

    Args:
        xyz: A numpy array of X, Y and Z point cloud coordinates
        block: Stops the running program until the windows is closed

    """
    plt.figure()
    plt.imshow(
        xyz[:, :, 2],
        vmin=np.nanmin(xyz[:, :, 2]),
        vmax=np.nanmax(xyz[:, :, 2]),
        cmap="viridis",
    )
    plt.colorbar()
    plt.title("Depth map")
    plt.show(block=block)

法线

由于Zivid SDK不支持法线的可视化，我们使用第三方库实现该功能：使用 PCL 在C++中实现，以及使用 Open3D 在Python中实现。

C++中的使用PCL实现

我们可以按下面的方式可视化法线。

跳转到源码

源码

std::cout << "Visualizing normals" << std::endl;
visualizePointCloudAndNormalsPCL(pointCloudPCL.makeShared(), pointCloudWithNormalsPCL.makeShared());

函数实现如下所示。

跳转到源码

源码

void visualizePointCloudAndNormalsPCL(
    const pcl::PointCloud<pcl::PointXYZRGB>::ConstPtr &pointCloud,
    const pcl::PointCloud<pcl::PointXYZRGBNormal>::ConstPtr &pointCloudWithNormals)
{
    auto viewer = pcl::visualization::PCLVisualizer("Viewer");

    int viewRgb(0);
    viewer.createViewPort(0.0, 0.0, 0.5, 1.0, viewRgb);
    viewer.addText("Cloud RGB", 0, 0, "RGBText", viewRgb);
    viewer.addPointCloud<pcl::PointXYZRGB>(pointCloud, "cloud", viewRgb);

    const int normalsSkipped = 10;
    std::cout << "Note! 1 out of " << normalsSkipped << " normals are visualized" << std::endl;

    int viewNormals(0);
    viewer.createViewPort(0.5, 0.0, 1.0, 1.0, viewNormals);
    viewer.addText("Cloud Normals", 0, 0, "NormalsText", viewNormals);
    viewer.addPointCloud<pcl::PointXYZRGBNormal>(pointCloudWithNormals, "cloudNormals", viewNormals);
    viewer.addPointCloudNormals<pcl::PointXYZRGBNormal>(
        pointCloudWithNormals, normalsSkipped, 1, "normals", viewNormals);

    viewer.setCameraPosition(0, 0, -100, 0, -1, 0);

    std::cout << "Press r to centre and zoom the viewer so that the entire cloud is visible" << std::endl;
    std::cout << "Press q to exit the viewer application" << std::endl;
    while(!viewer.wasStopped())
    {
        viewer.spinOnce(100);
        std::this_thread::sleep_for(std::chrono::milliseconds(100));
    }
}

在Python中使用Open3D

我们可以按下面的方式可视化法线。

跳转到源码

源码

normals_colormap = 0.5 * (1 - normals)

print("Visualizing normals in 2D")
display_rgb(rgb=rgba[:, :, :3], title="RGB image", block=False)
display_rgb(rgb=normals_colormap, title="Colormapped normals", block=True)

print("Visualizing normals in 3D")
display_pointcloud_with_downsampled_normals(point_cloud, zivid.PointCloud.Downsampling.by4x4)

函数实现如下所示。

跳转到源码

source

def display_pointcloud_with_downsampled_normals(
    point_cloud: PointCloud,
    downsampling: PointCloud.Downsampling,
) -> None:
    """Display point cloud with downsampled normals.

    Args:
        point_cloud: A Zivid point cloud handle
        downsampling: A valid Zivid downsampling factor to apply to normals

    """
    rgb = point_cloud.copy_data("rgba")[:, :, :3]
    xyz = point_cloud.copy_data("xyz")
    point_cloud.downsample(downsampling)
    normals = point_cloud.copy_data("normals")

    display_pointcloud(xyz=xyz, rgb=rgb, normals=normals)
def display_pointcloud(xyz: np.ndarray, rgb: np.ndarray, normals: Optional[np.ndarray] = None) -> None:
    """Display point cloud provided from 'xyz' with colors from 'rgb'.

    Args:
        rgb: RGB image
        xyz: A numpy array of X, Y and Z point cloud coordinates
        normals: Ordered array of normal vectors, mapped to xyz

    """
    xyz = np.nan_to_num(xyz).reshape(-1, 3)
    if normals is not None:
        normals = np.nan_to_num(normals).reshape(-1, 3)
    rgb = rgb.reshape(-1, 3)

    point_cloud_open3d = o3d.geometry.PointCloud(o3d.utility.Vector3dVector(xyz))
    point_cloud_open3d.colors = o3d.utility.Vector3dVector(rgb / 255)
    if normals is not None:
        point_cloud_open3d.normals = o3d.utility.Vector3dVector(normals)
        print("Open 3D controls:")
        print("  n: for normals")
        print("  9: for point cloud colored by normals")
        print("  h: for all controls")

    visualizer = o3d.visualization.Visualizer()  # pylint: disable=no-member
    visualizer.create_window()
    visualizer.add_geometry(point_cloud_open3d)

    if normals is None:
        visualizer.get_render_option().background_color = (0, 0, 0)
    visualizer.get_render_option().point_size = 1
    visualizer.get_render_option().show_coordinate_frame = True
    visualizer.get_view_control().set_front([0, 0, -1])
    visualizer.get_view_control().set_up([0, -1, 0])

    visualizer.run()
    visualizer.destroy_window()

结论

本教程展示了如何使用Zivid SDK在C++和C#中可视化点云，以及如何使用第三方库在Python中实现可视化功能。还展示了如何使用第三方库在C++、C#和Python中可视化彩色图像、深度图和法线。