可视化教程
介绍
本教程介绍如何使用Zivid SDK和第三方库来可视化Zivid相机捕获的3D和2D数据。
先决条件
安装 Zivid软件。
对于Python： 安装 Zivid-python
本教程从Zivid帧（frame）开始。查看 捕获教程 了解更多有关如何捕获帧的信息。
点云
在C++和C#中使用Zivid SDK
您可以通过帧（frame）可视化点云。
std::cout << "Setting up visualization" << std::endl;
Zivid::Visualization::Visualizer visualizer;
std::cout << "Visualizing point cloud" << std::endl;
visualizer.showMaximized();
visualizer.show(frame);
visualizer.resetToFit();
std::cout << "Running visualizer. Blocking until window closes" << std::endl;
visualizer.run();
Console.WriteLine("Setting up visualization");
var visualizer = new Zivid.NET.Visualization.Visualizer();
Console.WriteLine("Visualizing point cloud");
visualizer.Show(frame);
visualizer.ShowMaximized();
visualizer.ResetToFit();
Console.WriteLine("Running visualizer. Blocking until window closes");
visualizer.Run();
您也可以从点云对象来可视化点云。
std::cout << "Getting point cloud from frame" << std::endl;
auto pointCloud = frame.pointCloud();
std::cout << "Setting up visualization" << std::endl;
Zivid::Visualization::Visualizer visualizer;
std::cout << "Visualizing point cloud" << std::endl;
visualizer.showMaximized();
visualizer.show(pointCloud);
visualizer.resetToFit();
std::cout << "Running visualizer. Blocking until window closes" << std::endl;
visualizer.run();
Console.WriteLine("Getting point cloud from frame");
var pointCloud = frame.PointCloud;
Console.WriteLine("Setting up visualization");
var visualizer = new Zivid.NET.Visualization.Visualizer();
Console.WriteLine("Visualizing point cloud");
visualizer.Show(pointCloud);
visualizer.ShowMaximized();
visualizer.ResetToFit();
Console.WriteLine("Running visualizer. Blocking until window closes");
visualizer.Run();
在Python中使用Open3D
Zivid-Python 不支持点云可视化，这里我们使用 Open3D 来实现该功能。
下面展示了如何实现点云可视化的功能。
def display_pointcloud(xyz: np.ndarray, rgb: np.ndarray, normals: Optional[np.ndarray] = None) -> None:
"""Display point cloud provided from 'xyz' with colors from 'rgb'.
Args:
rgb: RGB image
xyz: A numpy array of X, Y and Z point cloud coordinates
normals: Ordered array of normal vectors, mapped to xyz
"""
xyz = np.nan_to_num(xyz).reshape(-1, 3)
if normals is not None:
normals = np.nan_to_num(normals).reshape(-1, 3)
rgb = rgb.reshape(-1, 3)
point_cloud_open3d = o3d.geometry.PointCloud(o3d.utility.Vector3dVector(xyz))
point_cloud_open3d.colors = o3d.utility.Vector3dVector(rgb / 255)
if normals is not None:
point_cloud_open3d.normals = o3d.utility.Vector3dVector(normals)
print("Open 3D controls:")
print(" n: for normals")
print(" 9: for point cloud colored by normals")
print(" h: for all controls")
visualizer = o3d.visualization.Visualizer() # pylint: disable=no-member
visualizer.create_window()
visualizer.add_geometry(point_cloud_open3d)
if normals is None:
visualizer.get_render_option().background_color = (0, 0, 0)
visualizer.get_render_option().point_size = 1
visualizer.get_render_option().show_coordinate_frame = True
visualizer.get_view_control().set_front([0, 0, -1])
visualizer.get_view_control().set_up([0, -1, 0])
visualizer.run()
visualizer.destroy_window()
彩色图像
由于Zivid SDK 和 Zivid-Python 不支持2D彩色图像可视化，我们使用第三方库实现该功能： 使 用 OpenCV 在C++和Python中实现，以及使 用 Matplotlib 在Python中实现。
C++中使用OpenCV实现
首先，我们将点云转换为OpenCV彩色图像。
下面展示了如何实现将点云转换为彩色图像的功能。
cv::Mat pointCloudToCvBGR(const Zivid::PointCloud &pointCloud)
{
auto rgb = cv::Mat(pointCloud.height(), pointCloud.width(), CV_8UC4);
pointCloud.copyData(reinterpret_cast<Zivid::ColorRGBA *>(rgb.data));
auto bgr = cv::Mat(pointCloud.height(), pointCloud.width(), CV_8UC4);
cv::cvtColor(rgb, bgr, cv::COLOR_BGR2RGB);
return bgr;
}
小技巧
也可以直接从Zivid 2D彩色图像中获取OpenCV彩色图像。
我们现在可以可视化彩色图像了。
Python中使用OpenCV实现
首先，我们将点云转换为OpenCV彩色图像。
下面展示了如何实现将点云转换为彩色图像的功能。
def _point_cloud_to_cv_bgr(point_cloud: zivid.PointCloud) -> np.ndarray:
"""Get bgr image from frame.
Args:
point_cloud: Zivid point cloud
Returns:
bgr: BGR image (HxWx3 ndarray)
"""
rgba = point_cloud.copy_data("rgba")
# Applying color map
bgr = cv2.cvtColor(rgba, cv2.COLOR_RGBA2BGR)
return bgr
小技巧
也可以直接从Zivid 2D彩色图像中获取OpenCV彩色图像。
我们现在可以可视化彩色图像了。
下面展示了如何实现可视化功能。
def _visualize_and_save_image(image: np.ndarray, image_file: str, window_name: str) -> None:
"""Visualize and save image to file.
Args:
image: BGR image (HxWx3 ndarray)
image_file: File name
window_name: OpenCV Window name
"""
cv2.namedWindow(window_name, cv2.WINDOW_NORMAL)
cv2.imshow(window_name, image)
print("Press any key to continue")
cv2.waitKey(0)
cv2.destroyWindow(window_name)
cv2.imwrite(image_file, image)
Python中通过Matplotlib实现
Matplotlib提供了一种更简单的在Python中可视化彩色图像的方法。
函数实现如下所示。
def display_rgb(rgb: np.ndarray, title: str = "RGB image", block: bool = True) -> None:
"""Display RGB image.
Args:
rgb: RGB image (HxWx3 ndarray)
title: Image title
block: Stops the running program until the windows is closed
"""
plt.figure()
plt.imshow(rgb)
plt.title(title)
plt.show(block=block)
深度图
由于Zivid SDK 和 Zivid-Python 不支持深度图的可视化功能，我们使用第三方库实现该功能： 使 用 OpenCV 在C++和Python中实现，以及使 用 Matplotlib 在Python中实现。
C++中使用OpenCV实现
首先，我们将点云转换为OpenCV深度图。
下面展示了将点云转换为深度图的函数。
cv::Mat pointCloudToCvZ(const Zivid::PointCloud &pointCloud)
{
cv::Mat z(pointCloud.height(), pointCloud.width(), CV_8UC1, cv::Scalar(0)); // NOLINT(hicpp-signed-bitwise)
const auto points = pointCloud.copyPointsZ();
// Getting min and max values for X, Y, Z images
const auto *maxZ = std::max_element(points.data(), points.data() + pointCloud.size(), isLesserOrNan);
const auto *minZ = std::max_element(points.data(), points.data() + pointCloud.size(), isGreaterOrNaN);
// Filling in OpenCV matrix with the cloud data
for(size_t i = 0; i < pointCloud.height(); i++)
{
for(size_t j = 0; j < pointCloud.width(); j++)
{
if(std::isnan(points(i, j).z))
{
z.at<uchar>(i, j) = 0;
}
else
{
z.at<uchar>(i, j) =
static_cast<unsigned char>((255.0F * (points(i, j).z - minZ->z) / (maxZ->z - minZ->z)));
}
}
}
// Applying color map
cv::Mat zColorMap;
cv::applyColorMap(z, zColorMap, cv::COLORMAP_VIRIDIS);
// Setting invalid points (nan) to black
for(size_t i = 0; i < pointCloud.height(); i++)
{
for(size_t j = 0; j < pointCloud.width(); j++)
{
if(std::isnan(points(i, j).z))
{
auto &zRGB = zColorMap.at<cv::Vec3b>(i, j);
zRGB[0] = 0;
zRGB[1] = 0;
zRGB[2] = 0;
}
}
}
return zColorMap;
}
我们现在可以可视化深度图了。
Python中使用OpenCV实现
首先，我们将点云转换为OpenCV深度图。
下面展示了将点云转换为深度图的函数。
def _point_cloud_to_cv_z(point_cloud: zivid.PointCloud) -> np.ndarray:
"""Get depth map from frame.
Args:
point_cloud: Zivid point cloud
Returns:
depth_map_color_map: Depth map (HxWx1 ndarray)
"""
depth_map = point_cloud.copy_data("z")
depth_map_uint8 = ((depth_map - np.nanmin(depth_map)) / (np.nanmax(depth_map) - np.nanmin(depth_map)) * 255).astype(
np.uint8
)
depth_map_color_map = cv2.applyColorMap(depth_map_uint8, cv2.COLORMAP_VIRIDIS)
# Setting nans to black
depth_map_color_map[np.isnan(depth_map)[:, :]] = 0
return depth_map_color_map
我们现在可以可视化深度图了。
函数实现如下所示。
def _visualize_and_save_image(image: np.ndarray, image_file: str, window_name: str) -> None:
"""Visualize and save image to file.
Args:
image: BGR image (HxWx3 ndarray)
image_file: File name
window_name: OpenCV Window name
"""
cv2.namedWindow(window_name, cv2.WINDOW_NORMAL)
cv2.imshow(window_name, image)
print("Press any key to continue")
cv2.waitKey(0)
cv2.destroyWindow(window_name)
cv2.imwrite(image_file, image)
Python中通过Matplotlib实现
Matplotlib提供了一种更简单的在Python中可视化深度图的方法。
函数实现如下所示。
def display_depthmap(xyz: np.ndarray, block: bool = True) -> None:
"""Create and display depthmap.
Args:
xyz: A numpy array of X, Y and Z point cloud coordinates
block: Stops the running program until the windows is closed
"""
plt.figure()
plt.imshow(
xyz[:, :, 2],
vmin=np.nanmin(xyz[:, :, 2]),
vmax=np.nanmax(xyz[:, :, 2]),
cmap="viridis",
)
plt.colorbar()
plt.title("Depth map")
plt.show(block=block)
法线
由于Zivid SDK不支持法线的可视化，我们使用第三方库实现该功能：使用 PCL 在C++中实现，以及使用 Open3D 在Python中实现。
C++中的使用PCL实现
我们可以按下面的方式可视化法线。
函数实现如下所示。
void visualizePointCloudAndNormalsPCL(
const pcl::PointCloud<pcl::PointXYZRGB>::ConstPtr &pointCloud,
const pcl::PointCloud<pcl::PointXYZRGBNormal>::ConstPtr &pointCloudWithNormals)
{
auto viewer = pcl::visualization::PCLVisualizer("Viewer");
int viewRgb(0);
viewer.createViewPort(0.0, 0.0, 0.5, 1.0, viewRgb);
viewer.addText("Cloud RGB", 0, 0, "RGBText", viewRgb);
viewer.addPointCloud<pcl::PointXYZRGB>(pointCloud, "cloud", viewRgb);
const int normalsSkipped = 10;
std::cout << "Note! 1 out of " << normalsSkipped << " normals are visualized" << std::endl;
int viewNormals(0);
viewer.createViewPort(0.5, 0.0, 1.0, 1.0, viewNormals);
viewer.addText("Cloud Normals", 0, 0, "NormalsText", viewNormals);
viewer.addPointCloud<pcl::PointXYZRGBNormal>(pointCloudWithNormals, "cloudNormals", viewNormals);
viewer.addPointCloudNormals<pcl::PointXYZRGBNormal>(
pointCloudWithNormals, normalsSkipped, 1, "normals", viewNormals);
viewer.setCameraPosition(0, 0, -100, 0, -1, 0);
std::cout << "Press r to centre and zoom the viewer so that the entire cloud is visible" << std::endl;
std::cout << "Press q to exit the viewer application" << std::endl;
while(!viewer.wasStopped())
{
viewer.spinOnce(100);
std::this_thread::sleep_for(std::chrono::milliseconds(100));
}
}
在Python中使用Open3D
我们可以按下面的方式可视化法线。
normals_colormap = 0.5 * (1 - normals)
print("Visualizing normals in 2D")
display_rgb(rgb=rgba[:, :, :3], title="RGB image", block=False)
display_rgb(rgb=normals_colormap, title="Colormapped normals", block=True)
print("Visualizing normals in 3D")
display_pointcloud_with_downsampled_normals(point_cloud, zivid.PointCloud.Downsampling.by4x4)
函数实现如下所示。
def display_pointcloud_with_downsampled_normals(
point_cloud: PointCloud,
downsampling: PointCloud.Downsampling,
) -> None:
"""Display point cloud with downsampled normals.
Args:
point_cloud: A Zivid point cloud handle
downsampling: A valid Zivid downsampling factor to apply to normals
"""
rgb = point_cloud.copy_data("rgba")[:, :, :3]
xyz = point_cloud.copy_data("xyz")
point_cloud.downsample(downsampling)
normals = point_cloud.copy_data("normals")
display_pointcloud(xyz=xyz, rgb=rgb, normals=normals)
def display_pointcloud(xyz: np.ndarray, rgb: np.ndarray, normals: Optional[np.ndarray] = None) -> None:
"""Display point cloud provided from 'xyz' with colors from 'rgb'.
Args:
rgb: RGB image
xyz: A numpy array of X, Y and Z point cloud coordinates
normals: Ordered array of normal vectors, mapped to xyz
"""
xyz = np.nan_to_num(xyz).reshape(-1, 3)
if normals is not None:
normals = np.nan_to_num(normals).reshape(-1, 3)
rgb = rgb.reshape(-1, 3)
point_cloud_open3d = o3d.geometry.PointCloud(o3d.utility.Vector3dVector(xyz))
point_cloud_open3d.colors = o3d.utility.Vector3dVector(rgb / 255)
if normals is not None:
point_cloud_open3d.normals = o3d.utility.Vector3dVector(normals)
print("Open 3D controls:")
print(" n: for normals")
print(" 9: for point cloud colored by normals")
print(" h: for all controls")
visualizer = o3d.visualization.Visualizer() # pylint: disable=no-member
visualizer.create_window()
visualizer.add_geometry(point_cloud_open3d)
if normals is None:
visualizer.get_render_option().background_color = (0, 0, 0)
visualizer.get_render_option().point_size = 1
visualizer.get_render_option().show_coordinate_frame = True
visualizer.get_view_control().set_front([0, 0, -1])
visualizer.get_view_control().set_up([0, -1, 0])
visualizer.run()
visualizer.destroy_window()
结论
本教程展示了如何使用Zivid SDK在C++和C#中可视化点云，以及如何使用第三方库在Python中实现可视化功能。还展示了如何使用第三方库在C++、C#和Python中可视化彩色图像、深度图和法线。