Benchmarking Your System

Introduction

Capture speed depends on more than the camera. The GPU, the CPU, the PCIe and network topology, the operating system, and the SDK version all contribute to how long a capture takes on your system. The numbers published in Capture Speed are measured on Zivid’s reference hardware, so the only way to know what your own system achieves is to measure it.

This tutorial shows how to benchmark your system, how to turn the results into a report, and how to share that report with Zivid Customer Success.

When to benchmark

Benchmarking your own system is useful when:

  • You are sizing hardware for a new application and need to know whether a given PC meets your cycle-time budget.

  • Your capture times are slower than the published Capture Speed numbers and you want to find out why.

  • Capture times have changed after a hardware, driver, operating system, or SDK upgrade.

  • Zivid Customer Success has asked you for benchmark data to help diagnose a performance issue.

Build and run the benchmark

The benchmark is the C++ sample ZividBenchmark.cpp. Build it as part of the C++ samples, as described in zivid-cpp-samples.

Close other GPU-heavy applications before running, so that they do not compete for the GPU while the benchmark is measuring.

Run it with a camera connected, timing the presets for that camera:

./ZividBenchmark --presets

ZividBenchmark.exe --presets

The presets are installed with the sample data, see Sample Data. The benchmark selects the ones belonging to the camera it is connected to. Every system running this measures the same settings, which is what lets one system be held against another. It is also what the published Capture Speed tables are measured with, so your numbers line up with those directly. Use this form when you send results to Zivid.

A full run takes several minutes. Each measurement is printed as it completes, and the results are stored in a CSV file in the working directory for the report to read.

The ambient-light variants of each preset are skipped, since they mostly repeat the same work with a different exposure. Add --all-ambient-variants to include them. That matters when your scene is lit at 50 or 60 Hz, because a preset and its 50 Hz variant are not interchangeable on time.

The report from such a run leads with those settings, ordered fastest first. Each bar is split into the time the camera spent acquiring and the time this PC spent turning that into a point cloud:

../../../_images/benchmark-report-presets-example.png

The acquisition half is set by the settings themselves - exposure, aperture, how many acquisitions - and changes little from one PC to another. The processing half is the part your hardware decides. Reading them separately tells you whether a slow capture is slow because of the settings you chose or because of the machine running them. That is the difference between tuning settings and buying a better GPU.

The same run also measures other metrics, such as 2D capture, connection time and saving to disk.

Measure your own settings

To time settings you have tuned yourself, save them as .yml files in a directory and point --settings-dir at it:

./ZividBenchmark --settings-dir ./my-settings

ZividBenchmark.exe --settings-dir .\my-settings

Every .yml file below that directory is timed over ten 3D captures and reported under the name of the file. Use this to pick the settings that meet your cycle-time budget, and to see what they cost on your hardware rather than on ours.

You can export a preset to a settings file from Zivid Studio, so a directory of candidates can mix presets you have adjusted with settings of your own.

Note

Each settings file adds roughly a minute to the run, more for the slower ones. Point --settings-dir at a handful of candidates rather than at every settings file you own.

Compare against the published numbers

The tables in Capture Speed are measured with the same presets that --presets times. A run made that way is therefore directly comparable: the same settings on the same camera model, measured on your hardware instead of ours.

Look up your camera model in those tables and compare preset for preset. A capture that is markedly slower on your system than in the table points at the system rather than the settings, and the split between acquisition and processing shows which half to look at.

Settings of your own have no row in those tables. Use Calculate 3D Capture Speed instead to estimate the acquisition time to expect. That estimate covers the acquisition half of each bar, the part the settings decide. The processing half is down to your hardware, so read it against the presets measured in the same run.

Generate a report

The Python sample visualize_benchmark_results.py turns the measurements into a report. It needs the Python sample dependencies installed, see zivid-python-samples.

Point it at the file the benchmark wrote:

python visualize_benchmark_results.py zivid_benchmark_results.csv

Run without arguments and it picks the most recent benchmark results in the working directory.

This writes benchmark_overview.png, leading with the settings files as shown above. A run made without --presets or --settings-dir opens with where a single capture spends its time instead, and the rest of the layout is unchanged.

Underneath the leading chart sit the supporting panels:

  • Connecting and 2D capture - the measurements the leading chart does not already cover. On a log scale, since connecting takes seconds and a 2D capture takes milliseconds.

  • Copying and saving data - what it costs to get a point cloud out of the SDK and onto disk. On a log scale, because saving XYZ and copying a single channel are orders of magnitude apart.

Every bar is a median. The benchmark repeats each measurement and reports the median rather than the mean, so one slow repetition does not move the number you read.

The settings files also carry an error bar: the standard deviation of the total capture time over the repetitions. A short bar means the setting is predictable on this machine. A long one means the same capture sometimes takes considerably longer, which is what a cycle time has to absorb. Thermal throttling is one cause: the benchmark captures back to back, and the camera pauses between acquisitions to keep the projector from overheating. See projector brightness for what drives it. A long bar is worth chasing before you commit to that setting.

To compare several systems - for example two PCs, two SDK versions, or the same PC before and after a change - pass several result files:

python visualize_benchmark_results.py system_a.csv system_b.csv

This writes benchmark_comparison.png, which places the configurations side by side and labels each series by whatever differs between them (operating system, camera model, SDK version, or GPU).

For a deeper look at a single dataset, add --detailed to also write one PNG per category with time series, per-settings breakdown and distribution plots. Use --summary-only to print the summary statistics without producing any plots.

Send the results to Zivid

If you want help interpreting the results, or you think your system is underperforming, send them to us at customersuccess@zivid.com.

Please include:

  • The result file or files produced by the benchmark. This is the most important part - it holds the raw measurements and your system information, and lets us repeat your analysis.

  • Every Zivid SDK log file from the same run. A run is often spread over several log files, and they are only useful to us as a set, so please include all of them rather than the newest or largest one. The report lists their full paths, so you do not have to go looking for them.

  • The generated PNG report, if you produced one.

  • A short description of what you are trying to achieve, including your target cycle time.

  • Anything unusual about the setup, for example a virtualized environment, a shared GPU, or a long or non-standard cable run.

If you cannot generate the report, send the result files and the logs on their own. Those are enough for us to do the analysis on our side.

Note

The benchmark results contain your camera serial number and information about your PC. They do not contain any point cloud data or images.

Further reading