Benchmarking and Profiling

Benchmarking is the process of measuring the performance of a program. It can be used to compare the execution time of two different algorithms. Profiling can be used to analyze where time is spent during the evaluation of an algorithm. We suggest to use criterion for benchmarking and flamegraph for profiling.

Benchmarking with Criterion

criterion supports statistical analysis of the runtime of your program. A plotting library has to be installed to generate graphs. You can find more information and help here:

• Criterion-rs GitHub
• Cargo-criterion GitHub
• Criterion Book (!Check your criterion version. As of writing this, the criterion book is not up-to-date.)

Commands for Criterion

a) cargo criterion <benchmark name regex>
Has to be installed with cargo install cargo-criterion.
Pros:

• You can remove features = ["html_reports"] from the Cargo.toml, leading to (slightly) faster compile times.
• criterion aims to move to just using cargo criterion
• The large Probability Density Function graph shows the samples and marks the outlier categorization borders.
• can use either gnuplot or plotters

b) cargo bench <benchmark name regex>
Pros:

• Can visualize the change in performance compared to the previous run or other baseline

Cons:

• can only use gnuplot

Profiling with Flamegraph

Flamegraphs provide insights into where execution time is spent. Thus, they enable efficient identification of bottlenecks in your code.

Details can be found here:

• Flamegraph GitHub.

Installation

Installing flamegraph in Linux and macOS is usually done by simply installing flamegraph using the following command.

cargo install flamegraph

In WSL you need some more steps, since you need to install a dependency called perf manually.

After running cargo install flamegraph, ensure your list of packages is up-to-date using sudo apt update and install several packages among build-essentials by running

sudo apt install -y build-essential libelf-dev libnuma-dev flex bison libdw-dev libunwind-dev libaudit-dev libslang2-dev libperl-dev python3-dev binutils-dev liblzma-dev libiberty-dev

Then you go into the home directory with cd ~, find the current kernel you’re running by executing uname -r, and download the appropriate linux kernel with an adjusted version of the following link.

wget - https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.15.5.tar.xz

Next, you extract the downloaded file with tar -xvf linux-5.15.5.tar.xz and move into perf with cd linux-5.15.5/tools/perf. In this directory, run make to compile it. The last step is to make perf visible / add it to your path, e.g., by running sudo cp ./perf /usr/local/bin/.

Command for Flamegraph

cargo flamegraph --freq 63300 --bench benchmarks -- --bench --profile-time 5 <benchmark name regex>
Generates a flamegraph that allows us to approximate how long each function executes. The accuracy of the approximation is better the more samples are produced. That can be improved by

• increasing the sampling frequency (--freq 63300). This frequency is throttled to the highest possible frequency, which depends on the CPU, CPU-temperature, power settings and much more…
• increasing profile-time (in seconds). That is how long the benchmark code will be executed. This parameter also disables the statistical analysis of criterion which prevents it from showing up in the graph. This parameter is optional but suggested.

The flamegraph can be overwhelming since it exposes a lot of internal workings of Rust, criterion, and more. The easiest way to find a specific function is to search with Ctrl + F. Simply enter a part of the Rust function name or regex (not the benchmark name).