Repository navigation
Out-of-process benchmarks for more stable wall time #127
Description
When running the WallTime reporting on dedicated hardware, I'm seeing more fluctuation than I believe that Criterion is actually reporting.
I'm a bit concerned about using the compatibility layer as well, so I started working on a fork that runs Criterion without a compatibility layer and instead ingests the log files directly and sends them to CodSpeed.
If that works, I'm curious why instead of using process wrappers at all (and compatibility layers), why not just ingest all metrics for benchmarks and profiles directly from the disk and upload them to CodSpeed, instead of wrapping them in a process?
Activity
jzombie commentedon Oct 6, 2025
Update: Got baseline working and posting to CodSpeed without using Criterion compatibility adapter.
jzombie#2
Still need to work on test name resolution, but getting it to post was a major hurdle as it was initially difficult to debug errors.
[-]Out-of-process benchmarks[/-][+]Out-of-process benchmarks for more stable wall time[/+]jzombie commentedon Oct 7, 2025
The number of fluctuations outside of my thresholds went from a dozen to two, so it seems to be more stable than the wrapped process.
What's currently not working is proper source file detection.
I am also curious about seeing if the
flamegraphcrate can be easily mapped to display profile info incodspeed.