Repository navigation
Different results running benchmarks with and without the action #127
Description
Hello! Thanks for this amazing project. Maybe I'm too new on this, but I'm trying to configure the codspeed action, I'm getting different results in some scenarios. I'm working on a project which mixes python and rust through pyo3, and I have a test here, Which behaves completely different when I run it with and without the action.
import time
import pytest
import numpy as np
from scipy.spatial.distance import cdist # type: ignore
from pymoors._pymoors import cross_euclidean_distances # type: ignore
def test_cdist_pymoors(benchmark):
matrix = np.random.rand(5000, 5000)
result = benchmark(cross_euclidean_distances, matrix, matrix)
assert result.shape == (5000, 5000)
def eculidean_distance_plus_time(matrix):
start_time = time.perf_counter()
result = cross_euclidean_distances(matrix, matrix)
end_time = time.perf_counter()
total = end_time - start_time
return result, total
@pytest.mark.parametrize("n", [1000, 1500, 2000, 2500])
def test_compare_scipy_cdist_vs_pymoors(benchmark, n):
matrix = np.random.rand(n, n)
result_pymoors, total_time = benchmark(eculidean_distance_plus_time, matrix)
# Measure the time for scipy's cdist using a timer
start_time = time.perf_counter()
result_scipy = cdist(matrix, matrix, metric="sqeuclidean")
end_time = time.perf_counter()
time_scipy = end_time - start_time
# Check the result is the same
np.testing.assert_allclose(result_pymoors, result_scipy, rtol=1e-5, atol=1e-8)
# Check the time
assert total_time < time_scipyIf I use the action the tests run like if I've deployed my package in debug mode, comparable with the time i got locally when i run them in debug mode
- name: Run benchmarks
uses: CodSpeedHQ/action@v3
with:
token: ${{ secrets.CODSPEED_TOKEN }}
run: |
pytest -s -vvv tests/benchmarks --codspeed
But when I remove the action, and run the benchmark directly, the tests run just fine, like in release mode
- name: Run benchmarks
run: pytest -s -vvv tests/benchmarks --codspeed
Is the action introducing a change in the runner machine? Is there a doc where I can see what is the action doing?
Thanks!
Activity
andresliszt commentedon Mar 9, 2025
I fixed it using
I can't find info related to the env var
CODSPEED_RUNNER_MODE, I found it looking at the runner codeadriencaccia commentedon Mar 10, 2025
Hey @andresliszt, thanks for taking the time to create an issue.
By default, the
CodSpeedHQ/actionwill run your code using theCPUinstrument. As explained in the docs, it will run your benchmark once on a simulated CPU and count its cycles, instead of measuring real time. This allow to have consistent measurements not subject to the noisyness of CI runners.The
CODPSEED_RUNNER_MODE=walltimeenv variable enables the Walltime instrument, which will measure the real wall time of your code. But this mode is subject to noisyness, this is why we recommend using our Macro Runners–bare-metal runners optimised for performanced provided and managed by CodSpeed– instead of the ones provided by GitHub.We did not document this env variable since using this mode does not make sense on regular runners, and is enabled by default on our Macro Runners.
I hope I clarified things a bit, let me know if something is not clear!
andresliszt commentedon Mar 10, 2025
Thank for answering this!. I have updated my test, the one showed in the issue is the one on main, working fine with
walltime. Note carefully that in the test i'm comparing my execution time given byagainst scipy execution time,
total_timecomes from the functioneculidean_distance_plus_time, which as I understand is completely independent from the codspeed stuff, so that time should be the same for both runner modes. My test with the default runner mode fails in the assertionassert total_time < time_scipy, where in that scenario my execution time is close to the debug execution time. With the walltime mode, my execution time is similar to the release one. Is that expected?adriencaccia commentedon Mar 10, 2025
Code that is being called passed in the
benchmarkfixture will be instrumented on the simulated CPU, so its execution will be slower than what it would be on a normal CPU.In your case, with the current benchmark code, you are comparing the time it took to run your implementation under the CodSpeed instrumentation to the time it took to run scipy's implementation without the CodSpeed instrumentation.
So for you assertion to be relevant, you would have to change your benchmarked code and wrap both the execution and time tracking of both your code and scipy's in the same function, and use the
benchmarkfixture on that function.That way both implementations will be instrumented on the simulated CPU, and you will have relevant things to compare.