perf(desktop-gpui): profile the macOS camera effect and tally Vision labels 3.6x faster by richiemcilroy · Pull Request #2867 · CapSoftware/Cap

richiemcilroy · 2026-10-22T00:14:14Z

Profiles the macOS live camera effect (blur and background removal) on synthetic person frames and fixes the one piece of waste that leaves the output provably identical. No camera, microphone or screen capture was used anywhere.

What changed

camera_blur/bench.rs (test-only, #[ignore]): drives the exact preview path, a 420v camera frame converted by VideoToolbox into the 720p BGRA IOSurface ring, then camera_blur::Worker::process (Vision instance mask request on its thread, guided filter, composite, BGRA surface, cutout snapshot) at 30 fps for 720p, 1080p and 4K sources. Frames are pans across public-domain NASA portrait photos (CAP_CAMERA_EFFECT_BENCH_IMAGES). Reports process time, worker CPU, CPU by thread, masks/s, mask age and footprint, plus a frame-synchronous output comparison (IoU, alpha PSNR, composite PSNR) against a saved reference.
vision.rs: the primary-person pick tallied the 512x384 instance label map with one areas[label] += 1 table. Nearly every label repeats its neighbour, so each increment waited on the previous store to the same counter. Four interleaved tables keep four independent chains: 0.36 ms -> 0.10 ms per mask on M4 Max. Counts are identical by construction (unit test against a plain count), and 90 frames of Remove output are bit-identical to main (IoU 1.0000, max alpha error 0).

Profile (M4 Max, macOS 27, local run of the CI-built bench binary)

1080p source, Remove, 30 fps:

stage	per mask (wall / CPU)
VT copy into Vision input (960x540)	0.44 / 0.09 ms
`VNGeneratePersonInstanceMaskRequest` perform (ANE)	6.7 / 0.8 ms
label tally (main)	0.36 / 0.36 ms
label tally (this PR)	0.10 / 0.10 ms
`generateMaskForInstances` + read	0.14 ms
motion sample + cleanup (holes, weak regions, smoothing)	0.54 ms

run	worker ms/frame p50	worker CPU ms/frame	whole process CPU, % of a core	masks/s
Remove 720p	2.07	0.68	10.6	21.1
Remove 1080p	2.04	0.64	10.1	20.6
Remove 4K	2.03	0.68	9.9	19.4
Heavy 1080p	1.91	0.51	10.0	20.8
Light 1080p	1.85	0.46	9.6	20.5

By thread (Remove 1080p): Vision internals 4.1%, our Vision thread 3.9%, blur worker 2.0%. The source resolution only changes the VideoToolbox conversion (0.56 / 0.98 / 3.1 ms wall for 720p / 1080p / 4K).

Findings:

The whole blur/removal path is about 10% of one core, not the ~48% measured in the app. Vision runs on the ANE (6.7 ms wall, 0.8 ms CPU) and already reuses its VNSequenceRequestHandler and request. The other ~38% is outside this path (preview painting and the cutout RenderImage upload on the main thread, capture, conversion, the mic meter); this bench cannot see those.
The ~440 MB is GPU driver memory, not blur buffers. On this machine any real GPU work by the process makes the driver allocate a "graphics" footprint and keep it until the GPU has been idle for about a second: a single 64 px wgpu clear is +390 MB, a buffer copy +165 MB, a raw Metal fillBuffer on a fresh MTLCommandQueue +165 MB. It is per process, not per queue: two more raw queues doing the same fill while the first allocation was still held added ~4 MB each, and the queue's max command buffer count (default, 64, 2048) makes no difference. The effect's own allocations are about 20-50 MB. The macOS 26 CI VM (paravirtual GPU) showed only +90 MB for the same first run. So the 747 MB in the app is mostly "something renders every frame", and turning blur off alone will not return it while the preview keeps painting.
Vision input size does not matter for cost. The instance mask is 512x384 for every input size, and Vision's CPU per call is the same at 1280x720, 960x540, 640x360 or 512x288. Smaller inputs only move the edge slightly (640x360: IoU 0.9993, alpha PSNR 47.9 dB, max alpha error 135 vs 960x540), so nothing to gain; left as is.
Forcing Vision to the GPU adds ~100 MB and to the CPU adds ~100 MB of malloc; the default (ANE) is already the best choice.
Masks run at ~21/s, not 30: the 33 ms inference interval against 33.3 ms frames with timer jitter skips about one frame in three. Not waste, just the effective rate.

HOLD for Richie (quality/latency trade-offs, not in this PR)

Segmentation rate while the bubble is idle, 1080p Remove, same bench:

inference interval	process CPU, % of a core	masks/s	mask age p50
33 ms (current)	10.3	20.8	36 ms
66 ms	7.1	12.4	65 ms
100 ms	5.8	8.5	74 ms

Halving the rate saves ~3 points of a core and doubles how far the cutout edge lags motion. Could be applied only while not recording, if that is acceptable.

Bigger follow-up outside this PR's files: Remove mode copies every 1280x720 cutout into a new RenderImage (0.15 ms on the worker) that the main thread then uploads to the sprite atlas, because gpui's BGRA surface shader forces alpha to 1. A gpui surface path that keeps alpha would let the cutout paint straight from the IOSurface like the other modes. Not touched here: it needs a gpui patch change and sits in camera_window.rs, where #2761 and #2806 are open.

Testing

cargo test --bin cap-gpui camera_blur::bench -- --ignored on Blacksmith macOS 26 (build, plus Vision on the VM's CPU fallback) and the same binary on an M4 Max for the numbers above; base and head binaries alternated three times.
label_tallies_match_a_plain_count unit test for the tally.

Conflicts: none with #2761, #2806 or #2849 (this touches camera_blur.rs, the new camera_blur/bench.rs and crates/camera-effects/src/vision.rs only).

Confidence Score: 5/5

The PR appears safe to merge, with one non-blocking reference-file check still incomplete.

Findings

Incomplete references look identical ▶

Fix with agent prompt

### Issue 1
apps/desktop-gpui/src/camera_blur/bench.rs:497-500
`compare` silently ignores missing frames and pixels because both loops use `zip`. A reference file missing its final frame can therefore report perfect scores against all 90 candidate frames. Check frame counts, frame lengths, and leftover reference bytes before computing scores.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds an ignored macOS benchmark for camera effects and speeds up Vision label counting with four independent tables.

The camera blur worker can be profiled with synthetic frames.
Remove-mode output can be compared frame by frame.
Vision picks the largest person with a faster label count.

Review was based on code inspection. The macOS benchmarks were not run.

_{Reviews (3) · Last reviewed commit: "fix(desktop-gpui): check compare sizes, ..." · Reviewed by Greptile}

…son frames

…d tables

vercel · 2026-10-22T00:14:17Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
cap-status	Ready	Preview	Oct 24, 2026 5:17am UTC

capy-ai

Capy found no issues.

View 3 other findings in Capy.

greptile-apps · 2026-10-23T21:59:34Z

+    for (reference, candidate) in reference.iter().zip(candidate) {
+        let (mut intersection, mut union) = (0u64, 0u64);
+        let (mut alpha_error, mut color_error) = (0f64, 0f64);
+        for (a, b) in reference.chunks_exact(4).zip(candidate.chunks_exact(4)) {


Incomplete references look identical

compare silently ignores missing frames and pixels because both loops use zip. A reference file missing its final frame can therefore report perfect scores against all 90 candidate frames. Check frame counts, frame lengths, and leftover reference bytes before computing scores.

Prompt To Fix With AI

This is a comment left during a code review. Path: apps/desktop-gpui/src/camera_blur/bench.rs Line: 497-500 Comment: **Incomplete references look identical** `compare` silently ignores missing frames and pixels because both loops use `zip`. A reference file missing its final frame can therefore report perfect scores against all 90 candidate frames. Check frame counts, frame lengths, and leftover reference bytes before computing scores. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fixed in 7e7e2c0. compare now asserts the frame counts match and each frame pair has the same byte length before scoring, so a truncated or oversized reference fails instead of reporting a perfect match.

greptile-apps · 2026-10-23T21:59:34Z

+        let cpu_start = thread_cpu_ns();
+        let start = Instant::now();
+        let job = converter.convert(frame_at(index), mode);
+        let converted = start.elapsed();
+        let output = worker.process(&runtime, job).unwrap();
+        let elapsed = start.elapsed();
+        let cpu = thread_cpu_ns() - cpu_start;


Worker CPU includes conversion

worker_cpu_ms includes Converter::convert, while process_ms explicitly excludes it. The reported worker CPU therefore includes conversion work, which changes with source resolution. Start a separate CPU timer immediately before Worker::process, or label the metric as conversion plus worker CPU.

Prompt To Fix With AI

This is a comment left during a code review. Path: apps/desktop-gpui/src/camera_blur/bench.rs Line: 331-337 Comment: **Worker CPU includes conversion** `worker_cpu_ms` includes `Converter::convert`, while `process_ms` explicitly excludes it. The reported worker CPU therefore includes conversion work, which changes with source resolution. Start a separate CPU timer immediately before `Worker::process`, or label the metric as conversion plus worker CPU. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fixed in 7e7e2c0. The thread CPU timer now starts right before Worker::process, after Converter::convert, so worker_cpu_ms covers the same span as process_ms.

greptile-apps · 2026-10-23T21:59:35Z

+    time.tv_sec as u64 * 1_000_000_000 + time.tv_nsec as u64
+}
+
+/// CPU time per live thread, keyed by thread id, with its name.


Comments repeat the code

The new comment above threads only describes what the function returns. AGENTS.md requires comments to explain non-obvious context and explicitly bans narrating code or restating types. Remove this comment or replace it with a needed explanation. The comment above camera_frames follows the same pattern. This repository requirement must be satisfied before merging.

Context Used: AGENTS.md (source)

Prompt To Fix With AI

This is a comment left during a code review. Path: apps/desktop-gpui/src/camera_blur/bench.rs Line: 204 Comment: **Comments repeat the code** The new comment above `threads` only describes what the function returns. `AGENTS.md` requires comments to explain non-obvious context and explicitly bans narrating code or restating types. Remove this comment or replace it with a needed explanation. The comment above `camera_frames` follows the same pattern. This repository requirement must be satisfied before merging. **Context Used:** AGENTS.md ([source](https://github.com/capsoftware/cap/blob/main/AGENTS.md)) --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fixed in 7e7e2c0. Removed the comments above threads and camera_frames.

richiemcilroy · 2026-10-24T05:08:37Z

@greptileai review

…rsion

capy-ai

Capy found no issues.

View 1 other finding in Capy.

richiemcilroy · 2026-10-24T05:39:20Z

@greptileai review

richiemcilroy added 3 commits October 22, 2026 06:25

test(desktop-gpui): benchmark the live camera effect on synthetic per…

12970d3

…son frames

perf(camera-effects): tally Vision instance labels in four interleave…

1d92609

…d tables

test(desktop-gpui): saturate the camera effect bench's stage subtraction

ff1442d

vercel Bot deployed to Preview – cap-status October 22, 2026 00:14

View deployment

vercel Bot deployed to Preview – cap-web October 22, 2026 00:16

View deployment

This comment has been minimized.

Sign in to view

richiemcilroy marked this pull request as ready for review October 23, 2026 21:55

Merge branch 'main' into perf/gpui-macos-camera-segmentation

57aed9d

vercel Bot deployed to Preview – cap-status October 23, 2026 21:56

View deployment

capy-ai Bot reviewed Oct 23, 2026

View reviewed changes

vercel Bot deployed to Preview – cap-web October 23, 2026 21:58

View deployment

greptile-apps Bot reviewed Oct 23, 2026

View reviewed changes

fix(desktop-gpui): check compare sizes, time worker CPU without conve…

7e7e2c0

…rsion

vercel Bot deployed to Preview – cap-status October 24, 2026 05:17

View deployment

vercel Bot deployed to Preview – cap-web October 24, 2026 05:18

View deployment

capy-ai Bot reviewed Oct 24, 2026

View reviewed changes

richiemcilroy merged commit eb682ee into main Oct 24, 2026
53 checks passed

Conversation

richiemcilroy commented Oct 22, 2026 • edited by greptile-apps Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed

Profile (M4 Max, macOS 27, local run of the CI-built bench binary)

HOLD for Richie (quality/latency trade-offs, not in this PR)

Testing

Confidence Score: 5/5

Findings

Summary

Uh oh!

vercel Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

capy-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

richiemcilroy Oct 24, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

richiemcilroy Oct 24, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

richiemcilroy Oct 24, 2026

Choose a reason for hiding this comment

Uh oh!

richiemcilroy commented Oct 24, 2026

Uh oh!

capy-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

richiemcilroy commented Oct 24, 2026

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

richiemcilroy commented Oct 22, 2026 •

edited by greptile-apps Bot

Loading

vercel Bot commented Oct 22, 2026 •

edited

Loading