Skip to content

perf(desktop-gpui): IDCT-scale JPEG library thumbnails - #2721

Open
kyzer1023 wants to merge 3 commits into
CapSoftware:mainfrom
kyzer1023:cursor/jpeg-thumbnail-idct-f70c
Open

kyzer1023 wants to merge 3 commits into
CapSoftware:mainfrom
kyzer1023:cursor/jpeg-thumbnail-idct-f70c

Conversation

@kyzer1023

@kyzer1023 kyzer1023 commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Library card thumbnails (thumbnail_rgba in apps/desktop-gpui/src/library.rs) decoded JPEGs at full resolution and then resampled to the 392×224 cover size. JPEGs now use jpeg_decoder::Decoder::scale to IDCT-scale toward the cover size first, then resize to the exact size. A 1920×1080 or 3840×2160 frame decodes at 480×270.

The sidebar's existing IDCT decode moved into a shared jpeg_thumbnail.rs, so decode_jpeg_thumbnail is the only IDCT decoder. The sidebar still asks for a square max; library cards pass the cover size. PNG is unchanged. Unsupported formats (CMYK, 16-bit gray) are rejected after read_info() and before any decode, so they go straight to the existing full-decode fallback.

The resize filter is picked from the decoded buffer's size against the cover target (thumbnail_resize), so the near 1:1 step after IDCT (for example 480×270 to 398×224) uses Triangle, while a full-size buffer still uses box sampling. Nothing is upscaled.

Cache keys are unchanged (thumbnail-<mtime>.jpg, preview-thumbnail-<mtime>.jpg, and the shared {hash}-{w}x{h}-<mtime>.jpg), so existing thumbnails stay valid and no orphaned files are left behind.

3 commits, 4 files, +246/-32. Based on main at 85219015.

Correctness

jpeg_thumbnail tests pass (5): square downsample, historical cover sizes, 1080p to 480×270, a complete 8×8 CMYK JPEG returns None, and the resize choice follows the decoded buffer rather than the original scale. Run through a standalone crate that path-includes the module; clippy and rustfmt are clean on the touched files. The full desktop-gpui app and the cache-slot tests in library.rs were not built here because gpui is patched to a local zed-cap checkout.

Benchmarks

Release, Linux VM, median of 7 samples after one warmup. Before is the old image full decode plus cover resample.

Source Before After Speedup Mean channel diff
Synthetic 1920×1080 7.12 ms 5.00 ms 1.42× 4.73
Synthetic 3840×2160 23.08 ms 12.73 ms 1.81× 9.96
sequoia-dark.jpg 4000×2500 36.07 ms 20.05 ms 1.80× 1.40
tahoe-dark.jpg 3840×2160 68.03 ms 54.30 ms 1.25× 1.58

A second run put synthetic 1080p at 1.67× and 4K at 1.87×. These numbers predate the switch to Triangle for the post-IDCT resize.

Overlap

Does not touch clip thumbnails (#2707) or recording recovery in library.rs (#2630).

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no actionable issues remain.

Summary

The PR shares JPEG thumbnail decoding between the sidebar and library. Library cards decode fewer pixels before resizing.

  • Library cards shrink JPEGs before building their thumbnail images.

This review checked the code; it did not run the app or tests.

Reviews (3) · Last reviewed commit: "address review feedback" · Reviewed by Greptile

Co-authored-by: Kyzer  <kyzer1023@users.noreply.github.com>
@vercel

vercel Bot commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

@kyzer1023 is attempting to deploy a commit to the Cap Software Inc Team on Vercel.

A member of the Team first needs to authorize it.

Comment thread apps/desktop-gpui/src/jpeg_thumbnail.rs Outdated
Comment on lines +101 to +105
let bytes = [
0xFF, 0xD8, 0xFF, 0xC0, 0x00, 0x14, 0x08, 0x00, 0x08, 0x00, 0x08, 0x04, 0x01, 0x11,
0x00, 0x02, 0x11, 0x00, 0x03, 0x11, 0x00, 0x04, 0x11, 0x00,
];
assert!(decode_jpeg_thumbnail(&bytes, |w, h| (w, h)).is_none());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 CMYK test misses its branch

cmyk_frame_returns_none uses a truncated JPEG header with no scan data. The test passes when decoding fails, even if CMYK rejection is removed. Use a complete CMYK JPEG so the test guards the unsupported-format branch.

Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/desktop-gpui/src/jpeg_thumbnail.rs
Line: 101-105

Comment:
**CMYK test misses its branch**

`cmyk_frame_returns_none` uses a truncated JPEG header with no scan data. The test passes when decoding fails, even if CMYK rejection is removed. Use a complete CMYK JPEG so the test guards the unsupported-format branch.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b519849. The test now builds a complete 8x8 CMYK JPEG, asserts the fixture itself decodes as CMYK32, then asserts the thumbnail path returns None.

Comment on lines +17 to +18
let pixels = decoder.decode().ok()?;
let decoded = match info.pixel_format {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Unsupported JPEGs decode twice

The shared helper decodes pixels before rejecting unsupported formats. Library thumbnails now pay for that discarded decode and then decode again through the fallback. Check info.pixel_format before calling decoder.decode() so CMYK and 16-bit gray files go straight to the fallback.

Suggested change
let pixels = decoder.decode().ok()?;
let decoded = match info.pixel_format {
if !matches!(
info.pixel_format,
jpeg_decoder::PixelFormat::RGB24 | jpeg_decoder::PixelFormat::L8
) {
return None;
}
let pixels = decoder.decode().ok()?;
let decoded = match info.pixel_format {
Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/desktop-gpui/src/jpeg_thumbnail.rs
Line: 17-18

Comment:
**Unsupported JPEGs decode twice**

The shared helper decodes pixels before rejecting unsupported formats. Library thumbnails now pay for that discarded decode and then decode again through the fallback. Check `info.pixel_format` before calling `decoder.decode()` so CMYK and 16-bit gray files go straight to the fallback.

```suggestion
    if !matches!(
        info.pixel_format,
        jpeg_decoder::PixelFormat::RGB24 | jpeg_decoder::PixelFormat::L8
    ) {
        return None;
    }
    let pixels = decoder.decode().ok()?;
    let decoded = match info.pixel_format {
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b519849. It calls read_info() first and returns None for anything other than RGB24 or L8 before scale or decode, so unsupported files only go through the fallback decode once.

Comment thread apps/desktop-gpui/src/jpeg_thumbnail.rs Outdated
Some((decoded, u32::from(info.width), u32::from(info.height)))
}

/// Cover size for a card. Never upscales.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Comment breaks the guide

The new cover_target comment only describes what the function does. The repository guide bans comments that narrate code and requires non-obvious context instead. Remove this comment or express the guarantee through a test. This repository requirement must be satisfied before merging.

Suggested change
/// Cover size for a card. Never upscales.

Context Used: AGENTS.md (source)

Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/desktop-gpui/src/jpeg_thumbnail.rs
Line: 34

Comment:
**Comment breaks the guide**

The new `cover_target` comment only describes what the function does. The repository guide bans comments that narrate code and requires non-obvious context instead. Remove this comment or express the guarantee through a test. This repository requirement must be satisfied before merging.

```suggestion

```

**Context Used:** AGENTS.md ([source](https://github.com/capsoftware/cap/blob/main/AGENTS.md))

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed in b519849, along with the other restating comment on decode_jpeg_thumbnail.

Co-authored-by: Kyzer  <kyzer1023@users.noreply.github.com>

Copy link
Copy Markdown
Contributor Author

@greptileai review

@richiemcilroy

Copy link
Copy Markdown
Member

Thanks @kyzer1023, the IDCT path looks right. Two small things: 1. Bumping the cache prefix (library.rs:812, 965 to 975) leaves the old thumbnail-<mtime>.jpg and preview-thumbnail-<mtime>.jpg inside every recording and screenshot bundle, plus the old entries in the shared thumbnail cache, for good, because remove_stale_variants only sweeps the new prefix. The old thumbnails are still valid at card size, so the simplest fix is to drop THUMBNAIL_CACHE_GENERATION and keep the existing keys; otherwise please also remove the legacy-prefix files when a new slot is stored. 2. In thumbnail_rgba (library.rs:1127) the filter is still picked from scale against the original size, so a 1080p JPEG now goes 480x270 to 398x224 through thumbnail_exact, the near 1:1 case the comment above says should use Triangle. Please choose the filter from decoded.width()/decoded.height() against the target.

Co-authored-by: Kyzer  <kyzer1023@users.noreply.github.com>

Copy link
Copy Markdown
Contributor Author

Thanks @richiemcilroy, both addressed in d16afcb.

  1. Dropped THUMBNAIL_CACHE_GENERATION. Bundle keys are back to thumbnail-<mtime>.jpg / preview-thumbnail-<mtime>.jpg and the shared cache key is unchanged, so existing thumbnails keep being used and nothing is orphaned. The cache-slot test asserts the original prefixes.
  2. thumbnail_rgba now picks the resize via thumbnail_resize, using the decoded buffer's size against the cover target. A 1080p JPEG (IDCT 480x270 to 398x224) goes through Triangle, a full-size buffer still uses box sampling, and nothing upscales. Added a test for that choice.

The jpeg_thumbnail tests (5) pass with clippy and rustfmt clean; the full desktop-gpui build wasn't run on my side since gpui points at a local zed-cap checkout.

Copy link
Copy Markdown
Contributor Author

@greptileai review

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants