rtspmultistream

The CI badge reflects the last completed run — if a page you're viewing shows it red, that's a stale cached image (GitHub/browser image caching), not current state; the live badge and Actions tab are the source of truth. The macOS/Jetson/Windows badges are hand-set from the status table below, not CI-derived — CI only proves the scripts parse cleanly, not that they work on real hardware (see why in that table).

Caution

Use this with extreme caution. This spins up an unauthenticated media server and captures directly from a camera device. Specifically:

• The RTSP server has no authentication. It's bound to loopback only by default, but any process/user on the local machine can read or publish to /rgb, /depth, /thermal with zero credentials — see "Bugs found in review" below for what this actually exposes, and what changing the bind address back to :8554 would additionally expose to your whole network.
• It opens real camera hardware and streams live video. Don't run it somewhere you wouldn't want a camera active.
• The Windows and Jetson scripts are unverified on real hardware (see the status table below) — reviewed and statically checked, never run against real dshow/v4l2/NVENC. Treat them as drafts, not trusted tooling, especially on a shared, production, or safety-relevant device.
• No warranty. Read the scripts before running them, particularly on anything other than a personal dev machine.

Status: what's actually verified vs. what's a reviewed draft

Platform	Capture/encode script	Status
macOS	stream_macos.sh	✅ Tested on real hardware. End-to-end automated test suite (test_macos.sh) passes: correct codec/resolution/30fps on all three streams, colorization verified quantitatively (not eyeballed), loopback-only network bind confirmed, clean teardown confirmed. Runs in CI too (static analysis only, no camera in CI).
Windows	stream_windows.ps1	⚠️ Not run on Windows. Parse-checked and statically analyzed (PSScriptAnalyzer) with zero errors; failure-handling path exercised for real on this machine using the actual ffmpeg binary (with a fake camera name, to safely trigger the error path without needing dshow). The actual dshow capture success path has never run against a real webcam.
Jetson Orin Nano / AGX Orin	stream_jetson.sh	⚠️ Not run on Jetson. Statically checked with shellcheck, zero warnings. Never executed — no v4l2 device or ARM hardware available here.

CI (.github/workflows/ci.yml) runs shellcheck on both bash scripts and a parse + PSScriptAnalyzer pass on the PowerShell script on every push — this catches syntax/lint regressions automatically, but cannot validate actual camera capture, RTSP publish, or hardware encode on any platform, since no CI runner (any OS) has a physical camera.

One MacBook camera → three synchronized RTSP streams (rgb, depth, thermal) for developing/demoing a multi-sensor ingestion pipeline without real depth/thermal hardware.

rtsp://localhost:8554/rgb
rtsp://localhost:8554/depth
rtsp://localhost:8554/thermal

Design decisions, and where I changed the original plan

The starting point for this was a ChatGPT-drafted plan (one RTSP server, three paths, VideoToolbox hardware encoding, cheap pseudo-depth/thermal transforms). That framing was right. Three things about how it proposed getting there were wrong or incomplete, and are done differently here:

1. "Capture once, fan out" was only a diagram, never actually implemented. The plan drew three boxes coming off one capture, then pivoted to a Python microservice (fusionbench-sensor-simulator) without showing how the fan-out itself would work. FFmpeg already does this natively with a single -filter_complex ... split=3 ... graph and multiple -map/output pairs in one process — no application code needed. That's what stream_macos.sh does: one avfoundation capture, one filter graph, three h264_videotoolbox encodes, three RTSP pushes, one process.

2. The proposed frame-sync mechanism (invented frame_id, propagated by hand) is solving a problem this architecture doesn't have. Because all three outputs are split from the same decoded frame inside one filter graph, they inherit the same source PTS lineage for free — perfect sync, zero code. Hand-rolled frame IDs only become necessary later, when rgb/depth/thermal come from physically separate sensors/processes. Not needed for this stage; noted as a real V2 concern when real hardware sensors are wired in, not implemented here.

3. The pseudo-depth/pseudo-thermal transforms are real FFmpeg filters, not a hand-wave. The plan described "pseudo depth mapping" and "thermal-like characteristics" conceptually but never named a mechanism. This uses FFmpeg's pseudocolor filter with the turbo preset (the standard depth-map colormap) for /depth, and the heat preset for /thermal, with a gblur/eq pre-pass to make depth look spatially smoothed and thermal look contrast-boosted. One non-obvious gotcha found while building this: pseudocolor needs a 3-plane input (yuv420p/rgb24) to have chroma planes to write the colormap into — feeding it single-plane format=gray first (the obvious thing to try) silently no-ops and passes the frame through as plain grayscale. stream_macos.sh feeds it yuv420p directly, which is why the colors actually show up.

4. Flagged, not silently ignored: VideoToolbox has a limited number of concurrent hardware encode sessions. Running three h264_videotoolbox sessions at once (one per branch) is untested territory the original plan didn't address. stream_macos.sh passes -allow_sw 1 on every branch so that if a hardware session is unavailable, that branch falls back to software H.264 instead of the whole process failing. In testing on this M2, three concurrent hardware sessions worked fine — but if you see stalls on your hardware, this is the first thing to check (-loglevel warning will show it).

5. The plan asserted "30 FPS" as simply achievable; it isn't, by default, on real hardware. Measured two real bugs while verifying actual output frame rate (not just declared/requested rate — always check both):

   • Leaving -pixel_format unset made avfoundation silently fall back to a different camera preset when the requested yuv420p@640x480@30fps combo wasn't available — the pipeline was actually capturing and publishing at ~120fps, 4x the intended rate, wasting CPU/bandwidth on all three branches. First fix pinned -pixel_format uyvy422, but that's specific to this MacBook's camera — tested directly that a recognized pixel format unsupported by a given camera's active mode silently free-falls the same way (only formats ffmpeg doesn't recognize at all fail loudly), so hardcoding one format doesn't generalize to other cameras. stream_macos.sh now probes a short capture per candidate format (uyvy422, yuyv422, nv12, 0rgb, bgr0) at startup and keeps the first one that isn't rejected and doesn't come back at a measured fps wildly higher than requested (the tell for a silent preset swap) — same failure signature the original bug had, now guarded against rather than just avoided for this one camera.
   • Even with the correct format pinned, this camera's auto-exposure caps real frame delivery at ~23fps in normal room light (AVFoundation lengthens exposure time per frame in lower light, capping max achievable fps below whatever you request — normal camera behavior, not a bug). stream_macos.sh forces -r "$FPS" on every output branch, which pads with duplicate frames so the published RTSP streams are an honest, correctly declared 30fps CFR — real temporal resolution is still ~23 unique frames/sec unless the room is brighter. Verified this holds for all three paths, not just one, by measuring actual delivered frame counts over a wall-clock window on each — declared frame rate in the SDP is not sufficient evidence on its own. The duplicate/real ratio isn't just a one-time manual measurement either: all three scripts now run with -stats, so ffmpeg's own dup=/drop= counters are visible in the running process's output whenever duplication is actually happening (it only prints them when nonzero, so this adds no noise the rest of the time) — no separate diagnostic tooling needed for something ffmpeg already tracks internally.

One thing kept as-is from the plan and correct as proposed: single RTSP server, one port, three paths — much cleaner than three separate RTSP servers/ports.

Deliberately not built yet: the Python fusionbench-sensor-simulator service. It's the right move later, specifically when /depth needs a real monocular depth-estimation model (CoreML/Metal) — FFmpeg can't run that. Until then it would just be more moving parts wrapping the same ffmpeg invocation, with the sync guarantee re-implemented instead of free. Swapping this for a Python service later is a drop-in change: same three RTSP URLs, same consumers.

Further review feedback: adopted, verified-and-rejected, or declined

A later round of feedback proposed several changes. Some restated what's already built above (frame sync via shared PTS, not an invented ID — already the design), some were genuinely useful and got implemented, and one was tested and found to make things worse, not better:

Adopted:

• The duplicate/unique-frame ratio should be an observable diagnostic, not just README prose. Already covered above via -stats.

Tested, and rejected because it's actually worse: the suggestion that scraping ffmpeg's human-readable stderr for the pixel-format probe is "fragile across versions" and should use structured output instead (e.g. ffprobe -show_entries stream=... -of json) is reasonable in the abstract. Tried it directly against this camera: ffprobe's r_frame_rate on a live avfoundation capture returned 1000000/1 (a nonsense placeholder) and avg_frame_rate returned 0/0 — both fields are simply undefined/unreliable for a live raw device the way they are for an already-muxed file with real container headers. The current approach — parsing ffmpeg's own dynamically measured summary line from an actual short capture — is the more reliable signal here, not the fragile one, for this specific case. Kept as-is. (The text format itself has also been stable across ffmpeg releases for well over a decade — it's not some volatile debug string.)

Declined, with reasoning:

• Restructuring this repo into a fusionbench/apps/services/sdk/tools/... monorepo skeleton, or moving these files into a tools/webcam-sensor-simulator/ subdirectory. There is no FusionBench platform code anywhere in this repo — it doesn't exist yet. Building empty directories for services/ingestion, services/synchronization, sdk/connectors etc. before any of that code exists is designing for a hypothetical future shape rather than the thing actually being built. This repo already is the equivalent of the tools/webcam-sensor-simulator being described; there's no larger monorepo around it to be nested inside of. If/when a consuming platform actually gets built, this can be pulled into a subdirectory of that repo then.
• Renaming the /depth//thermal RTSP paths to something like /sim_depth. The whole reason these are stable, real-sensor-shaped path names is the explicit design goal from the start of this project (see the fusionbench-sensor-simulator note above): a consumer built against rtsp://.../depth shouldn't need to change anything when that path is later backed by a real depth sensor instead of a webcam simulation. Renaming the path undermines the one design property this was built to have. The simulated nature is already documented extensively (this file, every script's header comments, the top-of-README disclaimer) — that's the right layer for this distinction, not the URL.
• Adopting the proposed AC-01–AC-10 acceptance criteria wholesale. Most (AC-04 reconnect, AC-06 UI, AC-07 modality labeling, AC-08 skew reporting) assume a consuming application, synchronizer, and UI that don't exist in this repo — they're acceptance criteria for a different, unbuilt system. AC-02/AC-03 (resolution/fps) are already covered by test_macos.sh. AC-01 (three streams staying up for a 30-minute soak) is a reasonable test this repo doesn't have yet — not added here since it's a genuinely separate, longer-running test rather than a quick fix, but a fair one to ask for explicitly if useful.

Restated, not new: the point about RTSP not automatically carrying a shared application-level frame_id once these become independent processes is correct and already the design — see point 2 above (PTS-based sync from the shared split, explicit IDs deferred to when there are real independent sensors). Worth restating as an explicit edge case for later: once /rgb, /depth, /thermal are three genuinely separate RTSP sessions (even from this same one-process setup, they're independent RTSP publishes), a consumer correlates frames across them via RTP/wall-clock timestamps, not a magic shared identifier — there's nothing in RTSP itself that stitches sessions together. True today, will still be true when real sensors replace the simulator.

Bugs found in review

Two review passes so far, both with a "verify empirically, don't just read the diff" bar.

Round 1 — after the first working version was pushed:

• Network exposure: mediamtx.yml originally bound rtspAddress: :8554, which is all interfaces (confirmed with lsof), with no auth and an open all_others dynamic path — anyone on the same LAN/WiFi could view, or even publish/hijack, /rgb, /depth, /thermal. Changed to rtspAddress: 127.0.0.1:8554. Verified: while running it only accepts connections via loopback; after stopping the process, lsof -iTCP:8554 shows nothing bound and no lingering state — this is a plain foreground process, not an installed service, so there's nothing left behind once it's killed. Also verified no other listener sneaks in — MediaMTX's api/metrics/pprof/playback servers all default to disabled, confirmed by checking what a running instance actually has open (lsof -p <pid>), not just reading the config. Trade-off: loopback-only also blocks watching the stream from another device (e.g. a phone) on your network — change back to :8554 if you want that, now with eyes open about what it exposes.
• Pixel-format portability (medium): stream_macos.sh hardcoded -pixel_format uyvy422, specific to this MacBook's camera. Fixed by probing candidate formats at startup instead (see design section above).

Round 2 — asked to be "extremely exigent" and re-check everything, including the two platforms I can't run:

• stream_macos.sh's probe only checked fps, not resolution. A silent fallback could in principle swap resolution instead of (or as well as) frame rate, and the probe wouldn't have caught it. Fixed: the probe now also requires the declared WIDTHxHEIGHT to appear in the candidate's probe output before accepting it. Regression-tested by reproducing the exact original bug (feeding pseudocolor single-plane gray again) and confirming test_macos.sh catches it — it does, colorfulness score 0.0 vs. a >15 threshold.
• stream_jetson.sh had no input-format probing at all — the exact same risk class already found and fixed on macOS, just not carried over. v4l2 has the same failure shape as avfoundation: many UVC webcams default to MJPEG-only at higher res/fps and ffmpeg won't necessarily pick a working format on its own. Fixed: added the same probe-and-verify pattern (yuyv422/mjpeg/nv12/uyvy422 candidates, checked against both measured fps and resolution). Can't verify this actually fixes anything on real Jetson hardware — only that the logic mirrors the pattern already proven correct on macOS, and it passes shellcheck clean.
• stream_windows.ps1 silently swallowed ffmpeg failures. $ErrorActionPreference = "Stop" does not catch a native .exe's non-zero exit code in PowerShell — that's a real, easy-to-miss language gotcha, not a bash-style guarantee carrying over. If ffmpeg.exe failed (wrong camera name, dshow can't open the device), the script would exit looking successful. Fixed with explicit $LASTEXITCODE checks. Verified for real: installed PowerShell on this Mac and ran the script against a fake camera name using the real ffmpeg binary (-f dshow doesn't exist on macOS, so this safely exercises the failure path) — before the fix this would have exited 0; after the fix it correctly reports the error and exits
  1. Also ran the script's argument-construction logic standalone to confirm the built ffmpeg argument array is well-formed (right element count, camera names with spaces stay as one argument, correct ordering).
• stream_windows.ps1 had no MJPEG fallback for the dshow equivalent of the same pixel-format risk (many USB webcams need -vcodec mjpeg at higher res/fps). Fixed: added a Test-DshowCapture probe that tries raw capture first, then MJPEG, before committing to the real run.
• Confirmed via pwsh's AST parser and PSScriptAnalyzer (installed locally to check for real rather than eyeballing): zero parse errors, zero Error-severity findings on stream_windows.ps1. The only findings are style-level Write-Host warnings, which is intentional here (interactive status output, not a value meant to be piped/captured).

Left open, low severity: every run logs Color range not set for yuv420p. Using MPEG range from the h264_videotoolbox encoders on macOS. Cosmetic today — output already verified correct — but the range is technically ambiguous to strict playback clients. Not fixed since pinning it risks a subtle shift in the verified-correct pseudocolor output; flagging rather than touching blind.

Also flagging as an accepted, not fixed, residual risk: even with the RTSP server loopback-bound, paths: all_others: still means any other local process or user account on the same machine can publish to or read from /rgb, /depth, /thermal with no credentials — the fix above closes the network-wide exposure, not local multi-user exposure. Fine for a personal single-user dev machine; worth revisiting (add publishUser/readUser to mediamtx.yml) before running this on anything shared.

Cross-platform: Windows / Jetson Orin Nano / Jetson AGX Orin

The architecture ports — MediaMTX ships static binaries for Windows and Linux/arm64, so mediamtx.yml is used unchanged on all three platforms. The capture+encode script does not, since avfoundation/h264_videotoolbox are macOS-only. Two more scripts exist for this:

• stream_windows.ps1 — dshow capture, encoder auto-selected from nvenc/qsv/amf/libx264 (whichever ffmpeg reports available), same split + pseudocolor filter graph.
• stream_jetson.sh — v4l2 capture (USB/UVC webcam only — a CSI/MIPI camera module needs NVIDIA's nvarguscamerasrc via GStreamer instead, a different pipeline this doesn't cover), software libx264 by default since JetPack's stock ffmpeg has no NVENC support out of the box (override with ENCODER=h264_nvmpi if you've built jetson-ffmpeg).

Both are unverified — written and reasoned through carefully, but there's no Windows machine or Jetson board in this environment to actually run them against, unlike stream_macos.sh which was tested end-to-end (frame rate measured, colorization inspected visually, network bind confirmed). Known risks going in:

• Windows dshow device names and supported pixel formats vary a lot by webcam driver — you'll likely need to check ffmpeg -list_devices true -f dshow -i dummy and possibly add -vcodec mjpeg if your webcam only exposes MJPEG at this resolution.
• JetPack's bundled ffmpeg version may predate the pseudocolor filter's turbo/heat presets. Both scripts probe for this at startup and degrade to plain grayscale for /depth and /thermal if the presets aren't available, rather than guessing at hand-rolled color math that can't be verified here — upgrade ffmpeg to get color back.
• If you hit a failure on either platform, send me the ffmpeg stderr output — I can't reproduce dshow/v4l2/NVENC behavior locally to debug blind.

Requirements

Already verified installed/working in this environment:

• ffmpeg (8.1.1, Homebrew) — has avfoundation, h264_videotoolbox, and pseudocolor with the turbo/heat presets built in.
• mediamtx (1.20.0, Homebrew) — brew install mediamtx.
• macOS camera permission granted to the terminal/app running ffmpeg (you'll get a one-time OS prompt the first time you capture).

Run it

Terminal 1 — start the RTSP server (project-local config, no system-wide install/service needed):

mediamtx ./mediamtx.yml

Terminal 2 — find your camera index (usually 0 for the built-in camera), then publish:

ffmpeg -f avfoundation -list_devices true -i ""   # confirm the index once
./stream_macos.sh

Override defaults via env vars if needed:

DEVICE=1 WIDTH=1280 HEIGHT=720 FPS=30 ./stream_macos.sh

On Windows: .\stream_windows.ps1 -CameraName "Integrated Camera" (find the exact name via ffmpeg -list_devices true -f dshow -i dummy).

On Jetson/Linux: DEVICE=/dev/video0 ./stream_jetson.sh. Both unverified — see the cross-platform section above before running.

Verify

ffprobe -rtsp_transport tcp rtsp://localhost:8554/rgb
ffprobe -rtsp_transport tcp rtsp://localhost:8554/depth
ffprobe -rtsp_transport tcp rtsp://localhost:8554/thermal

Or watch one live: ffplay -rtsp_transport tcp rtsp://localhost:8554/depth

Testing

./test_macos.sh

Real end-to-end test, macOS only (needs this machine's camera + mediamtx + ffmpeg). It starts the actual server and the actual capture script, then asserts on measured output rather than "did it not crash":

• MediaMTX bound to loopback only, and exactly one listener (no surprise extra ports from features we thought were off)
• All three streams: codec is h264, resolution is exactly 640x480, fps is exactly 30/1
• /depth and /thermal are actually colorized, checked quantitatively — a small embedded Python script reads a raw decoded frame and scores average |R-G|+|G-B|+|R-B| per pixel; true grayscale scores ~0, the real pseudocolor output scores 300+. This is the same check that catches the exact historical pseudocolor-fed-single-plane-gray regression (verified: reproducing that bug on purpose makes this test fail with a 0.0 score, confirming the check isn't a rubber stamp).
• Full teardown: port 8554 is completely released after stopping both processes.

No equivalent exists for Jetson or Windows — there's no matching hardware anywhere to run a real test against, not just in this session. .github/workflows/ci.yml covers what's actually possible without hardware: shellcheck on both bash scripts (test_macos.sh included) and a parse + PSScriptAnalyzer pass on the PowerShell script, on every push. See the status table at the top for exactly what that CI badge does and doesn't cover.

Files

• stream_macos.sh — the whole pipeline on macOS: one capture, one filter graph (split + per-branch transform), three encodes, three RTSP publishes. Verified end-to-end; see the review section above.
• stream_jetson.sh — Linux/v4l2 variant for Jetson Orin Nano/AGX Orin. Unverified, see status table and cross-platform section.
• stream_windows.ps1 — Windows/dshow variant. Unverified, see status table and cross-platform section.
• test_macos.sh — automated end-to-end test suite, macOS only. See Testing section above.
• mediamtx.yml — minimal RTSP-only server config, bound to loopback only (RTMP/HLS/WebRTC/SRT disabled since nothing here uses them; re-enable in the config if you want browser playback via HLS/WebRTC later). Same file works on all three platforms.
• .github/workflows/ci.yml — runs shellcheck + PowerShell static analysis on every push. Does not and cannot exercise real camera/RTSP behavior — no CI runner has a physical camera.