Skip to content

feat: e2e tests - #1501

Merged
dannyhw merged 11 commits into
callstack:mainfrom
MrMuzyk:e2e-ci
Oct 25, 2026
Merged

dannyhw merged 11 commits into
callstack:mainfrom
MrMuzyk:e2e-ci

Conversation

@MrMuzyk

@MrMuzyk MrMuzyk commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds an e2e lane that runs the MF2 tester app on a real emulator/simulator on CI, so Re.Pack bundling/runtime regressions are caught before merge instead of in a consumer app.

  • New workflow .github/workflows/e2e.yml — Android on an API 35 emulator, iOS on an iPhone 16 simulator. Each job starts both Re.Pack dev servers (host app :8081, MiniApp :8082), warms both platform compilers, builds the Debug app, then runs the suite against it. Android runs on PRs; iOS runs on merges to main (and dispatch), since it can't fit inside a PR's deadline yet — Phase 2 moves it onto a beefier runner.
  • Suite: apps/tester-federation-v2/e2e/smoke.e2e.ts — 3 deterministic tests (app boots, Home → Detail → back, Home → Gallery loads the MiniApp over the MF remote), via tester-army/e2e + @e2e-dev/mobile. No model credential needed. retries: 0 and workers: 1 are set explicitly: a flaked first attempt must read as a failure, not flaky, and the device is pinned (E2E_DEVICE) so a local run never lands on someone else's booted emulator.
  • Trigger: plain pull_request with paths: filters (app, packages, lockfile, the workflow itself), contents: read, no secrets, and the default merge-ref checkout — so GitHub's fork-workflow approval is the gate, no label needed, and PR edits to e2e.yml are self-tested before merge. A push to main never cancels, so the "main is healthy" signal survives a back-to-back merge. Telemetry off at workflow level (E2E_TELEMETRY_DISABLED: '1').
  • catalog: shared-version fix. All four MF configs now build their shared block from one new configs/sharedSingletons.mts helper, which takes each version from the installed package's own package.json.

Notes

  • e2e@0.19.0 / @e2e-dev/mobile@0.11.0 declare engines.node: ^22.22.3 || >=24.8.0 while the repo's root engines allows >=24, and there's no engine-strict — on Node 24.0–24.7 a contributor only gets a warning and then a confusing failure. CI pins 24.18.0.
  • tester-federation (v1) carries the same catalog: bug in its configs (incl. async-storage) — deliberately untouched here; worth a follow-up PR.
  • Building iOS locally on Xcode 27 needs IPHONEOS_DEPLOYMENT_TARGET=15.1 for xcodebuild (AsyncStorage's pod targets 13.0). CI on macos-15/Xcode 16.4 is unaffected.

Test plan

CI is the evidence — with the pull_request trigger, the Android lane runs on this PR's head; iOS can be proven per-PR via workflow_dispatch and runs on every merge to main. On failure, the e2e-android / e2e-ios artifacts carry junit + screenshots + logcat + both dev server logs.

Locally (needs Node ≥ 24.8 or 22.22.3; CI pins 24.18.0):

  1. Boot an emulator/simulator and install the Debug app (pnpm android / pnpm ios with the app's own scripts).
  2. cd apps/tester-federation-v2, start both servers: pnpm start:hostapp and pnpm start:miniapp (no flag needed anymore).
  3. E2E_DEVICE=<name or serial> pnpm exec e2e run --target android (or --target ios). With several devices booted, the pin keeps the run off the ones other work is using; unset, @e2e-dev/mobile picks up every booted device.

@vercel

vercel Bot commented Oct 16, 2026

Copy link
Copy Markdown

@MrMuzyk is attempting to deploy a commit to the Callstack Team on Vercel.

A member of the Team first needs to authorize it.

@changeset-bot

changeset-bot Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 2ccc592

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

# Conflicts:
#	apps/tester-federation-v2/package.json
#	pnpm-lock.yaml
@MrMuzyk
MrMuzyk marked this pull request as ready for review October 16, 2026 13:59

@dannyhw dannyhw left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your work on this. I ran the suite locally at 4feac606 on a dedicated API 35 emulator and an iOS 26.4 simulator: 3/3 on Android and 3/3 on iOS, with --workers 1 --retries 0, in a separate run per platform. So the tests themselves work.

Inline comments cover the details. The main points:

  1. Trigger model: I'd switch from pull_request_target to pull_request. See the comment on on:.
  2. The root cause from #1499 is still there. LogBox.ignoreAllLogs(true) hides the toast, but the MF2 shared versions still resolve to catalog:testers. Without E2E=true, the Gallery test still fails locally.
  3. Determinism: pin the device, set retries: 0 explicitly, and turn telemetry off.

Also:

  • e2e label: the repo doesn't have it yet (gh label list returns nothing). Without it, labelled fork runs are skipped silently. The description already lists this as a step before merge, so I'm noting it here so it doesn't get missed.
  • No CI run at this head. The green run linked in the description is a workflow_dispatch on c33c7bb3. The only difference from 4feac606 is that src/host/index.js loses the comment explaining the LogBox gating. If the gating stays, please keep that comment.
  • Running locally with Xcode 27: the iOS build needs IPHONEOS_DEPLOYMENT_TARGET=15.1 passed to xcodebuild, because AsyncStorage's pod targets 13.0. This doesn't affect CI on macos-15/Xcode 16.4.

Comment thread .github/workflows/e2e.yml Outdated
Comment thread .github/workflows/e2e.yml Outdated
Comment thread .github/workflows/e2e.yml Outdated
Comment thread .github/workflows/e2e.yml Outdated
Comment thread .github/workflows/e2e.yml Outdated
Comment thread apps/tester-federation-v2/e2e.config.ts Outdated
Comment thread apps/tester-federation-v2/src/host/index.js Outdated
Comment thread apps/tester-federation-v2/configs/rspack.host-app.mts Outdated
Comment thread apps/tester-federation-v2/package.json Outdated
Comment thread apps/tester-federation-v2/src/host/screens/MiniAppScreen.tsx Outdated
@MrMuzyk

MrMuzyk commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

@dannyhw Ive addressed the comments and moved away from the approach with 'e2e' label as implied by the comments in the review. It's ready for another round of review ;)

@MrMuzyk
MrMuzyk requested a review from dannyhw October 19, 2026 13:36
Comment thread .github/workflows/e2e.yml
Comment thread apps/tester-federation-v2/configs/rspack.host-app.mts Outdated
@MrMuzyk

MrMuzyk commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

@CristiCeban

Added manifest check to bot jobs; Added gathering ios device logs; Added a new helper for configs;

Ready for re-review :)

@MrMuzyk
MrMuzyk requested a review from CristiCeban October 20, 2026 11:09

@dannyhw dannyhw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work.
Thanks for your contribution 🙏

@dannyhw
dannyhw merged commit 171e3ec into callstack:main Oct 25, 2026
5 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants