Mobile Fastlane

App Testing agent (AI tests)Preview

Last updated 2026-10-07

The App Testing agent (preview, powered by Gemini in Firebase) runs natural-language test cases on real devices after each upload. You describe a goal such as "log in and open the patient list", and the agent drives the app to reach it.

Before you begin

Complete the Firebase setup page and use FIREBASE_TOKEN authentication. The build must be an APK.

Step 1: Opt in

The project owner goes to Firebase Console → App Distribution → your app → Test cases and accepts the App Testing / Gemini in Firebase terms.

Step 2: Create test cases

Create test cases in the console, with a goal, a hint and success criteria for each step. Copy each test case ID into fastlane/config/test_case_ids.yml.

Step 3: Choose devices

Choose devices in fastlane/config/testing.yml. Each test case runs once per device and orientation, and each run counts toward the quota.

Step 4: Use FIREBASE_TOKEN

Use FIREBASE_TOKEN from the account that opted in.

Warning: service accounts are rejected with 403 PERMISSION_DENIED, even with every IAM role.

Step 5: Test login (optional)

If a test signs in, set FIREBASE_TEST_USERNAME / FIREBASE_TEST_PASSWORD. Use a dedicated test account, never a real user's.

Step 6: Run and view results

my-app — zsh
~/my-app $ bundle exec fastlane android distribute environment:staging
[14:07:00]: --- Firebase App Testing Agent ---
[14:07:00]: Test cases: login-flow, first-test-case
[14:07:00]: Devices: Galaxy A54 (API 34, portrait, landscape), Pixel 9 (API 36, portrait)
[14:07:02]: Waiting for AI test results (blocking: true)...
[14:10:12]: ✅ login-flow       Galaxy A54 · Android 14 · Portrait   PASSED
[14:10:12]: ✅ login-flow       Galaxy A54 · Android 14 · Landscape  PASSED
[14:10:12]: ✅ first-test-case  Pixel 9 · Android 16 · Portrait      PASSED
[14:10:13]: Results: https://console.firebase.google.com/project/my-app/appdistribution/...
[14:10:14]: App Testing Agent report sent to Teams and e-mail.
With blocking: true, a failed test fails the lane.

Each run produces a Teams "App Testing Agent Report" Adaptive Card with:

  • a header with status (✅ PASSED / ❌ FAILED)
  • Total / Passed / Failed / Duration stat tiles
  • build details: platform, environment, version, build, artifact, branch, commit, triggered by
  • a per-test-case ✅/❌ list
  • a "Test Devices (N)" list with friendly names (e.g. Galaxy A54 · Android 14 · Portrait · Landscape)
  • failure details, when there are any
  • "View Firebase Results" and "View CI Run" buttons

The same report is also sent as an HTML e-mail.

App Testing Agent Report
✅ PASSED
4
Total
4
Passed
0
Failed
3m 12s
Duration
Platform
Android
Environment
staging
Version
1.0.0
Build
42
Artifact
MyApp-staging-v1.0.0-42.apk
Branch
main
Commit
a1b2c3d
Triggered by
GitHub Actions

Test cases

  • ✅ first-test-case
  • ✅ login-flow

Test Devices (2)

  • Galaxy A54 · Android 14 · Portrait · Landscape
  • Pixel 9 · Android 16 · Portrait
View Firebase ResultsView CI Run

Known limitations

Limitations:
  • Quota: 200 tests / month / project during preview. One test = one test case × one device, so 10 devices × 1 test case = 10 tests per distribution.
  • Timeout: 5 minutes per test.
  • FLAG_SECURE blocks the agent, which then sees a blank screen. If your app sets FLAG_SECURE, disable it for dev/staging builds.
  • APK only: AAB builds skip AI tests.
  • blocking: false uploads and starts the tests, but doesn't wait for (or fail on) results.

Preview features and quotas change. Check the official App Testing agent page for current limits.

Official documentation

Was this helpful?
PreviousSet up Firebase App DistributionNextBuild and distribute