Documentation

CI/CD Integration

Run SimuTest as a quality gate in your deployment pipeline to catch UX regressions before they reach production.

Overview

SimuTest integrates directly into CI/CD pipelines as a quality gate. On every pull request or deployment, SimuTest spins up AI-simulated users against your staging environment, scores the UX, and either passes or fails the build based on thresholds you define.

This lets you catch UX regressions the same way you catch functional bugs — automatically, before merge.

What SimuTest checks in CI

Task completion rate dropped below threshold
Overall UX score regressed from the base branch
New UX issues introduced by the change
Error rate on critical user flows increased

GitHub Actions

Use the official simutest/action GitHub Action. Add your API key as a repository secret named SIMUTEST_API_KEY under Settings → Secrets and variables → Actions.

name: SimuTest UX Quality Gate

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

jobs:
  simutest:
    name: Run SimuTest
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm ci

      - name: Start dev server
        run: npm run build && npm run start &
        env:
          NODE_ENV: production

      - name: Wait for server
        run: npx wait-on http://localhost:3000 --timeout 60000

      - name: Run SimuTest
        uses: simutest/action@v1
        with:
          api-key: ${{ secrets.SIMUTEST_API_KEY }}
          config: simutest.yaml
          github-token: ${{ secrets.GITHUB_TOKEN }}

      - name: Upload SimuTest report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: simutest-report
          path: simutest-report.json

Note: The github-token input is optional. Providing it enables SimuTest to post results as a PR comment automatically.

GitLab CI

For GitLab, use the SimuTest CLI directly. Add SIMUTEST_API_KEY as a CI/CD variable under Settings → CI/CD → Variables.

stages:
  - build
  - test

build:
  stage: build
  script:
    - npm ci
    - npm run build
  artifacts:
    paths:
      - .next/

simutest:
  stage: test
  image: node:20-alpine
  dependencies:
    - build
  before_script:
    - npm ci
    - npm run start &
    - npx wait-on http://localhost:3000 --timeout 60000
  script:
    - npx @simutest/cli run --config simutest.yaml
  variables:
    SIMUTEST_API_KEY: $SIMUTEST_API_KEY
  artifacts:
    when: always
    paths:
      - simutest-report.json
    reports:
      junit: simutest-junit.xml

Quality Gates

Quality gates define the thresholds that determine whether SimuTest passes or fails your CI build. Configure them in simutest.yaml.

Field	Options	Description
metric	score, completion_rate, error_rate, avg_time	The metric to evaluate
threshold	number	Minimum acceptable value (or maximum for error_rate and avg_time)
action	fail, warn	fail exits with a non-zero code; warn logs the issue but passes

# simutest.yaml
url: https://your-site.com
task: "Find and complete a purchase on the checkout page"
sessions: 30
model: claude-sonnet
viewport: [desktop, mobile]

quality_gates:
  - metric: score
    threshold: 70
    action: fail
  - metric: completion_rate
    threshold: 60
    action: fail
  - metric: error_rate
    threshold: 20
    action: warn

PR Comments

When a github-token is provided (GitHub Actions) or post_pr_comment: true is set in your config, SimuTest automatically posts a summary comment on the pull request. The comment compares this PR's results against the base branch so reviewers can immediately see whether UX improved or regressed.

Example PR comment:

## SimuTest Results

| Metric | This PR | Base branch | Change |
|--------|---------|-------------|--------|
| UX Score | 82 | 78 | +4 ✅ |
| Completion Rate | 91% | 86% | +5% ✅ |
| Avg. Time on Task | 42s | 49s | -7s ✅ |
| Error Rate | 8% | 12% | -4% ✅ |

**Top issues found:**
- 3 sessions struggled to locate the CTA on mobile
- 1 session encountered a form validation error with no clear message

[View full report →](https://app.simutest.dev/tests/test_abc123)

Custom Configuration

All CI settings — URL, task, session count, quality gates, and reporting options — live in a simutest.yaml file at the root of your repository. The URL field supports environment variable substitution, which is useful for pointing tests at a dynamic preview deployment URL.

# simutest.yaml
url: ${DEPLOY_URL:-http://localhost:3000}
task: "Find the pricing page and sign up for the Pro plan"
sessions: 50
model: claude-sonnet
viewport: [desktop, mobile]

quality_gates:
  - metric: score
    threshold: 75
    action: fail
  - metric: completion_rate
    threshold: 70
    action: fail

ci:
  post_pr_comment: true
  upload_artifacts: true
  artifact_path: ./simutest-report.json

For the full list of configuration options, see the YAML Config reference.

On this page