Documentation

YAML Configuration

Declaratively define all your UX tests, quality gates, and authentication in a single simutest.yaml file.

Overview

The YAML config format lets you define all your SimuTest tests as code. Check the file into your repository alongside your application and run simutest run --config simutest.yaml in CI to validate UX on every pull request.

A starter config is available at simutest.example.yaml in the root of this repository. Copy it to simutest.yaml and edit to match your application.

File Structure

A config file has four top-level keys:

version: 1          # required, must be 1

defaults:            # applied to all tests unless overridden
  model: claude-sonnet-4-20250514
  sessions: 100
  viewport: [desktop, mobile]
  timeout: 120

tests:               # list of test definitions
  - name: "..."
    url: "..."
    task: "..."

quality_gates:       # thresholds that trigger pass/fail/warn
  - metric: overall_score
    threshold: 7.0
    action: fail

Key	Required	Description
version	Yes	Config schema version — must be `1`
defaults	No	Default settings applied to all tests
tests	Yes	List of test definitions
quality_gates	No	Score thresholds that trigger pass/warn/fail

Defaults Block

Settings in the defaults block are applied to every test. Any test can override them by specifying the same key directly.

defaults:
  model: claude-sonnet-4-20250514   # AI model for all tests
  sessions: 100                      # sessions per test
  viewport: [desktop, mobile]        # run on both viewports
  timeout: 120                       # seconds before a session times out

Key	Type	Description
model	string	AI model to use (e.g. claude-sonnet-4-20250514)
sessions	number	Number of simulated sessions per test
viewport	string \| string[]	'desktop' \| 'mobile' \| 'tablet' or an array of viewports
timeout	number	Per-session timeout in seconds

Test Definitions

Each entry in the tests array defines one UX test. Settings here override the defaults block.

tests:
  - name: "Homepage UX"
    url: http://localhost:3000
    task: "Find the pricing page"
    criteria: [cta_findability, navigation_clarity]

  - name: "Checkout Flow"
    url: http://localhost:3000/checkout
    task: "Complete a purchase with the default cart items"
    sessions: 200                    # override the default
    criteria: [task_completion, trust_credibility, error_handling]

Key	Required	Description
name	Yes	Human-readable test name shown in reports
url	Yes	URL of the page under test
task	Yes	Natural language task for the AI agent to complete
sessions	No	Override the default session count for this test
criteria	No	List of evaluation metrics (e.g. `task_completion`)
personas	No	`auto-diverse` or a list of custom persona definitions
auth	No	Authentication configuration for protected pages

Quality Gates

Quality gates define score thresholds. When a test run completes, SimuTest checks every gate and exits with a non-zero code if any fail gate is not met — suitable for blocking CI builds.

quality_gates:
  - metric: overall_score
    threshold: 7.0
    action: fail      # exit with non-zero code

  - metric: task_completion
    threshold: 6.5
    action: fail

  - metric: trust_credibility
    threshold: 6.0
    action: warn      # print warning but do not fail

Key	Description
metric	Score to check: `overall_score` or any criteria name
threshold	Minimum acceptable score (0–10)
action	`fail` — exit non-zero \| `warn` — print warning and continue

Auth in YAML

SimuTest supports five authentication methods. Sensitive values are always read from environment variables — never hard-coded in the YAML file.

credentials — form login

Navigates to a login page, fills in username and password fields, and waits for a redirect.

- name: "Dashboard UX (credentials auth)"
  url: https://app.example.com/dashboard
  task: "Find and export the monthly analytics report"
  auth:
    method: credentials
    login_url: https://app.example.com/login
    username_env: TEST_USERNAME       # env var name (not the value)
    password_env: TEST_PASSWORD
    success_url: https://app.example.com/dashboard

token — inject into storage

Writes an authentication token directly into localStorage, sessionStorage, or a cookie before loading the page.

- name: "Admin Panel (token auth)"
  url: https://app.example.com/admin
  task: "Navigate to user management and search for a user"
  auth:
    method: token
    token_env: TEST_AUTH_TOKEN        # env var holding the token
    storage_type: localStorage        # localStorage | sessionStorage | cookie
    key: authToken                    # key to write the token into

http_basic — HTTP Basic Auth

Adds HTTP Basic Auth headers to every request. Used for staging environments behind basic auth.

- name: "Staging Site (HTTP basic auth)"
  url: https://staging.example.com
  task: "Verify the homepage loads correctly"
  auth:
    method: http_basic
    username_env: STAGING_USERNAME
    password_env: STAGING_PASSWORD

cookies — inject cookies

Injects one or more cookies before the session starts. Useful for cookie-based session tokens.

- name: "Cookie-authenticated page"
  url: https://app.example.com/dashboard
  task: "View the analytics dashboard"
  auth:
    method: cookies
    cookies:
      - name: session_id
        value_env: SESSION_COOKIE_VALUE
        domain: app.example.com
        path: /
        http_only: true

storage_state — Playwright state

Restores a full browser storage state (cookies + localStorage) from a PlaywrightstorageState() JSON snapshot. Useful when you already have Playwright E2E tests.

- name: "Playwright storage state"
  url: https://app.example.com/dashboard
  task: "Export the weekly report"
  auth:
    method: storage_state
    storage_state_env: STORAGE_STATE_JSON   # JSON string of Playwright storage state

Full Example

The complete annotated simutest.yaml showing all features together:

version: 1

defaults:
  model: claude-sonnet-4-20250514
  sessions: 100
  viewport: [desktop, mobile]
  timeout: 120

tests:
  - name: "Homepage UX"
    url: http://localhost:3000
    task: "Find the pricing page"
    criteria: [cta_findability, navigation_clarity]

  - name: "Checkout Flow"
    url: http://localhost:3000/checkout
    task: "Complete a purchase with the default cart items"
    sessions: 200
    criteria: [task_completion, trust_credibility, error_handling]

  - name: "Dashboard UX (credentials auth)"
    url: https://app.example.com/dashboard
    task: "Find and export the monthly analytics report"
    auth:
      method: credentials
      login_url: https://app.example.com/login
      username_env: TEST_USERNAME
      password_env: TEST_PASSWORD
      success_url: https://app.example.com/dashboard

  - name: "Admin Panel (token auth)"
    url: https://app.example.com/admin
    task: "Navigate to user management and search for a user"
    auth:
      method: token
      token_env: TEST_AUTH_TOKEN
      storage_type: localStorage
      key: authToken

  - name: "Staging Site (HTTP basic auth)"
    url: https://staging.example.com
    task: "Verify the homepage loads correctly"
    auth:
      method: http_basic
      username_env: STAGING_USERNAME
      password_env: STAGING_PASSWORD

quality_gates:
  - metric: overall_score
    threshold: 7.0
    action: fail
  - metric: task_completion
    threshold: 6.5
    action: fail
  - metric: trust_credibility
    threshold: 6.0
    action: warn

On this page