VibeTesting skill (coding agents)

The VibeTesting skill is a repo template folder you install beside your codebase so a local coding agent in any vibecoding tool or IDE that supports agent skills can plan flows from repository context and start cloud executions via the same External API you use from CI/CD.

Example prompt — you describe the run in natural language; the agent turns it into a plan and invokes the CLI (after you approve):

/vibetesting run 2 tests, max 20 steps each — Test 1: verify login and logout.
Test 2: verify error messages on a wrong login attempt. Use test user my-user1
where login is needed, and send the report to my email.

• Locally: your agent summarizes what to test (natural-language goals), optional correlation metadata, and run limits.
• In the cloud: VibeTesting runs the browser and executes steps — consumes credits when steps run (see Pricing & Billing).

Official API reference remains the site docs: /docs/api/.

---

Folder layout

After you unpack the skill template you get one skill directory (location depends on your tool; e.g. .cursor/skills/vibetesting/ or similar) containing:

Item	Purpose
SKILL.md	Instructions the agent reads (skill name/description in YAML frontmatter)
references/	Deeper references bundled with the skill (API field notes, flow guidelines)
scripts/	Python CLI (run_vibetesting.py, preflight.py, setup_vibetesting.py, client helpers)
config.json	Non-secret defaults: run limits, target mode (see API token storage)
runtime/	Gitignored. last-run/last_run.json, optional plans/*.json

Do not put API tokens in the skill directory. Config files live at the skill root next to SKILL.md. The CLI resolves the skill home as the directory that contains scripts/, unless you pass --skill-home.

---

Prerequisites

• Python 3.9+ (stdlib only — no pip packages required)
• A VibeTesting account and an API token from the dashboard (API section), stored as described under API Token Storage below
• Your app URL configured / verified on the dashboard (see Core Concepts and target modes below)

---

Setup

API Token Storage

VibeTesting API tokens should not be stored in the skill folder, repo files, config.json, or chat.

For local agent skill usage, the token is read from:

• VIBETESTING_API_TOKEN
• ~/.vibetesting/credentials.json

On Windows:

• %USERPROFILE%\.vibetesting\credentials.json

Recommended setup:

python scripts/setup_vibetesting.py

Important notes:

• Do not paste API tokens into chat.
• Do not commit API tokens.
• Do not store tokens in the skill folder.
• config.json should contain runtime defaults only, not secrets.
• If credentials are missing, the skill preflight should ask the user to run setup before the first run.

Keep runtime/ out of git — it holds last-run metadata and local plans (adjust path prefixes to your install location). A pattern like .cursor/skills/*/runtime/ works for multiple skills.

Preflight

python scripts/preflight.py validates credentials, the bundled HTTPS API endpoint in config, and that runtime artifacts are not committed. run and status run preflight automatically; use standalone preflight for first-time setup or troubleshooting.

---

Configuration

All CLI calls hit VibeTesting’s hosted External API you already use from automation—there is no separate “your API URL” to figure out for normal setups. config.json may include an api_base_url field for the template tooling to stay in sync with the product default; leave it as shipped unless support tells you otherwise. Secrets belong in environment variables or ~/.vibetesting/credentials.json (see API token storage), not in config.json.

Defaults

Common fields under defaults include:

• mode — guided or exploration (guided is typical for scripted plans)
• flows_count — exploration parallelism (exploration mode)
• max_steps_per_flow — step cap per flow (same naming as chat and API)
• allow_parallel, should_stop_existing, should_send_report — forwarded to run-tests when not overridden in the plan

Target URL: dashboard_default vs runtime_override

• target.mode: dashboard_default (default): runs use the application URL verified in the VibeTesting dashboard. If target.target_url is set anyway, it is not sent to the API until you switch modes — the CLI warns or prompts so this is never silent.

• target.mode: runtime_override: sends target_url from config/plan if your token allows server-side overrides.

Non-interactive automation when dashboard_default ignores a configured URL: set environment variable VT_CONFIRM_DASHBOARD_DEFAULT_IGNORE_TARGET_URL=1 to proceed, or clear target_url, or use runtime_override.

Login

Prefer dashboard Test Users (see also FAQ login): in JSON plans use "login": {"login_needed": true, "test_user": "<name>"} or "test_user_id": "<uuid>" from Settings → Test Users. Secrets stay in the dashboard; do not paste passwords into flow steps.

If you are not using a dashboard Test User, you can supply a temporary login payload by saving the same JSON shape as the API’s login object in a file on disk and passing --login-json path/to/file.json when you run scripts/run_vibetesting.py. That file stays local—gitignore it, do not commit it, and do not paste or log its contents in chat or tooling output.

---

CLI usage

Run commands with working directory set to the skill folder (contains scripts/ and config.json). In tooling that supports it, prefer a working-directory option instead of cd … && so shells like PowerShell stay compatible.

From skill folderFrom repo root (--skill-home)

python scripts/preflight.py

python scripts/run_vibetesting.py run --plan runtime/plans/your_plan.json
python scripts/run_vibetesting.py run --goal "Open Settings, change a preference, save, verify after refresh." \
  --name "Settings smoke" --max-steps 20 --risk medium

python scripts/run_vibetesting.py status --latest
python scripts/run_vibetesting.py status --latest --batch
python scripts/run_vibetesting.py stop --latest

python path/to/skill-folder/scripts/run_vibetesting.py run --skill-home path/to/skill-folder --goal "..."

• run: --plan (path to JSON plan) or --goal (single guided flow) — mutually exclusive.
• status: --latest uses runtime/last-run/last_run.json, or --execution-id.
• stop: --latest stops IDs from last run, or --execution-id, or no id for app-wide stop behavior per API.

On success, run prints JSON including execution_ids, previous_execution_ids, and fresh_execution. Avoid reusing stale client_run_id values — generate a fresh id per attempt (the CLI fills one when omitted).

The skill HTTP client sends auth only as Authorization: api-key <your-token> (it does not use the query-string api-key= style).

---

Execution modes & plans

Guided (default)

• Plan: "execution_type": "guided" and a tests array with name + steps (steps may be a string or list).
• Locally you may attach risk, reason, login_required per test for planning — the runner sends only name and steps to the API; a plan_snapshot may be stored under runtime/last-run/ for your review.

Exploration

• Plan: execution_type: exploration, flows_count, general_instructions, optional limits — no tests[].

Minimal guided example (flows.json)

{
  "execution_type": "guided",
  "tests": [
    {
      "name": "Settings preference persists",
      "steps": "Open Settings, change a notification preference, save, refresh, and verify the preference remains selected.",
      "reason": "User-visible regression check.",
      "risk": "medium",
      "login_required": true
    }
  ],
  "login": {
    "login_needed": true,
    "test_user": "admin-ci"
  },
  "max_steps_per_flow": 20
}

Add optional metadata, client_run_id, target_url, general_instructions as needed. Keep payloads minimal: distilled goals only — not source trees, raw diffs, tokens, or credentials in steps.

---

Agent best practices

1. Explicit consent before run — summarize target (dashboard vs override), mode, flows, login mode, caps, credit impact; wait for approval before any API call.
2. No auto-polling — after starting a run, tell the user executions began; call status only when they ask.
3. Secrets — API token only via VIBETESTING_API_TOKEN or ~/.vibetesting/credentials.json (see API token storage); never log tokens or login files.
4. Shell-safe invocation — set tool working directory to the skill folder; on Windows avoid Bash-only cd … &&.
5. Write user-visible steps — e.g. "Submit checkout and verify confirmation" rather than REST paths or selectors.
6. Credit-aware runs — choose realistic flows_count and max_steps_per_flow (see What consumes credits).

For flow authoring nuance (anti-patterns, exploration scope, mapping diffs to journeys), mirror the bundled references in your template (references/flow_generation_guidelines.md) and align with Guided docs: Running Tests → Guided Mode.

---

Learn more on this site

• External API
• Running Tests overview
• Scheduling
• Troubleshooting
• FAQ

The template pack may ship additional references/ files alongside SKILL.md; keep those in sync with this site’s API docs when fields change.