Do mocks ever get allowed?

No. This skill enforces zero mocks and real dependencies to validate features.

Where should I store and track scenarios?

Write and run them in .scratch/, and promote patterns to scenarios.jsonl as committed documentation.

How do I ensure a scenario can run independently?

Design it to be standalone with its own data setup; avoid dependencies on prior runs; run in CI/CD with parallelization.

scenario-testing

Scanned

npx machina-cli add skill aiskillstore/marketplace/scenario-testing --openclaw

Files (1)

SKILL.md

3.2 KB

Scenario-Driven Testing for AI Code Generation

Core Principle

The Iron Law: "NO FEATURE IS VALIDATED UNTIL A SCENARIO PASSES WITH REAL DEPENDENCIES"

Mocks create false confidence. Only scenarios exercising real systems validate that code works.

The Truth Hierarchy

Scenario tests (real system, real data) = truth
Unit tests (isolated) = human comfort only
Mocks = lies hiding bugs

As stated in the principle: "A test that uses mocks is not testing your system. It's testing your assumptions about how dependencies behave."

When to Use This Skill

Validating new functionality
Before declaring work complete
When tempted to use mocks
After fixing bugs requiring verification
Any time you need to prove code works

Required Practices

1. Write Scenarios in `.scratch/`

Use any language appropriate to the task
Exercise the real system end-to-end
Zero mocks allowed
Must be in .gitignore (never commit)

2. Promote Patterns to `scenarios.jsonl`

Extract recurring scenarios as documented specifications
One JSON line per scenario
Include: name, description, given/when/then, validates
This file IS committed

3. Use Real Dependencies

External APIs must hit actual services (sandbox/test mode acceptable). Mocking any dependency invalidates the scenario.

4. Independence Requirement

Each scenario must run standalone without depending on prior executions. This enables:

Parallel execution
Prevents hidden ordering dependencies
Reliable CI/CD integration

What Makes a Scenario Invalid

A scenario is invalid if it:

Contains any mocks whatsoever
Uses fake data instead of real storage
Depends on another scenario running first
Never actually executed to verify it passes

Common Violations to Avoid

Reject these rationalizations:

"Just a quick unit test..." - Unit tests don't validate features
"Too simple for end-to-end..." - Integration breaks simple things
"I'll mock for speed..." - Speed doesn't matter if tests lie
"I don't have API credentials..." - Ask your human partner for real ones

Definition of Done

A feature is complete only when:

✅ A scenario in .scratch/ passes with zero mocks
✅ Real dependencies are exercised
✅ .scratch/ remains in .gitignore
✅ Robust patterns extracted to scenarios.jsonl

Example Workflow

Write scenario - Create .scratch/test-user-registration.py
Use real dependencies - Hit real database, real auth service (test mode)
Run and verify - Execute scenario, confirm it passes
Extract pattern - Document in scenarios.jsonl
Keep .scratch ignored - Never commit scratch scenarios

Why This Matters

Unit tests verify isolated logic
Integration tests verify components work together
Scenario tests verify the system actually works

Only scenario tests prove your feature delivers value to users.

Source

git clone https://github.com/aiskillstore/marketplace/blob/main/skills/2389-research/scenario-testing/SKILL.mdView on GitHub

Overview

Scenario-driven testing validates features by running end-to-end scenarios against real systems. It rejects mocks as false confidence and requires real dependencies in the .scratch/ directory, with recurring patterns exported to scenarios.jsonl. This approach ensures features actually function under real conditions.

How This Skill Works

Create scenarios in .scratch/ using any language suitable for the task and exercise the real system end-to-end with zero mocks. Then promote repeatable cases by exporting them to scenarios.jsonl as one JSON line per scenario, including name, description, given/when/then, and validates. Keep .scratch/ ignored in git to ensure scratch work never lands in production.

When to Use It

Validating new functionality
Before declaring work complete
When tempted to use mocks
After fixing bugs requiring verification
Any time you need to prove code works

Quick Start

Step 1: Create .scratch/test-<scenario>.py (or .js, .ts, etc.) with the end-to-end scenario
Step 2: Run the scenario against real dependencies in sandbox/test mode and confirm it passes
Step 3: Export a pattern to scenarios.jsonl (one JSON line per scenario) and ensure .scratch/ is ignored by git

Best Practices

Write all scenarios in .scratch/ using the task's language
Exercise the real system end-to-end with zero mocks
Keep .scratch/ in .gitignore and never commit scratch scenarios
Promote recurring scenarios to scenarios.jsonl with one JSON line each
Design each scenario to run standalone so it can run in parallel

Example Use Cases

Test user registration against a real auth service in sandbox mode
Validate order placement by hitting the actual payment service in test mode
End-to-end product checkout with real inventory systems
Verify a new feature flag rollout against a live config service
Validate a data ingestion pipeline against a real storage sink in test data

Frequently Asked Questions

Add this skill to your agents