aiconfig-variations
Scannednpx machina-cli add skill launchdarkly/agent-skills/aiconfig-variations --openclawAI Config Variations
You're using a skill that will guide you through testing and optimizing AI configurations through variations. Your job is to design experiments, create variations, and systematically find what works best.
Prerequisites
- Existing AI Config (use
aiconfig-createfirst) - LaunchDarkly API access token or MCP server
- Clear hypothesis about what to test
Core Principles
- Test One Thing at a Time: Change model OR prompt OR parameters, not all at once
- Have a Hypothesis: Know what you're trying to improve
- Measure Results: Use metrics to compare variations
- Verify via API: The agent fetches the config to confirm variations exist
API Key Detection
- Check environment variables —
LAUNCHDARKLY_API_KEY,LAUNCHDARKLY_API_TOKEN,LD_API_KEY - Check MCP config — If applicable
- Prompt user — Only if detection fails
Workflow
Step 1: Identify What to Optimize
What's the problem? Cost, quality, speed, accuracy? How will you measure success?
Step 2: Design the Experiment
| Goal | What to Vary |
|---|---|
| Reduce cost | Cheaper model (e.g., gpt-4o-mini) |
| Improve quality | Better model or prompt |
| Reduce latency | Faster model, lower max_tokens |
| Increase accuracy | Different model (Claude vs GPT-4) |
Step 3: Create Variations
Follow API Quick Start:
POST /projects/{projectKey}/ai-configs/{configKey}/variations- Include modelConfigKey (required for UI)
- Keep everything else constant except what you're testing
Step 4: Set Up Targeting
Use aiconfig-targeting skill to control distribution (e.g., 50/50 split for A/B test).
Step 5: Verify
-
Fetch config:
GET /projects/{projectKey}/ai-configs/{configKey} -
Confirm variations exist with correct model and parameters
-
Report results:
- ✓ Variations created
- ✓ Models and parameters correct
- ⚠️ Flag any issues
modelConfigKey
Required for models to show in UI. Format: {Provider}.{model-id} — e.g., OpenAI.gpt-4o, Anthropic.claude-sonnet-4-5.
What NOT to Do
- Don't test too many things at once
- Don't forget modelConfigKey
- Don't make decisions on small sample sizes
Related Skills
aiconfig-create— Create the initial configaiconfig-targeting— Control who gets which variationaiconfig-update— Refine based on learnings
References
Source
git clone https://github.com/launchdarkly/agent-skills/blob/main/skills/ai-configs/aiconfig-variations/SKILL.mdView on GitHub Overview
AI Config Variations guides you through testing and optimizing AI configurations using controlled variations. It teaches you to identify what to optimize, design experiments, and create variations, then systematically compare results to find the best setup.
How This Skill Works
Start with a clear hypothesis about cost, quality, speed, or accuracy, then design experiments that vary only one factor at a time (model, prompt, or parameters). Use the API to create variations (POST /projects/{projectKey}/ai-configs/{configKey}/variations), ensure modelConfigKey is provided, then route traffic with aiconfig-targeting and verify via GET /projects/{projectKey}/ai-configs/{configKey}.
When to Use It
- You want to reduce cost by testing cheaper models while keeping results stable
- You aim to improve quality by swapping in better models or refining prompts
- You need to reduce latency by selecting faster models or lowering max_tokens
- You want to increase accuracy by comparing models (e.g., Claude vs GPT-4) and tuned prompts
- You are validating that variations exist and want to report outcomes after a controlled test
Quick Start
- Step 1: Identify what to optimize (cost, quality, speed, accuracy) and define success metrics
- Step 2: Design the experiment and decide what to vary (model, prompt, or parameters)
- Step 3: Create variations via API: POST /projects/{projectKey}/ai-configs/{configKey}/variations and set a modelConfigKey
Best Practices
- Test one thing at a time: vary only model, prompt, or parameters, not all at once
- State a clear hypothesis before starting
- Define measurable metrics and compare variations against a baseline
- Verify variations via API and fetch the config to confirm changes
- Document results and version variations to enable reproducibility
Example Use Cases
- Reduce cost by testing a cheaper model (e.g., gpt-4o-mini) while keeping prompts constant
- Improve quality by swapping in a better model or refining prompts and instructions
- Cut latency by selecting faster models and lowering max_tokens while preserving accuracy
- Increase accuracy by comparing models (e.g., Claude vs GPT-4) and tuned prompts
- Validate that variations exist by fetching the config and confirming modelConfigKey and parameters
