Review and Promote Traces

Use this skill for the judgment and promotion loop after trace evaluation.

Interactive Q&A protocol (mandatory)

If context is unclear, ask one question at a time with lettered options.

Example:

What do you want to review first?
A) One specific run
B) Pending queue triage across runs

Rules:

• Ask one question per message.
• Use lettered options whenever practical.
• Ask one follow-up when needed, then continue.

Workflow

1. If needed, flag evaluated run:
   • Use flag_review_item with run_id.
2. Retrieve review items:
   • Use list_review_items with pagination.
3. Collect human judgments:
   • Present review items to the user and ask for judgment one item at a time when needed.
   • Capture judgment_value and optional notes for each item.
   • Present lettered options that match the live evaluation setup:
     • Binary example:
       • A) Pass
       • B) Fail
     • Categorical example:
       • A) <option 1>
       • B) <option 2>
       • C) <option 3>
     • Continuous example:
       • A) Enter numeric value (within configured range)
       • B) Skip this item for now
   • If valid options are unclear, look them up before asking:
     1. Inspect list_review_items payload for item result details and expected value hints.
     2. Use get_result(run_id) for run-level context and chain outputs.
     3. If evaluation id is available in run metadata, call get_evaluation(evaluation_id) and read config/judgment criteria to derive valid judgment options.
   • When options remain ambiguous after lookup, ask one clarification question with lettered options before proceeding.
4. Submit judgments:
   • For each target item, call judge_review_item.
   • Pass the user-provided judgment_value and optional notes into judge_review_item.
   • Include notes when judgment context matters.
5. Promote judged outputs:
   • Use add_reviewed_items_to_dataset with run_id.
6. Report result:
   • number of items judged
   • promotion status and row counts
   • any skipped or blocked items

Queue-wide triage guidance

• Group by run and status first.
• Prioritize high-impact runs or oldest pending runs.
• Keep an audit trail of judgment rationale in notes.

Scopes reference

• list_review_items requires review:read
• flag_review_item, judge_review_item, and add_reviewed_items_to_dataset require review:write

If a scope error occurs, ask the user to create a key with the missing scope in Truesight Settings.