Kai Ase Siren
Skip to content

MCP loop tasks

What a task file under housecast/mcpeval/tasks/ is, and what it withholds.

A capability, not a case

transcribe-video.yaml carries twenty prompts. Past ten, prompts cost machine time rather than review time, and a sign test needs 15 of 20 to move where it needs 9 of 10.

What the model never sees

Nothing in a task file is sent to the model, which sees the prompt text and the subject's own tool descriptions only. rules is the grader's, and a rule naming the expected calls would measure whether the agent can read the answer key. Top-level rules apply to every prompt, and each prompt overrides them.

The edge case

p13 and p14 point at a file with no audio track. The correct behaviour is to discover that and tell the user, not to transcribe.