Troubleshooting scenarios

Each scenario is a problem someone reported. Pick the most likely diagnosis. The modules behind each answer are named in its explanation.

1. "Tier 2 always says skipped, even though I set ANTHROPIC_API_KEY."

   The Anthropic key has expired Tier 2 needs a non-Anthropic embedding provider and its key Tier 2 only runs on pull requests The skill has no description, so there is nothing to embed
2. "Our Tier 3 run reports the agent never used any tools, but I watched it call the shell tool repeatedly."

   The skill is broken and should be rewritten The judge model is too weak A gap in the evaluator's tool-recognition heuristic; read its source first Docker is blocking tool calls
3. "A brand-new skill fails Tier 1 with an author_format error, and I did write an author."

   The metadata: block is missing from the frontmatter The author must be a GitHub username The skill needs a CHANGELOG.md Tier 1 requires a version field
4. "Tier 1 flags skill.yaml and CHANGELOG.md as unexpected files."

   Delete both files; they are not allowed The skill fails certification until they are removed Rename them to SKILL.yaml and SKILL.md A non-blocking advisory; move them into an allowed subdirectory or set SKILL_EVAL_SCHEMA_ALLOWED_DIRS
5. "Our judge model quietly matches whichever model the agent uses, and we never configured that."

   SKILL_EVAL_JUDGE_MODEL was never set, so the judge fell back to the agent's model SkillEvaluator always uses the same model for both roles The agent overrides the judge at run time The judge credential is shared with the agent by design
6. "A Tier 3 sandbox run cannot find our repository's own tool bridge script."

   The bridge script needs to be listed in SKILL.md The --copy-repo flag is missing Docker images cannot run Python The evaluator does not support custom tools
7. "We want a cheap way to validate Tier 3 wiring without spending $15-45 per skill."

   Run Tier 1 instead; it measures the same thing Run a Haiku-tier quick pass and treat its Skill Lift as certification evidence Skip the without-skill arm to halve the cost Run a Haiku-tier quick pass to check the wiring, but not as certification evidence
8. "Our Tier 4 script mines completed Tier 3 trials, but a rerun of the identical command produced a different directory structure."

   A real, observed quirk; make the script handle both layouts The second run is corrupted and should be discarded The pinned evaluator version changed between runs The grader wrote into the trial directory