Final exam | SkillEvaluator Mastery

Fifteen questions across all eight modules. Score 80% (12 of 15) or more to pass. Answer every question, then check your answers.

What does Skill Lift measure?
With-skill score minus without-skill score in Tier 3 The improvement in Tier 1 quality score after fixes How many more tools the agent calls with the skill The judge's agreement with a human grader
Which tier is free and offline by default?
Tier 1 Tier 2 Tier 3 Tier 4
Why does Tier 2 need an OpenAI (or compatible) key even for an all-Claude Tier 3 run?
Tier 2 uses GPT as its judge Anthropic does not provide an embedding provider The skill catalog is stored with OpenAI SkillEvaluator is built on the OpenAI SDK
What does Tier 4 not supply?
A harness to grade against A trajectory from Tier 3 Domain-specific correctness judgment A hook for a custom grader
Which environment variable sets a separate judge model?
SKILL_EVAL_LLM_PROVIDER SKILL_EVAL_EMBEDDING_PROVIDER ANTHROPIC_MODEL SKILL_EVAL_JUDGE_MODEL
Which easy-to-forget flag leaves the sandbox unable to find the repository's own tool bridge?
--no-dedup --copy-repo --env-mode docker --results-dir
Why pin an exact SkillEvaluator commit rather than install @main?
Real, breaking behavioral differences have occurred between versions @main is not publicly available Pinned commits include extra tiers Pinning avoids the need for Docker
What happens if Tier 2's embedding provider is missing?
The run fails loudly It silently reports "skipped" rather than erroring It falls back to Tier 1 It uses cached embeddings
What does framework/adapters/nvidia_skillevaluator.py do?
Installs the pinned evaluator Runs Tier 3 in Docker Normalizes raw NVIDIA report JSON into a stable internal schema Uploads reports to GitHub Pages
What support level should you expect from SkillEvaluator?
Enterprise, with an SLA Experimental, community-supported with no SLA Beta, with vendor support Long-term support
Which frontmatter field does the author_format check need?
author: at the top level metadata: author: ... owner: in skill.yaml maintainers: in CHANGELOG.md
In this repository's CI, which jobs block a pull request?
tier3-fast and domain-graders Every job, including placeholders tier1, and similarity on EXACT_DUPLICATE None; CI is advisory only
Roughly what did a full 150-trial Tier 3 matrix cost per skill in this repository's runs?
Under $1 and two minutes $1-3 and 12 minutes $15-45 and 30-55 minutes $500 and a full day
What bug did this repository patch locally in the pinned evaluator?
A judge token-truncation bug (max_tokens=1024, no retry) A Docker networking bug A missing embedding provider A YAML parsing bug in SKILL.md
Why is a Haiku-tier quick pass not certification evidence?
It skips the judge Haiku cannot use tools It only covers five eval cases A weaker agent compresses Skill Lift toward zero on both arms