SkillEvaluator Mastery
A short course on NVIDIA SkillEvaluator, the open-source framework for testing agent skills, taught from what this repository learned by running it on twelve portfolio-management skills: the real costs, the real bugs, and the settings that are easy to get wrong.
It is for engineers and product owners who write agent skills and want to know whether a skill is well formed, whether it duplicates one that already exists, and whether it actually makes an agent better.
How the course works
- Read the eight modules in order. Each one covers one part of SkillEvaluator and ends with a short multiple-choice check. Your answers are marked as soon as you submit them.
- Work through the troubleshooting scenarios. Eight problems people actually hit, each with a diagnosis to choose.
- Take the final exam. Fifteen questions across every module; 80% passes.
The hands-on tutorial has you scaffold a toy skill and run Tier 1 against it for free. The reference collects every command, setting, and finding in one place.
Reading the course costs nothing. Running SkillEvaluator's Tier 1 is also free and offline, but a full Tier 3 run against a live agent cost this repository $15-45 per skill, so the modules say clearly which steps need credentials or money.
Your progress and quiz scores are saved in this browser only. Nothing is sent anywhere.
The modules
- Module 1What it is and whyThe four tiers, what each one costs, and what SkillEvaluator leaves to you.
- Module 2Tier 1: construction and securityFree, offline checks for schema, quality, and security on every pull request.
- Module 3Tier 2: similarityCatching near-duplicate skills, and the embedding-provider trap.
- Module 4Tier 3: live-agent evaluationMeasuring Skill Lift with a live agent and judge, and what that costs.
- Module 5Tier 4: domain gradersWriting reusable graders that judge domain correctness.
- Module 6Installing and runningInstall options, pinning an exact commit, and checking the install.
- Module 7Skill package layoutThe folder layout and frontmatter that Tier 1 checks.
- Module 8Reports, CI/CD, and real bugsReading reports, wiring CI, and four problems this repository hit.
Other ways to take it
- As a ten-slide overview. The Skills Ledger is a crash course on why skill governance matters and what this repository's real runs found.
- As one file. Download the course as a single HTML page that works offline.
- With a coding agent. The same material is an agent skill,
skills/skillevaluator-mastery/, that teaches it as a conversation and writes fresh questions each time. In a checkout of this repository, sayskillevalexpertto Claude Code, Codex, or GitHub Copilot.
SkillEvaluator Mastery — A From-Scratch, Hands-On Tutorial
This is the from-scratch companion to
docs/15_SKILLS_AND_SKILLEVALUATOR_REFERENCE.md.
Doc 15 is the reference to look things up in; this document is the
progressive, hands-on exercise to actually build the muscle memory —
scaffold a toy skill, watch each tier fire, break it on purpose, and read a
real report. It assumes zero prior SkillEvaluator experience.
The same material is also a course with quizzes, troubleshooting scenarios,
and a scored final exam, on the
course site or as
one downloadable file.
It is also an agent skill,
skills/skillevaluator-mastery/, that
teaches it as a conversation: in a checkout of this repository, say
"skillevalexpert" to Claude Code, Codex, or GitHub Copilot. To use the
skill elsewhere, copy that folder to .github/skills/, .claude/skills/,
.agents/skills/, or ~/.copilot/skills/. This document is the
read-it-yourself version.
0. Before you start
You need: uv (or pip), Python 3.12/3.13, git. Everything through
step 3 below is free and needs no API key, no Docker, and no live model.
Steps 4+ are clearly marked with their real cost/credential requirements —
do not run them without reading the caveat first.
Step 1 — Install Tier 1 only
uv tool install --python 3.13 "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git"
skillevaluator --version
skillevaluator health-check
If you only want Tier 1 (no security scanning), the plain skillevaluator
extra is enough — see docs/15_... §4 for all the install variants and why
this repository pins an exact commit instead of @main.
Step 2 — Scaffold a toy skill from scratch
mkdir -p /tmp/my-first-skill/evals
Create /tmp/my-first-skill/SKILL.md:
---
name: my-first-skill
description: Say hello in a specific format.
metadata:
author: Your Name <you@example.com>
---
# My First Skill
## Use when
Use this skill when asked to greet someone.
## Procedure
1. Greet the user by name if given.
2. Otherwise greet generically.
This is intentionally minimal — you'll break it on purpose in Step 4.
Step 3 — Run Tier 1 and read the report
skillevaluator quality-check /tmp/my-first-skill
skillevaluator validate /tmp/my-first-skill --no-dedup
Look at:
- The overall quality score and which dimensions it scored on.
- Any advisory findings about missing recommended sections (## Do not use
when, ## Examples, etc. — see docs/15_... §5 for the full list).
- Whether the security scan step ran (it needs gitleaks/SkillSpector
installed separately — docs/15_... §4).
Checkpoint: you should see a passing schema check (the frontmatter has
name, description, and a metadata.author) but likely several advisory
quality findings, because the body is minimal.
Step 4 — Break it on purpose, then fix it
Remove the metadata: block entirely and rerun quality-check. This is a
real, observed failure mode in this repository — the author_format
check fails outright, not just an advisory finding (docs/15_... §5). Put
the block back, then remove the ## Use when section and rerun — this time
notice it's an advisory finding, not a hard failure. This distinction (hard
schema failure vs. advisory quality finding) is the single most useful
thing to internalize about Tier 1.
Now add a skill.yaml at the skill root with placeholder content and rerun
quality-check again — Tier 1's schema check may flag it as an "unexpected"
root file (docs/15_... §5); move it into an allowed layout, or note the
SKILL_EVAL_SCHEMA_ALLOWED_DIRS escape hatch, without treating it as a bug.
Step 5 — Inspect a real report from this repository
Rather than running Tier 3 yourself (see the cost caveat in Step 7), read an already-generated report:
ls reports/*.json
cat skills/portfolio-overview/BENCHMARK.md
Compare the raw JSON's shape (agents, dimensions_with_skill, lift,
pass_at_k, or quality_summary for Tier 1-only reports) against
framework/adapters/nvidia_skillevaluator.py's parse_nvidia_report —
run this in a Python shell to see the normalized shape produced:
uv run python -c "
from framework.adapters.nvidia_skillevaluator import parse_nvidia_report_file
import json
result = parse_nvidia_report_file(
'reports/skillevaluator-output-20260904020239.json',
skill_id='pm.demo', skill_name='Demo', skill_version='0.0.0')
print(json.dumps(result, indent=2)[:2000])
"
Checkpoint: you should be able to say, in one sentence each, what the
raw NVIDIA shape looks like and what this repository's normalized adapter
produces from it — and why the distinction matters (docs/15_... §7).
Step 6 — Understand Tier 2 without spending money
Tier 2 needs a working embedding provider that is not Anthropic. Rather
than configuring one, read catalogs/skill-catalog.json's structure (an
entries list, each with an embedding vector and a content_fingerprint)
and policies/similarity.yaml's governance actions
(EXACT_DUPLICATE → block, HIGH_SIMILARITY → advisory review). Then read
framework/certification/check_similarity.py to see how this repository
turns a raw similarity score into one of those actions.
Step 7 — Understand Tier 3 without spending money (unless you choose to)
Real cost warning, read before running anything here: a full Tier 3
matrix in this repository's own experience took 30-55 minutes and roughly
$15-45 per skill (docs/15_... §2-3, docs/MILESTONE_4_PERFORMANCE_ATTRIBUTION.md).
Instead of running it, read the milestone doc above end-to-end. It walks
through: the exact command used, the four real bugs found (an
agent-compatibility gap, a judge truncation bug, a missing --copy-repo
flag, and API credit exhaustion), and how the final Skill Lift number
(+0.1253) was computed and interpreted. If and when you do choose to run a
real Tier 3 matrix, use the exact command in docs/15_... §4c, and budget
time, Docker, an agent credential, and a separate judge-model credential
via SKILL_EVAL_JUDGE_MODEL.
Step 8 — Understand Tier 4 and certification, on top
Read graders/finance/portfolio_overview.py alongside
skills/portfolio-overview/evals/grader.py to see the thin-hook pattern,
then policies/certification.yaml and
framework/certification/profile_resolver.py to see how this repository
turns raw Tier 1-4 evidence into a pass/fail certification decision that is
explicitly independent of SkillEvaluator's own opinion (docs/02_TARGET_ARCHITECTURE.md,
docs/04_EVALUATION_AND_CERTIFICATION.md).
Self-test checklist
Before considering yourself "SkillEvaluator-literate," you should be able to answer, without looking anything up:
- What does each of the four tiers check, and which are free vs. costly?
- What's the difference between a schema failure and a quality finding?
- Why does Tier 2 need a non-Anthropic embedding provider?
- What does "Skill Lift" mean, and what does a real number (e.g. +0.1253) tell you?
- What are the three easy-to-forget Tier 3 requirements
(
--copy-repo,SKILL_EVAL_JUDGE_MODEL, judge vs. agent credential)? - What's the difference between the raw NVIDIA report shape and this repository's normalized adapter output, and why does that boundary exist?
- Where does SkillEvaluator's job end and this repository's own certification logic begin?
To test yourself on the same material with marked quizzes, scenarios, and a scored final exam, take the course.
Skills & SkillEvaluator — One-Stop Reference
This document is the reference after completing the local tutorial. Start with
docs/00_TUTORIAL.md for a no-credentials exercise. This
document is the single place to point someone who asks "what are agent
skills, what does NVIDIA SkillEvaluator actually do, is it worth adopting,
and how do I get it running on my own repo?" It is written to stand alone —
combining the official upstream documentation with this project's own
hands-on findings (including the real bugs, costs, and gotchas discovered by
actually running it, not just reading its docs). Every claim about this
project's experience links to the evidence doc it came from; every claim
about the external ecosystem cites its source.
1. What SkillEvaluator is, and what it helps achieve
NVIDIA SkillEvaluator is an open-source, multi-tier framework for evaluating AI agent artifacts — starting with agent skills — through deterministic quality gates, semantic overlap detection, synthetic evaluation-dataset generation, and live agent evaluation that measures how a skill actually changes agent behavior (NVIDIA docs; GitHub; NVIDIA technical blog). Its support level is explicitly Experimental — community-supported on a best-effort basis through GitHub Issues, no SLA. That matters: this project found and worked around real bugs in it (§4 below), which is exactly the kind of thing "Experimental" should lead you to expect.
The problem it solves. As a team accumulates AI skills (packaged instructions that teach an agent to do a specific job), review-by-eyeball stops scaling. SkillEvaluator answers, with numbers instead of opinions:
- Is the skill well-formed and safe? (Tier 1)
- Is it a near-duplicate of something that already exists? (Tier 2)
- Does having the skill actually change agent behavior for the better — and by how much? (Tier 3 — "Skill Lift": the with-skill score minus the without-skill score)
- Is the agent's own answer, with the skill, actually correct? (Tier 4 — domain-specific graders you write; SkillEvaluator supplies the harness, not the finance/legal/domain judgment)
What it does not do: it does not build skills for you, does not run
your production agent, does not manage credentials or a data pipeline, and
does not (by itself) decide certification policy — that is what this
project's own framework/certification/engine.py and
policies/certification.yaml add on top. See
docs/02_TARGET_ARCHITECTURE.md for how this project draws that line.
2. What our own tests actually show
Two skills in this repository have been taken all the way through the real
pipeline — live agent, live judge model, real Docker sandbox, real
certification policy. Both results are real, both are FAIL, and both
failures are precisely diagnosed rather than vague:
| Skill | Trials | Skill Lift | Certification | Why |
|---|---|---|---|---|
| Performance Attribution | 150 (75/75 both arms) | +0.1253 | FAIL | Discoverability 0.8862 < 0.90 — a metric-scoping artifact (two eval cases whose correct behavior is to use no tools at all), not a real defect. Full trail: docs/MILESTONE_4_PERFORMANCE_ATTRIBUTION.md. |
| Portfolio Overview | 150 (75/75 both arms) | +0.1316 | FAIL | The same discoverability shortfall (0.8942), independently corroborated by SkillEvaluator's own report — plus a newly discovered certification policy gap: a minimum metric (reconciliation) specific to one skill's grader auto-fails every other skill regardless of quality. Evidence: skills/portfolio-overview/BENCHMARK.md. |
Getting Performance Attribution's result required finding and fixing four real bugs — not toy issues, actual blockers:
- An agent-compatibility gap in the evaluator's Tier 3 execution heuristic
(Codex's shell tool self-reports as
exec, a string the evaluator's recognized-tool-name list didn't include). - A token-budget bug in the evaluator's own judge code (
max_tokens=1024, no retry, on two of three judge functions — the third had already fixed the identical bug, the fix just wasn't propagated). Patched locally, tracked inpatches/. - A missing
--copy-repoCLI flag (this project's own operator error, not an evaluator bug) that left the sandbox unable to find the repo's own tool bridge. - API credit exhaustion mid-run, twice — a real operating-cost lesson, not a bug.
Beyond those two full certifications: Milestone 6 deliberately introduced
six synthetic defects (vague description, duplicate skill, missing
derivatives, mismatched dates, weak/no-value skill, unauthorized data
source) and confirmed the framework catches five of them directly with real
evidence (a Tier 2 similarity score of 0.9468 on the deliberate duplicate;
existing grader regression tests for the domain-correctness defects) — see
docs/MILESTONE_6_DELIBERATE_DEFECTS.md. Milestone 9 put a real Tier 1
check into CI (§6 below), and its first live run against a real PR passed
in 54 seconds.
The honest summary: no skill in this catalog has cleared certification
outright yet, and that is reported plainly rather than smoothed over —
see docs/14_EXECUTIVE_SUMMARY_AND_WALKTHROUGH.md for the full cold-read
walkthrough with an honest pros/cons assessment.
3. Feasibility: how hard is it to bring this to a real repo?
Grounded in this project's actual experience, not the marketing pitch:
What's genuinely easy:
- Tier 1 (schema, security, PII, quality scoring) is entirely offline and
free — no API key, no Docker, no sandbox. skillevaluator quality-check
<skill> and skillevaluator validate <skill> (Tier 1 only) run in
seconds and are safe to run constantly, including in CI on every PR (this
project's Milestone 9 job does exactly this).
- The pinned-version install is a single pip/uv command plus one line
for an optional security scanner (§5).
- The tool is genuinely agent-agnostic at the CLI level — switching Tier 3's
agent (Codex → Claude Code) was a one-flag change once the compatibility
gap was diagnosed.
What's genuinely hard, and costs real money and time:
- Tier 2 needs a working embedding provider, and Anthropic does not
provide one — you need SKILL_EVAL_EMBEDDING_PROVIDER=openai (or
nv_build / an OpenAI-compatible endpoint) plus that provider's key, even
if your Tier 3 agent is entirely Claude-based. This project only
discovered this because Tier 2 silently reported "skipped" rather than
erroring, on every run, until an OpenAI key was wired in.
- Tier 3 is expensive and slow. A full 25-case × 3-attempt × 2-arm
matrix (150 trials) against claude-sonnet-5 as both agent and judge
took this project 30-55 minutes and roughly $15-45 per skill, and
exhausted a modest prepaid API balance mid-run more than once in one day
of iterative debugging. A cheap Haiku-tier "quick pass" (same 150 trials,
weaker model) ran in ~12 minutes for ~$1-3 — useful for validating
wiring, not a substitute for real certification evidence: Haiku's
quick-pass numbers on this project's skills failed nearly every
certification threshold that Sonnet cleared, because a weaker agent
compresses Skill Lift toward zero on both arms, not because the skill
itself is worse.
- Tier 3 needs Docker (or a local/cloud sandbox), the selected agent's
own credentials, and — separately, and easy to forget — a judge-model
credential and the SKILL_EVAL_JUDGE_MODEL override, or the judge
silently falls back to whichever model the agent under test uses,
coupling judge cost to agent cost for no reason.
- The evaluator itself has real bugs at "Experimental" support level —
budget time to read its source when something looks wrong rather than
assuming the skill is broken. Both real bugs this project found (§2) were
root-caused by reading skillevaluator's own vendored source, not by
guessing.
- Directory layouts can differ between runs of the same tool — this
project observed two different completed-trial directory structures
across otherwise-identical invocations of skillevaluator validate
--agent-eval; any custom post-hoc analysis script (like the Tier 4
extraction pipeline in §7) needs to defend against that rather than
assume a fixed layout.
Bottom line: Tier 1 is worth adopting almost unconditionally — it's
free and catches real, fixable structural problems (see the SKILL.md
schema/quality findings in every Tier 1 report this project has generated).
Tier 2 needs one extra credential decision. Tier 3 is a genuine cost/time
investment that should be reserved for skills where you actually want
certification-grade evidence, not run reflexively on every change — this is
exactly why this project's own Milestone 7 right-sizing decision limits full
Tier 3 depth to three flagship skills and keeps the other nine at Tier 1/2
only (docs/10_DEVELOPMENT_ROADMAP_AND_PROGRESS.md).
4. Installing SkillEvaluator — step by step
4a. The general, upstream-recommended install
Per the official installation guide:
# Prerequisites: uv (or pip), Python 3.12 or 3.13, git; Docker only if you'll run Tier 3
uv tool install --python 3.13 "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git"
# Smaller footprints, if you don't need everything:
# skillevaluator -- Tier 1 only, no security scanning
# skillevaluator[security] -- + Bandit/pip-audit
# skillevaluator[tier2,tier3] -- + dedup and live-agent eval
# Verify
skillevaluator --version
skillevaluator health-check
External security tools are installed separately to avoid dependency conflicts:
brew install gitleaks # or: go install / release binary
uv tool install git+https://github.com/NVIDIA/SkillSpector.git
4b. How this project pins and installs it (reproducible, recommended for any real adoption)
Installing @main is fine for exploration; for anything you intend to
certify against, pin an exact commit — SkillEvaluator's own release
notes and this project's Milestone 4 both found real, breaking behavioral
differences worth pinning against
(docs/13_NVIDIA_EVALUATOR_UPGRADE_POLICY.md):
uv venv .venv --python 3.13
uv pip install --python .venv/bin/python \
'skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git@009aa300be7925c7ba75760592baeb941cc29ba8'
uv pip install --python .venv/bin/python \
'git+https://github.com/NVIDIA/SkillSpector.git@29b0dc8c39424e8e31ca055fa027adf8ba8f9650'
brew install gitleaks
# This project's local tracked patch for a real judge-truncation bug found
# in the pinned version -- see §2 and patches/README.md. Skip this line if
# you're pinning a version where NVIDIA has since fixed it upstream.
patch -p1 -d .venv/lib/python3.13/site-packages < patches/skillevaluator-0.2.1-judge-max-tokens.patch
Full reasoning for pinning (not just installing latest) and the upgrade
process when you do want to move the pin: docs/13_NVIDIA_EVALUATOR_UPGRADE_POLICY.md.
4c. Running each tier
# Tier 1 -- free, offline, seconds. Do this on every skill, constantly.
skillevaluator quality-check skills/my-skill
skillevaluator validate skills/my-skill --no-dedup
# Tier 2 -- needs an embedding provider (not Anthropic; see §3)
SKILL_EVAL_EMBEDDING_PROVIDER=openai OPENAI_API_KEY=... \
skillevaluator similarity-check skills/ --type skill
# Tier 3 -- needs Docker, agent credentials, a judge model, and real time/$
SKILL_EVAL_LLM_PROVIDER=anthropic ANTHROPIC_API_KEY=... \
SKILL_EVAL_JUDGE_MODEL=claude-sonnet-5 \
skillevaluator validate skills/my-skill \
--agent-eval -a claude-code --agent-model claude-code=claude-sonnet-5 \
--env-mode docker --copy-repo \
--results-dir reports/my-skill-tier3 -r cli,json,markdown
--copy-repo and SKILL_EVAL_JUDGE_MODEL are both easy to forget and both
silently degrade your results rather than error loudly — see §3 and
docs/MILESTONE_4_PERFORMANCE_ATTRIBUTION.md for exactly how each one bit
this project the first time.
5. Repository and skill-package layout requirements
SkillEvaluator's Tier 1 schema check is opinionated about layout. Grounded in real findings from every Tier 1 report this project has generated:
skills/<skill-name>/
├── SKILL.md # required
├── skill.yaml # optional metadata this project's own governance layer reads
├── CHANGELOG.md # optional
├── evals/
│ ├── config.yml # harbor/tier3 run parameters, grading mode
│ ├── evals.json # the eval case dataset
│ └── grader.py # custom Tier 4 grading hook
└── tests/ # optional local unit tests for the grader
SKILL.md frontmatter — the schema and quality checks specifically
look for:
---
name: my-skill # required; must match the directory name
description: One sentence, plus WHEN # required; missing "when to use" language
# is flagged as a discoverability finding
metadata:
author: Team Name <team@example.com> # required for the author_format check
# tags: [finance, reporting] # recommended, not required
# version: 1.0.0 # recommended, not required
---
A missing metadata: block is a real, observed failure mode in this
project — several skills started without one and failed Tier 1's
author_format check until it was added.
Recommended body sections (each an advisory finding if missing, not a
hard fail, but each genuinely moves the quality score): ## Use when,
## Do not use when, ## Procedure, ## Instructions or ## Usage,
## Examples, prerequisites/limitations/troubleshooting sections.
A real, observed quirk: by default, Tier 1's schema check flags files
like skill.yaml and CHANGELOG.md sitting in the skill root as
"unexpected" — it expects the skill root to contain only SKILL.md plus a
small set of subdirectories (agents/, assets/, config/, evals/,
references/, scripts/, tests/, tools/). These are non-blocking
advisories in this project's configuration, not hard failures, but if you
want a clean report, either move such files into an allowed subdirectory or
set SKILL_EVAL_SCHEMA_ALLOWED_DIRS.
evals/evals.json shape (this project's convention, one case shown):
{
"skill_name": "my-skill",
"evals": [
{
"id": "my-skill-001",
"category": "explicit-positive",
"prompt": "...",
"expected_output": "...",
"assertions": ["Do not fabricate facts.", "..."],
"expected_skill": "my-skill"
}
]
}
evals/config.yml (real, working example from this repo):
schema_version: 1
harbor:
n_attempts: 3
pass_threshold: 0.80
n_concurrent: 4
agent_runtime_preflight: true
base_image_mode: reuse
pre_agent_setup:
- >-
if test -r /workspace/skills/my-skill/SKILL.md; then
printf '%s\n' '# Required skill bootstrap' \
'Before answering, read /workspace/skills/my-skill/SKILL.md with cat.' \
> /workspace/AGENTS.md;
fi
skill_workspace:
mode: group
grading:
mode: default_plus_custom
evals/grader.py is a thin hook — real Tier 4 domain logic belongs in a
shared, reusable module (this project's pattern: graders/finance/*.py,
composed from generic building blocks — see Milestone 8 in
docs/10_DEVELOPMENT_ROADMAP_AND_PROGRESS.md for why reusability across
skills was made an explicit exit criterion):
from graders.finance.my_skill import grade as grade_evidence
def grade(case, trajectory=None, expected=None):
evidence = trajectory or expected or {}
result = grade_evidence(evidence)
return {"skill": "my-skill", "metrics": result["metrics"],
"score": result["score"], "notes": []}
6. CI/CD integration
This project has a real, working example rather than a theoretical
recipe: .github/workflows/skills-quality.yml's tier1 job. It installs
the pinned evaluator (plus SkillSpector and gitleaks), diffs the PR against
its base SHA to find changed skills/* directories, and runs
skillevaluator validate <skill> --no-dedup on each — a non-zero exit
blocks the PR (given branch protection requires the check). Its first real
run, against this project's own PR #4, passed in 54 seconds. No API keys
required, because Tier 1 is LLM-free by default.
Extending this to Tier 2 in CI needs an embedding-provider secret
(OPENAI_API_KEY or equivalent) and a central approved-skill catalog to
diff new/changed skills against (catalogs/ in this repo) — SkillEvaluator's
own docs describe this as a standard "merge gate with exit codes" pattern,
and mention a maintained GitHub Actions recipe as part of progressive
adoption.
Extending to Tier 3 in CI needs a Docker-capable runner, the target
agent's credentials, a separately-configured judge model, and — given the
real cost data in §3 — a firm opinion on when it runs. This project's own
recommendation, informed by real experience: keep Tier 3 advisory, not
blocking, on every PR (too slow and expensive to gate routine merges), and
reserve blocking Tier 3 runs for an explicit release/certification workflow
that runs on demand or on a schedule, not on every commit. See the
similarity, tier3-fast, and domain-graders jobs in this project's own
workflow file for the placeholder shape that pattern would fill in, and
docs/10_DEVELOPMENT_ROADMAP_AND_PROGRESS.md's Milestone 9 section for why
this project deliberately stopped at the Tier 1 gate rather than building
the full pipeline.
7. What this project built on top (reusable patterns)
Two pieces of this project's own code are worth knowing about if you're adopting SkillEvaluator elsewhere, because they solve problems the tool itself doesn't:
- A normalized adapter (
framework/adapters/nvidia_skillevaluator.py) that translates NVIDIA's raw report JSON into a stable internal schema, so nothing downstream (certification logic, reporting) parses vendor-specific fields directly — this is what makes an evaluator version bump a contained, testable event instead of a library-wide breaking change. Seedocs/02_TARGET_ARCHITECTURE.md. - A free, post-hoc Tier 4 extraction pipeline
(
skills/*/evals/{tier3_trial_extractor,aggregate_tier4,generate_benchmark}.py) that mines already-completed Tier 3 trial data on disk to compute domain-specific correctness metrics and hard gates (regression pass rate, authorization/permission-denial scanning) without any new API calls — built because SkillEvaluator's own Tier 4 is bring-your-own-grader, and because re-running live evaluation just to compute a metric you already have the raw data for is exactly the kind of unnecessary spend this project's real credit-exhaustion experience (§2) made a priority to avoid.
8. References
What agent skills are, and the open standard
- Anthropic: Agent Skills open standard, agentskills.io — published as an open spec on December 18, 2025; Microsoft, OpenAI, Atlassian, Figma, Cursor, and GitHub have adopted it.
- agentskills/agentskills — the specification repo
- anthropics/skills — Anthropic's own 17 official skills, spec included
- Anthropic docs: Agent Skills — the SKILL.md format, progressive disclosure, loading behavior.
- Simon Willison: Agent Skills — independent, widely-cited explainer on why the format matters.
- Strapi: What Are Agent Skills and How To Use Them
MCP vs. Skills — a distinction worth understanding before designing a skill
- MCP vs Agent Skills: Understanding the Difference — the core framing: MCP is the agent's sensory/action interface to external systems; Skills are the agent's procedural memory (how to do a task once it has the tools).
- The Agentic Stack: Agent Skills vs. MCP
NVIDIA SkillEvaluator
- Official documentation
- Installation guide
- GitHub repository
- NVIDIA technical blog: Evaluating AI Agent Skill Performance with NVIDIA SkillEvaluator
- Scan Agent Skills Before Installation — NVIDIA's own security-scanning guidance for third-party skills, relevant background for this project's Tier 1 security/PII checks.
How other companies approach skills, tools, and agent extensibility
- OpenAI — Agents SDK documentation: Agents, Tools, Handoffs, Guardrails as the primitive vocabulary; tool/function calling as the mechanism closest to what Anthropic calls a "skill" plus MCP combined. Python SDK / JS SDK.
- GitHub — About agent skills (GitHub Docs): Copilot's own Agent Skills, interoperable with the same
anthropics/skillsformat; GitHub documents six customization primitives (agents, skills, hooks, plugins, extensions, instruction files). github/awesome-copilot skills collection.
Tutorials, courses, and a podcast episode
- Anthropic Academy: Introduction to Agent Skills — official, free course.
- Anthropic Academy: Claude Code 101 and Claude Code in Action.
- Claude Agent SDK — Full Workshop, Thariq Shihipar (Anthropic), YouTube
- "Claude Skills explained: How to create reusable AI workflows" — How I AI podcast (Claire Vo), Apple Podcasts / written version, Lenny's Newsletter — walks through building and using Skills in Claude Code and Cursor for recurring workflows.
This project's own evidence, referenced throughout this document
| Topic | Document |
|---|---|
| Full architecture and the adapter/certification boundary | docs/02_TARGET_ARCHITECTURE.md |
| NVIDIA version pinning and upgrade process | docs/13_NVIDIA_EVALUATOR_UPGRADE_POLICY.md |
| Milestone 4 — first real certification attempt, all four bugs, in detail | docs/MILESTONE_4_PERFORMANCE_ATTRIBUTION.md |
| Milestone 5 — second certified skill, the reconciliation policy-gap finding | docs/10_DEVELOPMENT_ROADMAP_AND_PROGRESS.md, skills/portfolio-overview/BENCHMARK.md |
| Milestone 6 — deliberate defect demonstration, in detail | docs/MILESTONE_6_DELIBERATE_DEFECTS.md |
| Full milestone status and the 2026-08-30 scope decision | docs/10_DEVELOPMENT_ROADMAP_AND_PROGRESS.md |
| Standalone cold-read executive summary | docs/14_EXECUTIVE_SUMMARY_AND_WALKTHROUGH.md |
Module 1: What SkillEvaluator Is, and Why
Based on the reference §1.
The core idea
NVIDIA SkillEvaluator is an open-source, multi-tier framework for evaluating AI agent artifacts — starting with agent skills — through deterministic quality gates, semantic overlap detection, synthetic evaluation-dataset generation, and live agent evaluation that measures how a skill actually changes agent behavior.
Sources: NVIDIA docs, GitHub, NVIDIA technical blog.
Support level
Experimental — community-supported on a best-effort basis through GitHub Issues, no SLA. This matters: this repository found and worked around real bugs in it (see Module 8) — exactly what "Experimental" should lead you to expect.
The four tiers, one line each
| Tier | Question it answers | Cost |
|---|---|---|
| Tier 1 | Is the skill well-formed and safe? | Free, offline, seconds |
| Tier 2 | Is it a near-duplicate of something that already exists? | Needs an embedding provider |
| Tier 3 | Does having the skill change agent behavior for the better, and by how much ("Skill Lift")? | Needs Docker, agent + judge credentials, real time/money |
| Tier 4 | Is the agent's answer, with the skill, actually correct for this domain? | Bring-your-own grader; SkillEvaluator supplies the harness, not domain judgment |
What it explicitly does NOT do
It does not build skills for you, does not run your production agent, does
not manage credentials or a data pipeline, and does not by itself decide
certification policy — that is what this repository's own
framework/certification/engine.py and policies/certification.yaml add on
top. See docs/02_TARGET_ARCHITECTURE.md.
Check your understanding
5 questions on this module. Get 4 right to complete it.
-
Tier 3 runs a live agent with and without the skill and reports the difference as Skill Lift.
-
It is Experimental, which is why this repository budgeted time for finding and working around real bugs in it.
-
Similarity detection is Tier 2. Certification policy is what this repository adds on top, in framework/certification/engine.py and policies/certification.yaml.
-
Tier 1 needs no API key, no Docker, and no sandbox, so it can run on every pull request.
-
Tier 4 is bring-your-own-grader. SkillEvaluator supplies the harness, not the domain judgment.
Module 2: Tier 1 — Construction & Security
Based on the reference §3-5.
What Tier 1 checks
- Schema validity of
SKILL.mdfrontmatter and body sections. - A quality score across advisory dimensions (discoverability language, recommended body sections, author format, etc.).
- Security scanning (Bandit/pip-audit for code, gitleaks for secrets, SkillSpector for skill-specific risks) — installed as separate external tools to avoid dependency conflicts.
Commands
skillevaluator quality-check skills/my-skill # scoring only
skillevaluator validate skills/my-skill --no-dedup # Tier 1 only, skips Tier 2
Both run in seconds, need no API key, no Docker, no sandbox — safe to run constantly, including on every PR.
Why it's worth adopting "almost unconditionally"
Free and catches real, fixable structural problems. This repository's own Tier 1 reports have surfaced real findings across nearly every skill drafted (see Module 7 for exact layout requirements that trip this check).
A real CI example
.github/workflows/skills-quality.yml's tier1 job installs the pinned
evaluator plus SkillSpector and gitleaks, diffs the PR against its base SHA
to find changed skills/* directories, and runs
skillevaluator validate <skill> --no-dedup on each — non-zero exit blocks
the PR. First real run (PR #4) passed in 54 seconds, no API keys required.
Check your understanding
5 questions on this module. Get 4 right to complete it.
-
Embedding comparison is Tier 2. Tier 1 is schema, quality scoring, and security scanning.
-
--no-dedupskips Tier 2.quality-checkis the other Tier 1 command, for scoring only. -
That is what makes it safe to run constantly, including on every pull request.
-
The security scanners are composed from outside the package so their dependencies cannot conflict with it.
-
The tier1 job in .github/workflows/skills-quality.yml runs validate --no-dedup on each changed skill; a non-zero exit blocks the merge.
Module 3: Tier 2 — Semantic Similarity / Deduplication
Based on the reference §3, §6.
What it does
Embeds a candidate skill's description/content and compares it against a central catalog to catch near-duplicate skills before merge (e.g. two teams independently building "explain portfolio performance" under different names).
The gotcha: embedding provider
Tier 2 needs a working embedding provider, and Anthropic does not provide
one. You need SKILL_EVAL_EMBEDDING_PROVIDER=openai (or nv_build / an
OpenAI-compatible endpoint) plus that provider's key — even if Tier 3's agent
is entirely Claude-based. If this is missing, Tier 2 silently reports
"skipped" rather than erroring — a real trap this project actually hit.
Command
SKILL_EVAL_EMBEDDING_PROVIDER=openai OPENAI_API_KEY=... \
skillevaluator similarity-check skills/ --type skill
This repository's governance layer on top
Tier 2 alone gives a similarity score. This repository's own
framework/certification/check_similarity.py and policies/similarity.yaml
turn that score into governance actions — e.g. EXACT_DUPLICATE blocks a
PR, HIGH_SIMILARITY routes to an advisory architecture_review with an
assigned owner/SLA, rather than blocking outright. This is this repository's
own addition, not a SkillEvaluator feature.
Check your understanding
5 questions on this module. Get 4 right to complete it.
-
Tier 2 embeds skills, so it needs SKILL_EVAL_EMBEDDING_PROVIDER set to openai, nv_build, or an OpenAI-compatible endpoint.
-
A silent skip rather than an error, a real trap this project hit.
-
Set it with that provider's key, for example SKILL_EVAL_EMBEDDING_PROVIDER=openai with OPENAI_API_KEY.
-
Only EXACT_DUPLICATE blocks. HIGH_SIMILARITY goes to an advisory architecture_review.
-
Tier 2 alone gives a score; turning it into block or review actions is this repository's own addition.
Module 4: Tier 3 — Live-Agent Evaluation & Skill Lift
Based on the reference §2-4 and
docs/MILESTONE_4_PERFORMANCE_ATTRIBUTION.md.
The core mechanic
Run the same eval cases twice — once with the skill loaded, once without — against a live agent and a live judge model, in a sandbox (Docker/Harbor). Skill Lift = with-skill score minus without-skill score. This is the central number Tier 3 produces.
Real numbers from this repository
| Skill | Trials | Skill Lift |
|---|---|---|
| Performance Attribution | 150 (75/75 both arms) | +0.1253 |
| Portfolio Overview | 150 (75/75 both arms) | +0.1316 |
A full 25-case × 3-attempt × 2-arm matrix against claude-sonnet-5 as both
agent and judge took 30-55 minutes and roughly $15-45 per skill. A
cheap Haiku-tier "quick pass" ran in ~12 minutes for ~$1-3 — useful for
validating wiring, not a substitute for certification-grade evidence
(a weaker agent compresses Skill Lift toward zero on both arms).
Requirements easy to forget
- Docker (or another sandbox), the agent's own credentials.
- A separate judge-model credential and
SKILL_EVAL_JUDGE_MODELoverride — otherwise the judge silently falls back to the agent's own model, coupling judge cost to agent cost for no reason. --copy-repo— without it, the sandbox can't find the repo's own tool bridge (a real operator-error bug this project hit).
Command shape
SKILL_EVAL_LLM_PROVIDER=anthropic ANTHROPIC_API_KEY=... \
SKILL_EVAL_JUDGE_MODEL=claude-sonnet-5 \
skillevaluator validate skills/my-skill \
--agent-eval -a claude-code --agent-model claude-code=claude-sonnet-5 \
--env-mode docker --copy-repo \
--results-dir reports/my-skill-tier3 -r cli,json,markdown
Metrics beyond Skill Lift
pass_at_k (reliability across repeated attempts) and per-dimension
dimensions_with_skill scores (e.g. discoverability, tool-use correctness)
also appear in the raw report — see Module 8 for the report shape.
Check your understanding
5 questions on this module. Get 4 right to complete it.
-
Tier 3 runs the same cases twice, with and without the skill, against a live agent and judge.
-
That was 25 cases, three attempts, and two arms against claude-sonnet-5 as agent and judge. The $1-3 figure is the Haiku-tier quick pass.
-
That couples judge cost to agent cost for no reason, so set a separate judge model and credential.
-
Without it the sandbox cannot see the repository's files, an operator error this project actually made.
-
It is useful for checking the wiring, but a low lift there says more about the agent than the skill.
Module 5: Tier 4 — Domain Graders
Based on the reference §7 and Milestone 8
(docs/10_DEVELOPMENT_ROADMAP_AND_PROGRESS.md).
What Tier 4 is
SkillEvaluator supplies the harness (a Tier 3 trajectory to grade against),
not the domain judgment. You write evals/grader.py per skill; real domain
logic should live in a shared, reusable module, not duplicated per skill.
This repository's pattern
from graders.finance.my_skill import grade as grade_evidence
def grade(case, trajectory=None, expected=None):
evidence = trajectory or expected or {}
result = grade_evidence(evidence)
return {"skill": "my-skill", "metrics": result["metrics"],
"score": result["score"], "notes": []}
Composed from generic building blocks in graders/finance/*.py, reused
across all 12+ PM skills — reusability across skills was made an explicit
exit criterion in Milestone 8.
A free, post-hoc extraction pipeline
This repository built skills/*/evals/{tier3_trial_extractor,aggregate_tier4,
generate_benchmark}.py — a pipeline that mines already-completed Tier 3
trial data on disk to compute domain-specific correctness metrics and hard
gates (e.g. regression pass rate, authorization/permission-denial scanning)
without any new API calls, because Tier 4 itself is bring-your-own-grader
and SkillEvaluator doesn't do this for you.
A real observed quirk to defend against
Directory layouts can differ between runs of the same tool — this project
observed two different completed-trial directory structures across
otherwise-identical skillevaluator validate --agent-eval invocations. Any
custom post-hoc analysis script needs to defend against that, not assume a
fixed layout.
Check your understanding
4 questions on this module. Get 4 right to complete it.
-
SkillEvaluator supplies the harness; you write evals/grader.py and the domain logic it calls.
-
Each skill's grader.py is a thin wrapper; reuse across skills was an explicit Milestone 8 exit criterion.
-
tier3_trial_extractor, aggregate_tier4, and generate_benchmark compute correctness metrics and hard gates from trials already on disk.
-
This is a real, observed quirk of the tool; analysis scripts should not assume a fixed layout.
Module 6: Installing & Running SkillEvaluator
Based on the reference §4 and
docs/13_NVIDIA_EVALUATOR_UPGRADE_POLICY.md.
General upstream install
# Prerequisites: uv (or pip), Python 3.12 or 3.13, git; Docker only for Tier 3
uv tool install --python 3.13 "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git"
skillevaluator --version
skillevaluator health-check
Smaller footprints exist: skillevaluator (Tier 1 only), [security]
(+ Bandit/pip-audit), [tier2,tier3] (+ dedup and live-agent eval).
External security tools install separately to avoid dependency conflicts:
brew install gitleaks
uv tool install git+https://github.com/NVIDIA/SkillSpector.git
Why pin an exact commit
Installing @main is fine for exploration; for anything you intend to
certify against, pin an exact commit. SkillEvaluator's own release notes
and this repository's own experience both found real, breaking behavioral
differences worth pinning against.
uv venv .venv --python 3.13
uv pip install --python .venv/bin/python \
'skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git@<pinned-sha>'
This repository also tracks a local patch for a real judge-truncation bug
found in its pinned version (patches/skillevaluator-0.2.1-judge-max-tokens.patch)
— apply it only if your pinned version predates NVIDIA's own upstream fix.
Per-tier run commands
See Modules 2-4 for the exact Tier 1/2/3 commands and their required environment variables.
Check your understanding
5 questions on this module. Get 4 right to complete it.
-
Both appear in the upstream install steps in Module 6 and the tutorial's Step 1.
-
@main is fine for exploring; anything you intend to certify against should be pinned.
-
Plain skillevaluator is Tier 1 only, and [security] adds Bandit and pip-audit. [all] installs everything.
-
The patch fixes a judge token-truncation bug in the pinned 0.2.1 build.
-
Tier 3 runs the agent in a sandbox; the other tiers do not need Docker.
Module 7: Skill Package Layout
Based on the reference §5.
Required layout
skills/<skill-name>/
├── SKILL.md # required
├── skill.yaml # optional metadata this repo's governance layer reads
├── CHANGELOG.md # optional
├── evals/
│ ├── config.yml # harbor/tier3 run parameters, grading mode
│ ├── evals.json # the eval case dataset
│ └── grader.py # custom Tier 4 grading hook
└── tests/ # optional local unit tests for the grader
SKILL.md frontmatter Tier 1 actually checks
---
name: my-skill # required; must match the directory name
description: One sentence, plus WHEN # required; missing "when to use" language
# is flagged as a discoverability finding
metadata:
author: Team Name <team@example.com> # required for the author_format check
---
A missing metadata: block is a real, observed failure mode — several
skills in this repository started without one and failed Tier 1's
author_format check until it was added.
A real, observed quirk
By default, Tier 1's schema check flags files like skill.yaml and
CHANGELOG.md sitting in the skill root as "unexpected" — it expects the
root to contain only SKILL.md plus a small allowed set of subdirectories
(agents/, assets/, config/, evals/, references/, scripts/,
tests/, tools/). These are non-blocking advisories, not hard failures —
move such files into an allowed subdirectory or set
SKILL_EVAL_SCHEMA_ALLOWED_DIRS for a clean report.
Recommended body sections
## Use when, ## Do not use when, ## Procedure, ## Instructions/
## Usage, ## Examples, prerequisites/limitations/troubleshooting — each
an advisory finding if missing, not a hard fail, but each genuinely moves
the quality score.
Check your understanding
5 questions on this module. Get 4 right to complete it.
-
Everything else in the layout is optional or read only by this repository's governance layer.
-
Tier 1 checks that name matches the folder the skill lives in.
-
Several skills in this repository failed this way until the metadata block was added.
-
The schema check expects only SKILL.md plus a small set of allowed subdirectories in the root.
-
Recommended sections are advisory, not hard fails, but each one moves the quality score.
Module 8: Reading Reports, CI/CD, and Real Bugs
Based on the reference §2, §6-7 and
framework/adapters/nvidia_skillevaluator.py.
Two different report shapes to know
- Raw NVIDIA JSON — Tier 3 ("agent-eval") reports carry
agents(agent name →dimensions_with_skill,lift,pass_at_k); Tier 1-only reports instead carryquality_summary(per-dimension scores) and aresultslist including a "Security Scan" validator entry. - This repository's normalized contract — produced by
framework/adapters/nvidia_skillevaluator.py'sparse_nvidia_report, which converts either raw shape into a stable internal schema (generic_metrics,skill_lift,reliability,findings,certification: {status: "UNASSESSED"}). Downstream certification logic never touches vendor-specific fields directly — this is what makes a future evaluator version bump a contained, testable event instead of a library-wide breaking change.
Never conflate the two — if asked "what does the report look like," clarify which one is meant.
CI/CD integration this repository actually runs
.github/workflows/skills-quality.yml:
- tier1 — blocks a PR on Tier 1 validation failure across changed skills;
also runs a real ownership-enforcement check
(framework/certification/check_ownership.py) that fails on the literal
domain-owner-required placeholder. No API keys required.
- similarity — real as of this repo's own history, applies
policies/similarity.yaml governance actions against the central
catalog, blocking only on EXACT_DUPLICATE.
- tier3-fast, domain-graders — documented placeholders, not fully built;
this repository's own recommendation is to keep Tier 3 advisory, not
blocking, on every PR (too slow/expensive to gate routine merges) and
reserve blocking Tier 3 for an explicit release/certification workflow.
Real bugs this repository found and worked around
- An agent-compatibility gap in the evaluator's Tier 3 execution heuristic
(a shell tool self-reporting as
exec, a string the recognized-tool-name list didn't include). - A judge token-truncation bug (
max_tokens=1024, no retry) in the evaluator's own judge code, found by reading its vendored source — patched locally, tracked inpatches/. - A missing
--copy-repoCLI flag (an operator error, not an evaluator bug) that left the sandbox unable to find the repo's own tool bridge. - API credit exhaustion mid-run — an operating-cost lesson, not a bug.
Lesson: at "Experimental" support level, budget time to read the evaluator's own source when something looks wrong rather than assuming the skill under test is broken.
Check your understanding
5 questions on this module. Get 4 right to complete it.
-
quality_summary belongs to Tier 1-only reports; generic_metrics and certification belong to this repository's normalized contract.
-
Downstream certification logic never reads vendor-specific fields, so only parse_nvidia_report has to change.
-
tier3-fast and domain-graders are documented placeholders; the recommendation is to keep Tier 3 advisory on pull requests.
-
Credit exhaustion was an operating-cost lesson, and the other two were real bugs in the evaluator.
-
Two of this repository's four problems were found by reading the evaluator's vendored source.
Troubleshooting scenarios
Each scenario is a problem someone reported. Pick the most likely diagnosis. The modules behind each answer are named in its explanation.
-
Anthropic does not provide embeddings. Set SKILL_EVAL_EMBEDDING_PROVIDER=openai with OPENAI_API_KEY, or an OpenAI-compatible endpoint. (Module 3)
-
This repository hit exactly this with a shell tool that reports itself as exec. (Module 8)
-
The author line is usually there, just not inside metadata. (Module 7)
-
The schema check expects a fixed set of subdirectories in the skill root. (Module 7)
-
Set a separate judge model and credential; otherwise judge cost is coupled to agent cost. (Module 4)
-
An easy-to-forget operator error, not an evaluator bug. (Module 4)
-
A weaker agent compresses Skill Lift toward zero on both arms, so the number says little about the skill. (Module 4)
-
Directory layouts can differ between runs of the same tool. (Module 5)
Final exam
Fifteen questions across all eight modules. Score 80% (12 of 15) or more to pass. Answer every question, then check your answers.
-
Module 4.
-
Module 2.
-
Module 3.
-
Module 5.
-
Module 4.
-
Module 4.
-
Module 6.
-
Module 3.
-
Downstream logic never parses vendor-specific fields directly. (Module 8)
-
Module 1.
-
Module 7.
-
tier3-fast and domain-graders are advisory placeholders. (Module 8)
-
Module 4.
-
The patch lives in patches/. (Modules 6 and 8)
-
The low lift reflects the agent, not the skill. (Module 4)