SkillEvaluator Mastery NVIDIA SkillEvaluator

SkillEvaluator Mastery

A short course on NVIDIA SkillEvaluator, the open-source framework for testing agent skills, taught from what this repository learned by running it on twelve portfolio-management skills: the real costs, the real bugs, and the settings that are easy to get wrong.

It is for engineers and product owners who write agent skills and want to know whether a skill is well formed, whether it duplicates one that already exists, and whether it actually makes an agent better.

How the course works

  1. Read the eight modules in order. Each one covers one part of SkillEvaluator and ends with a short multiple-choice check. Your answers are marked as soon as you submit them.
  2. Work through the troubleshooting scenarios. Eight problems people actually hit, each with a diagnosis to choose.
  3. Take the final exam. Fifteen questions across every module; 80% passes.

The hands-on tutorial has you scaffold a toy skill and run Tier 1 against it for free. The reference collects every command, setting, and finding in one place.

Reading the course costs nothing. Running SkillEvaluator's Tier 1 is also free and offline, but a full Tier 3 run against a live agent cost this repository $15-45 per skill, so the modules say clearly which steps need credentials or money.

Your progress and quiz scores are saved in this browser only. Nothing is sent anywhere.

The modules

  1. Module 1What it is and whyThe four tiers, what each one costs, and what SkillEvaluator leaves to you.
  2. Module 2Tier 1: construction and securityFree, offline checks for schema, quality, and security on every pull request.
  3. Module 3Tier 2: similarityCatching near-duplicate skills, and the embedding-provider trap.
  4. Module 4Tier 3: live-agent evaluationMeasuring Skill Lift with a live agent and judge, and what that costs.
  5. Module 5Tier 4: domain gradersWriting reusable graders that judge domain correctness.
  6. Module 6Installing and runningInstall options, pinning an exact commit, and checking the install.
  7. Module 7Skill package layoutThe folder layout and frontmatter that Tier 1 checks.
  8. Module 8Reports, CI/CD, and real bugsReading reports, wiring CI, and four problems this repository hit.

Other ways to take it

  • As a ten-slide overview. The Skills Ledger is a crash course on why skill governance matters and what this repository's real runs found.
  • As one file. Download the course as a single HTML page that works offline.
  • With a coding agent. The same material is an agent skill, skills/skillevaluator-mastery/, that teaches it as a conversation and writes fresh questions each time. In a checkout of this repository, say skillevalexpert to Claude Code, Codex, or GitHub Copilot.

SkillEvaluator Mastery — A From-Scratch, Hands-On Tutorial

This is the from-scratch companion to docs/15_SKILLS_AND_SKILLEVALUATOR_REFERENCE.md. Doc 15 is the reference to look things up in; this document is the progressive, hands-on exercise to actually build the muscle memory — scaffold a toy skill, watch each tier fire, break it on purpose, and read a real report. It assumes zero prior SkillEvaluator experience.

The same material is also a course with quizzes, troubleshooting scenarios, and a scored final exam, on the course site or as one downloadable file. It is also an agent skill, skills/skillevaluator-mastery/, that teaches it as a conversation: in a checkout of this repository, say "skillevalexpert" to Claude Code, Codex, or GitHub Copilot. To use the skill elsewhere, copy that folder to .github/skills/, .claude/skills/, .agents/skills/, or ~/.copilot/skills/. This document is the read-it-yourself version.


0. Before you start

You need: uv (or pip), Python 3.12/3.13, git. Everything through step 3 below is free and needs no API key, no Docker, and no live model. Steps 4+ are clearly marked with their real cost/credential requirements — do not run them without reading the caveat first.


Step 1 — Install Tier 1 only

uv tool install --python 3.13 "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git"
skillevaluator --version
skillevaluator health-check

If you only want Tier 1 (no security scanning), the plain skillevaluator extra is enough — see docs/15_... §4 for all the install variants and why this repository pins an exact commit instead of @main.

Step 2 — Scaffold a toy skill from scratch

mkdir -p /tmp/my-first-skill/evals

Create /tmp/my-first-skill/SKILL.md:

---
name: my-first-skill
description: Say hello in a specific format.
metadata:
  author: Your Name <you@example.com>
---

# My First Skill

## Use when

Use this skill when asked to greet someone.

## Procedure

1. Greet the user by name if given.
2. Otherwise greet generically.

This is intentionally minimal — you'll break it on purpose in Step 4.

Step 3 — Run Tier 1 and read the report

skillevaluator quality-check /tmp/my-first-skill
skillevaluator validate /tmp/my-first-skill --no-dedup

Look at: - The overall quality score and which dimensions it scored on. - Any advisory findings about missing recommended sections (## Do not use when, ## Examples, etc. — see docs/15_... §5 for the full list). - Whether the security scan step ran (it needs gitleaks/SkillSpector installed separately — docs/15_... §4).

Checkpoint: you should see a passing schema check (the frontmatter has name, description, and a metadata.author) but likely several advisory quality findings, because the body is minimal.

Step 4 — Break it on purpose, then fix it

Remove the metadata: block entirely and rerun quality-check. This is a real, observed failure mode in this repository — the author_format check fails outright, not just an advisory finding (docs/15_... §5). Put the block back, then remove the ## Use when section and rerun — this time notice it's an advisory finding, not a hard failure. This distinction (hard schema failure vs. advisory quality finding) is the single most useful thing to internalize about Tier 1.

Now add a skill.yaml at the skill root with placeholder content and rerun quality-check again — Tier 1's schema check may flag it as an "unexpected" root file (docs/15_... §5); move it into an allowed layout, or note the SKILL_EVAL_SCHEMA_ALLOWED_DIRS escape hatch, without treating it as a bug.

Step 5 — Inspect a real report from this repository

Rather than running Tier 3 yourself (see the cost caveat in Step 7), read an already-generated report:

ls reports/*.json
cat skills/portfolio-overview/BENCHMARK.md

Compare the raw JSON's shape (agents, dimensions_with_skill, lift, pass_at_k, or quality_summary for Tier 1-only reports) against framework/adapters/nvidia_skillevaluator.py's parse_nvidia_report — run this in a Python shell to see the normalized shape produced:

uv run python -c "
from framework.adapters.nvidia_skillevaluator import parse_nvidia_report_file
import json
result = parse_nvidia_report_file(
    'reports/skillevaluator-output-20260904020239.json',
    skill_id='pm.demo', skill_name='Demo', skill_version='0.0.0')
print(json.dumps(result, indent=2)[:2000])
"

Checkpoint: you should be able to say, in one sentence each, what the raw NVIDIA shape looks like and what this repository's normalized adapter produces from it — and why the distinction matters (docs/15_... §7).

Step 6 — Understand Tier 2 without spending money

Tier 2 needs a working embedding provider that is not Anthropic. Rather than configuring one, read catalogs/skill-catalog.json's structure (an entries list, each with an embedding vector and a content_fingerprint) and policies/similarity.yaml's governance actions (EXACT_DUPLICATE → block, HIGH_SIMILARITY → advisory review). Then read framework/certification/check_similarity.py to see how this repository turns a raw similarity score into one of those actions.

Step 7 — Understand Tier 3 without spending money (unless you choose to)

Real cost warning, read before running anything here: a full Tier 3 matrix in this repository's own experience took 30-55 minutes and roughly $15-45 per skill (docs/15_... §2-3, docs/MILESTONE_4_PERFORMANCE_ATTRIBUTION.md).

Instead of running it, read the milestone doc above end-to-end. It walks through: the exact command used, the four real bugs found (an agent-compatibility gap, a judge truncation bug, a missing --copy-repo flag, and API credit exhaustion), and how the final Skill Lift number (+0.1253) was computed and interpreted. If and when you do choose to run a real Tier 3 matrix, use the exact command in docs/15_... §4c, and budget time, Docker, an agent credential, and a separate judge-model credential via SKILL_EVAL_JUDGE_MODEL.

Step 8 — Understand Tier 4 and certification, on top

Read graders/finance/portfolio_overview.py alongside skills/portfolio-overview/evals/grader.py to see the thin-hook pattern, then policies/certification.yaml and framework/certification/profile_resolver.py to see how this repository turns raw Tier 1-4 evidence into a pass/fail certification decision that is explicitly independent of SkillEvaluator's own opinion (docs/02_TARGET_ARCHITECTURE.md, docs/04_EVALUATION_AND_CERTIFICATION.md).


Self-test checklist

Before considering yourself "SkillEvaluator-literate," you should be able to answer, without looking anything up:

  • What does each of the four tiers check, and which are free vs. costly?
  • What's the difference between a schema failure and a quality finding?
  • Why does Tier 2 need a non-Anthropic embedding provider?
  • What does "Skill Lift" mean, and what does a real number (e.g. +0.1253) tell you?
  • What are the three easy-to-forget Tier 3 requirements (--copy-repo, SKILL_EVAL_JUDGE_MODEL, judge vs. agent credential)?
  • What's the difference between the raw NVIDIA report shape and this repository's normalized adapter output, and why does that boundary exist?
  • Where does SkillEvaluator's job end and this repository's own certification logic begin?

To test yourself on the same material with marked quizzes, scenarios, and a scored final exam, take the course.

Skills & SkillEvaluator — One-Stop Reference

This document is the reference after completing the local tutorial. Start with docs/00_TUTORIAL.md for a no-credentials exercise. This document is the single place to point someone who asks "what are agent skills, what does NVIDIA SkillEvaluator actually do, is it worth adopting, and how do I get it running on my own repo?" It is written to stand alone — combining the official upstream documentation with this project's own hands-on findings (including the real bugs, costs, and gotchas discovered by actually running it, not just reading its docs). Every claim about this project's experience links to the evidence doc it came from; every claim about the external ecosystem cites its source.


1. What SkillEvaluator is, and what it helps achieve

NVIDIA SkillEvaluator is an open-source, multi-tier framework for evaluating AI agent artifacts — starting with agent skills — through deterministic quality gates, semantic overlap detection, synthetic evaluation-dataset generation, and live agent evaluation that measures how a skill actually changes agent behavior (NVIDIA docs; GitHub; NVIDIA technical blog). Its support level is explicitly Experimental — community-supported on a best-effort basis through GitHub Issues, no SLA. That matters: this project found and worked around real bugs in it (§4 below), which is exactly the kind of thing "Experimental" should lead you to expect.

The problem it solves. As a team accumulates AI skills (packaged instructions that teach an agent to do a specific job), review-by-eyeball stops scaling. SkillEvaluator answers, with numbers instead of opinions:

  • Is the skill well-formed and safe? (Tier 1)
  • Is it a near-duplicate of something that already exists? (Tier 2)
  • Does having the skill actually change agent behavior for the better — and by how much? (Tier 3 — "Skill Lift": the with-skill score minus the without-skill score)
  • Is the agent's own answer, with the skill, actually correct? (Tier 4 — domain-specific graders you write; SkillEvaluator supplies the harness, not the finance/legal/domain judgment)

What it does not do: it does not build skills for you, does not run your production agent, does not manage credentials or a data pipeline, and does not (by itself) decide certification policy — that is what this project's own framework/certification/engine.py and policies/certification.yaml add on top. See docs/02_TARGET_ARCHITECTURE.md for how this project draws that line.


2. What our own tests actually show

Two skills in this repository have been taken all the way through the real pipeline — live agent, live judge model, real Docker sandbox, real certification policy. Both results are real, both are FAIL, and both failures are precisely diagnosed rather than vague:

Skill Trials Skill Lift Certification Why
Performance Attribution 150 (75/75 both arms) +0.1253 FAIL Discoverability 0.8862 < 0.90 — a metric-scoping artifact (two eval cases whose correct behavior is to use no tools at all), not a real defect. Full trail: docs/MILESTONE_4_PERFORMANCE_ATTRIBUTION.md.
Portfolio Overview 150 (75/75 both arms) +0.1316 FAIL The same discoverability shortfall (0.8942), independently corroborated by SkillEvaluator's own report — plus a newly discovered certification policy gap: a minimum metric (reconciliation) specific to one skill's grader auto-fails every other skill regardless of quality. Evidence: skills/portfolio-overview/BENCHMARK.md.

Getting Performance Attribution's result required finding and fixing four real bugs — not toy issues, actual blockers:

  1. An agent-compatibility gap in the evaluator's Tier 3 execution heuristic (Codex's shell tool self-reports as exec, a string the evaluator's recognized-tool-name list didn't include).
  2. A token-budget bug in the evaluator's own judge code (max_tokens=1024, no retry, on two of three judge functions — the third had already fixed the identical bug, the fix just wasn't propagated). Patched locally, tracked in patches/.
  3. A missing --copy-repo CLI flag (this project's own operator error, not an evaluator bug) that left the sandbox unable to find the repo's own tool bridge.
  4. API credit exhaustion mid-run, twice — a real operating-cost lesson, not a bug.

Beyond those two full certifications: Milestone 6 deliberately introduced six synthetic defects (vague description, duplicate skill, missing derivatives, mismatched dates, weak/no-value skill, unauthorized data source) and confirmed the framework catches five of them directly with real evidence (a Tier 2 similarity score of 0.9468 on the deliberate duplicate; existing grader regression tests for the domain-correctness defects) — see docs/MILESTONE_6_DELIBERATE_DEFECTS.md. Milestone 9 put a real Tier 1 check into CI (§6 below), and its first live run against a real PR passed in 54 seconds.

The honest summary: no skill in this catalog has cleared certification outright yet, and that is reported plainly rather than smoothed over — see docs/14_EXECUTIVE_SUMMARY_AND_WALKTHROUGH.md for the full cold-read walkthrough with an honest pros/cons assessment.


3. Feasibility: how hard is it to bring this to a real repo?

Grounded in this project's actual experience, not the marketing pitch:

What's genuinely easy: - Tier 1 (schema, security, PII, quality scoring) is entirely offline and free — no API key, no Docker, no sandbox. skillevaluator quality-check <skill> and skillevaluator validate <skill> (Tier 1 only) run in seconds and are safe to run constantly, including in CI on every PR (this project's Milestone 9 job does exactly this). - The pinned-version install is a single pip/uv command plus one line for an optional security scanner (§5). - The tool is genuinely agent-agnostic at the CLI level — switching Tier 3's agent (Codex → Claude Code) was a one-flag change once the compatibility gap was diagnosed.

What's genuinely hard, and costs real money and time: - Tier 2 needs a working embedding provider, and Anthropic does not provide one — you need SKILL_EVAL_EMBEDDING_PROVIDER=openai (or nv_build / an OpenAI-compatible endpoint) plus that provider's key, even if your Tier 3 agent is entirely Claude-based. This project only discovered this because Tier 2 silently reported "skipped" rather than erroring, on every run, until an OpenAI key was wired in. - Tier 3 is expensive and slow. A full 25-case × 3-attempt × 2-arm matrix (150 trials) against claude-sonnet-5 as both agent and judge took this project 30-55 minutes and roughly $15-45 per skill, and exhausted a modest prepaid API balance mid-run more than once in one day of iterative debugging. A cheap Haiku-tier "quick pass" (same 150 trials, weaker model) ran in ~12 minutes for ~$1-3 — useful for validating wiring, not a substitute for real certification evidence: Haiku's quick-pass numbers on this project's skills failed nearly every certification threshold that Sonnet cleared, because a weaker agent compresses Skill Lift toward zero on both arms, not because the skill itself is worse. - Tier 3 needs Docker (or a local/cloud sandbox), the selected agent's own credentials, and — separately, and easy to forget — a judge-model credential and the SKILL_EVAL_JUDGE_MODEL override, or the judge silently falls back to whichever model the agent under test uses, coupling judge cost to agent cost for no reason. - The evaluator itself has real bugs at "Experimental" support level — budget time to read its source when something looks wrong rather than assuming the skill is broken. Both real bugs this project found (§2) were root-caused by reading skillevaluator's own vendored source, not by guessing. - Directory layouts can differ between runs of the same tool — this project observed two different completed-trial directory structures across otherwise-identical invocations of skillevaluator validate --agent-eval; any custom post-hoc analysis script (like the Tier 4 extraction pipeline in §7) needs to defend against that rather than assume a fixed layout.

Bottom line: Tier 1 is worth adopting almost unconditionally — it's free and catches real, fixable structural problems (see the SKILL.md schema/quality findings in every Tier 1 report this project has generated). Tier 2 needs one extra credential decision. Tier 3 is a genuine cost/time investment that should be reserved for skills where you actually want certification-grade evidence, not run reflexively on every change — this is exactly why this project's own Milestone 7 right-sizing decision limits full Tier 3 depth to three flagship skills and keeps the other nine at Tier 1/2 only (docs/10_DEVELOPMENT_ROADMAP_AND_PROGRESS.md).


4. Installing SkillEvaluator — step by step

Per the official installation guide:

# Prerequisites: uv (or pip), Python 3.12 or 3.13, git; Docker only if you'll run Tier 3
uv tool install --python 3.13 "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git"

# Smaller footprints, if you don't need everything:
#   skillevaluator              -- Tier 1 only, no security scanning
#   skillevaluator[security]    -- + Bandit/pip-audit
#   skillevaluator[tier2,tier3] -- + dedup and live-agent eval

# Verify
skillevaluator --version
skillevaluator health-check

External security tools are installed separately to avoid dependency conflicts:

brew install gitleaks                                       # or: go install / release binary
uv tool install git+https://github.com/NVIDIA/SkillSpector.git

Installing @main is fine for exploration; for anything you intend to certify against, pin an exact commit — SkillEvaluator's own release notes and this project's Milestone 4 both found real, breaking behavioral differences worth pinning against (docs/13_NVIDIA_EVALUATOR_UPGRADE_POLICY.md):

uv venv .venv --python 3.13
uv pip install --python .venv/bin/python \
  'skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git@009aa300be7925c7ba75760592baeb941cc29ba8'
uv pip install --python .venv/bin/python \
  'git+https://github.com/NVIDIA/SkillSpector.git@29b0dc8c39424e8e31ca055fa027adf8ba8f9650'
brew install gitleaks

# This project's local tracked patch for a real judge-truncation bug found
# in the pinned version -- see §2 and patches/README.md. Skip this line if
# you're pinning a version where NVIDIA has since fixed it upstream.
patch -p1 -d .venv/lib/python3.13/site-packages < patches/skillevaluator-0.2.1-judge-max-tokens.patch

Full reasoning for pinning (not just installing latest) and the upgrade process when you do want to move the pin: docs/13_NVIDIA_EVALUATOR_UPGRADE_POLICY.md.

4c. Running each tier

# Tier 1 -- free, offline, seconds. Do this on every skill, constantly.
skillevaluator quality-check skills/my-skill
skillevaluator validate skills/my-skill --no-dedup

# Tier 2 -- needs an embedding provider (not Anthropic; see §3)
SKILL_EVAL_EMBEDDING_PROVIDER=openai OPENAI_API_KEY=... \
  skillevaluator similarity-check skills/ --type skill

# Tier 3 -- needs Docker, agent credentials, a judge model, and real time/$
SKILL_EVAL_LLM_PROVIDER=anthropic ANTHROPIC_API_KEY=... \
SKILL_EVAL_JUDGE_MODEL=claude-sonnet-5 \
  skillevaluator validate skills/my-skill \
  --agent-eval -a claude-code --agent-model claude-code=claude-sonnet-5 \
  --env-mode docker --copy-repo \
  --results-dir reports/my-skill-tier3 -r cli,json,markdown

--copy-repo and SKILL_EVAL_JUDGE_MODEL are both easy to forget and both silently degrade your results rather than error loudly — see §3 and docs/MILESTONE_4_PERFORMANCE_ATTRIBUTION.md for exactly how each one bit this project the first time.


5. Repository and skill-package layout requirements

SkillEvaluator's Tier 1 schema check is opinionated about layout. Grounded in real findings from every Tier 1 report this project has generated:

skills/<skill-name>/
├── SKILL.md              # required
├── skill.yaml            # optional metadata this project's own governance layer reads
├── CHANGELOG.md          # optional
├── evals/
│   ├── config.yml        # harbor/tier3 run parameters, grading mode
│   ├── evals.json        # the eval case dataset
│   └── grader.py         # custom Tier 4 grading hook
└── tests/                # optional local unit tests for the grader

SKILL.md frontmatter — the schema and quality checks specifically look for:

---
name: my-skill                          # required; must match the directory name
description: One sentence, plus WHEN     # required; missing "when to use" language
                                          # is flagged as a discoverability finding
metadata:
  author: Team Name <team@example.com>   # required for the author_format check
  # tags: [finance, reporting]           # recommended, not required
# version: 1.0.0                        # recommended, not required
---

A missing metadata: block is a real, observed failure mode in this project — several skills started without one and failed Tier 1's author_format check until it was added.

Recommended body sections (each an advisory finding if missing, not a hard fail, but each genuinely moves the quality score): ## Use when, ## Do not use when, ## Procedure, ## Instructions or ## Usage, ## Examples, prerequisites/limitations/troubleshooting sections.

A real, observed quirk: by default, Tier 1's schema check flags files like skill.yaml and CHANGELOG.md sitting in the skill root as "unexpected" — it expects the skill root to contain only SKILL.md plus a small set of subdirectories (agents/, assets/, config/, evals/, references/, scripts/, tests/, tools/). These are non-blocking advisories in this project's configuration, not hard failures, but if you want a clean report, either move such files into an allowed subdirectory or set SKILL_EVAL_SCHEMA_ALLOWED_DIRS.

evals/evals.json shape (this project's convention, one case shown):

{
  "skill_name": "my-skill",
  "evals": [
    {
      "id": "my-skill-001",
      "category": "explicit-positive",
      "prompt": "...",
      "expected_output": "...",
      "assertions": ["Do not fabricate facts.", "..."],
      "expected_skill": "my-skill"
    }
  ]
}

evals/config.yml (real, working example from this repo):

schema_version: 1
harbor:
  n_attempts: 3
  pass_threshold: 0.80
  n_concurrent: 4
  agent_runtime_preflight: true
  base_image_mode: reuse
  pre_agent_setup:
    - >-
      if test -r /workspace/skills/my-skill/SKILL.md; then
        printf '%s\n' '# Required skill bootstrap' \
          'Before answering, read /workspace/skills/my-skill/SKILL.md with cat.' \
          > /workspace/AGENTS.md;
      fi
skill_workspace:
  mode: group
grading:
  mode: default_plus_custom

evals/grader.py is a thin hook — real Tier 4 domain logic belongs in a shared, reusable module (this project's pattern: graders/finance/*.py, composed from generic building blocks — see Milestone 8 in docs/10_DEVELOPMENT_ROADMAP_AND_PROGRESS.md for why reusability across skills was made an explicit exit criterion):

from graders.finance.my_skill import grade as grade_evidence

def grade(case, trajectory=None, expected=None):
    evidence = trajectory or expected or {}
    result = grade_evidence(evidence)
    return {"skill": "my-skill", "metrics": result["metrics"],
            "score": result["score"], "notes": []}

6. CI/CD integration

This project has a real, working example rather than a theoretical recipe: .github/workflows/skills-quality.yml's tier1 job. It installs the pinned evaluator (plus SkillSpector and gitleaks), diffs the PR against its base SHA to find changed skills/* directories, and runs skillevaluator validate <skill> --no-dedup on each — a non-zero exit blocks the PR (given branch protection requires the check). Its first real run, against this project's own PR #4, passed in 54 seconds. No API keys required, because Tier 1 is LLM-free by default.

Extending this to Tier 2 in CI needs an embedding-provider secret (OPENAI_API_KEY or equivalent) and a central approved-skill catalog to diff new/changed skills against (catalogs/ in this repo) — SkillEvaluator's own docs describe this as a standard "merge gate with exit codes" pattern, and mention a maintained GitHub Actions recipe as part of progressive adoption.

Extending to Tier 3 in CI needs a Docker-capable runner, the target agent's credentials, a separately-configured judge model, and — given the real cost data in §3 — a firm opinion on when it runs. This project's own recommendation, informed by real experience: keep Tier 3 advisory, not blocking, on every PR (too slow and expensive to gate routine merges), and reserve blocking Tier 3 runs for an explicit release/certification workflow that runs on demand or on a schedule, not on every commit. See the similarity, tier3-fast, and domain-graders jobs in this project's own workflow file for the placeholder shape that pattern would fill in, and docs/10_DEVELOPMENT_ROADMAP_AND_PROGRESS.md's Milestone 9 section for why this project deliberately stopped at the Tier 1 gate rather than building the full pipeline.


7. What this project built on top (reusable patterns)

Two pieces of this project's own code are worth knowing about if you're adopting SkillEvaluator elsewhere, because they solve problems the tool itself doesn't:

  • A normalized adapter (framework/adapters/nvidia_skillevaluator.py) that translates NVIDIA's raw report JSON into a stable internal schema, so nothing downstream (certification logic, reporting) parses vendor-specific fields directly — this is what makes an evaluator version bump a contained, testable event instead of a library-wide breaking change. See docs/02_TARGET_ARCHITECTURE.md.
  • A free, post-hoc Tier 4 extraction pipeline (skills/*/evals/{tier3_trial_extractor,aggregate_tier4,generate_benchmark}.py) that mines already-completed Tier 3 trial data on disk to compute domain-specific correctness metrics and hard gates (regression pass rate, authorization/permission-denial scanning) without any new API calls — built because SkillEvaluator's own Tier 4 is bring-your-own-grader, and because re-running live evaluation just to compute a metric you already have the raw data for is exactly the kind of unnecessary spend this project's real credit-exhaustion experience (§2) made a priority to avoid.

8. References

What agent skills are, and the open standard

MCP vs. Skills — a distinction worth understanding before designing a skill

NVIDIA SkillEvaluator

How other companies approach skills, tools, and agent extensibility

Tutorials, courses, and a podcast episode

This project's own evidence, referenced throughout this document

Topic Document
Full architecture and the adapter/certification boundary docs/02_TARGET_ARCHITECTURE.md
NVIDIA version pinning and upgrade process docs/13_NVIDIA_EVALUATOR_UPGRADE_POLICY.md
Milestone 4 — first real certification attempt, all four bugs, in detail docs/MILESTONE_4_PERFORMANCE_ATTRIBUTION.md
Milestone 5 — second certified skill, the reconciliation policy-gap finding docs/10_DEVELOPMENT_ROADMAP_AND_PROGRESS.md, skills/portfolio-overview/BENCHMARK.md
Milestone 6 — deliberate defect demonstration, in detail docs/MILESTONE_6_DELIBERATE_DEFECTS.md
Full milestone status and the 2026-08-30 scope decision docs/10_DEVELOPMENT_ROADMAP_AND_PROGRESS.md
Standalone cold-read executive summary docs/14_EXECUTIVE_SUMMARY_AND_WALKTHROUGH.md

Module 1: What SkillEvaluator Is, and Why

Based on the reference §1.

The core idea

NVIDIA SkillEvaluator is an open-source, multi-tier framework for evaluating AI agent artifacts — starting with agent skills — through deterministic quality gates, semantic overlap detection, synthetic evaluation-dataset generation, and live agent evaluation that measures how a skill actually changes agent behavior.

Sources: NVIDIA docs, GitHub, NVIDIA technical blog.

Support level

Experimental — community-supported on a best-effort basis through GitHub Issues, no SLA. This matters: this repository found and worked around real bugs in it (see Module 8) — exactly what "Experimental" should lead you to expect.

The four tiers, one line each

Tier Question it answers Cost
Tier 1 Is the skill well-formed and safe? Free, offline, seconds
Tier 2 Is it a near-duplicate of something that already exists? Needs an embedding provider
Tier 3 Does having the skill change agent behavior for the better, and by how much ("Skill Lift")? Needs Docker, agent + judge credentials, real time/money
Tier 4 Is the agent's answer, with the skill, actually correct for this domain? Bring-your-own grader; SkillEvaluator supplies the harness, not domain judgment

What it explicitly does NOT do

It does not build skills for you, does not run your production agent, does not manage credentials or a data pipeline, and does not by itself decide certification policy — that is what this repository's own framework/certification/engine.py and policies/certification.yaml add on top. See docs/02_TARGET_ARCHITECTURE.md.

Check your understanding

5 questions on this module. Get 4 right to complete it.

  1. Which tier answers "does having the skill change agent behavior for the better, and by how much"?
  2. What support level does NVIDIA SkillEvaluator have?
  3. Which of these does SkillEvaluator itself do?
  4. Which tier is free, offline, and finishes in seconds?
  5. What does Tier 4 provide out of the box?

Module 2: Tier 1 — Construction & Security

Based on the reference §3-5.

What Tier 1 checks

  • Schema validity of SKILL.md frontmatter and body sections.
  • A quality score across advisory dimensions (discoverability language, recommended body sections, author format, etc.).
  • Security scanning (Bandit/pip-audit for code, gitleaks for secrets, SkillSpector for skill-specific risks) — installed as separate external tools to avoid dependency conflicts.

Commands

skillevaluator quality-check skills/my-skill   # scoring only
skillevaluator validate skills/my-skill --no-dedup   # Tier 1 only, skips Tier 2

Both run in seconds, need no API key, no Docker, no sandbox — safe to run constantly, including on every PR.

Why it's worth adopting "almost unconditionally"

Free and catches real, fixable structural problems. This repository's own Tier 1 reports have surfaced real findings across nearly every skill drafted (see Module 7 for exact layout requirements that trip this check).

A real CI example

.github/workflows/skills-quality.yml's tier1 job installs the pinned evaluator plus SkillSpector and gitleaks, diffs the PR against its base SHA to find changed skills/* directories, and runs skillevaluator validate <skill> --no-dedup on each — non-zero exit blocks the PR. First real run (PR #4) passed in 54 seconds, no API keys required.

Check your understanding

5 questions on this module. Get 4 right to complete it.

  1. Which of these is not part of a Tier 1 check?
  2. Which command runs Tier 1 only?
  3. What does a Tier 1 run need beyond the install?
  4. Why are gitleaks and SkillSpector installed as separate tools?
  5. In this repository's CI, what happens when Tier 1 validation fails on a changed skill?

Module 3: Tier 2 — Semantic Similarity / Deduplication

Based on the reference §3, §6.

What it does

Embeds a candidate skill's description/content and compares it against a central catalog to catch near-duplicate skills before merge (e.g. two teams independently building "explain portfolio performance" under different names).

The gotcha: embedding provider

Tier 2 needs a working embedding provider, and Anthropic does not provide one. You need SKILL_EVAL_EMBEDDING_PROVIDER=openai (or nv_build / an OpenAI-compatible endpoint) plus that provider's key — even if Tier 3's agent is entirely Claude-based. If this is missing, Tier 2 silently reports "skipped" rather than erroring — a real trap this project actually hit.

Command

SKILL_EVAL_EMBEDDING_PROVIDER=openai OPENAI_API_KEY=... \
  skillevaluator similarity-check skills/ --type skill

This repository's governance layer on top

Tier 2 alone gives a similarity score. This repository's own framework/certification/check_similarity.py and policies/similarity.yaml turn that score into governance actions — e.g. EXACT_DUPLICATE blocks a PR, HIGH_SIMILARITY routes to an advisory architecture_review with an assigned owner/SLA, rather than blocking outright. This is this repository's own addition, not a SkillEvaluator feature.

Check your understanding

5 questions on this module. Get 4 right to complete it.

  1. Why does Tier 2 need an OpenAI (or compatible) key even when the Tier 3 agent is entirely Claude?
  2. What happens if Tier 2's embedding provider is missing?
  3. Which environment variable selects the embedding provider?
  4. Under this repository's similarity policy, what does a HIGH_SIMILARITY result do?
  5. Where do the governance actions on top of a similarity score come from?

Module 4: Tier 3 — Live-Agent Evaluation & Skill Lift

Based on the reference §2-4 and docs/MILESTONE_4_PERFORMANCE_ATTRIBUTION.md.

The core mechanic

Run the same eval cases twice — once with the skill loaded, once without — against a live agent and a live judge model, in a sandbox (Docker/Harbor). Skill Lift = with-skill score minus without-skill score. This is the central number Tier 3 produces.

Real numbers from this repository

Skill Trials Skill Lift
Performance Attribution 150 (75/75 both arms) +0.1253
Portfolio Overview 150 (75/75 both arms) +0.1316

A full 25-case × 3-attempt × 2-arm matrix against claude-sonnet-5 as both agent and judge took 30-55 minutes and roughly $15-45 per skill. A cheap Haiku-tier "quick pass" ran in ~12 minutes for ~$1-3 — useful for validating wiring, not a substitute for certification-grade evidence (a weaker agent compresses Skill Lift toward zero on both arms).

Requirements easy to forget

  • Docker (or another sandbox), the agent's own credentials.
  • A separate judge-model credential and SKILL_EVAL_JUDGE_MODEL override — otherwise the judge silently falls back to the agent's own model, coupling judge cost to agent cost for no reason.
  • --copy-repo — without it, the sandbox can't find the repo's own tool bridge (a real operator-error bug this project hit).

Command shape

SKILL_EVAL_LLM_PROVIDER=anthropic ANTHROPIC_API_KEY=... \
SKILL_EVAL_JUDGE_MODEL=claude-sonnet-5 \
  skillevaluator validate skills/my-skill \
  --agent-eval -a claude-code --agent-model claude-code=claude-sonnet-5 \
  --env-mode docker --copy-repo \
  --results-dir reports/my-skill-tier3 -r cli,json,markdown

Metrics beyond Skill Lift

pass_at_k (reliability across repeated attempts) and per-dimension dimensions_with_skill scores (e.g. discoverability, tool-use correctness) also appear in the raw report — see Module 8 for the report shape.

Check your understanding

5 questions on this module. Get 4 right to complete it.

  1. What is Skill Lift?
  2. Roughly what did a full 150-trial Tier 3 matrix cost per skill in this repository's runs?
  3. What happens if SKILL_EVAL_JUDGE_MODEL is not set?
  4. What does the --copy-repo flag fix?
  5. Why is a cheap Haiku-tier quick pass not certification evidence?

Module 5: Tier 4 — Domain Graders

Based on the reference §7 and Milestone 8 (docs/10_DEVELOPMENT_ROADMAP_AND_PROGRESS.md).

What Tier 4 is

SkillEvaluator supplies the harness (a Tier 3 trajectory to grade against), not the domain judgment. You write evals/grader.py per skill; real domain logic should live in a shared, reusable module, not duplicated per skill.

This repository's pattern

from graders.finance.my_skill import grade as grade_evidence

def grade(case, trajectory=None, expected=None):
    evidence = trajectory or expected or {}
    result = grade_evidence(evidence)
    return {"skill": "my-skill", "metrics": result["metrics"],
            "score": result["score"], "notes": []}

Composed from generic building blocks in graders/finance/*.py, reused across all 12+ PM skills — reusability across skills was made an explicit exit criterion in Milestone 8.

A free, post-hoc extraction pipeline

This repository built skills/*/evals/{tier3_trial_extractor,aggregate_tier4, generate_benchmark}.py — a pipeline that mines already-completed Tier 3 trial data on disk to compute domain-specific correctness metrics and hard gates (e.g. regression pass rate, authorization/permission-denial scanning) without any new API calls, because Tier 4 itself is bring-your-own-grader and SkillEvaluator doesn't do this for you.

A real observed quirk to defend against

Directory layouts can differ between runs of the same tool — this project observed two different completed-trial directory structures across otherwise-identical skillevaluator validate --agent-eval invocations. Any custom post-hoc analysis script needs to defend against that, not assume a fixed layout.

Check your understanding

4 questions on this module. Get 4 right to complete it.

  1. What does Tier 4 not supply?
  2. Where should real domain grading logic live, in this repository's pattern?
  3. What does this repository's post-hoc extraction pipeline do?
  4. Two identical --agent-eval runs produced different completed-trial directory layouts. What is the right response?

Module 6: Installing & Running SkillEvaluator

Based on the reference §4 and docs/13_NVIDIA_EVALUATOR_UPGRADE_POLICY.md.

General upstream install

# Prerequisites: uv (or pip), Python 3.12 or 3.13, git; Docker only for Tier 3
uv tool install --python 3.13 "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git"

skillevaluator --version
skillevaluator health-check

Smaller footprints exist: skillevaluator (Tier 1 only), [security] (+ Bandit/pip-audit), [tier2,tier3] (+ dedup and live-agent eval).

External security tools install separately to avoid dependency conflicts:

brew install gitleaks
uv tool install git+https://github.com/NVIDIA/SkillSpector.git

Why pin an exact commit

Installing @main is fine for exploration; for anything you intend to certify against, pin an exact commit. SkillEvaluator's own release notes and this repository's own experience both found real, breaking behavioral differences worth pinning against.

uv venv .venv --python 3.13
uv pip install --python .venv/bin/python \
  'skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git@<pinned-sha>'

This repository also tracks a local patch for a real judge-truncation bug found in its pinned version (patches/skillevaluator-0.2.1-judge-max-tokens.patch) — apply it only if your pinned version predates NVIDIA's own upstream fix.

Per-tier run commands

See Modules 2-4 for the exact Tier 1/2/3 commands and their required environment variables.

Check your understanding

5 questions on this module. Get 4 right to complete it.

  1. Which two commands verify an install?
  2. Why pin an exact SkillEvaluator commit instead of installing @main?
  3. Which install extra adds deduplication and live-agent evaluation?
  4. When should you apply this repository's judge-max-tokens patch?
  5. Which prerequisite is needed only for Tier 3?

Module 7: Skill Package Layout

Based on the reference §5.

Required layout

skills/<skill-name>/
├── SKILL.md              # required
├── skill.yaml            # optional metadata this repo's governance layer reads
├── CHANGELOG.md          # optional
├── evals/
│   ├── config.yml        # harbor/tier3 run parameters, grading mode
│   ├── evals.json        # the eval case dataset
│   └── grader.py         # custom Tier 4 grading hook
└── tests/                # optional local unit tests for the grader

SKILL.md frontmatter Tier 1 actually checks

---
name: my-skill                          # required; must match the directory name
description: One sentence, plus WHEN     # required; missing "when to use" language
                                          # is flagged as a discoverability finding
metadata:
  author: Team Name <team@example.com>   # required for the author_format check
---

A missing metadata: block is a real, observed failure mode — several skills in this repository started without one and failed Tier 1's author_format check until it was added.

A real, observed quirk

By default, Tier 1's schema check flags files like skill.yaml and CHANGELOG.md sitting in the skill root as "unexpected" — it expects the root to contain only SKILL.md plus a small allowed set of subdirectories (agents/, assets/, config/, evals/, references/, scripts/, tests/, tools/). These are non-blocking advisories, not hard failures — move such files into an allowed subdirectory or set SKILL_EVAL_SCHEMA_ALLOWED_DIRS for a clean report.

## Use when, ## Do not use when, ## Procedure, ## Instructions/ ## Usage, ## Examples, prerequisites/limitations/troubleshooting — each an advisory finding if missing, not a hard fail, but each genuinely moves the quality score.

Check your understanding

5 questions on this module. Get 4 right to complete it.

  1. Which file is required in a skill folder?
  2. What must the frontmatter name field match?
  3. A new skill fails the author_format check even though you wrote an author. What is the likely cause?
  4. Tier 1 flags skill.yaml and CHANGELOG.md in the skill root as unexpected. How serious is that?
  5. What happens if SKILL.md has no ## Use when section?

Module 8: Reading Reports, CI/CD, and Real Bugs

Based on the reference §2, §6-7 and framework/adapters/nvidia_skillevaluator.py.

Two different report shapes to know

  • Raw NVIDIA JSON — Tier 3 ("agent-eval") reports carry agents (agent name → dimensions_with_skill, lift, pass_at_k); Tier 1-only reports instead carry quality_summary (per-dimension scores) and a results list including a "Security Scan" validator entry.
  • This repository's normalized contract — produced by framework/adapters/nvidia_skillevaluator.py's parse_nvidia_report, which converts either raw shape into a stable internal schema (generic_metrics, skill_lift, reliability, findings, certification: {status: "UNASSESSED"}). Downstream certification logic never touches vendor-specific fields directly — this is what makes a future evaluator version bump a contained, testable event instead of a library-wide breaking change.

Never conflate the two — if asked "what does the report look like," clarify which one is meant.

CI/CD integration this repository actually runs

.github/workflows/skills-quality.yml: - tier1 — blocks a PR on Tier 1 validation failure across changed skills; also runs a real ownership-enforcement check (framework/certification/check_ownership.py) that fails on the literal domain-owner-required placeholder. No API keys required. - similarity — real as of this repo's own history, applies policies/similarity.yaml governance actions against the central catalog, blocking only on EXACT_DUPLICATE. - tier3-fast, domain-graders — documented placeholders, not fully built; this repository's own recommendation is to keep Tier 3 advisory, not blocking, on every PR (too slow/expensive to gate routine merges) and reserve blocking Tier 3 for an explicit release/certification workflow.

Real bugs this repository found and worked around

  1. An agent-compatibility gap in the evaluator's Tier 3 execution heuristic (a shell tool self-reporting as exec, a string the recognized-tool-name list didn't include).
  2. A judge token-truncation bug (max_tokens=1024, no retry) in the evaluator's own judge code, found by reading its vendored source — patched locally, tracked in patches/.
  3. A missing --copy-repo CLI flag (an operator error, not an evaluator bug) that left the sandbox unable to find the repo's own tool bridge.
  4. API credit exhaustion mid-run — an operating-cost lesson, not a bug.

Lesson: at "Experimental" support level, budget time to read the evaluator's own source when something looks wrong rather than assuming the skill under test is broken.

Check your understanding

5 questions on this module. Get 4 right to complete it.

  1. Which top-level key appears in a raw NVIDIA Tier 3 report?
  2. Why does this repository normalize NVIDIA reports through an adapter?
  3. Which CI jobs block a pull request in this repository?
  4. Which of the four problems this repository hit was an operator error, not an evaluator bug?
  5. At an Experimental support level, what should you do when a result looks wrong?

Troubleshooting scenarios

Each scenario is a problem someone reported. Pick the most likely diagnosis. The modules behind each answer are named in its explanation.

  1. "Tier 2 always says skipped, even though I set ANTHROPIC_API_KEY."
  2. "Our Tier 3 run reports the agent never used any tools, but I watched it call the shell tool repeatedly."
  3. "A brand-new skill fails Tier 1 with an author_format error, and I did write an author."
  4. "Tier 1 flags skill.yaml and CHANGELOG.md as unexpected files."
  5. "Our judge model quietly matches whichever model the agent uses, and we never configured that."
  6. "A Tier 3 sandbox run cannot find our repository's own tool bridge script."
  7. "We want a cheap way to validate Tier 3 wiring without spending $15-45 per skill."
  8. "Our Tier 4 script mines completed Tier 3 trials, but a rerun of the identical command produced a different directory structure."

Final exam

Fifteen questions across all eight modules. Score 80% (12 of 15) or more to pass. Answer every question, then check your answers.

  1. What does Skill Lift measure?
  2. Which tier is free and offline by default?
  3. Why does Tier 2 need an OpenAI (or compatible) key even for an all-Claude Tier 3 run?
  4. What does Tier 4 not supply?
  5. Which environment variable sets a separate judge model?
  6. Which easy-to-forget flag leaves the sandbox unable to find the repository's own tool bridge?
  7. Why pin an exact SkillEvaluator commit rather than install @main?
  8. What happens if Tier 2's embedding provider is missing?
  9. What does framework/adapters/nvidia_skillevaluator.py do?
  10. What support level should you expect from SkillEvaluator?
  11. Which frontmatter field does the author_format check need?
  12. In this repository's CI, which jobs block a pull request?
  13. Roughly what did a full 150-trial Tier 3 matrix cost per skill in this repository's runs?
  14. What bug did this repository patch locally in the pinned evaluator?
  15. Why is a Haiku-tier quick pass not certification evidence?