Teaching an AI agent
a new job — and proving
it actually learned it.

01 · First principles

What is an "agent skill," really?

Think of onboarding a new hire. You don't re-explain your whole business every time — you hand them a one-page SOP: "when a client asks X, do Y, cite Z, never do W." They read it once, keep it handy, and follow it when the situation matches.

A skill is that SOP, written for an AI agent instead of a person. Concretely, it's a folder with a SKILL.md file — a short description of when to use it, plus step-by-step instructions the agent loads only when a task actually matches. This "load only when relevant" trick is called progressive disclosure, and it's now an open standard (agentskills.io) that Anthropic, OpenAI, GitHub, and others have all adopted.

The problem this repo tackles isn't "how do I write one skill" — it's "how do I know if the fifty skills my org has accumulated are any good, still needed, and not quietly wrong."

02 · Why this exists

Skill libraries grow faster than anyone can review them.

Two teams independently build "explain portfolio performance" under different names. A skill reads well but never actually changes what the agent does. A finance skill gives an answer that sounds right but reconciles to the wrong number. Nobody notices until it's in front of a client.

Generic testing doesn't catch most of this. A skill can pass every code review and still be a duplicate, still add zero real value, still be confidently wrong about a number a domain expert would catch instantly. You need three different kinds of checking — structural, behavioral, and domain-correctness — and a way to run all three without it costing a fortune every time.

03 · Scope

A governance layer, not a skills platform.

This project does not try to be a generic, build-your-own-skill platform. It leans entirely on NVIDIA SkillEvaluator to do the hard evaluation engineering, and adds exactly four things a real organization needs on top:

• Ownership enforcement — no named business owner and domain reviewer, no certification.
• Duplicate detection — every new skill checked against a central catalog before merge.
• Deterministic domain grading — finance-specific correctness checks generic evaluation can't see.
• Risk-tiered certification — an informational skill and a decision-support skill aren't held to the same bar.

And — reaffirmed partway through this project — the deeper goal isn't even "ship a certified 12-skill catalog." It's building real, hands-on fluency with how this class of evaluation tooling actually behaves, so that fluency transfers to whatever skill library you touch next.

04 · The mechanism

Four tiers, each answering a different question.

05 · Where it fits

It's CI/CD and code review — for instructions, not code.

Map it onto a development lifecycle you already know:

This project's own CI (Milestone 9) does exactly the cheap end of this today — a real GitHub Actions job blocks a PR when a changed skill fails Tier 1. First live run, against a real PR: 54 seconds, no API keys.

06 · What it's actually for

The goals this helps you hit in the agent dev lifecycle.

• Replace vibes with numbers. "Skill Lift" is a measured with-vs-without delta, not a demo.
• Catch domain-wrong answers generic eval can't see. Reconciliation errors, stale dates, missing coverage.
• Stop duplicate skill sprawl before it compounds. One catalog, checked automatically.
• Right-size rigor to risk, instead of one flat bar that's too strict for some skills and too loose for others.
• Keep "certified" meaningful even as the underlying vendor tool changes underneath you.
• Leave an audit trail — a benchmark tied to an exact skill version, dataset, agent, and model, not a verbal assurance.

07 · What we actually tested

Two skills, taken all the way through — for real.

Not demoed. Live agent, live judge model, real Docker sandbox, real certification policy. Both came back an honest fail, for precisely diagnosed reasons — which is the point: a governance layer that always says pass isn't doing anything.

Both fails trace to the same discoverability metric-scoping artifact — independently corroborated by SkillEvaluator's own report on the second run. The second run also surfaced a genuine certification-policy gap (one metric silently penalizes every skill except the first one it was written for) — found only by actually running the real pipeline twice.

08 · What "using it for real" surfaced

Four real bugs — mostly not ours.

09 · The honest maturity read

What's actually feasible, based on the numbers above.

10 · Closing

Key takeaways.

"A governance layer that always says pass isn't doing anything."