Skip to main content
Back to timeline
NVIDIA/SkillEvaluatorSource publication:

NVIDIA open-sources SkillEvaluator: agent skills pass a static quality gate first, then a with-and-without skill run comparison

Synopsis

NVIDIA released SkillEvaluator, an open-source command-line tool that checks AI agent skills and measures their effect on task results. A skill is a folder of instructions and supporting files defined by the Agent Skills specification. The tool runs a static quality check first, then compares runs with and without a skill; the example skill text-analyzer scores 86.2/100 (B), exits 0, and the default passing threshold is 70. That score covers static quality only, and publication evidence stays marked incomplete.

Interpretation

An AI agent skill can now pass an automated static quality check before it enters a run-level comparison of its effect. The static quality check covers the skill itself, publication evidence stays marked incomplete at that point, and a skill's effect on task results is left to the run comparison stage. The example skill text-analyzer scores 86.2/100 (B) in the quality check, exits 0, and the default passing threshold is 70, with the report saved as JSON.

The check runs through one command line and needs no API key, scanner, GPU or Docker to complete the static quality stage. The quality check reads a skill directory and writes results in both command-line and JSON report formats. After installing uv and Git, cloning the repository and letting uv provision Python 3.13 and dependencies, the quality check runs on the example skill directory.

A skill's effect on task results requires actually running an agent, and a static score does not replace that stage. The static quality check covers the skill itself, and publication evidence remains marked incomplete at that point. Security assessment needs scanners and agent-impact assessment needs live runs, and both sit in tiers separate from the static quality check.

Perspective

The public material shows a static quality check on the example skill text-analyzer, plus installation, CI setup and tiered evaluation paths; security scanning and agent-run evaluation require additional scanner and sandbox configuration.

A static quality score is not publication evidence, and the material states that publication evidence remains incomplete; security assessment depends on scanners and agent-impact assessment depends on live runs, and a scanner that cannot run marks the result incomplete and exits nonzero.