Skip to main content
Back to timeline
NVIDIA/SkillEvaluatorSource publication:

NVIDIA releases SkillEvaluator: static quality checks for agent skills, then a with-and-without skill comparison

Synopsis

NVIDIA has released SkillEvaluator, a tool that checks the quality of AI agent skills and measures their effect on task results. It runs quality checks first, then compares runs with and without a skill; a skill is a folder of instructions and supporting files defined by the Agent Skills specification. The bundled text-analyzer example scores 88.8/100 (B) on the static quality check against a default passing threshold of 70, while publication evidence remains marked INCOMPLETE.

AI-generated editorial illustration: NVIDIA / SkillEvaluator

Interpretation

AI agent skills now have a dedicated checker: SkillEvaluator first runs quality checks on a skill, then measures the skill's effect on task results by comparing runs with and without it. Skills are folders of instructions and supporting files packaged under the Agent Skills specification, and SkillEvaluator is part of the NVIDIA Verified Skills pipeline, putting quality checks and the with-and-without comparison behind one command line. The bundled text-analyzer example scores 88.8/100 (B) on the static quality check, exits 0, and saves results to reports/first-quality/skillevaluator-quality.json, with a default passing threshold of 70.

The tool organizes checks into tiers, each independent, with the matching extra added to the command when needed. Tier 1 fetches pinned, verified SkillSpector and Gitleaks releases when missing, with Semgrep optional; Tier 2 reports whether inter-skill comparison ran; Tier 3 autopilot reuses sources or creates one skill-grounded case with limited coverage. The example's static check needs no API key, scanner, GPU or Docker; rubric-eval sends skill documentation to a model and saves feedback without an agent runtime or Docker, while security needs scanners and agent impact needs live evaluation.

Perspective

The public material shows the static quality-check result for the bundled text-analyzer example and the command-line usage of the tiered checks; security scanning needs scanners and agent impact needs live evaluation, and publication evidence remains marked INCOMPLETE.

The material states that static checks are not publication evidence, which remains INCOMPLETE; completed evaluations can still fail rubric thresholds; Tier 3 autopilot cases have limited coverage.