Skip to main content
Back to timeline
NVIDIA/SkillEvaluatorSource publication:

NVIDIA's SkillEvaluator checks agent skills before merge, and its sample run shows 2 of 6 harnesses failing to load one skill

Synopsis

NVIDIA released SkillEvaluator, a tool that checks AI agent skills and plugins before merge and measures a skill's effect on task results. It puts quality, schema, version, PII, license, Unicode and script checks alongside harness compatibility in one command-line pipeline, then optionally compares runs with and without a skill. A sample output shows 2 of 6 harnesses failing to load the same skill, one because its description exceeded the 1024-unit limit.

AI-generated editorial illustration: NVIDIA / SkillEvaluator

Interpretation

An AI agent skill can now be checked automatically before it is merged into a repository, with the skill folder itself as the object of inspection rather than a failure discovered later during an agent run. The earlier path was to let the agent load the skill and surface problems during a task; SkillEvaluator moves that step to submission time and reports, harness by harness, whether the skill loads. The sample output lists 6 harnesses for one skill: opencode, hermes and openclaw load it, claude-code does not load it because .agents/skills is not scanned, copilot-cli does not load it because the description is 1,028 UTF-16 units, 4 over the 1,024 limit, and codex is degraded because the description was cut to 1,021 characters plus an ellipsis, for 2 of 6 not loaded.

The tool splits checks into three tiers, and the first tier needs no API key, scanner, GPU or Docker, printing a score out of 100 with a grade and the findings behind it. Tier 1 places the harness compatibility check alongside quality, schema, version, PII, license, Unicode and script linting; security and code integrity go to SkillSpector, Gitleaks, Bandit and optional Semgrep, and dependency auditing uses pip-audit. The quality check's passing threshold defaults to 70, and that score is static quality only, with publication evidence still marked incomplete and security and agent impact requiring additional checks; connected to CI, 11 checks run on changed skills in GitHub or GitLab, backed by more than 500 static rules and detection patterns.

Perspective

The demonstrated scope is skill folders organized under the Agent Skills specification, invoked from a terminal, CI or an agent; the sample output is a versioned summary, and full output can include notes and schema diagnostics. Installation targets Python 3.12 to 3.14, and Tier 3 evaluation requires Docker running.

The quality-check score reflects static quality only, publication evidence stays marked incomplete, and security and agent impact need additional checks to be covered. Scores can differ by release, and the sample output is described as a versioned summary.