Breaking

Anthropic Adds Built-In Plugin Evaluation Command to Claude Code, Scoring Extensions Against a No-Plugin Baseline

September 12, 2026 at 11:06 EDT

Software Dev & Coding
AI Agents

Anthropic has added a built-in testing command to Claude Code that lets developers put a number on whether a plugin or skill is actually helping — or just adding clutter. The feature, invoked as claude plugin eval, shipped in Claude Code version 2.1.269 and lets plugin authors run their extensions against realistic prompts, grade the results, and compare them against how Claude performs with no plugin loaded at all.

SEP 2026 · ANTHROPIC

A Report Card for Claude Code Plugins

A new claude plugin eval command scores Claude Code plugins against a no-plugin baseline, showing which extensions earn their place and which ones Claude quietly ignores.

v2.1.269+

Minimum Claude Code version required

3 + 3

Default runs per case: with the plugin, then without

Grader types built in: 4 free, 2 billed

$0.41

Example cost for one six-run case

Same Request, Plugin On Vs. Off

One example case score from the documentation, on a 0-to-1 scale

1.00

WITH plugin

0.33

W/OUT plugin

Δ +0.67

The share of the score the plugin itself is responsible for

How An Eval Run Works

STEP 1

Write test cases: a prompt plus graders

→

STEP 2

Run three times with the plugin loaded

→

STEP 3

Run three times with no plugin at all

→

STEP 4

Score both runs and read the delta

What Developers Welcome

Catches skills that are installed but never actually fire
At least one plugin maintainer adopted it the day it went public

What Still Needs Work

No built-in history of past runs is kept
Every run and judge check is billed at list price

Continue reading

The rest of this article is for AI News Blitz readers. Choose an option below to keep reading.