Agentic Commerce Benchmark · v4

A year ago, “can an AI build a shop?” was a fair question. Today it’s the wrong one. Every frontier model can emit a plausible product page. What separates them now is whether they can ship working software across the messy reality of commerce: a polished storefront, an on-device AI that actually answers, a real ERP integration, a model that learns from returns, and an agent that completes a purchase without going off the rails.

So we stopped scoring one beautiful artifact and started measuring a capability profile. The v4 suite asks each model to do five very different jobs, and the result is a far more honest picture of where these systems are strong, where they bluff, and — increasingly — where open weights have caught up.

From one exam to five

Each task targets a different capability axis. A model can no longer win by being a great front-end stylist; it has to prove it can also reason about data contracts, run inference in the browser, learn online, and plan multi-step actions against a live API.

A · Premium Storefront — Frontend & Commerce Craft

Build the most convincing, modern product experience for a single hero product: 3D configurator, variants, live stock, multi-step checkout, an in-page AI assistant, accessibility and motion. The one task that still leans on a design-quality judge.

B · Client-side WebGPU Product Q&A — Applied On-device AI

Ship a page that loads a small language model in the browser and answers grounded questions about a product catalog — no server round-trips. Pinned weights, cached locally, with a WASM/CPU fallback when no GPU is present.

C · Microsoft Dynamics 365 Order Integration — Integration Engineering

Transfer shop orders into Dynamics 365 through a mock D365 API: field mapping, auth, idempotency, retries and error handling. Scored against a gold mapping and the API’s real server-side state — not against how the code looks.

D · In-browser Fashion Fit Estimation — On-device ML & Continual Learning

From a body photo, estimate the clothing metrics that matter for fit, run entirely client-side, and learn from returns online so predictions improve as feedback arrives. The hardest task in the suite.

E · Autonomous Buying Agent — Agentic Planning & Tool Use

A runnable agent that, given a budget and intent, plans and completes a purchase against a mock store API — searching, adding to cart, and checking out — while respecting constraints and recovering from tool errors.

How v4 is scored

Every task ships a frozen prompt and a standalone harness that runs the deliverable and produces an objective autoScore by checking real effects: did the cart actually persist, did the order land in the mock ERP’s state with the correct fields, did the agent reach a valid checkout, did the on-device model return grounded answers. An LLM judge still grades the things only humans care about — craft, clarity, UX — but for tasks B through E the objective share now dominates.

Three principles keep the leaderboard honest:

  • Verify effects, not vibes. Harnesses inspect server-side truth state (the mock D365 store, the mock shop API) rather than grepping the source for the right-looking strings. Stubs don’t score.
  • Graded partial credit. Research-adjacent tasks award tiers — full · high · mid · low · 0 — so a half-working answer is distinguishable from a no-op, and the board keeps its resolution.
  • Reproducibility with real runtimes. Tasks that need external models or libraries may fetch them, but only from pinned URLs with SHA-256 hashes, cached locally behind an allowlist.

Two honest constraints up front: tasks B and D were evaluated on the WASM/CPU fallback tier (no GPU in the runner), so on-device absolute numbers reflect that path; and the storefront task keeps its v4 scale where craft bonuses can push a score slightly above 100.

The leaderboard

The global index is the mean of a model’s five task scores, each capped at 100. Here is where the nine runs landed:

A wrinkle worth being transparent about: Claude Opus 4.8 ran as two partial entries — a storefront-only “manual” run (100, one task) and a “high” run covering the four new tasks (99, four tasks). GLM 5.2 is the only model that completed all five tasks itself, posting a 98.6 — and it is the only entry to reach tier full on every single task. Below it, Cursor Composer 2.5 (94.8) leads the rest of the field, while Kimi K2.5 (40.1) trails after shipping nothing usable on two tasks.

Where each model actually shines

The single index hides the interesting story. Read across the rows and the capability profiles jump out — strong front-end stylists that fall apart on online learning, integration specialists that can’t plan, and one open model that is simply consistent everywhere.

The on-device cliff (Task D)

Fashion fit with online learning split the field cleanly. Only four models cleared it at full — Opus, GLM 5.2, Composer and GPT-5.5 (97–99). Grok, Sonnet, Gemini and Kimi all dropped to low (42–54): they produced a fit estimator but never wired up the hard part — learning from returns. Continual learning in the browser remains a genuine differentiator.

The agent that some models simply failed (Task E)

Six of nine models completed the autonomous buying agent at full. But GPT-5.5 and Kimi K2.5 scored a flat 0 — they did not ship an agent that could complete a checkout against the mock store. A reminder that “agentic” is still a capability you have to verify by running it, not by reading the plan.

Integration is the great equalizer (Task C)

Dynamics 365 was the most uniformly solved task: seven models landed full (94–99), with only Kimi stuck at mid. Mapping orders into an ERP is, it turns out, exactly the kind of well-specified contract today’s models are good at.

The headline: GLM 5.2 is open weights at the frontier

For years the assumption was that the very best agentic and coding work would stay locked inside proprietary frontier models. The v4 suite is the clearest evidence yet that the gap has closed. GLM 5.2 — open weights — matches Claude Opus 4.8 across the board, and it does so as the only model that finished every task at the top tier.

GLM 5.2 · the open-weights standout

Why this matters beyond bragging rights: for merchants and platforms, an open-weights model that performs at this level changes the build-vs-buy maths entirely. It can be self-hosted, fine-tuned, audited, and run inside your own data boundary — no per-token dependency on a single vendor, no data leaving your perimeter. When the open option is no longer a compromise on capability, “sovereign by default” stops being a trade-off and becomes the obvious starting point.

Open weights is not automatically a win, though: the other open model in the suite, Kimi K2.5, finished last at 40.1 with two zero-scoring tasks. The story here is specifically GLM 5.2, not open models in general.

What we took away

One number can’t describe a model anymore. The same system can top the storefront beauty contest and faceplant on online learning. Capability profiles, not single scores, are how you actually pick a model for a job.

“Agentic” is still earned at runtime. Two strong models shipped zero working buying agents. The only way to know if an agent works is to run it against a live API and check the resulting state.

On-device AI is the new frontier. Browser-side inference (Task B) and continual learning from feedback (Task D) separated the field more than anything else. This is where the next year of differentiation will happen.

Open weights arrived. GLM 5.2 isn’t “good for an open model” — it is competitive with the best proprietary system in the suite, and more consistent across tasks than any of them.

The full suite — every prompt, harness, rubric, model deliverable, screenshot and live preview — is published. Browse the leaderboard, open any model’s run, and judge the artifacts yourself.

→ Explore the v4 benchmark and live model deliverables

#AgenticCommerce #LLM #OpenSource #Ecommerce #Shopware #GLM