Checked with Moonshot's own verifier

Summary

Across the six benchmark runs the endpoint served 8,190 requests at 50 concurrent for about 7 hours, with 2 errors (0.02%), 1 truncation, and no manual intervention. Aggregate throughput on the official-config MMMU run was about 1,600 tok/s (≈1,350 tok/s of output).

Setup

MMMU Pro Vision: within noise of the reference

1,730 college-level questions across 29 subjects, one or more images each, 10 options. Reference 0.82.

The 0.8-point gap is smaller than one standard error, so it cannot be distinguished from the reference. It sits in the same band as the hosts Moonshot lists in the same table (Nebius 0.814, DigitalOcean 0.816, Modal 0.817, Inferact 0.818; Baseten 0.804). The official-config run produced 8.57 M output tokens, about 4,950 tokens of reasoning and answer per question, and every question finished with stop.

By subject (official config; green = above the 81.2% mean)

Agriculture, diagnostics and music are the weakest subjects for every model on the public MMMU Pro leaderboard, not specific to this deployment.

OCRBench: 94% of the reference, and where the gap is

1,000 images in 10 categories. Reference 0.89.

By category (official config; green = above the 83.4% overall)

The six categories that cover receipts, forms, screenshots and documents (75% of the set) score 87.5–98%. The gap to 0.89 comes from the small subsets. All three modes side by side:

Two things stand out and we are not going to hide either. Handwritten math expressions score 49% in all three runs: that is consistent, so it is a real weakness of this deployment or of the model at this precision, and it is the item we are investigating first (image preprocessing and maximum pixel budget on the vision path are the leading hypotheses). The other small subsets swing widely between runs (artistic text 58% / 94% / 92%, handwriting 64% / 80% / 86%, non-semantic text 98% / 90% / 48%): with 50 images per subset one image is 2 points, and sampling at temperature 1.0 makes single-run subset numbers noisy. We report the official-config run as the headline number because that is what the reference was measured with, not because it is the best one on every row.

Tool-call JSON Schema: 402 of 408

The suite sends each of 204 JSON Schemas as a tool's parameters, forces the model to call the tool, and validates the returned arguments locally with jsonschema. Every case runs once non-streaming and once streaming.

All six failures are the same defect, and we reproduced it against the public endpoint on 17 September. The three schemas carry a $id whose value contains a fragment ("#user", "https://example.com/schemas/user#/definitions/name", "…#/$defs/name"). Our inference layer validates tool parameters against the JSON Schema 2020-12 metaschema, which forbids non-empty fragments in $id, and rejects the request with HTTP 400 before it reaches the model:

Tool 0 function has invalid 'parameters' schema: '#user' does not match '^[^#]*#?$'
Failed validating 'pattern' in metaschema['allOf'][0]['properties']['$id']

Moonshot's own API accepts these schemas, so this is a strict-validation difference on our side, not a model capability gap. The fix (normalising $id before validation) is scheduled; this page will be updated when the six cases pass. Every other schema type in the suite (nested objects, arrays, enums, const, numeric ranges, $ref/$defs, additionalProperties, anyOf, multi-tool choice) passed in both modes.

Parameter constraints: 18 of 18

The params suite checks that temperature, top_p, presence_penalty, frequency_penalty and n are accepted, ignored or pinned exactly as the reference does under think-mode=opensource. All 18 assertions passed in 77 seconds.

Load, as a by-product

Six full benchmark runs at 50 concurrent requests are a 7-hour soak test of the exact traffic shape agent workloads produce: long reasoning outputs, multi-image inputs, sustained concurrency.

For per-request latency from outside China, see the status page, which probes this endpoint every 10 minutes from a US runner.

What did not pass, in one place

• OCRBench 0.834 vs 0.89. Handwritten math at 49% is consistent and under investigation; artistic text, handwriting and non-semantic text are noisy 50-image subsets.
• 6 tool-call cases rejected with 400 because of strict $id validation. Reproduced 2026-09-17; fix scheduled.
• 2 errors and 1 truncation in 8,190 requests.
• Not run: BEAM (1M-context memory) and DeepSWE (113 coding-agent tasks), the two optional suites most hosts in Moonshot's table have also left as pending.

Reproduce it

Nothing here needs our cooperation. Clone the verifier, set KIMI_BASE_URL=https://microquickjs.com/v1 and KIMI_API_KEY to your key, and run the commands we ran:

# OCRBench, official config
uv run python eval.py ocrbench --model kimi/Kimi-K3 --max-tokens 16384 --thinking --think-mode kimi --stream \
  --max-connections 50 --temperature 1.0 --top-p 0.95 --thinking-effort high

# MMMU Pro Vision, official config
uv run python eval.py mmmu --model kimi/Kimi-K3 --max-tokens 98304 --thinking --think-mode kimi --stream \
  --max-connections 50 --temperature 1.0 --top-p 0.95 --thinking-effort high

# Tool-call JSON Schema and parameter suites
uv run pytest tests/tool_call_json_schema --base-url https://microquickjs.com/v1 --api-key $KIMI_API_KEY --smoke-model Kimi-K3 --think-mode opensource -v
uv run pytest tests/params --base-url https://microquickjs.com/v1 --api-key $KIMI_API_KEY --smoke-model Kimi-K3 --think-mode opensource -v

A full run costs roughly 35 M tokens. If you want to run it against us at 50 concurrent, email us for a concurrency bump and test credit; the per-sample inspect-ai logs of our own runs are available to teams on request.

Kimi K3 is a model by Moonshot AI, used under the Kimi K3 License. Kimi Vendor Verifier is Moonshot's project; we have no affiliation with Moonshot beyond serving their released weights. Reference values are Moonshot's published self-evaluation.