QuickSilver Pro sells a dozen image-generation models behind one OpenAI-compatible endpoint, /v1/images/generations. They are billed per image, and the price per image ranges widely: $0.003 for FLUX.1 Schnell, $0.07 for Seedream 5.0 Pro. The question is what the extra money buys.

So on 2026-10-20 we gave 11 of them the same three prompts and looked at every image.

Nano Banana Pro is also in the catalog; it is not part of this test.

How we tested

Three prompts, each testing something different:

1. Text on packaging: Product photo of a matte black coffee bag standing on a wooden table. The front label reads "NORTHWIND ROASTERS" in large white capital letters, and below it, in smaller text, "Ethiopia Yirgacheffe" and "250 g". Soft studio lighting, shallow depth of field.
2. Exact composition: Flat vector illustration on a plain white background: exactly three red apples on the left side of the image, and exactly one blue cup on the right side. Nothing else in the image.
3. Photoreal scene: Photorealistic photograph of a small wooden cabin beside a mountain lake at dawn, mist over the water, a red canoe tied to a wooden dock in the foreground, pine trees, no people.

Every request used the same parameters: size="1024x1024", n=1, nothing else. Each model ran each prompt once. We then ran the composition prompt a second time on every model, one request at a time, to get a cleaner latency figure and a second sample. We judged every image by eye: for the label, whether all three lines are spelled exactly right; for the illustration, whether there are exactly three apples on the left and one cup on the right; for the scene, whether every element in the prompt is there and physically plausible.

Results

Latency is the median of four requests per model, measured from one client. The second composition run was not saved for Seedream 4, Seedream 5.0 Pro and Bria FIBO 1.5, so those have one sample. All 33 requests in the main run succeeded.

What stands out:

• Text is mostly solved, except at the cheap end. Eight of the eleven models wrote all three lines of the label correctly. FLUX.1 Schnell and Bria FIBO 1.5 got the brand name right and garbled or misspelled the smaller lines; SDXL Turbo produced lettering that is not words.
• Counting is where models differ. Only the three GPT Image models drew three apples and one cup in both runs. Qwen Image 3.0, FLUX.2 Pro and FLUX.1 Schnell got it right once and wrong once, so a single good image does not mean the model follows counts reliably. FLUX.2 Klein drew two apples both times.
• The photoreal scene was easy for nearly everyone. Eight of eleven images had every element in a plausible arrangement. The failures were physical: a canoe merged into the dock, a canoe lying on top of it.
• Latency spans nearly two orders of magnitude. FLUX.1 Schnell answered in under a second; Seedream 5.0 Pro took 28 to 51 seconds per image.

Text on packaging

GPT Image 2.5 Flare	GPT Image 2.5 Sunburst	Qwen Image 3.0
FLUX.2 Klein	Bria FIBO 1.5	SDXL Turbo

Three apples on the left, one cup on the right

GPT Image 2.5 Sunburst	Seedream 5.0 Pro	Bria FIBO 1.5
Qwen Image 3.0 (first run)	FLUX.2 Pro (first run)	SDXL Turbo

Cabin, lake and red canoe

GPT Image 2.5 Flare	Seedream 5.0 Pro	FLUX.2 Pro
Qwen Image 3.0	FLUX.2 Klein	SDXL Turbo

What the API actually returns

The models do not all answer a request the same way, and code that works with one can break on another:

• Size is not always what you ask for. We asked for 1024x1024 every time. Seedream 4 returned 1280x1024 every time we checked. GPT Image 2 picks its own aspect ratio at about 1.57 megapixels: 1774x887 for the apples (twice), 1536x1024 for the lake, 1312x1199 for the coffee bag. Every other model returned 1024x1024.
• Some models return a link instead of the image. GPT Image, Qwen Image, FLUX and SDXL Turbo put the image in b64_json. Seedream 4, Seedream 5.0 Pro and Bria FIBO 1.5 return a download URL in url instead, and still did when we passed response_format="b64_json". We did not test how long those links stay valid, so download the file straight away.
• The format differs too. FLUX, SDXL Turbo and the Seedream models returned JPEG; GPT Image, Qwen Image and Bria returned PNG. Check the file's first bytes rather than assuming an extension.

Code that handles both response shapes:

import base64
import urllib.request

from openai import OpenAI

client = OpenAI(
    base_url="https://api.quicksilverpro.io/v1",
    api_key="YOUR_QSP_KEY",
)

result = client.images.generate(
    model="gpt-image-2.5-flare",
    prompt="Flat vector illustration: three red apples on the left, one blue cup on the right, plain white background.",
    size="1024x1024",
    n=1,
)

image = result.data[0]
if image.b64_json:
    data = base64.b64decode(image.b64_json)
else:  # Seedream and Bria return a download link instead
    data = urllib.request.urlopen(image.url).read()

with open("apples.png", "wb") as f:
    f.write(data)

Which one to use

• GPT Image 2.5 Flare if the image has to match the prompt: correct text and correct counts in every run, at $0.018 per image and about 11 seconds. Sunburst gave the same accuracy here, more slowly, and added a line of text we did not ask for; GPT Image 2 is cheaper at $0.01 but slower and does not keep the size you request.
• Qwen Image 3.0 or FLUX.2 Pro for good text and photoreal scenes in five to nine seconds, if you can retry when a count comes out wrong.
• FLUX.2 Klein or FLUX.1 Schnell when speed and volume matter more than precision: drafts, thumbnails, many variations to choose from. Klein even got the label text right, in about a second.
• Seedream 5.0 Pro produced accurate, natural-looking images, but at $0.07 and 28 to 51 seconds per image it is the most expensive and slowest model in this test.

The whole test, about 50 images including reruns and checks, cost about $1.45 at QuickSilver Pro prices on 2026-10-20.

Limits of this test

One or two images per model per prompt is a very small sample: the composition prompt showed that the same model can get it right on one run and wrong on the next, so treat every cell in the table as one observation, not a rate. Quality was judged by one person looking at the images, and "looks natural" is a matter of taste. We used default settings only; several of these models accept extra parameters (quality, seeds, prompt rewriting) that we did not try. Latency was measured from one client on one day, and in the first run all eleven models were working at the same time. Run your own prompts before choosing: every image request appears in your dashboard with its cost.

For setup in other languages and SDKs, see the quickstart and the model list.

Kimi K3.1 Pro Highspeed is the same model as Kimi K3.1 Pro, served on Moonshot's high-speed endpoint. On QuickSilver Pro it costs $4.00 / $20.00 per million tokens, twice the $2.00 / $10.00 of the standard version. The usual pitch is "about three times faster". We wanted to know how much faster it is end to end, and when the extra money buys anything.

We tested both on 2026-11-02 through the QuickSilver Pro API, in two ways: single API requests of three shapes, and a coding agent working on six tasks with hidden tests. Short answer:

• Generation speed: Highspeed generated text 2 to 3 times faster.
• Time to first token: about the same for both models. Highspeed does not start answering any sooner.
• End to end: a short answer arrived 1.7 times faster and a long code answer 2 times faster. A question about a 52,000-token log came back only 1.25 times faster.
• Coding agent: every task finished 2.5 times faster on average, for 1.5 times the cost. Both versions passed every hidden test.

Test 1: single API requests

How we tested

• Requests: streamed /v1/chat/completions calls with default settings, sent by a small Python script. Each model got three request types:
  • Short answer: "In three sentences, explain what a database index is and when it hurts write performance." About 110 input tokens.
  • Long code answer: an LRU cache module with per-entry TTL, thread safety and size-based eviction, plus a pytest file. About 150 input tokens; both models wrote about 11,700 output tokens, of which about 9,400 were reasoning.
  • Long input: a 52,614-token service log with one line naming the root cause of an incident. The question asked the model to quote that line and explain it.
• Runs: 8 per model and request type, 48 requests in total. The two models alternated going first, so both saw the same network and server conditions.
• No cache: every request started with a unique tag, so no request was served from the prompt cache.
• Measured: time to the first token (reasoning or answer), total time, output tokens per second after the first token, and the cost QuickSilver Pro returned in each response's usage block.

Results

Medians of 8 runs per model and request type.

Both models found the root-cause line in all 16 long-input runs and quoted it exactly.

What the numbers say

The speedup is in generation, not in getting started. Once text started flowing, Highspeed produced 2 to 3 times as many tokens per second. Time to first token was the same, or slightly longer on Highspeed: 2.8 seconds against 2.1 for the short answer. On the long log both models spent about 7 seconds reading the input before the first token.

So the end-to-end gain depends on the shape of the request. Total time is the wait for the first token plus the generation time. The more of a request is generation, the closer it gets to the full speedup:

• Short answer: 1.7 times faster. Waiting for the first token is a big share of a 10-second request.
• Long code answer: 2 times faster, or 2 minutes 51 seconds saved per answer.
• 52K-token log: 1.25 times faster. Most of the time goes to reading the input, so Highspeed saved about 2.5 seconds for twice the cost.

Twice the price meant twice the cost per request. Both models wrote about the same number of tokens for the same request, so each Highspeed request cost almost exactly double.

Kimi reasons a lot by default. Between 60% and 80% of the output tokens were reasoning, depending on the request type. The long code answer varied from 6,200 to 15,300 output tokens across runs of the same prompt, so the cost of a single request can swing by a factor of two either way.

Test 2: a coding agent on six tasks

How we tested

• Agent: OpenCode 1.19.4, non-interactive (opencode run --auto), with QuickSilver Pro added as an OpenAI-compatible provider. The only thing that changed between runs was -m quicksilverpro/kimi-k3.1-pro or -m quicksilverpro/kimi-k3.1-pro-highspeed.
• Agent settings: a clean OpenCode config with no MCP servers, plugins or extra skills, and OpenCode's defaults otherwise. Both models got the same system prompt and tools.
• Tasks: the six tasks from our Gemini 4 Argon vs Claude Opus 5.5 test. Each is a throwaway Python repository with a one-paragraph prompt, a spec or bug report, one visible test, and a separate hidden test suite that we ran after the agent finished:
  1. Two-file bug fix in a shop module. 18 hidden tests.
  2. A duration parser implemented from a page of rules. 84 hidden tests.
  3. A refactor that must not change behaviour. 317 hidden tests.
  4. Sign-up validation with interacting rules. 84 hidden tests.
  5. A config-format parser with references and cycle detection. 57 hidden tests.
  6. A Mustache template renderer, graded against the official Mustache spec suite. 136 hidden tests.
• Runs: each task twice per model, 24 runs in total. The two models ran side by side.
• Cost: the per-request cost that QuickSilver Pro returned, summed over every request in a run.

Results

Both models passed every hidden test in every run. Averages over two runs per model and task:

All twelve runs per model together took 82 minutes on Kimi K3.1 Pro and 33 minutes on Highspeed.

What the numbers say

In an agent, Highspeed cost 1.5 times as much, not 2. About 90% of each run's input tokens were cache reads. An agent sends the whole conversation again on every request, and the repeated part comes from the prompt cache. Cache reads cost $0.20 per million tokens on both versions, so only the new input and the output pay the doubled price.

Saving four minutes cost about eleven cents. On average a task took 4 minutes less on Highspeed and cost $0.107 more.

The Mustache gap is partly luck. In one of its two Mustache runs, Kimi K3.1 Pro started a sub-agent and made 41 requests, against 11 to 18 in the other Mustache runs. That run took 18 minutes and cost $0.65. Without it, the Mustache times were 7 minutes against 4. All four Mustache runs downloaded the official spec tests and ran them before finishing. Our hidden tests for that task are the same files, so the Mustache scores are not independent.

When Highspeed is worth it

• Someone is waiting: chat, coding assistants, interactive agents. Two to three times faster generation is easy to feel. An agent task that takes 7 minutes takes about 3.
• Long answers: code generation and long-form writing, where generation is most of the time.
• Agents with long sessions: the prompt cache keeps the extra cost to about 1.5 times.

Stay on Kimi K3.1 Pro for:

• Batch jobs and pipelines, where nobody watches the clock. You pay twice as much for time nobody uses.
• Long input, short output: summarising, extraction and search over big documents. On our 52,000-token log, Highspeed saved 2.5 seconds per request at twice the price.

Limits of this test

• One day of measurements (2026-11-02). Serving speed depends on load, so another day or time could give different numbers, for either version.
• Eight runs per API request type and two runs per agent task. The agent times varied by up to 2.5 times between two runs of the same task, mostly because of how much the model chose to reason or check its own work.
• Times include network time from our client to QuickSilver Pro and on to the provider.
• Default settings only. We did not change the reasoning effort or limit reasoning tokens. Either would change speed and cost for both versions.
• One agent, OpenCode, and small Python tasks of 1.5 to 18 minutes each, not a multi-hour change in a large codebase.

Run it yourself

Stream a request to each model and watch the difference:

export QSP_API_KEY=YOUR_QSP_KEY

for model in kimi-k3.1-pro kimi-k3.1-pro-highspeed; do
  echo "== $model"
  time curl -sN https://api.quicksilverpro.io/v1/chat/completions \
    -H "Authorization: Bearer $QSP_API_KEY" \
    -H "Content-Type: application/json" \
    -d "{\"model\": \"$model\", \"stream\": true, \"messages\": [{\"role\": \"user\", \"content\": \"Write a Python LRU cache with TTL.\"}]}" \
    > /dev/null
done

To try both in OpenCode, add QuickSilver Pro as a provider in opencode.json:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "quicksilverpro": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "QuickSilver Pro",
      "options": {
        "baseURL": "https://api.quicksilverpro.io/v1",
        "apiKey": "{env:QSP_API_KEY}"
      },
      "models": {
        "kimi-k3.1-pro": { "name": "Kimi K3.1 Pro" },
        "kimi-k3.1-pro-highspeed": { "name": "Kimi K3.1 Pro Highspeed" }
      }
    }
  }
}

opencode run --auto -m quicksilverpro/kimi-k3.1-pro "Fix the failing test in tests/"
opencode run --auto -m quicksilverpro/kimi-k3.1-pro-highspeed "Fix the failing test in tests/"

Every request shows up in your QuickSilver Pro dashboard with its model, token counts and cost, so you can compare the two versions on your own work. Prices for every model are on the pricing page.

DeepSeek V4.1 Flash and Claude Haiku 5.5 are the two models people reach for when a job runs thousands of times a day: classification, extraction, small code edits, the background calls of an agent. Both are cheap per token. The question is which one is cheaper per finished task, and whether either gets things wrong more often.

So we measured it, on 2026-10-20, through the QuickSilver Pro API.

Update, 2026-11-07: DeepSeek V4.1 Flash's price fell on 2026-11-06, from $0.125 / $0.55 to $0.0558 / $0.2231 per million tokens. Haiku's price did not change. Below, costs are shown at both prices, computed from the same measured tokens; the test itself was not rerun.

How we tested

17 tasks, every one with an answer a script can check:

• 7 Python coding tasks (duration parser, interval merging, LRU cache, strict Roman numerals, an expression evaluator with C-style division, semantic-version ordering, shell-style word splitting). The returned code is run against a fixed set of unit tests; it passes only if every assertion holds.
• 4 extraction tasks (an invoice from an email, attendees from meeting notes, log-line counts, the final state of an order after a chat full of changes). The returned JSON is compared to the expected object field by field.
• 6 reasoning tasks with known numeric or one-word answers (counting arrangements, a pipes-and-tank problem, a weekday two years out, a step-by-step simulation, an elimination puzzle, a digit-sum count).

Each model got the same single user message per task, through the OpenAI-compatible /v1/chat/completions endpoint, non-streaming, max_tokens 8,000, no system prompt and no temperature setting. Each task ran three times, so 51 requests per model. Cost is the billed input and output tokens multiplied by the QuickSilver Pro prices on 2026-10-20. DeepSeek V4.1 Flash thinks before it answers by default, and those reasoning tokens are billed as output, so they are included.

Results

Costs are computed from the measured tokens: the first row at QuickSilver Pro prices on 2026-10-20, the second with DeepSeek V4.1 Flash's price from 2026-11-06.

Haiku's input count is higher than Flash's for the same prompts mainly because of overhead, not the prompts themselves. Every Claude request on QuickSilver Pro carries about 690 extra input tokens: an identity instruction the gateway adds, plus a system block added upstream. A one-line "say hi" to Claude Opus 5.5 reported about 690 input tokens in our earlier test. Those tokens are billed and are included in Haiku's cost above, about $0.02 of its $0.13 per 1,000 tasks. Without them Haiku would be slightly cheaper, so the comparison does not change.

What stands out:

• Same pass rate, very different bill. Both models passed 50 of 51. Haiku did it for about one seventh of the cost at the prices we tested with, and about a third at Flash's new price. It is cheaper per token on both input and output, and it wrote fewer tokens: Flash averaged 1,730 output tokens per task, 88% of them reasoning the reader never sees.
• Haiku answered faster on every task type. Its median was 2.8 seconds against 9.9 for Flash, and its slowest answers came back much sooner: on the expression evaluator, the shell splitter and the elimination puzzle, Flash usually took 30 to 60 seconds, against 2 to 18 seconds for Haiku.
• The misses were different. Flash gave a wrong number on the elimination puzzle once (the other two runs were right). Haiku's one miss was a response that ended in the middle of the function; the other two runs of that task passed.

Turning off Flash's reasoning

DeepSeek V4.1 Flash can skip the thinking step. On QuickSilver Pro you pass reasoning: {"enabled": false} in the request body; we checked that the response then reports zero reasoning tokens. With reasoning off, Flash became three times cheaper and almost as fast as Haiku, and dropped three tasks: the elimination puzzle twice and the duration parser once. At 2026-10-20 prices it was still about 2.5 times Haiku's cost per task; at the 2026-11-06 price the two cost about the same, $0.13 per 1,000 tasks each.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.quicksilverpro.io/v1",
    api_key="YOUR_QSP_KEY",
)

response = client.chat.completions.create(
    model="deepseek-v4.1-flash",
    messages=[{"role": "user", "content": "Extract the invoice number from: ..."}],
    extra_body={"reasoning": {"enabled": False}},
)
print(response.choices[0].message.content)

Which one to use

• Claude Haiku 5.5 for high-volume work with short prompts: extraction, classification, small functions, routing. On this test it matched Flash's accuracy at about a third of the cost of Flash with default settings, and answered faster.
• DeepSeek V4.1 Flash if your prompts are long. Haiku's price rises to $0.125 / $0.625 for any request whose prompt is longer than 200K tokens, while Flash keeps one rate across its 1M-token context. We did not test long prompts, so compare the two prices for your own prompt sizes.
• If you do use Flash for short tasks, try it with reasoning off first. On our tasks it gave up a little accuracy for a much smaller bill and much faster answers. At the 2026-11-06 price it costs about what Haiku does, but Haiku was still faster (2.8 against 3.6 seconds median) and passed two more runs.

Both models are starter models, so the $0.05 free credit a new account gets covers a test like this one: one pass of all 17 tasks on both models cost us about two cents.

Limits of this test

17 tasks and three runs per model is a small sample. One task more or less changes a pass rate by two points, so read 50/51 versus 48/51 as "about the same", not as a ranking. The tasks are short, self-contained and in English; long documents, multi-turn conversations, tool calls and other languages were not tested. Latency was measured from one client on one day and depends on load. Run your own prompts before you switch a production workload: every request appears in your dashboard with its tokens and cost, which makes that comparison easy.

For the full Claude price list, see Claude API pricing in 2026.

On QuickSilver Pro, Claude Opus 5.5 costs $1.00 / $5.00 per million tokens and GPT-6.1 Sol costs $2.00 / $10.00. Both are sold for agentic coding, so for anyone running a coding agent the question is which one gets the work done, how fast, and for how much.

We tested this on 2026-10-27. Both models ran in the same agent, OpenCode, on six small but real coding tasks, and we graded every run with tests the agent never saw. Short answer: both models passed every hidden test in every run. Opus finished a task in 82 seconds on average and Sol in 152. Per task they cost about the same: $0.094 for Opus and $0.098 for Sol.

How we tested

• Agent: OpenCode 1.19.4, non-interactive (opencode run --auto), with QuickSilver Pro added as an OpenAI-compatible provider. The only thing that changed between runs was -m quicksilverpro/claude-opus-5-5 or -m quicksilverpro/gpt-6.1-sol. We could not use Claude Code here. QuickSilver Pro serves GPT-6.1 Sol only on the OpenAI-compatible endpoints (/v1/chat/completions and /v1/responses), and Claude Code needs the Anthropic Messages API, so we used one agent that can drive both models.
• Agent settings: a clean OpenCode config with no MCP servers, plugins or extra skills, and OpenCode's defaults otherwise. OpenCode sent no reasoning-effort setting, so each model ran at its default. OpenCode chooses its system prompt and file-editing tools by model family, and we left that as shipped. With Opus it used a 9,300-character system prompt and edit / write tools. With Sol it used a 5,100-character prompt and an apply_patch tool. Each model therefore ran the way OpenCode normally drives it.
• Tasks: six throwaway Python repositories, each with a one-paragraph prompt, a spec or bug report in the repo, and one visible test. Every task had a separate hidden test suite that we ran after the agent finished. We wrote five of the suites before any run and checked that each passed against our own reference solution and failed against the starting code. The sixth is the official Mustache spec suite.
• Runs: each task twice per model, same inputs, 24 runs in total. Opus and Sol ran each task at the same time, so both saw the same network and server conditions.
• Graded on: the share of hidden tests passing. We also recorded wall-clock time, number of API requests, input / cached / output tokens, and cost.
• Cost: the per-request cost that QuickSilver Pro returns in each response's usage block, summed over every request in a run, at QuickSilver Pro prices on 2026-10-27. We recorded it with a small local pass-through proxy. The same figures appear per request in the dashboard.

The tasks:

1. Two-file bug fix. A shop module whose totals were off by a cent and whose discount was applied twice. The fix needed changes in both files, plus a rule from the README that the bug report did not mention: a minimum order for one discount code. 18 hidden tests.
2. Implement from a spec. A duration parser ("1h30m", "2 hours and 5 min", "PT1H30M") with a page of rules on separators, unit order and what must be rejected, including non-ASCII digits. 84 hidden tests.
3. Refactor without changing behaviour. Split a long function full of copy-pasted code into helpers. The hidden tests compared the new output with the original on 300 random inputs and 16 edge cases, including invalid options and missing fields. They also checked that the function shrank to at most half its original length. 317 hidden tests.
4. Add input validation. A sign-up function with a written rule set: per-field error codes, rule order, and fields that interact. For example, a password must not contain the username, but only if the username is valid. 84 hidden tests.
5. Parser for a config format. Sections, quoted values with escapes, inline comments, line continuation with correct error line numbers, and ${section.key} references with cycle detection. 57 hidden tests.
6. A Mustache template renderer from scratch, graded against the official Mustache spec test suite: interpolation, sections, inverted sections, comments, partials and delimiter changes. 136 hidden tests.

Results

Averages over two runs per model and task. Cost is per run, at QuickSilver Pro prices on 2026-10-27.

Where the tokens and the money went, averaged over all 12 runs per model:

Sol's uncached-input line includes cache writes: QuickSilver Pro billed some of Sol's uncached input at the cache-write rate rather than the plain input rate. Opus's uncached input was billed at the plain input rate.

What the numbers say

Quality was a tie. Neither model failed a hidden test in 24 runs. That includes all 136 cases of the official Mustache suite and the 316 comparisons with the original function in the refactor. These tasks did not find a ceiling for either model.

Opus was faster on every task. On average Sol took 1.9 times as long, from 1.5 times as long on the Mustache renderer to 4 times as long on the bug fix. Two things account for it. Opus used fewer requests per task (7.9 against 12.1), so it made fewer round trips. Its requests were also quicker, with a median of 7.6 seconds per request against 11.7 for Sol. Sol also split its work into more small steps. Over 12 runs it made 54 calls to OpenCode's file-read tool and 43 to-do-list updates. Opus made 6 and none.

The cost came out even, for two different reasons. Opus has the lower per-token prices, but it wrote almost twice as much output (7,800 tokens per task against 4,200). It also got less of its input from the prompt cache. On every Opus request the cache covered the same 8,028 tokens, about the size of OpenCode's system prompt and tool definitions. The rest of the conversation was billed at the full input rate each time. Sol's cache hits grew with the conversation and covered 81% of its input. Sol was cheaper on the two longest tasks, the config parser and the Mustache renderer. Opus was cheaper on the other four.

Both checked their own work the same way. In both refactor runs, each model saved a copy of the original function and wrote a throwaway script that compared old and new output on random inputs. On the Mustache task, both models downloaded the official spec test files from GitHub in both runs and ran them before finishing: Opus with curl, Sol with OpenCode's web-fetch tool and a Python one-liner. Our hidden tests for that task are the same files, so neither model's Mustache score is independent of what it downloaded.

When to use which

Opus 5.5 is the better default for interactive work on tasks like these. You wait about half as long for the same result, at the same price per task.

GPT-6.1 Sol is just as good at these tasks and was the cheaper of the two on the two longest ones. If your agent runs unattended and wall time matters less than the bill, the two are close enough that your own workload should decide. We did not find a task in this set that either model failed, so we cannot tell you which one solves more on harder work.

Both models are on the same QuickSilver Pro key and balance, so trying the other one is a change of -m.

Limits of this test

• Six tasks, two runs per model. Small, self-contained Python tasks of 30 seconds to 3.5 minutes each, not a multi-hour change in a large codebase.
• We wrote five of the six hidden test suites, so they reflect our reading of each spec. We checked each against a reference solution before running the agents.
• One agent, OpenCode, with its default per-model prompts and tools. Another agent, or a different reasoning-effort setting, could change both the speed and the cost. Opus reported no reasoning tokens in any run, so it worked without extended thinking. Sol used about 700 reasoning tokens per task at its default effort.
• Wall time includes network and server time. Two runs of the same model on the same task differed by up to 47%, so read the time column as a rough guide.
• Every Claude request reported about 690 more input tokens than the text sent. OpenCode's short title request, which is the same text for every model, came to 1,593 input tokens on Opus against 593 on Sol, and a one-line "say hi" to Opus reported 690. At 7.9 requests per task that adds at most about $0.005 to an Opus run. It does not change the comparison.
• One of Sol's 133 agent requests got an upstream error and OpenCode retried it. The error was not billed and the run passed.

Run it yourself

The OpenCode config we used, pointed at the public endpoint. Save it as opencode.json in your project:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "quicksilverpro": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "QuickSilver Pro",
      "options": {
        "baseURL": "https://api.quicksilverpro.io/v1",
        "apiKey": "{env:QSP_API_KEY}"
      },
      "models": {
        "claude-opus-5-5": { "name": "Claude Opus 5.5", "limit": { "context": 1000000, "output": 64000 } },
        "gpt-6.1-sol": { "name": "GPT-6.1 Sol", "limit": { "context": 1000000, "output": 64000 } }
      }
    }
  }
}

Then run the same prompt with each model:

export QSP_API_KEY=YOUR_QSP_KEY

opencode run --auto -m quicksilverpro/claude-opus-5-5 "Fix the failing test in tests/"
opencode run --auto -m quicksilverpro/gpt-6.1-sol "Fix the failing test in tests/"

--auto approves the agent's file edits and shell commands without asking, so use it only in a project you can reset. OpenCode shows a cost of 0 for a custom provider. The real cost of each request, with its model and token counts, is in your QuickSilver Pro dashboard. For more on the setup, see Use QuickSilver Pro in OpenCode and Crush. For the same six-task method with Claude Code, see Claude Opus 5.5 vs Sonnet 5.5 for coding agents. Claude Opus 5.5 is 75% below Anthropic's list price of $4.00 / $20.00.

Gemini 4 Argon is Google's new flagship reasoning model. On QuickSilver Pro it costs $2.00 / $10.00 per million tokens, against $1.00 / $5.00 for Claude Opus 5.5. We wanted to know whether it can stand in for Opus in a coding agent, and what that costs.

We tested this on 2026-10-27. Both models ran in the same agent, OpenCode, on six small but real coding tasks, and we graded every run with tests the agent never saw. Short answer: both passed every hidden test in every run. Gemini 4 Argon was slower and more expensive on every task: 134 seconds and $0.142 per task on average, against 82 seconds and $0.094 for Opus.

How we tested

• Agent: OpenCode 1.19.4, non-interactive (opencode run --auto), with QuickSilver Pro added as an OpenAI-compatible provider. The only thing that changed between runs was -m quicksilverpro/gemini-4-argon or -m quicksilverpro/claude-opus-5-5. QuickSilver Pro serves Gemini 4 Argon on the OpenAI-compatible endpoints (/v1/chat/completions and /v1/responses), not on the Anthropic Messages API that Claude Code needs. So we used one agent that can drive both models.
• Agent settings: a clean OpenCode config with no MCP servers, plugins or extra skills, and OpenCode's defaults otherwise. OpenCode sent no reasoning-effort setting, so each model ran at its default. OpenCode chooses its system prompt by model family, and we left that as shipped. Gemini got a 16,400-character system prompt and Opus a 9,300-character one. Both got the same tools.
• Tasks: six throwaway Python repositories, each with a one-paragraph prompt, a spec or bug report in the repo, and one visible test. Every task had a separate hidden test suite that we ran after the agent finished. We wrote five of the suites before any run and checked that each passed against our own reference solution and failed against the starting code. The sixth is the official Mustache spec suite.
• Runs: each task twice per model, same inputs, 24 runs in total. Both models ran each task at the same time, so both saw the same network and server conditions.
• Graded on: the share of hidden tests passing. We also recorded wall-clock time, number of API requests, input / cached / output tokens, and cost.
• Cost: the per-request cost that QuickSilver Pro returns in each response's usage block, summed over every request in a run, at QuickSilver Pro prices on 2026-10-27. We recorded it with a small local pass-through proxy. The same figures appear per request in the dashboard.

The tasks:

1. Two-file bug fix. A shop module whose totals were off by a cent and whose discount was applied twice. The fix needed changes in both files, plus a rule from the README that the bug report did not mention: a minimum order for one discount code. 18 hidden tests.
2. Implement from a spec. A duration parser ("1h30m", "2 hours and 5 min", "PT1H30M") with a page of rules on separators, unit order and what must be rejected, including non-ASCII digits. 84 hidden tests.
3. Refactor without changing behaviour. Split a long function full of copy-pasted code into helpers. The hidden tests compared the new output with the original on 300 random inputs and 16 edge cases, including invalid options and missing fields. They also checked that the function shrank to at most half its original length. 317 hidden tests.
4. Add input validation. A sign-up function with a written rule set: per-field error codes, rule order, and fields that interact. For example, a password must not contain the username, but only if the username is valid. 84 hidden tests.
5. Parser for a config format. Sections, quoted values with escapes, inline comments, line continuation with correct error line numbers, and ${section.key} references with cycle detection. 57 hidden tests.
6. A Mustache template renderer from scratch, graded against the official Mustache spec test suite: interpolation, sections, inverted sections, comments, partials and delimiter changes. 136 hidden tests.

Results

Averages over two runs per model and task. Cost is per run, at QuickSilver Pro prices on 2026-10-27.

Where the tokens and the money went, averaged over all 12 runs per model:

What the numbers say

Quality was a tie. Neither model failed a hidden test in 24 runs. That includes all 136 cases of the official Mustache suite and the 316 comparisons with the original function in the refactor.

Gemini got there with less checking. On the refactor, both Opus runs saved a copy of the original function and compared old and new output on random inputs before finishing. Neither Gemini run did that; Gemini relied on the one visible test and still matched the original on all 316 hidden comparisons. On the Mustache task, both Opus runs downloaded the official spec test files from GitHub and ran them. Our hidden tests for that task are the same files, so Opus's score there is not independent. Gemini did not download them and passed all 136 anyway. What Gemini did instead was run the test suite more often: 24 pytest runs over its 12 runs, against 14 for Opus.

Gemini was slower on every task, from 1.2 times as long on the Mustache renderer to 2.3 times on the config parser. Its typical request was actually quicker than Opus's, with a median of 6.2 seconds against 7.6. But it made more requests (10.2 per task against 7.9), and its slow requests were slower: one in ten took over 34 seconds, and the slowest took 109. The config parser shows the effect most: the second Gemini run took 18 requests and almost five minutes.

Gemini cost more on every task, about 1.5 times as much on average. It has the higher per-token prices on QuickSilver Pro, and it sent more input per task because it made more requests. It wrote fewer output tokens than Opus, but at its output price they still cost more ($0.056 against $0.039 per task). Gemini's caching did better than Opus's: 69% of its input came from the prompt cache. On every Opus request the cache covered the same 8,028 tokens, about the size of OpenCode's system prompt and tool definitions. The rest of the conversation was billed at the full input rate. Even so, Opus came out cheaper.

When Gemini 4 Argon makes sense

On these tasks, Opus 5.5 is the better choice for a coding agent. It solved the same tasks, in about 60% of the time, for about two thirds of the cost.

Gemini 4 Argon solved every task too, so it is a workable second model. Use it if you want a non-Anthropic model in the loop, for example to cross-check a change, or if you are already building on Gemini elsewhere. We did not find a task in this set that either model failed, so we cannot tell you which one solves more on harder work. On this evidence, Gemini 4 Argon does not save you money over Opus 5.5.

Limits of this test

• Six tasks, two runs per model. Small, self-contained Python tasks of 30 seconds to 5 minutes each, not a multi-hour change in a large codebase.
• We wrote five of the six hidden test suites, so they reflect our reading of each spec. We checked each against a reference solution before running the agents.
• One agent, OpenCode, with its default per-model system prompts. Another agent, or a different reasoning-effort setting, could change both the speed and the cost. Gemini used about 900 reasoning tokens per task at its default effort. Opus reported none, so it worked without extended thinking.
• The prompt told the agents not to edit the existing tests. In four runs Gemini added its own tests to the end of the existing test file instead of creating a new one. It did not change or remove the existing test, and the hidden tests are separate, so this did not affect the scores.
• Wall time includes network and server time. Two runs of the same model on the same task differed by up to 47%, so read the time column as a rough guide.
• Every Claude request reported about 690 more input tokens than the text sent. OpenCode's short title request, which is the same text for every model, came to 1,593 input tokens on Opus against 639 on Gemini. At 7.9 requests per task that adds at most about $0.005 to an Opus run, which does not change the comparison.

Run it yourself

The OpenCode config we used, pointed at the public endpoint. Save it as opencode.json in your project:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "quicksilverpro": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "QuickSilver Pro",
      "options": {
        "baseURL": "https://api.quicksilverpro.io/v1",
        "apiKey": "{env:QSP_API_KEY}"
      },
      "models": {
        "gemini-4-argon": { "name": "Gemini 4 Argon", "limit": { "context": 1000000, "output": 64000 } },
        "claude-opus-5-5": { "name": "Claude Opus 5.5", "limit": { "context": 1000000, "output": 64000 } }
      }
    }
  }
}

Then run the same prompt with each model:

export QSP_API_KEY=YOUR_QSP_KEY

opencode run --auto -m quicksilverpro/gemini-4-argon "Fix the failing test in tests/"
opencode run --auto -m quicksilverpro/claude-opus-5-5 "Fix the failing test in tests/"

--auto approves the agent's file edits and shell commands without asking, so use it only in a project you can reset. OpenCode shows a cost of 0 for a custom provider. The real cost of each request, with its model and token counts, is in your QuickSilver Pro dashboard. Model details are on the Gemini 4 Argon and Claude Opus 5.5 pages. For the OpenCode setup, see Use QuickSilver Pro in OpenCode and Crush. For Opus against GPT-6.1 Sol on the same tasks, see Claude Opus 5.5 vs GPT-6.1 Sol for coding agents.

GitHub Codespaces and QuickSilver Machines both give you a Linux box in the browser that a coding agent can treat as its own. We already compared their list prices and features. This time we ran the same Claude Code task on both, three times each, on 2026-11-05, and timed everything from creating the box to deleting it.

Short version:

• The task itself ran at the same speed on both. Three runs of a 317-test refactoring task averaged 24.0 seconds on Codespaces and 24.5 seconds on Machines, and every run passed all hidden tests. The model and the API were the same, so the box made no real difference.
• Codespaces started faster. About 19 seconds from gh codespace create to a working shell, against 35 seconds from clicking Create to a running Machine with its terminal open.
• Machines needed no setup. On Codespaces we installed Claude Code, created an API key and wrote a settings file before the first task. On the Machine, claude worked as soon as the terminal opened.
• Pause kept our processes only on Machines. A Codespace stop shut the container down: files stayed, the background job was gone. A Machine pause took 11 seconds, resume 7, and the job was still running afterwards.
• Price depends on GitHub's free quota. Codespaces is free until a personal account's monthly quota runs out, then $0.18 per hour for 2 cores. Machines is $0.02 per running hour from the first minute.

How we tested

Both boxes sent their AI requests to the same QuickSilver Pro API with the same model, so any difference in task time comes from the box, not the model. The Codespace first came up in Singapore, the default for our location. We deleted it and created it again in West US 2, so that both boxes were on the US west coast.

The task is the refactoring task from our coding agent benchmark: a long, copy-pasted build_report function has to be split into small helpers without changing its output for any input. A hidden suite of 317 tests, which the agent never sees, compares the result with the original function. One script ran it 3 times on each box, each time on a fresh copy of the repository:

claude -p --dangerously-skip-permissions --model claude-sonnet-5-5 --output-format json "$PROMPT"

The script also timed a fixed single-threaded Python loop, to compare raw CPU speed.

Getting to the first task

Codespaces. gh codespace create returned in 6 seconds, and the shell answered over SSH 14 seconds later, 19 seconds in total (17 for the first attempt in Singapore). Claude Code is not in the default image, so we installed it with the official installer, which took 32 seconds. Claude Code then needs a model provider. We created a QuickSilver Pro API key in the dashboard and wrote it to ~/.claude/settings.json:

{
  "env": {
    "ANTHROPIC_BASE_URL": "https://api.quicksilverpro.io",
    "ANTHROPIC_AUTH_TOKEN": "YOUR_QSP_KEY",
    "ANTHROPIC_DEFAULT_OPUS_MODEL": "claude-opus-5-5",
    "ANTHROPIC_DEFAULT_SONNET_MODEL": "claude-sonnet-5-5",
    "ANTHROPIC_DEFAULT_HAIKU_MODEL": "claude-haiku-5-5"
  }
}

With an Anthropic account you would sign in instead of using a key. Either way, it is a step you do once per Codespace, or once in a dev container that you maintain. One thing to watch: our first check, claude -p "Reply with ok", went to Claude Opus 5.5, Claude Code's default, and cost $0.030 for a one-word reply. Pass --model or set a default model if you do not want Opus.

Machines. We clicked Create Machine, and 35 seconds later the machine was Running with the terminal open. Claude Code, Codex, OpenCode and Hermes were already installed and signed in with the machine's own key, so we went straight to the task. The setup steps are in our Machines walkthrough.

Results

What the numbers say

The box is not the bottleneck. On Codespaces, Claude Code's own timing put 21 to 22.5 seconds of each 23 to 25-second run in API calls, and the hidden tests ran in a third of a second on both boxes. A faster box would not have made the agent noticeably faster. This holds for agent tasks like this one: a large build or test suite would feel the difference in CPU and RAM, and Codespaces has twice the RAM.

AI cost is a property of the model, not the box. All six runs cost 2 to 3 cents on Claude Sonnet 5.5. The two dearer Codespaces runs differ from the rest in their first request ($0.016 against $0.005 to $0.006), which looks like a prompt-cache write rather than anything the box did.

Setup is the real difference. On Codespaces we had to install the agent, get a key and configure it. A dev container can do this for you on every new Codespace, but someone has to write and maintain it, and the key or sign-in is still yours to manage. On Machines, the agents and the key come with the box, and machine time and AI usage come from one QuickSilver balance.

Pause behaves differently. Stopping a Codespace shuts the container down: your files survive, running processes do not. Pausing a Machine saves a checkpoint, so a long-running agent or dev server is still there when you come back. In our test the background job's log simply had a gap for the paused minute.

What it costs

Machines charges $0.02 per running hour, per minute, from your QuickSilver Pro balance, and nothing while paused. Our Machine was billed $0.0013 for about 4 minutes of running time.

Codespaces charges $0.18 per hour for a 2-core machine and $0.07 per GB-month of storage, but personal accounts get a monthly quota first: 120 core-hours and 15 GB-month on GitHub Free, 180 core-hours and 20 GB-month on GitHub Pro. A 2-core Codespace uses 2 core-hours per hour, so GitHub Free covers 60 hours a month of it. Our Codespace ran about 15 minutes, half a core-hour of the quota, and cost us nothing. Organizations get no free quota.

For an agent box used 2 hours a day on 22 working days (44 hours a month):

AI usage is extra in every case. With Machines it comes from the same balance; with Codespaces it goes to whichever provider your agent uses.

Which one to pick

Codespaces makes sense when:

• your use fits in the free quota of a personal account;
• your work starts from a GitHub repository and you want VS Code in the browser;
• your team already has dev containers that install the tools you need.

Machines makes sense when:

• you would pay for Codespaces anyway, because the quota is used up or an organization pays: $0.02 against $0.18 an hour;
• you want Claude Code, Codex, OpenCode and Hermes ready without installing anything or handling keys;
• you want a full Linux desktop with Chromium next to the terminal;
• you pause and resume often and want running processes to survive.

Limits of this test

• One task, three runs per box, on one day. Times through a shared API vary from run to run.
• The two boxes are not the same size: Codespaces has 8 GB of RAM, Machines 4 GB.
• We timed Codespaces from the command line (gh codespace create and gh codespace ssh) and Machines from the browser. Opening VS Code in the browser on Codespaces takes longer than an SSH shell.
• The Codespace stop-and-start test was done over SSH, so we checked a background job but not a shell variable in an open terminal.
• Machines is in beta. It has one size, an 8-hour session limit and a 200-hour monthly cap, and no backup guarantee.

Run it yourself

The task is described in our coding agent benchmark; any task from your own repository works the same way with the claude -p command above. To try the Machine side, open Machines, create one and type claude in the terminal. To run Claude Code on your own computer or in a Codespace instead, follow our Claude Code setup guide.

The usual reason to choose an open model over Claude is price. Now that every Claude model on QuickSilver Pro costs 75% less than Anthropic's list price, that reason needs checking again, and so does the question of what you give up in quality. So we put three large open models from our catalog and two Claude models through the same set of tasks, on 2026-10-20.

How we tested

17 tasks, each with an answer a script can check, in two groups:

• 13 standard tasks: six small Python functions (a duration parser, interval merging, an LRU cache, strict Roman numerals, an expression evaluator with C-style division, semantic-version ordering), three extractions to JSON (an invoice from an email, attendees from meeting notes, log-line counts) and four short maths and logic questions.
• 4 harder tasks: shell-style word splitting with quoting rules, the final state of an order after a chat full of changes and reversals, a counting-out puzzle where the count alternates between 3 and 4, and the number of integers up to 100,000 whose digits sum to 20.

Code passes only if it survives a fixed set of unit tests; JSON passes only if it equals the expected object; maths answers must match exactly. Every model got the same single user message per task through the OpenAI-compatible /v1/chat/completions endpoint, non-streaming, max_tokens 8,000, no system prompt, no temperature setting. Each task ran three times: 51 requests per model.

With these defaults the three open models think before answering and bill those reasoning tokens as output; the two Claude models answered directly, without extended thinking. Cost below is the billed input and output tokens multiplied by QuickSilver Pro prices on 2026-10-20.

Results

Costs are at QuickSilver Pro prices on 2026-10-20, computed from the measured tokens.

Where the open models were good enough

• Routine extraction. All three open models got the invoice, the attendee list and the log counts right in all nine of their runs. If your workload is "turn this text into this JSON", any of them will do it.
• Small, well-specified functions. Kimi K3 passed every coding task in every run. GLM 5.3 missed two of 21 coding runs: once it accepted an empty string as a Roman numeral, once it rejected a valid quoted string in the shell splitter.
• GLM 5.3 on price and speed. It was the only open model that was both cheaper than Sonnet 5.5 (about 30% less per task) and as fast (the same 3.7-second median). On the 13 standard tasks alone it passed 37 of 39 for $0.85 per 1,000 tasks, against 39 of 39 for $1.59 on Sonnet.

Where they were not

• Multi-step reasoning. The counting-out puzzle needs about forty careful steps. Sonnet 5.5 and Opus 5.5 solved it in all six runs. Kimi K3 solved it once in three, GLM 5.3 once (its other answers were 37 and 13), DeepSeek V4 Pro never.
• Getting an answer out at all. DeepSeek V4 Pro returned an empty answer 8 times out of 51: it spent its tokens reasoning and stopped before writing anything, five times by reaching the 8,000-token limit. A ninth request had not finished after ten minutes. Kimi K3's two misses were the same pattern. A larger max_tokens might have let them finish, at a higher cost per task; we did not test that.
• Following the output format. In one run GLM 5.3 wrote the correct JSON object and then appended its reasoning after it, which breaks any parser that expects JSON only.
• Latency. Kimi K3's median was 26.6 seconds and DeepSeek V4 Pro's 110 seconds, with DeepSeek's median ranging from 24 seconds in one run to 233 seconds in another. Fine for a batch job, not for anything a user waits on.
• Price. At 75% off list, Sonnet 5.5 cost less per task than both Kimi K3 (ten times less) and DeepSeek V4 Pro (a third), and it passed every task. Even Opus 5.5 was cheaper per task than both on this test.

So which should you use?

• For extraction and simple functions where cost matters most, GLM 5.3 held up: cheaper than Sonnet and just as fast, with a few more misses. Validate its output, as you would with any model in a pipeline.
• For anything with several reasoning steps, or where a wrong or empty answer is expensive, Claude Sonnet 5.5 was the better buy on this test: every task passed, a few seconds each, at a lower cost per task than two of the three open models.
• Kimi K3 was accurate on the standard tasks but slow and, at its price, by far the most expensive model here per task.
• DeepSeek V4 Pro had the lowest pass rate and the slowest answers on this harness; if you use it, allow a large max_tokens and retry empty responses.

Limits of this test

17 tasks and three runs per model is a small sample: one task moves a pass rate by two points, and the harder group has only four tasks. The tasks are short, single-turn and in English. We did not test long contexts, tool use, agentic coding, other languages, or the open models with reasoning turned down or Claude with extended thinking turned on, and any of those could change the picture. Latency was measured from one client on one day. Treat the table as a starting point and run your own prompts: the request code is the same for every model, so switching is a one-word change.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.quicksilverpro.io/v1",
    api_key="YOUR_QSP_KEY",
)

for model in ["glm-5.3", "kimi-k3", "deepseek-v4-pro", "claude-sonnet-5-5"]:
    response = client.chat.completions.create(
        model=model,
        max_tokens=8000,
        messages=[{"role": "user", "content": "Your task here"}],
    )
    print(model, response.usage.completion_tokens, response.choices[0].message.content)

For the full Claude price list, see Claude API pricing in 2026. For requesting output that follows a JSON schema, see the JSON schema guide.

Claude Opus 5.5 costs $1.00 / $5.00 per million tokens on QuickSilver Pro, against $0.50 / $2.50 for Claude Sonnet 5.5. The question for anyone running a coding agent is whether Opus buys anything for that.

We tested it on 2026-10-20 with Claude Code, on six small but real coding tasks, and graded every run with tests the agent never saw. Short answer: on these tasks, no. Both models passed every hidden test in every run. Opus was slower and, per task, cost about two and a half times as much.

How we tested

• Agent: Claude Code 2.1.301, non-interactive (claude -p), default settings, with --allowedTools "Read,Edit,Write,Bash,Glob,Grep" so it could read and edit files and run shell commands. The only thing that changed between runs was --model claude-sonnet-5-5 or --model claude-opus-5-5. ANTHROPIC_DEFAULT_HAIKU_MODEL was set to claude-haiku-5-5.
• Tasks: six throwaway Python repositories, each with a one-paragraph prompt, a spec or bug report in the repo, and one visible test. Every task had a separate hidden test suite that we ran after the agent finished. Five suites were written by us before any run and checked against our own reference solutions; the sixth is the official Mustache spec suite.
• Runs: each task twice per model, same inputs, 24 runs in total.
• Graded on: the share of hidden tests passing. Also recorded: wall-clock time, number of API requests, input / cached / output tokens, and cost.
• Cost: computed from the token counts of every request each run made, at QuickSilver Pro prices on 2026-10-20 — input, output, cache reads and cache writes each at their own rate.

The tasks:

1. Two-file bug fix. A shop module whose totals were off by a cent and whose discount was applied twice. The fix needed changes in both files plus a rule from the README that the bug report did not mention (a minimum order for one code). 17 hidden tests.
2. Implement from a spec. A duration parser ("1h30m", "2 hours and 5 min", "PT1H30M") with a page of rules on separators, unit order and what must be rejected. 57 hidden tests.
3. Refactor without changing behaviour. Split a long function full of duplicated code into helpers. The hidden tests compared the new output with the original on 300 random inputs and a set of edge cases, and checked that the function actually got shorter. 312 hidden tests.
4. Add input validation. A sign-up function with a written rule set: per-field error codes, rule order, and fields that interact (a password must not contain the username, but only if the username is valid). 52 hidden tests.
5. Parser for a config format. Sections, quoted values with escapes, inline comments, line continuation with correct error line numbers, and ${section.key} references with cycle detection. 34 hidden tests.
6. A Mustache template renderer from scratch, graded against the official Mustache spec test suite: interpolation, sections, inverted sections, comments, partials and delimiter changes. 136 hidden tests.

Results

Averages over two runs per model and task. Cost is per run, at QuickSilver Pro prices on 2026-10-20.

Where the tokens went, averaged over all 12 runs per model:

What the numbers say

Quality was a tie. Neither model failed a single hidden test in 24 runs, including the 136 cases of the official Mustache suite, which include the fiddly rules for whitespace around standalone tags and indented partials. These tasks did not find a ceiling for Sonnet 5.5, so they cannot show Opus clearing it.

Opus did not write much more. The two models used about the same number of requests and output tokens per task. The model-only cost difference (2.2 times) comes almost entirely from Opus's higher per-token price.

Opus checked its own work more, and that is where its extra time and cost went. It ran more shell commands. On the refactor, both Opus runs wrote a throwaway script that compared the old and new function on random inputs; one of the two Sonnet runs did the same, the other relied on the existing test. On the Mustache task, both Opus runs downloaded the official spec test files from GitHub and ran them before finishing — one of them started a sub-agent to do it — which made that task four times as expensive as with Sonnet. (Our hidden tests for that task are the same files, so Opus's score there is not independent. Sonnet passed all 136 without them.)

Opus was slower in every task, from about the same speed on the validation task to 2.7 times as long on the refactor. Averaged over all runs it took 1.6 times as long.

The extra requests you are billed for

Every run also sent requests to Claude Sonnet 5, a model we never selected. We logged one of them: it is Claude Code's own safety check, a "security monitor" prompt that receives the conversation so far and grades the shell command the agent is about to run. Each check sent about 49,000 input tokens and returned a few tokens of output.

Three things about these checks matter for your bill:

• They do not appear in Claude Code's own usage summary, which in our runs listed only the model we picked. They are ordinary API requests made with your key, so they are billed like any other.
• In our runs they cost about as much as the Sonnet 5.5 work itself ($0.036 against $0.034 per task).
• They scale with how many shell commands the agent runs, not with the model you chose. Opus ran more commands, so it triggered almost three times as many checks; the two Opus Mustache runs, which downloaded files from GitHub, sent 9 and 15.

When Opus earns its price, and when Sonnet is enough

Sonnet 5.5 is enough for the kind of work we measured: bug fixes across a couple of files, implementing a function from a written spec, behaviour-preserving refactors, validation rules, parsers of a few hundred lines. It solved all of them, every time, at about 40% of Opus's cost per task and in less time.

Opus 5.5 is worth trying when Sonnet fails a task — and, judging by these runs, when you want the agent to go out of its way to verify its work, as it did by fuzz-testing its refactors and fetching the reference test suite. We did not find a task in this set that Sonnet failed, so we cannot tell you how much more Opus solves on harder work. Treat that as a reason to keep Opus one /model opus away, not to run it by default.

A practical setup: map Claude Code's sonnet alias to claude-sonnet-5-5 and its opus alias to claude-opus-5-5 (see Claude Code with QuickSilver Pro), work in Sonnet, and switch with /model opus for the task where Sonnet keeps going round in circles.

Limits of this test

• Six tasks, two runs per model. Small, self-contained Python tasks of 20 seconds to 2.5 minutes each, not a multi-hour change in a large codebase.
• Five of the six hidden test suites were written by us, so they reflect our reading of each spec. We checked each against a reference solution before running the agents.
• Wall time includes network and server time, and two runs of the same model on the same task differed by as much as 55%. Read the time column as a rough guide.
• Claude Code ran with its default settings, including the default effort level for each model.

Run it yourself

The settings we used, pointed at the public endpoint:

export ANTHROPIC_BASE_URL=https://api.quicksilverpro.io
export ANTHROPIC_AUTH_TOKEN=YOUR_QSP_KEY
export ANTHROPIC_DEFAULT_HAIKU_MODEL=claude-haiku-5-5

claude -p "Fix the failing test in tests/" --model claude-sonnet-5-5 --allowedTools "Read,Edit,Write,Bash,Glob,Grep"
claude -p "Fix the failing test in tests/" --model claude-opus-5-5 --allowedTools "Read,Edit,Write,Bash,Glob,Grep"

Each request shows up in your QuickSilver Pro dashboard with its model, token counts and cost, so you can compare the two models on your own tasks. Prices for every Claude model are in Claude API pricing in 2026.