Forty Questions: Identifying a Language Model from One-Word Answers
A cheap 40-question test caught an API provider serving Gemma under the name Llama 4 Maverick and billing accordingly.
A new arXiv paper shows that an LLM behind an API can be fingerprinted from its answers to 40 everyday one-word questions, correctly identifying the exact model 92.5% of the time across 94 models, versus 1.1% by chance. Applying this to 96 OpenRouter endpoints of 13 open models found 92 matched their claimed identity, but one endpoint billed as "Llama 4 Maverick" actually self-identified as, and was billed as, Gemma 4, and billing analysis also revealed inconsistent hidden token overhead across providers like Grok 4.7 versus GPT-4o.
Why it matters
This is a cheap, reproducible audit method for a trust problem that's otherwise invisible to API consumers: whether the model you're calling and paying for is actually the model advertised. It surfaces real instances of model substitution and inconsistent billing/context overhead in a widely used model marketplace (OpenRouter), which has direct cost and reliability implications for anyone building on third-party model APIs.
What is verified
A method using 40 one-word answers to everyday questions identifies the exact model behind an API 92.5% of the time across a panel of 94 models, versus 1.1% by chance. [1]
Profiles built through OpenRouter correctly identified all ten models served by DeepInfra under blinded labels. [1]
Applying the method to 96 endpoints of 13 open models on OpenRouter, 92 identified as the model they claimed to be. [1]
One endpoint served what it called Llama 4 Maverick but self-identified as Gemma 4 and was billed Gemma 4's token counts on every message. [1]
A system prompt lowers identification accuracy by 6 to 16 points. [1]
Billing analysis found Grok 4.7 billed 1,247 input tokens for a one-line question that cost GPT-4o only 12 tokens, and five models stated the current date without being billed for it, indicating hidden context injection. [1]
What is not
- The paper is a single-author arXiv preprint; it is not stated whether it has been peer reviewed or independently replicated.
- No detail is given on which specific providers or endpoints were mislabeled beyond the one Gemma/Llama case, limiting ability to assess scope of the practice.
- It is not established whether model substitution found here was deliberate provider misconduct, a proxy/routing artifact, or a labeling error.
- No information on how widespread this kind of substitution is across the broader API ecosystem beyond the 96 endpoints tested.
The case for
The method is cheap (cents per endpoint), has a clear statistical baseline (92.5% vs 1.1% chance), and produced a concrete, checkable finding: one endpoint billed as Llama 4 Maverick was actually Gemma 4. This is a verifiable, actionable audit technique rather than a speculative claim.
The case against
This is a single-author preprint with no independent replication noted, a sample of only 96 endpoints and 13 open models, and no explanation of whether the mislabeling found was intentional fraud, a proxy artifact, or an error, which limits how much can be concluded about how widespread or deliberate the practice is across the industry.
What to do with this
Treat this as a useful specialist audit tool worth trying against any client-facing model routing setup, but do not treat the single found instance of mislabeling as evidence of a widespread industry pattern without further data.
Sources
- 1arXiv cs.CLarxiv.org · Primary source
Original source published 20 October 2026 at 16:33.
An AI-assisted briefing by Coding Partners. We link the evidence behind the reporting so you can explore the original work and assess the claims. How CP Signal works.