Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing
Rerunning the same LLM query at temperature 0 flips up to a quarter of row-level answers, undermining how benchmarks compare…
Researchers re-executed three semantic query benchmarks (SemBench, LRO-Bench, UDA-Bench) across 1.2 million model calls and found substantial run-to-run variance even at temperature 0, with up to 24% of rows changing output on one task. In SemBench's own published results, 21% of text results changed quality between executions, and most of LRO-Bench's implementation recommendations were not actually settled by the benchmark's executions.
Why it matters
Semantic query engines and many LLM-based data pipelines are evaluated and compared using benchmarks that assume a handful of runs at temperature 0 represent stable ground truth; this work shows that assumption is often false, meaning published system comparisons and "best practice" recommendations may not be statistically settled by the evidence benchmarks actually generated.
What is verified
At temperature 0, between 0.07% and 10.2% of rows change their operator output over five executions depending on the model, and up to 24.0% on one task. [1]
On UDA-Bench, a few percent of extracted values change but about two thirds of the answers do, consistent with a possible-worlds model predicting exponential amplification across rows in a plan. [1]
In SemBench's published results, 21% of text results change quality between executions, including on engines whose plans are deterministic. [1]
16 of 28 tests of LRO-Bench's implementation recommendations are not settled by the executions the benchmarks run. [1]
The study logged 1.2 million model calls by re-executing the text and tabular workloads of SemBench, LRO-Bench, and UDA-Bench through their own harnesses. [1]
Voting reduces variance but does not remove it, and a calibrated cascade keeps its accuracy guarantee while the rows it returns still change. [1]
What is not
- Which specific models were tested is not detailed in the abstract, only that variance depends on the model used.
- The paper's proposed reporting and design rules for benchmarks/engines are not detailed beyond being mentioned.
- No independent replication beyond the authors' own re-execution of existing benchmark harnesses is described.
- Real-world production impact on deployed semantic query or RAG-style systems outside benchmark settings is not measured.
Sources
- 1arXiv cs.CRarxiv.org · Primary source
Original source published 9 October 2026 at 18:34.
An AI-assisted briefing by Coding Partners. We link the evidence behind the reporting so you can explore the original work and assess the claims. How CP Signal works.