For the past couple of years, the pressure inside most companies has pointed one direction: use more AI. Employees were encouraged, sometimes measured, on how much they used it. Adoption dashboards tracked prompts sent, tokens burned, queries run, as if usage itself were the output worth optimizing for. That mindset earned its own name: "token maxing." The underlying assumption was simple: more tokens in means more value out.
That assumption is starting to crack. Companies are realizing that the volume of AI being used isn't the same thing as the value it's producing. The metric that matters is efficiency: how much you get back per token spent, not how many tokens you spent. Context Analytics maintains a corpus of more than 3 million parsed SEC documents, mapped down to the company level.
At that volume, a token difference that looks trivial in a demo (a few hundred tokens per call) stops being a rounding error. It becomes a real line item, and it's exactly the kind of cost that "just use more AI" never asked anyone to look at.
So, we ran an experiment. We took ten large-cap 10-Ks, asked the same six questions about each one two different ways (once against the original document, once against our own parsed, structured version of the same disclosure), and measured what changed: tokens, cost, response time, accuracy, and hallucination. This post is what we found, including the parts that didn't turn out the way we expected.
The Collection Work Is Already Done
Before getting into the experiment, it's worth noting that the collection phase of this process is already done on our end. Most token-maxing conversations start from the assumption that getting a document in front of the model is the easy part, and the only question is what to do with it once it's there. That's not where the work is.
An SEC filing does not arrive as a clean question-and-answer document. A 10-K is a multi-hundred-page HTML file wrapped in an inline-XBRL viewer and split across dozens of exhibits. The footnote you want, say, segment reporting, is buried inside a rendered sub-document you must know to look for. You can build a pipeline to fetch, unwrap, and parse that yourself. You can pay for a vendor subscription that already has. Either way, getting from "filing exists on EDGAR" to "here is the segment-reporting note as clean text, mapped to the right ticker" is real infrastructure work. It just happens invisibly, before anyone opens a chat window and starts asking questions.
That's the work behind the "B" condition in this experiment. It's also the work behind our 3-million-document corpus. The question we wanted to answer is whether that upfront structuring work pays for itself downstream, at the point where someone is running an LLM against it.
The Experiment
Universe: Ten large-cap 10-K filings: FedEx, Oracle, NIKE, Conagra Brands, Darden Restaurants, Paychex, RPM International, Seagate Technology, J.M. Smucker, and Cintas.
Conditions: A, Original document: the segment-reporting footnote pulled directly from the original HTML document retrieved directly from EDGAR (via the filing's rendered sub-document), narrative text only. We refer to this as the "original document" throughout the rest of this post.
B, Parsed data: the equivalent section from our own parsed, structured extraction of the same filing.
Both conditions were deliberately restricted to prose. We stripped every numeric table from both sides before running anything. That was a specific choice: this test is about narrative-extraction efficiency, not about whether an LLM can read a table. A different, and equally valid, experiment would test table-heavy questions; that's not this one.
Task: Six questions per filing, asked of both conditions, covering things stated (or not stated) in segment-reporting prose: who the decision-maker is, how many reportable segments exist, whether segment structure changed, customer concentration, geographic revenue description, and impairment charges. The model was explicitly instructed to answer "blank" rather than guess whenever a fact wasn't in the text; this was designed as a hallucination stress test as much as an accuracy test.
Scale: 10 companies × 6 questions × 2 conditions = 120 graded answers, run with Claude Sonnet.
Isolation. Each of the 20 calls (one per company, per condition) ran in its own independent, isolated session: no web access, no tool use, and no memory of any other call. The model was given only the text block for that single company and condition and instructed not to search the web or use outside knowledge. That matters for this experiment specifically, since the whole point was testing whether the model answers from the text in front of it rather than from something else. An answer that happened to be correct because the model recognized the company and recalled its segment structure from training, or because information leaked in from a prior call in the same session, wouldn't tell us anything about token efficiency. Isolating each call rules that out and keeps the "blank when not stated" results honest.
What we're not claiming.
Ten filings is a small sample. It's enough to detect a token-usage effect; it is not enough to reliably detect a modest accuracy or speed difference, even if one exists. Token counts are estimated using a ~4-characters-per-token heuristic, not exact API-reported usage. They're directionally right, not precise.Response time reflects single-call latency in a shared environment, not a controlled, repeated-trial benchmark; treat it as one measurement, not a stable average.One model, one provider, one question style (segment-reporting facts). We're not claiming this generalizes to every filing section or every LLM.
Results
*Estimated using Sonnet list pricing ($3/M input, $15/M output) applied to the token estimates above; directional, not measured billing.
Parsed data used about 5% fewer input tokens per call, on average, across the ten filings. Paired against the same filing (Wilcoxon signed-rank test): p = 0.008. That's a real, reproducible effect, not noise. Structured extraction reliably trims the padding that survives in raw narrative text.
Response time: not significant.
Difference: 0.02 seconds. p = 0.94. There is no detectable speed advantage in either direction. Fewer tokens in, in this experiment, did not mean a faster answer out. Latency here was dominated by ordinary call-to-call variance, not input size.
Accuracy and hallucination: not significant, and one honest miss.
Zero hallucinations in either condition. Across 120 answers, the model never fabricated a fact it wasn't given. Given the explicit instruction to leave answers blank, that's the result we hoped for going in.
On accuracy, the two conditions agreed on the correct answer in 59 of 60 paired questions. The lone disagreement: Paychex's original document disclosed a small (~1%) European revenue figure the model missed; separately, a parsed excerpt for one company omitted a customer-concentration fact that its original-document counterpart included. With only one discordant pair, a McNemar test returns p = 1.0; there is no statistical basis to call either condition more accurate. The raw percentages (96.7% vs. 98.3%) look like a story, but with this sample size, they aren't one yet.
What This Supports
The headline that survives scrutiny is narrower than "token maxing is dead," but it's a real one: structured data cut token usage by a measurable, statistically significant margin, with no detectable cost to accuracy. It didn't make the model faster, and we can't yet say it made it more accurate, but it didn't make it worse, either. It did make it cheaper.
Run that 5% savings across ten filings and it's a rounding error. Run it across a corpus of 3 million documents, at the volume of a real filings pipeline processing thousands of questions a day, and the same percentage stops being a curiosity. It's the difference between a cost model that scales linearly with your usage and one that doesn't.
The Savings This Experiment Didn’t Even Measure
This test only compared two ways of handing a model text and asking it to reason over the whole thing. That's the least efficient way to use structured data, not the most efficient.
Because our data is parsed down to the section and field level, you don't have to ask an LLM "does this filing mention a segment change" at all. You can query for the segment-reporting note directly and skip the rest of the document entirely, before a single model token is spent.
You can run a keyword search or lightweight NLP pass to pre-filter which filings even need a model call. And for fields that don't require judgment (single segment vs. multiple, for instance), you can write a deterministic script against structured fields instead of asking an LLM to infer something that's just a lookup. Every one of those moves saves more tokens than the comparison in this post, because it avoids the API call altogether rather than shrinking it.
What’s Next: The Harder Version of This Test
Everything in this experiment ran on SEC filings, which, for all their quirks, are semi-structured HTML. There are tags, sections, an EDGAR filing index, and a rendering structure you can lean on even before any parsing work begins.
Our global filings coverage doesn't get that scaffolding. Non-U.S. disclosure regimes are frequently unstructured PDFs: scanned pages, inconsistent layouts, no reliable section markers, sometimes no consistent language. The gap between "raw" and "structured" is much wider there than it is between an EDGAR HTML page and a JSON file. If the token-savings case holds up in that environment, it holds up in the harder case that matters at global scale. That's the next post in this series.
Learn more about how Context Analytics powers next-generation investment research at www.contextanalytics-ai.com.
Token counts are estimates based on a ~4-characters-per-token heuristic; no live tokenizer was available in the test environment. Cost figures are estimates based on Claude Sonnet list pricing, not measured API billing. All original documents sourced from SEC EDGAR; parsed data sourced from Context Analytics' internal corporate filings corpus.
