Retrieval quality measurement
Arcus is building one thing. A way to answer that question with a number instead of an impression, for a team already running retrieval over its own corpus. It measures what came back and compares it against what a person said was relevant. It writes no answers of its own.
The question
A generation step fails loudly and gets blamed. The retrieval step in front of it fails silently and does not.
A retrieval system takes a question, searches an index and hands back a ranked list of passages. Something downstream then reads that list. It might be a language model writing an answer, it might be a search results page, it might be a person scrolling. Whatever it is, it can only work with what the list contains.
When the output is wrong, the argument in the room is almost always about the model. Change the prompt. Try a larger model. Add a grounding instruction. Sometimes that is the right argument. Often the model was never given the passage it needed, answered from whatever it did receive, and the whole debate is being held about the wrong component.
Telling those cases apart is not a matter of judgement or of reading enough transcripts. It is a measurement, and it is a measurement that has been well defined in the information retrieval literature for sixty years. The awkward part is that very few teams running a retrieval pipeline in 2026 are taking it.
| What actually happened | What the user sees | Where the fault is | What a measurement would say |
|---|---|---|---|
| The right passage was retrieved and was in the context, and the answer was still wrong | A wrong answer | The generation step | Recall at k is high for this question. Stop blaming retrieval and go and look at the prompt |
| The right passage was never retrieved at all | A confident wrong answer, or a refusal | Retrieval | Recall at k is zero for this question, and no amount of prompting will fix it |
| The right passage was retrieved at rank eighteen, and the pipeline passes eight | The same wrong answer, and a harder one to believe | Ranking, or the cut off | Recall at twenty is high, recall at eight is zero. The document was found and then thrown away |
| No passage in the index answers the question | A refusal, or an invention | The corpus | No labelled document for this question is returned under any configuration. This is a content problem wearing a software costume |
Those four rows need four different people to fix them, and three of the four are invisible if the only thing being examined is the final answer. Separating them is the entire product.
The measures
Scroll the shelf sideways. None of these are ours, and that is the point of using them.
M01
recall@k
Of the documents a person marked relevant for this question, what share appeared anywhere in the top k. A pipeline cannot use what it never fetched, so this number is the ceiling on everything downstream of it.
The ceiling on the whole pipeline
M02
precision@k
Of the k documents returned, what share were actually relevant. Low precision fills a context window with material that has to be discarded, and in a generation pipeline the thing doing the discarding is a model that will never mention it.
What the wrong ones cost
M03
nDCG@k
Rank sensitive and grade sensitive. A relevant document at position one counts for more than the same document at position nine, discounted logarithmically, then divided by the best ordering that was available for that question.
Position and grade together
M04
MRR
One divided by the rank of the first relevant document, averaged across the query set. The right number to read when the reader stops at the first useful hit and everything below it is decoration.
First useful hit only
M05
success@k
The blunt one. What share of questions returned at least one relevant document at all. It earns its place because an average hides the fact that one question in twelve comes back with nothing usable.
How often nothing usable
M06
coverage
What share of the documents in the index are ever returned by any question in the set. A document that no question reaches is paying for storage and answering nobody, and no per query measure above can see it.
The corpus, not the query
M07
redundancy@k
What share of a result set is near duplicate passages from a single source, sitting in slots a second source could have used. High recall with high redundancy is a result set that looks healthy and says one thing.
Slots spent saying it twice
M08
split@k
How often the passage that actually answers the question was cut across two chunks, so that neither half scores well enough on its own to be retrieved. A chunking fault that presents as a retrieval fault.
A chunking fault in disguise
M09
rewrite-overlap
Where the pipeline rewrites a question before retrieval, how far the result set for the rewritten question overlaps the result set for the original. Rewriting is meant to help, and this is the number that says whether it did.
Did the rewrite wander
M10
stability
Whether the same question returns the same documents after the index is rebuilt, the embedding model is swapped, or the chunker is retuned. Change that nobody measured is change your users will find first.
What moved when you rebuilt
M11
kappa
Where two people judged the same question and document pair, how often they agreed once agreement by chance is taken out. Every number above is worth exactly as much as the labels underneath it.
Measuring the ruler itself
Eleven measures. The shelf scrolls sideways.
Recall and precision at a cut off, mean reciprocal rank and discounted cumulative gain are standard information retrieval measures. The experimental method they belong to goes back to the Cranfield studies of the 1960s and has been run annually at scale since the first Text Retrieval Conference in 1992. Normalised discounted cumulative gain in the form most people use was set out by Jarvelin and Kekalainen in 2002.
A company that claimed to have invented a new measure of retrieval quality would deserve a long sceptical look, and would need to publish the definition, the derivation and a reproduction before anyone believed it. We are making no such claim. The measures are public, old and well understood. What is missing on most teams is not a new measure. It is a labelled query set, a harness that runs it, and somebody looking at the result on a Tuesday.
Every number on that shelf needs relevance judgements underneath it, and relevance judgements are made by people reading documents. We have no trick for making that cheap, and any vendor who tells you they do should be asked to show the agreement rate between their labels and yours. That is what M11 exists for.
Boundaries
Three denials worth making in plain words, because the category is crowded with things that sound similar.
Arcus is not a chatbot. It is not a vector database. It is not a model provider. It measures, and it does not generate. There is no point in the design at which it composes a sentence for a user to read as an answer.
| Not this | Why the confusion happens | What does that job |
|---|---|---|
| A chatbot or an assistant | Two letters in the company name | A model provider behind whatever interface your team already chose. Nothing here produces natural language for an end user. The output is a table of numbers and a list of the questions that did worst |
| A vector database or a search index | Measuring retrieval sounds close to storing the things retrieved | The index you already run, whichever it is. This is designed to read result lists made of document identifiers, ranks and scores, and it has nowhere to put a corpus even if it wanted one |
| A model provider | Everything in this field is assumed to be selling inference | Whoever you already pay for embeddings and generation. Nothing here is trained, nothing is hosted, and no inference is served |
| An observability or performance product | Both of them watch a system that is running | Your existing tracing, logging and uptime tooling. Latency, error rate and cost per query are real numbers that matter, and none of them is retrieval quality. Confusing the two is how a slow pipeline gets described as an inaccurate one |
| A document store | Evaluating documents sounds like holding documents | Wherever your documents live now. The design intent is to hold identifiers, ranks, scores and labels, and never the document text |
| A public leaderboard | Benchmarks are the visible half of this field | Public benchmarks, which are useful for comparing methods and nearly useless for deciding whether your pipeline works. The measures here are computed on your questions against your corpus, which is the only comparison that decides anything on your team |
The pieces
Nothing below has been built. The chips say so individually so that no reader has to take the summary on trust.
Somewhere to keep the questions and the human judgements about which documents answer them.
The thing that replays a query set against a retrieval endpoint and records what came back.
The per question and aggregate table that the eleven measures are written into.
Re-run the same query set after a change and show what moved.
The measure that points at our own instrument rather than at your pipeline.
Grading the text a language model produced at the end of the pipeline.
Status
A status section belongs near the top of a page, not in small type at the bottom of one.
ARCUS AI PTY LTD was registered in New South Wales in 2026. It has published no software. There is nothing to download, nothing to sign up for, no private beta, no waiting list and no early access programme. Those things are frequently invented at this stage to make a page feel busier, and inventing one would be the first dishonest thing on this site.
What exists is a clear specification of the measures, a decision about where the product boundary sits, and prototype code that computes several of the measures over a query set we wrote ourselves. That code has never run against anybody else's corpus, has never been reviewed by anybody outside the company, and is not published.
| Question | Answer |
|---|---|
| Is there a product to buy | No |
| Is there code to download | No |
| Are there customers | None |
| Is there revenue | None |
| Is there outside investment | None |
| Are there published benchmark results | None. The prototype has only been run against a query set we wrote, which proves nothing about anybody else's corpus |
| Are there certifications | None held. See the note below |
| Is there a paper | No |
| What can you actually do here today | Read this, disagree with it, and write to us |
ARCUS AI PTY LTD does not hold ISO/IEC 27001 certification, a SOC 2 Type I or Type II report, an IRAP assessment or an Essential Eight attestation, and will not represent otherwise until one is genuinely held. It has published no benchmark results, no paper and no reproducible experiment. It has no customers, no revenue and no outside investment.
Conceptual
Written in the conditional throughout, because none of it runs today.
What follows describes a design. It is not a feature list, it is not a screenshot of anything, and no step below can be performed by anyone today.
| Would be recorded | Would never be recorded | Why the line sits there |
|---|---|---|
| Document identifiers and their ranks | Document text or passage contents | Every measure on the shelf is computed from identifiers and ranks. Document text adds nothing to the arithmetic and adds a great deal to what a breach would cost |
| Retrieval scores as returned by your system | Embedding vectors | A score is a number your system already produced. A vector can often be inverted back towards the text it came from |
| A stable identifier for each question | The question text, unless you choose to include it | Real user questions are frequently personal information. The join works on identifiers, so the text is optional by construction rather than by policy |
| Timing of the run and the configuration label | Credentials, connection strings or endpoint secrets | The harness would run where your credentials already are and would have no reason to carry them anywhere |
This is stated now, in advance, because a data boundary written before the code is a constraint and a data boundary written afterwards is a description. If any of it turns out to be unworkable, this page will say which line moved and why, rather than quietly changing.
Contact
There is no form on this website. There is an address, and a person reads it.
If you run a retrieval system and you already measure it, we would like to know what you measure and what you found. If you run one and you do not measure it, we would like to know what stopped you, because the answer is usually the labelling cost and we would rather hear that from you than assume it.
If you think the premise of this page is wrong, that is the most useful message we could receive. It costs nothing to send and it is worth more than agreement.
Write to contact@arcusai.fyi. Replies go out within five business days. Nothing about writing to us subscribes you to anything, because there is no list to be subscribed to.