Wiki

Production Search Evaluation

How the podcast archive evaluates production search with relevance checks, RAG quality, business metrics, A/B tests, and feedback loops.

Related Wiki Pages

Search Search, RAG, and Knowledge Systems Information Retrieval Vector Database vs Search Engine Evaluation

Production search evaluation is the practice of proving that a search system returns useful results under real product constraints. In the podcast archive, that means evaluating candidate retrieval and ranking. It also means checking hybrid signals, generated answers, latency, and business impact.

Use Search for the broader topic and Evaluation for model and product measurement across the archive. Here, the boundary is search systems that need to work in production, including vector search and RAG.

Link Map

Start with these related wiki pages:

The main podcast discussions are:

The guest experts are:

Daniel Svonava on search architecture and business metrics
Atita Arora on RAG evaluation and human review
Reem Mahmoud on production ML search systems

Common Definition

The archive converges on one definition: production search evaluation isn’t a single relevance score. Teams first check whether the retriever found plausible candidates. Then they check whether the ranker ordered them well. Finally, they check whether users and the business got the outcome they came for.

In Building Search Systems, Daniel Svonava separates fast candidate generation from ranking around 12:45. That split matters for evaluation because a ranker can’t fix documents that retrieval never returned. A good candidate set can still fail when ranking, filters, or business rules order it badly.

In Modern Search Systems, Atita Arora applies the same evaluation split to RAG. From 42:49 to 48:09 she separates model choice, chunking, retrieval count, and prompt context. She also treats citations, answer quality and end-user feedback as separate checks.

Guest Perspectives

Daniel pushes teams toward business metrics. At 1:01:25, he argues that search metrics become more valuable when they connect to the outcome a company cares about. Examples include revenue, contacts, clicks, and orders. Around 1:03:50, he adds offline tests and A/B tests. He also adds control groups and engineer-accessible metrics for fast iteration (Building Search Systems).

In Modern Search Systems, Atita starts one layer lower (40:01-48:09).

She checks these pieces of the system:

chunking strategy and overlap
embedding model and retrieval count
references, generated responses, and human review

Teams can use this checklist when they don’t yet have enough live traffic for reliable online experiments.

In Production ML Search, Reem Mahmoud emphasizes product constraints that ordinary semantic similarity can miss. Around 34:00-45:11, the episode covers recency, popularity, and filters. It also covers feature fusion and query-time weights. Teams need to evaluate those signals because each can improve one use case while harming another.

Evaluate Retrieval and Ranking Separately

Production search first retrieves a small candidate set, then ranks it. Daniel describes retrieval as narrowing the haystack. Ranking is a more model-heavy decision about the probability that a query and result match (Building Search Systems, 12:45).

Teams should test retrieval and ranking separately. Retrieval evaluation asks whether relevant items are present in the candidate set at all. Ranking evaluation asks whether the best items are near the top after scoring, reranking, filtering, or personalization. Mixing the two can hide failures: a team may tune ranking metrics while the retriever silently drops the most useful documents.

Tie Search Metrics to Product Outcomes

The archive warns against dashboards that matter only to the search team. In Building Search Systems, Daniel says around 1:01:25 that funding and trust improve when metrics connect to business performance. A marketplace can translate search success into contacts, clicks, orders, or other events that can be assigned business value.

This doesn’t mean every evaluation starts with revenue. Early-stage systems may use offline relevance judgments and click-through proxies. They may also use successful-event counts, latency targets, or human labels. Teams should make the proxy explicit, then test whether it predicts the product outcome.

Use Offline Tests Before Online Experiments

Offline tests give teams repeatable checks before they expose changes to users. Atita describes preparing representative questions and expected answers for RAG evaluation. Around 48:09, she compares pipeline responses with human review and semantic similarity (Modern Search Systems).

Daniel’s production-search discussion says offline tests give engineers fast feedback before or alongside A/B tests (Building Search Systems, 1:03:50). Teams can use offline tests for speed and diagnosis. Then they can use online experiments when the question is whether the change actually moves user behavior.

Evaluate Hybrid Signals as Tradeoffs

Hybrid search adds relevance signals beyond text similarity, including filters, recency, and popularity. It can also include behavioral data, personalization, and business constraints. The Production ML Search episode shows why this is hard around 34:00. A strict recency filter can remove a relevant article that’s just outside the cutoff, while a pure semantic match can return stale or unpopular results.

Teams should therefore check tradeoffs, not only aggregate relevance. Freshness, exact-match behavior, and semantic recall can point in different directions. Latency, diversity, personalization, and commercial constraints can pull the ranking in other directions. Guests distinguish hard requirements from soft scoring signals. Teams can then test query-time weights or ranking rules by context.

Evaluate RAG Answers Beyond Retrieval

RAG systems add another layer because the answer can fail even when retrieval succeeds. Atita’s podcast-transcript chatbot example retrieves chunks and augments a prompt. It asks the LLM to answer from context and can attach references to build user trust (Modern Search Systems, 42:49). Teams then need to look at chunk choice and citation usefulness. They also need to check answer correctness, unsupported claims, “I don’t know” behavior, and user feedback.

This is why Search, RAG, and Knowledge Systems treats search quality and RAG answer quality as related but separate. A search system can retrieve the right evidence and still generate a weak answer. A generative answer can sound fluent while hiding a retrieval miss.

These pages cover the surrounding search and evaluation topics: