eval/README.md

RAG Evaluation Module

This folder contains a lightweight retrieval-evaluation harness for the project.

Supported Steps

1. beir/scifact
2. beir/fiqa
3. open-ragbench
4. t2-ragbench
5. local-options

Each run builds a temporary Chroma index under eval/indexes/ and writes reports under eval/reports/.

Smoke Tests

uv --cache-dir .uv-cache run python -m eval.rag_eval --dataset beir/scifact --max-corpus-docs 200 --max-queries 10 --rebuild
uv --cache-dir .uv-cache run python -m eval.rag_eval --dataset beir/fiqa --max-corpus-docs 500 --max-queries 10 --rebuild
uv --cache-dir .uv-cache run python -m eval.rag_eval --dataset open-ragbench --max-corpus-docs 50 --max-queries 10 --rebuild
uv --cache-dir .uv-cache run python -m eval.rag_eval --dataset t2-ragbench --max-corpus-docs 50 --max-queries 10 --rebuild
uv --cache-dir .uv-cache run python -m eval.rag_eval --dataset local-options --max-queries 3 --rebuild

Run The Whole Suite

uv --cache-dir .uv-cache run python -m eval.run_eval_suite --rebuild

By default, the suite runs:

• beir/scifact
• beir/fiqa
• open-ragbench
• local-options

Useful options:

# Accurate run after changing PDF parsing, chunking, embedding, retrieval code, or sampling parameters.
uv --cache-dir .uv-cache run python -m eval.run_eval_suite --rebuild

# Faster run that reuses existing indexes.
uv --cache-dir .uv-cache run python -m eval.run_eval_suite

# Run only selected datasets.
uv --cache-dir .uv-cache run python -m eval.run_eval_suite --datasets local-options,beir/fiqa

# Override shared parameters for all selected datasets.
uv --cache-dir .uv-cache run python -m eval.run_eval_suite --top-k 10 --max-queries 20 --max-corpus-docs 1000

# Save a stable suite-level report name.
uv --cache-dir .uv-cache run python -m eval.run_eval_suite --output-name latest_rag_eval

The suite writes per-dataset reports and one aggregate report under eval/reports/.

Common Commands

Run with the default multilingual embedding model:

uv --cache-dir .uv-cache run python -m eval.run_eval_suite --rebuild

Use a custom embedding model for experiments:

RAG_EMBED_MODEL=intfloat/multilingual-e5-base \
uv --cache-dir .uv-cache run python -m eval.run_eval_suite \
  --datasets local-options \
  --top-k 5 \
  --output-name local_options_e5_base \
  --rebuild

Run the fastest local check while developing PDF parsing or chunking:

uv --cache-dir .uv-cache run python -m eval.run_eval_suite \
  --datasets local-options \
  --max-queries 3 \
  --top-k 5 \
  --rebuild

Run only the standard public retrieval smoke tests:

uv --cache-dir .uv-cache run python -m eval.run_eval_suite \
  --datasets beir/scifact,beir/fiqa \
  --rebuild

Run the financial benchmark only:

uv --cache-dir .uv-cache run python -m eval.run_eval_suite \
  --datasets beir/fiqa \
  --max-corpus-docs 1000 \
  --max-queries 50 \
  --top-k 5 \
  --rebuild

Run the PDF-like benchmark only:

uv --cache-dir .uv-cache run python -m eval.run_eval_suite \
  --datasets open-ragbench \
  --max-corpus-docs 100 \
  --max-queries 20 \
  --top-k 5 \
  --rebuild

Compare different top-k values:

uv --cache-dir .uv-cache run python -m eval.run_eval_suite \
  --datasets local-options \
  --top-k 3 \
  --output-name local_options_top3 \
  --rebuild

uv --cache-dir .uv-cache run python -m eval.run_eval_suite \
  --datasets local-options \
  --top-k 10 \
  --output-name local_options_top10 \
  --rebuild

Compare retrieval with and without reranker:

uv --cache-dir .uv-cache run python -m eval.run_eval_suite \
  --datasets local-options \
  --top-k 5 \
  --output-name local_options_no_reranker \
  --rebuild

uv --cache-dir .uv-cache run python -m eval.run_eval_suite \
  --datasets local-options \
  --top-k 5 \
  --use-reranker \
  --reranker-candidates 25 \
  --output-name local_options_with_reranker \
  --rebuild

Use a custom reranker model:

uv --cache-dir .uv-cache run python -m eval.run_eval_suite \
  --datasets beir/fiqa \
  --use-reranker \
  --reranker-model cross-encoder/ms-marco-MiniLM-L-6-v2 \
  --reranker-candidates 50 \
  --top-k 5 \
  --rebuild

Compare different chunk settings:

uv --cache-dir .uv-cache run python -m eval.run_eval_suite \
  --datasets local-options \
  --chunk-size 384 \
  --chunk-overlap 64 \
  --output-name local_options_chunk384 \
  --rebuild

uv --cache-dir .uv-cache run python -m eval.run_eval_suite \
  --datasets local-options \
  --chunk-size 768 \
  --chunk-overlap 128 \
  --output-name local_options_chunk768 \
  --rebuild

Run a larger, slower evaluation before reporting results:

uv --cache-dir .uv-cache run python -m eval.run_eval_suite \
  --datasets beir/scifact,beir/fiqa,open-ragbench,local-options \
  --max-corpus-docs 2000 \
  --max-queries 100 \
  --top-k 5 \
  --output-name full_rag_eval \
  --rebuild

Stop immediately when one dataset fails:

uv --cache-dir .uv-cache run python -m eval.run_eval_suite \
  --datasets beir/scifact,beir/fiqa,open-ragbench,local-options \
  --fail-fast \
  --rebuild

Run a single dataset directly without the suite wrapper:

uv --cache-dir .uv-cache run python -m eval.rag_eval \
  --dataset local-options \
  --max-queries 3 \
  --top-k 5 \
  --rebuild

Suggested Workflow

1. During development, run local-options with a small query count.
2. After changing PDF extraction, chunking, embeddings, or retrieval code, add --rebuild.
3. Before comparing two versions, use the same --datasets, --max-queries, --max-corpus-docs, --top-k, --chunk-size, and --chunk-overlap.
4. Use --output-name to save stable report names for before/after comparison.
5. When testing reranker, compare the same dataset once without --use-reranker and once with --use-reranker.

Metrics

• hit_at_1
• hit_at_3
• hit_at_5
• hit_at_k
• mrr
• ndcg_at_k

The public benchmarks test whether the eval pipeline works on standard datasets. The local-options benchmark is the project-specific check for PDF parsing, formula extraction, and section-aware chunking.