# RAG Evaluation Module This folder contains a lightweight retrieval-evaluation harness for the project. ## Supported Steps 1. `beir/scifact` 2. `beir/fiqa` 3. `open-ragbench` 4. `t2-ragbench` 5. `local-options` Each run builds a temporary Chroma index under `eval/indexes/` and writes reports under `eval/reports/`. ## Smoke Tests ```bash uv --cache-dir .uv-cache run python -m eval.rag_eval --dataset beir/scifact --max-corpus-docs 200 --max-queries 10 --rebuild uv --cache-dir .uv-cache run python -m eval.rag_eval --dataset beir/fiqa --max-corpus-docs 500 --max-queries 10 --rebuild uv --cache-dir .uv-cache run python -m eval.rag_eval --dataset open-ragbench --max-corpus-docs 50 --max-queries 10 --rebuild uv --cache-dir .uv-cache run python -m eval.rag_eval --dataset t2-ragbench --max-corpus-docs 50 --max-queries 10 --rebuild uv --cache-dir .uv-cache run python -m eval.rag_eval --dataset local-options --max-queries 3 --rebuild ``` ## Run The Whole Suite ```bash uv --cache-dir .uv-cache run python -m eval.run_eval_suite --rebuild ``` By default, the suite runs: - `beir/scifact` - `beir/fiqa` - `open-ragbench` - `local-options` Useful options: ```bash # Accurate run after changing PDF parsing, chunking, embedding, retrieval code, or sampling parameters. uv --cache-dir .uv-cache run python -m eval.run_eval_suite --rebuild # Faster run that reuses existing indexes. uv --cache-dir .uv-cache run python -m eval.run_eval_suite # Run only selected datasets. uv --cache-dir .uv-cache run python -m eval.run_eval_suite --datasets local-options,beir/fiqa # Override shared parameters for all selected datasets. uv --cache-dir .uv-cache run python -m eval.run_eval_suite --top-k 10 --max-queries 20 --max-corpus-docs 1000 # Save a stable suite-level report name. uv --cache-dir .uv-cache run python -m eval.run_eval_suite --output-name latest_rag_eval ``` The suite writes per-dataset reports and one aggregate report under `eval/reports/`. ## Common Commands Run with the default multilingual embedding model: ```bash uv --cache-dir .uv-cache run python -m eval.run_eval_suite --rebuild ``` Use a custom embedding model for experiments: ```bash RAG_EMBED_MODEL=intfloat/multilingual-e5-base \ uv --cache-dir .uv-cache run python -m eval.run_eval_suite \ --datasets local-options \ --top-k 5 \ --output-name local_options_e5_base \ --rebuild ``` Run the fastest local check while developing PDF parsing or chunking: ```bash uv --cache-dir .uv-cache run python -m eval.run_eval_suite \ --datasets local-options \ --max-queries 3 \ --top-k 5 \ --rebuild ``` Run only the standard public retrieval smoke tests: ```bash uv --cache-dir .uv-cache run python -m eval.run_eval_suite \ --datasets beir/scifact,beir/fiqa \ --rebuild ``` Run the financial benchmark only: ```bash uv --cache-dir .uv-cache run python -m eval.run_eval_suite \ --datasets beir/fiqa \ --max-corpus-docs 1000 \ --max-queries 50 \ --top-k 5 \ --rebuild ``` Run the PDF-like benchmark only: ```bash uv --cache-dir .uv-cache run python -m eval.run_eval_suite \ --datasets open-ragbench \ --max-corpus-docs 100 \ --max-queries 20 \ --top-k 5 \ --rebuild ``` Compare different `top-k` values: ```bash uv --cache-dir .uv-cache run python -m eval.run_eval_suite \ --datasets local-options \ --top-k 3 \ --output-name local_options_top3 \ --rebuild uv --cache-dir .uv-cache run python -m eval.run_eval_suite \ --datasets local-options \ --top-k 10 \ --output-name local_options_top10 \ --rebuild ``` Compare retrieval with and without reranker: ```bash uv --cache-dir .uv-cache run python -m eval.run_eval_suite \ --datasets local-options \ --top-k 5 \ --output-name local_options_no_reranker \ --rebuild uv --cache-dir .uv-cache run python -m eval.run_eval_suite \ --datasets local-options \ --top-k 5 \ --use-reranker \ --reranker-candidates 25 \ --output-name local_options_with_reranker \ --rebuild ``` Use a custom reranker model: ```bash uv --cache-dir .uv-cache run python -m eval.run_eval_suite \ --datasets beir/fiqa \ --use-reranker \ --reranker-model cross-encoder/ms-marco-MiniLM-L-6-v2 \ --reranker-candidates 50 \ --top-k 5 \ --rebuild ``` Compare different chunk settings: ```bash uv --cache-dir .uv-cache run python -m eval.run_eval_suite \ --datasets local-options \ --chunk-size 384 \ --chunk-overlap 64 \ --output-name local_options_chunk384 \ --rebuild uv --cache-dir .uv-cache run python -m eval.run_eval_suite \ --datasets local-options \ --chunk-size 768 \ --chunk-overlap 128 \ --output-name local_options_chunk768 \ --rebuild ``` Run a larger, slower evaluation before reporting results: ```bash uv --cache-dir .uv-cache run python -m eval.run_eval_suite \ --datasets beir/scifact,beir/fiqa,open-ragbench,local-options \ --max-corpus-docs 2000 \ --max-queries 100 \ --top-k 5 \ --output-name full_rag_eval \ --rebuild ``` Stop immediately when one dataset fails: ```bash uv --cache-dir .uv-cache run python -m eval.run_eval_suite \ --datasets beir/scifact,beir/fiqa,open-ragbench,local-options \ --fail-fast \ --rebuild ``` Run a single dataset directly without the suite wrapper: ```bash uv --cache-dir .uv-cache run python -m eval.rag_eval \ --dataset local-options \ --max-queries 3 \ --top-k 5 \ --rebuild ``` ## Suggested Workflow 1. During development, run `local-options` with a small query count. 2. After changing PDF extraction, chunking, embeddings, or retrieval code, add `--rebuild`. 3. Before comparing two versions, use the same `--datasets`, `--max-queries`, `--max-corpus-docs`, `--top-k`, `--chunk-size`, and `--chunk-overlap`. 4. Use `--output-name` to save stable report names for before/after comparison. 5. When testing reranker, compare the same dataset once without `--use-reranker` and once with `--use-reranker`. ## Metrics - `hit_at_1` - `hit_at_3` - `hit_at_5` - `hit_at_k` - `mrr` - `ndcg_at_k` The public benchmarks test whether the eval pipeline works on standard datasets. The `local-options` benchmark is the project-specific check for PDF parsing, formula extraction, and section-aware chunking.