diff --git a/.cursor/plans/ensemble_refactor_benchmark_be904eec.plan.md b/.cursor/plans/ensemble_refactor_benchmark_be904eec.plan.md new file mode 100644 index 0000000000000000000000000000000000000000..1b7b481f6c30ed5047d527157e32dc9edfd3ab60 --- /dev/null +++ b/.cursor/plans/ensemble_refactor_benchmark_be904eec.plan.md @@ -0,0 +1,267 @@ +--- +name: Ensemble refactor benchmark +overview: Compare Option A vs Option B, then reorganize the three research scripts into a shared `research/ensemble/` package (Option B) and add a clear run + benchmark workflow from CPU smoke tests through real-model ablations. +todos: + - id: scaffold-package + content: "Create research/ensemble/ package: pyproject.toml, workspace member, ensemble dependency group" + status: completed + - id: extract-shared + content: Extract shared modules (backends, memory, jepa, bridge, eval/metrics) from the three existing scripts + status: completed + - id: migrate-jepa + content: Move Ensemble + demo into jepa_ensemble.py; migrate eval_harness to eval/jepa_harness.py + status: completed + - id: migrate-world + content: Move WorldEnsemble into world_ensemble.py; add world_model.py and energy.py + status: completed + - id: world-harness + content: Add eval/world_harness.py with energy vs random vs oracle selector comparison + status: completed + - id: benchmark-data + content: Add research/data/benchmark-qa.jsonl and benchmark-kb.jsonl from lesson topics + status: completed + - id: run-docs + content: Add research/ensemble/README.md, scripts/smoke.sh, and .env.example ensemble vars with Tier 1–3 commands + status: completed + - id: cleanup-shims + content: Remove or thin re-export old top-level research/*.py files; verify smoke + toy harness pass + status: completed +isProject: false +--- + +# Ensemble package (Option B) + run/benchmark plan + +## Option A vs Option B + +| | **Option A** (folder split only) | **Option B** (shared package) | +|---|---|---| +| **Layout** | `research/ensemble/jepa/` + `research/ensemble/world_model/` — move files, fix imports | `research/ensemble/` with shared `backends.py`, `jepa.py`, `memory.py`, etc. | +| **Effort** | ~1 hour | ~3–4 hours | +| **Duplication** | Keeps ~400 lines duplicated (two LLM backends, two JEPAs, two VectorStores) | Single implementation; bugfixes apply to both ensembles | +| **Imports** | Still fragile (`from jepa.ensemble import ...`) | Clean: `from ensemble.jepa_ensemble import Ensemble` | +| **Benchmarking** | Works for JEPA only ([`research/eval_harness.py`](research/eval_harness.py)); world model has no harness | Shared [`research/ensemble/eval/metrics.py`](research/ensemble/eval/metrics.py) + harnesses for both tracks | +| **Future agent hook** | Harder — two divergent codepaths | One package to import from `libs/agent` later if needed | +| **When to pick A** | You only need tidy folders for a demo and will not touch world-model eval | — | +| **When to pick B** | You will run ablations, add world-model benchmarks, or train bridge checkpoints | **Your choice** | + +**Recommendation:** Go straight to **Option B under `research/ensemble/`** (experiments stay out of `libs/`). Option A is a reasonable **fallback** if time runs out: do the folder split and defer deduplication. + +```mermaid +flowchart LR + subgraph shared [research/ensemble shared] + backends[backends.py] + memory[memory.py] + jepa_mod[jepa.py] + bridge[bridge.py] + metrics[eval/metrics.py] + end + + subgraph jepa_track [JEPA track] + jepa_ens[jepa_ensemble.py] + jepa_eval[eval/jepa_harness.py] + end + + subgraph world_track [World-model track] + world_mod[world_model.py] + energy[energy.py] + world_ens[world_ensemble.py] + world_eval[eval/world_harness.py] + end + + backends --> jepa_ens + memory --> jepa_ens + jepa_mod --> jepa_ens + bridge --> jepa_ens + jepa_ens --> jepa_eval + metrics --> jepa_eval + + backends --> world_ens + memory --> world_ens + jepa_mod --> world_ens + world_mod --> world_ens + energy --> world_ens + bridge --> world_ens + world_ens --> world_eval + metrics --> world_eval +``` + +--- + +## Target layout + +``` +research/ + ensemble/ + pyproject.toml # workspace member, optional deps + README.md # run + benchmark commands + ensemble/ + __init__.py + backends.py # TinyBackend + HFBackend (from jepa file) + memory.py # Embedder, VectorStore, Router + jepa.py # JEPA module + bridge.py + jepa_ensemble.py # Ensemble class (from llm_emb_jepa_ensemble_pluggable.py) + world_model.py # WorldModel + energy.py # EnergyModel + world_ensemble.py # WorldEnsemble (from world_model_ensemble.py) + eval/ + __init__.py + metrics.py # EM, F1, paired_bootstrap (from eval_harness) + jepa_harness.py # ablation ladder + best-of-N (from eval_harness) + world_harness.py # NEW: energy vs random vs oracle on shared drafts + scripts/ + smoke.sh # toy CPU checks for both ensembles + data/ + education-lesson-chat.jsonl # existing + benchmark-qa.jsonl # NEW: short QA for eval (derived from lesson topics) + benchmark-kb.jsonl # NEW: 1–2 sentence facts per topic for RAG + finetune.py # unchanged +``` + +Delete after migration (or leave thin re-export shims for one release): +- [`research/llm_emb_jepa_ensemble_pluggable.py`](research/llm_emb_jepa_ensemble_pluggable.py) +- [`research/eval_harness.py`](research/eval_harness.py) +- [`research/world_model_ensemble.py`](research/world_model_ensemble.py) + +--- + +## Package wiring + +1. Add [`research/ensemble/pyproject.toml`](research/ensemble/pyproject.toml) with `name = "ensemble"`, `torch` required, `transformers`/`peft`/`accelerate` optional (same pattern as root [`finetune`](pyproject.toml) group). +2. Extend root [`pyproject.toml`](pyproject.toml): + - `[tool.uv.workspace] members` → add `"research/ensemble"` + - `[dependency-groups] ensemble = [...]` mirroring `finetune` + - Optional root dep: `"ensemble"` for `uv run` convenience +3. Add ensemble env vars to [`.env.example`](.env.example) (model path, QA/KB paths, checkpoint path). + +--- + +## How to try the models (3 tiers) + +### Tier 1 — Smoke (CPU, no HF download, ~30s) + +Validates imports and inference plumbing after refactor. + +```bash +uv sync --group ensemble +uv run --package ensemble python -m ensemble.jepa_ensemble tiny # train 50 steps + answer +uv run --package ensemble python -m ensemble.world_ensemble tiny # train 60 steps + answer +bash research/ensemble/scripts/smoke.sh +``` + +Uses `TinyBackend` / `TinyLLM` — random weights, synthetic segments. Confirms modules load and forward/generate paths work. + +### Tier 2 — Micro demo (real small model, ~2–5 min) + +Quick “does it run on GPU/CPU with a real tokenizer?” + +```bash +uv run --package ensemble python -m ensemble.jepa_ensemble Qwen/Qwen2.5-0.5B-Instruct +uv run --package ensemble python -m ensemble.world_ensemble Qwen/Qwen2.5-0.5B-Instruct +``` + +Or a local/finetuned path from [`models.yaml`](models.yaml) / `FINETUNE_OUT`: + +```bash +uv run --package ensemble python -m ensemble.jepa_ensemble ./models/finetuned/minicpm5-1b-lora-merged +``` + +### Tier 3 — Benchmark (ablation + significance) + +**JEPA ablation ladder** (existing logic, moved to `eval/jepa_harness.py`): + +```bash +# Toy benchmark (no download) +uv run --package ensemble python -m ensemble.eval.jepa_harness \ + --llm tiny --toy --limit 20 --n_drafts 8 + +# Real model + project-aligned QA +uv run --package ensemble python -m ensemble.eval.jepa_harness \ + --llm Qwen/Qwen2.5-0.5B-Instruct \ + --qa research/data/benchmark-qa.jsonl \ + --kb research/data/benchmark-kb.jsonl \ + --limit 50 --n_drafts 8 + +# With bridge-trained checkpoint (C5) +uv run --package ensemble python -m ensemble.eval.jepa_harness \ + --llm ./models/finetuned/minicpm5-1b-lora-merged \ + --qa research/data/benchmark-qa.jsonl \ + --kb research/data/benchmark-kb.jsonl \ + --ckpt ./checkpoints/ensemble_bridge.pt +``` + +**World-model benchmark** (new `eval/world_harness.py`, parallel API): + +- Same QA/KB inputs +- Compare selectors on shared drafts: `first | random | energy | oracle` +- Report mean energy gap and `P(energy > random)` via shared bootstrap + +**Optional continual-forgetting flag** (JEPA only): `--continual` as today. + +--- + +## Benchmark data + +Add small files derived from lesson topics in [`research/data/education-lesson-chat.jsonl`](research/data/education-lesson-chat.jsonl): + +**`benchmark-qa.jsonl`** (~8–10 rows): + +```json +{"question": "What is photosynthesis?", "answer": "Plants make food using sunlight, water, and carbon dioxide.", "domain": "science"} +``` + +**`benchmark-kb.jsonl`** (~8–10 rows): + +```json +{"text": "Photosynthesis: plants use sunlight, water, and CO2 to make glucose and release oxygen."} +``` + +Keeps eval aligned with the hackathon domain (education) without needing a large external QA set. + +--- + +## What “good” benchmark output looks like + +**JEPA harness** (from existing [`eval_harness.py`](research/eval_harness.py)): + +``` +config EM F1 lat(s) +C1_base 0.120 0.180 0.05 +C2_rag 0.240 0.310 0.06 +C3_rag_router 0.260 0.330 0.06 +C4_full_jepa 0.320 0.400 0.45 + +best-of-N selector comparison: + first EM=0.28 + random EM=0.31 + jepa EM=0.38 + oracle EM=0.52 + P(jepa > random) = 0.97 JEPA critic WORKS +``` + +On **toy + untrained weights**, deltas will be noise — smoke tier only checks the pipeline runs. Meaningful numbers need Tier 2/3 with a real LLM and/or a trained bridge checkpoint. + +**World harness** (new): same table shape but selectors `first | random | energy | oracle`. + +--- + +## Implementation notes (minimal scope) + +- Preserve existing public classes: `Ensemble`, `WorldEnsemble`, same `answer_ids` / `answer` APIs. +- `HFBackend` in JEPA track keeps LoRA adapter bank + router; world track keeps simpler `HFLLM` (no router) — unify only where behavior matches. +- Add `if __name__ == "__main__"` entrypoints on `jepa_ensemble.py` and `world_ensemble.py` (move demos from current files). +- No integration with Gradio/`libs/agent` in this pass — research-only per your choice. +- No new pytest suite unless requested; `smoke.sh` + toy harness are the acceptance check. + +--- + +## If time is short (Option A fallback) + +Skip deduplication; only: + +``` +research/ensemble/jepa/{ensemble.py, eval_harness.py} +research/ensemble/world_model/{ensemble.py} +``` + +Fix imports, add `README.md` with Tier 1–3 commands. Revisit shared modules post-hackathon. diff --git a/.cursor/plans/lm-eval_harness_integration_34af70c7.plan.md b/.cursor/plans/lm-eval_harness_integration_34af70c7.plan.md new file mode 100644 index 0000000000000000000000000000000000000000..af0dff84744866e73267e6af93e471d1c31b089b --- /dev/null +++ b/.cursor/plans/lm-eval_harness_integration_34af70c7.plan.md @@ -0,0 +1,339 @@ +--- +name: lm-eval harness integration +overview: Add EleutherAI lm-evaluation-harness as the primary academic benchmark track for finetuned HF/LoRA models and ensemble checkpoints, with preset-aware CLI, fair-comparison configs, and structured results — building on existing slm-evals and ensemble harnesses without replacing them. +todos: + - id: deps-lm-eval + content: Add lm-eval[hf] optional dependency group to root pyproject.toml and slm-evals extras + status: completed + - id: preset-resolver + content: Implement preset_resolver.py mapping models.yaml presets to lm-eval model_args (base, LoRA, merged) + status: completed + - id: ensemble-lm-backend + content: Implement ensemble-lm custom LM class (generate_until + loglikelihood via underlying HF LLM) + status: completed + - id: slm-lm-eval-cli + content: Build run_lm_eval.py CLI with YAML config, simple_evaluate(), results + summary.md + --compare-to + status: completed + - id: lm-eval-configs + content: Add lm_eval_minicpm5.yaml, lm_eval_smoke.yaml, lm_eval_compare_study.yaml under research/evals/configs/ + status: completed + - id: docs-workflows + content: Document finetune + ensemble lm-eval workflows in research/evals/USAGE.md and research/USAGE.md + status: completed + - id: finetune-hook + content: "Optional: add --lm-eval-after to finetune.py after CLI smoke tests pass" + status: completed + - id: phase2-verify + content: "Deferred: slm-compare, education_qa, harness JSON export from verification pipeline plan" + status: cancelled +isProject: false +--- + +# lm-evaluation-harness Integration for research/ Models + +## Context: what you already have + +Your repo runs **three eval tracks** today; none use standard academic benchmarks (GSM8K, ARC, HellaSwag): + +| Track | Tool | Best for | +|-------|------|----------| +| Agentic | [`slm-benchmark`](research/evals/src/slm_evals/run_benchmark.py) | BFCL, τ-bench, GAIA, SWE | +| Finetune training | [`finetune.py`](research/finetune.py) | eval_loss / perplexity only | +| Ensemble components | [`jepa_harness`](research/ensemble/src/ensemble/eval/jepa_harness.py) | RAG/router/JEPA ablation on custom QA | + +The existing [Model Verification Pipeline plan](.cursor/plans/model_verification_pipeline_ed9d35ab.plan.md) defers lm-eval; per your choice, **lm-eval comes first**. `slm-compare` / `education_qa` stay Phase 2. + +```mermaid +flowchart TB + subgraph models [Model artifacts] + basePreset[models.yaml preset] + loraDir[LoRA adapter dir] + mergedDir[Merged checkpoint] + ensCkpt[Ensemble manifest checkpoint] + end + + subgraph newTrack [NEW: lm-eval track] + resolver[PresetResolver] + slmLmEval[slm-lm-eval CLI] + hfLm[lm_eval hf backend] + ensLm[ensemble-lm custom backend] + tasks[Standard tasks] + results[results/lm_eval/] + end + + subgraph existing [Existing tracks unchanged] + agentic[slm-benchmark] + ablation[jepa_harness] + end + + basePreset --> resolver + loraDir --> resolver + mergedDir --> resolver + resolver --> hfLm + ensCkpt --> ensLm + slmLmEval --> hfLm + slmLmEval --> ensLm + hfLm --> tasks + ensLm --> tasks + tasks --> results +``` + +--- + +## Benchmark matrix (what to run for each claim) + +Match tasks to model size (~1B) and claim type: + +| Model | lm-eval tasks (primary) | Existing complement | +|-------|-------------------------|---------------------| +| **Base preset** (`minicpm5-1b`, `gemma4-e2b-mobile`) | `arc_easy`, `arc_challenge`, `hellaswag`, `piqa`, `boolq`, `gsm8k` | `slm-benchmark --benchmarks bfcl,tau_bench` | +| **Finetuned LoRA** (`minicpm5-1b-lesson-lora`) | Same tasks; `pretrained=base,peft=adapter` | Same agentic suite | +| **Merged finetune** (`minicpm5-1b-lesson-merged`, `gemma-merged-local`) | Same tasks; `pretrained=merged_path` | Same | +| **Ensemble checkpoint** | Same generative + MC tasks via custom backend | `jepa_harness` for component ablation; `slm-benchmark --model-type ensemble` for agentic E2E | + +**Smoke profile** (CPU/GPU quick check, ~5–15 min): `--tasks arc_easy,hellaswag --num_fewshot 0 --limit 50` + +**Full profile** (reportable): `--tasks arc_easy,arc_challenge,hellaswag,piqa,boolq,gsm8k --num_fewshot 5` (gsm8k uses 8-shot per harness default) + +**Fair comparison rules** (from your guide, enforced in YAML): +- Identical `tasks`, `num_fewshot`, `limit`, `seed`, `batch_size` +- Same base tokenizer (preset resolution guarantees this for LoRA) +- `temperature=0` / greedy decoding (lm-eval default for MC; gsm8k generative) +- Never compare `training_results.json` `result_score` to lm-eval accuracy + +--- + +## Phase 1 — Dependencies and install surface + +**Root [`pyproject.toml`](pyproject.toml)** — add optional group: + +```toml +lm-eval = [ + "lm-eval[hf]>=0.4.9", +] +``` + +Install: `uv sync --group evals --group lm-eval --group finetune` + +Keep lm-eval **optional** so agentic-only workflows stay lightweight. + +--- + +## Phase 2 — Preset → lm-eval model_args resolver + +New module: `research/evals/src/slm_evals/lm_eval/preset_resolver.py` + +Reuse [`inference.config.get_model_config`](libs/inference/src/inference/config.py) (same as [`finetune.py`](research/finetune.py)): + +| Preset shape | lm-eval invocation | +|--------------|-------------------| +| `model_id` only (base) | `pretrained={model_id},trust_remote_code=True` | +| `model_id` + `adapter_path` (LoRA) | `pretrained={model_id},peft={adapter_path},trust_remote_code=True` | +| Local merged dir | `pretrained={model_id},trust_remote_code=True` | +| Ensemble (`jepa-ensemble-lesson` or path with `manifest.json`) | `--model ensemble-lm` (custom backend) | + +Reject multimodal / llama_cpp presets with a clear error (same rule as finetune). + +--- + +## Phase 3 — CLI wrapper `slm-lm-eval` + +New entry: `research/evals/src/slm_evals/run_lm_eval.py` +Register in [`research/evals/pyproject.toml`](research/evals/pyproject.toml): + +```toml +[project.scripts] +slm-lm-eval = "slm_evals.run_lm_eval:main" +``` + +**Flags:** + +```bash +uv run --package slm-evals slm-lm-eval \ + --config research/evals/configs/lm_eval_minicpm5.yaml \ + --preset minicpm5-1b \ + --experiment-name minicpm5-1b__lm-eval-baseline +``` + +| Flag | Purpose | +|------|---------| +| `--preset` | Resolve from [`models.yaml`](models.yaml) | +| `--model` | Override path/Hub id (merged dir or ensemble ckpt) | +| `--adapter` | Override LoRA path (alternative to preset) | +| `--config` | YAML: tasks, num_fewshot, limit, seed, device, batch_size | +| `--tasks` | CLI override of task list | +| `--compare-to` | Path to prior `results.json` → print delta table (lightweight pre-`slm-compare`) | +| `--experiment-name` / `--output-dir` | Write under `results/lm_eval/{name}/` | + +Implementation: subprocess or programmatic call to `lm_eval.simple_evaluate()` (preferred over shelling to `lm_eval` CLI — easier to inject custom model registration). + +Output artifacts per run: +- `results.json` (lm-eval native) +- `summary.md` (human-readable table: task → acc/score) +- `run_meta.json` (preset, base_model, adapter_path, tasks, seed, git hash optional) + +--- + +## Phase 4 — Ensemble custom backend + +New file: `research/evals/src/slm_evals/lm_eval/ensemble_lm.py` + +Register with lm-eval: + +```python +@register_model("ensemble-lm") +class EnsembleLM(LM): + ... +``` + +Load via existing [`load_ensemble_model`](research/evals/src/slm_evals/utils/model_loader.py) / `ensemble.checkpoint.load_checkpoint`. + +| lm-eval method | Implementation | +|----------------|----------------| +| `generate_until` | `ens.generate_text(prompt, max_new_tokens=..., temperature=0)` — full JEPA+RAG+router stack | +| `loglikelihood` | Delegate to underlying `ens.llm` HF model (default adapter index 0) for MC tasks; document that this evaluates **base LLM head**, not selector — pair with `generate_until` scores for full-stack generative tasks | + +This split is intentional: MC benchmarks need token logprobs; the ensemble’s value on generative QA shows up in `generate_until` (gsm8k) and in [`jepa_harness`](research/ensemble/src/ensemble/eval/jepa_harness.py). + +Import side-effect: ensure `ensemble_lm` is imported before `simple_evaluate()` so registration runs. + +--- + +## Phase 5 — Experiment configs + +Add under `research/evals/configs/`: + +**`lm_eval_minicpm5.yaml`** — baseline template: + +```yaml +tasks: + - arc_easy + - arc_challenge + - hellaswag + - piqa + - boolq + - gsm8k +num_fewshot: 5 # gsm8k harness may override to 8 internally +limit: null # null = full; 100 for dev +seed: 42 +batch_size: auto +device: auto +dtype: bfloat16 +trust_remote_code: true +``` + +**`lm_eval_smoke.yaml`** — `limit: 25`, tasks `[arc_easy, hellaswag]` + +**`lm_eval_compare_study.yaml`** — documents baseline + candidate preset names and shared settings + +--- + +## Phase 6 — End-to-end workflows (finetune + ensemble) + +### Finetuned model verification + +```bash +# 1. Baseline (same config, pinned seed) +uv run --package slm-evals slm-lm-eval \ + --config research/evals/configs/lm_eval_minicpm5.yaml \ + --preset minicpm5-1b \ + --experiment-name minicpm5-1b__baseline + +# 2. Train +uv run python research/finetune.py --preset minicpm5-1b --mode lora --epochs 3 + +# 3. Candidate — LoRA via preset (no merge required) +uv run --package slm-evals slm-lm-eval \ + --config research/evals/configs/lm_eval_minicpm5.yaml \ + --preset minicpm5-1b-lesson-lora \ + --experiment-name minicpm5-1b-lora__v1 \ + --compare-to results/lm_eval/minicpm5-1b__baseline/results.json +``` + +For Gemma: use `gemma-lora-local` / `gemma-merged-local` presets after notebook or `finetune.py` training. + +### Ensemble verification + +```bash +# Component ablation (domain QA) — existing +uv run --package ensemble python -m ensemble.eval.jepa_harness \ + --llm openbmb/MiniCPM5-1B \ + --qa research/data/benchmark-qa.jsonl \ + --kb research/data/benchmark-kb.jsonl + +# Academic benchmarks on saved ensemble +uv run --package slm-evals slm-lm-eval \ + --config research/evals/configs/lm_eval_minicpm5.yaml \ + --model ./models/ensemble/jepa-lesson-pretrain \ + --experiment-name ensemble-jepa__lm-eval +``` + +### Optional post-finetune hook (minimal, Phase 1.5) + +Add to [`finetune.py`](research/finetune.py) only after CLI stabilizes: + +- `--lm-eval-after` + `--lm-eval-config` → subprocess `slm-lm-eval` on output checkpoint +- Append `lm_eval_summary` path to `training_results.json` + +Defer `--eval-baseline` auto-compare until Phase 2 `slm-compare` exists; use `--compare-to` on lm-eval outputs in the meantime. + +--- + +## Phase 7 — Documentation + +Update: +- [`research/evals/USAGE.md`](research/evals/USAGE.md) — lm-eval section, PEFT notes, task profiles +- [`research/USAGE.md`](research/USAGE.md) — unified “verify finetune / ensemble” checklist +- [`.env.example`](.env.example) — optional `LM_EVAL_TASKS`, `LM_EVAL_SEED` (low priority) + +Include the verification checklist from your guide (seeds, fair comparison, no val-set leakage) mapped to concrete commands. + +--- + +## File change summary + +| File | Change | +|------|--------| +| [`pyproject.toml`](pyproject.toml) | `lm-eval` dependency group | +| [`research/evals/pyproject.toml`](research/evals/pyproject.toml) | Optional `lm-eval[hf]` extra; `slm-lm-eval` script | +| `research/evals/src/slm_evals/lm_eval/__init__.py` | Package init | +| `research/evals/src/slm_evals/lm_eval/preset_resolver.py` | Preset → model_args | +| `research/evals/src/slm_evals/lm_eval/ensemble_lm.py` | Custom `ensemble-lm` backend | +| `research/evals/src/slm_evals/run_lm_eval.py` | Main CLI | +| `research/evals/configs/lm_eval_*.yaml` | Baseline, smoke, compare templates | +| [`research/finetune.py`](research/finetune.py) | Optional `--lm-eval-after` (after CLI stable) | +| [`research/evals/USAGE.md`](research/evals/USAGE.md), [`research/USAGE.md`](research/USAGE.md) | Workflows | + +--- + +## Testing plan + +1. **Import smoke**: `uv run --package slm-evals python -c "import lm_eval; import slm_evals.lm_eval.ensemble_lm"` +2. **HF base**: `slm-lm-eval --config lm_eval_smoke.yaml --preset minicpm5-1b --limit 10` +3. **LoRA**: same with `--preset minicpm5-1b-lesson-lora` (or `--adapter ./models/finetuned/...`) +4. **Ensemble**: `slm-lm-eval --model ./models/ensemble/... --config lm_eval_smoke.yaml` +5. **Compare**: baseline + candidate runs produce `--compare-to` delta without task/seed mismatch warnings +6. **Regression**: existing `slm-benchmark` and `jepa_harness --toy` still pass unchanged + +--- + +## Phase 2 (deferred — existing verification plan) + +After lm-eval stabilizes, implement from [model_verification_pipeline plan](.cursor/plans/model_verification_pipeline_ed9d35ab.plan.md): + +- `slm-compare` with paired bootstrap across **both** lm-eval and slm-benchmark JSON +- `education_qa` domain benchmark in slm-evals +- PEFT loading in `slm-benchmark` model_loader (reuse `preset_resolver`) +- Harness JSON export (`jepa_harness --output-dir`) + +This gives you: **academic generalization** (lm-eval) + **agentic capability** (slm-evals) + **ensemble component proof** (jepa harness) with shared statistical comparison. + +--- + +## Expected limitations (document, don’t hide) + +- **1B models** will score low on gsm8k/mmlu — use for **relative** baseline vs finetune deltas, not SOTA claims +- **Ensemble loglikelihood** uses underlying LLM, not full selector stack — report both modes in docs +- **MiniCPM / Gemma** need `trust_remote_code=True` +- **First lm-eval run** downloads datasets; pin `HF_HOME` / cache for reproducibility +- **Multi-seed training** (3–5 seeds) is manual until Phase 2 compare aggregates runs diff --git a/.cursor/plans/model_verification_pipeline_ed9d35ab.plan.md b/.cursor/plans/model_verification_pipeline_ed9d35ab.plan.md new file mode 100644 index 0000000000000000000000000000000000000000..fd6f9a844976516d94f96d408283b761808d4a67 --- /dev/null +++ b/.cursor/plans/model_verification_pipeline_ed9d35ab.plan.md @@ -0,0 +1,280 @@ +--- +name: Model Verification Pipeline +overview: Operationalize model verification for both fine-tuned SLMs and ensemble research by adding comparison tooling, preset-aware eval, a lesson-domain benchmark, reproducible experiment configs, and optional post-finetune eval — all grounded in your existing `slm-evals` and ensemble harness infrastructure. +todos: + - id: eval-infra + content: Add preset resolution, PEFT loading, eval_seed sample pinning, and education_qa benchmark to slm-evals + status: pending + - id: compare-cli + content: Implement slm-compare with paired bootstrap, delta tables, and subset mismatch warnings + status: pending + - id: finetune-hook + content: Add --eval-after / --eval-baseline flags to finetune.py with subprocess eval + compare + status: pending + - id: ensemble-json + content: Persist jepa/world harness results to JSON (--output-dir) using shared stats helpers + status: pending + - id: docs-configs + content: Add baseline_minicpm5.yaml template and verification checklist to research/ USAGE docs + status: pending +isProject: false +--- + +# Model Verification Pipeline for research/ + +## Current state + +Your repo already has **three parallel eval tracks** with no unified comparison: + +| Track | Tool | Metrics | Results | Stats | +|-------|------|---------|---------|-------| +| Fine-tuned SLM | [`slm-benchmark`](research/evals/src/slm_evals/run_benchmark.py) | BFCL, τ-bench, GAIA, SWE pass rate | `results//results.json` | None | +| Finetune training | [`finetune.py`](research/finetune.py) | eval_loss, perplexity, `result_score` | `training_results.json` | None | +| Ensemble | [`jepa_harness`](research/ensemble/src/ensemble/eval/jepa_harness.py) / [`world_harness`](research/ensemble/src/ensemble/eval/world_harness.py) | EM, F1, ablation ladder | stdout only | [`paired_bootstrap`](research/ensemble/src/ensemble/eval/metrics.py) | + +The manual before/after loop in [`research/evals/USAGE.md`](research/evals/USAGE.md) works but lacks significance testing, preset resolution, and a domain-aligned benchmark for lesson fine-tuning. + +```mermaid +flowchart TB + subgraph today [Today] + baseline[slm-benchmark baseline] + finetune[finetune.py] + reeval[slm-benchmark new model] + manual[Manual diff results.csv] + baseline --> manual + reeval --> manual + finetune --> reeval + end + + subgraph target [Target pipeline] + preset[models.yaml preset] + evalRun[slm-benchmark + eval_seed] + compare[slm-compare] + report[comparison_report.md] + preset --> evalRun + evalRun --> compare + compare --> report + end +``` + +--- + +## Verification strategy (mapped to your repo) + +### 1. Fair comparison checklist (enforce via shared eval config) + +Create a **single YAML experiment config** per comparison study (copy from [`experiment_001.yaml`](research/evals/configs/experiment_001.yaml)) and reuse it for baseline + candidate runs: + +- Same `benchmarks`, `max_samples`, `benchmark_overrides`, `temperature: 0.0`, `max_new_tokens` +- Same `eval_seed` (new field) so sample subsets are identical across models +- Baseline = base preset from [`models.yaml`](models.yaml) (e.g. `minicpm5-1b`); candidate = LoRA/merged preset (e.g. `minicpm5-1b-lesson-lora`) + +**Do not** compare `training_results.json` `result_score` against BFCL pass rate — they measure different things. + +### 2. Benchmark selection for your use cases + +| Claim | Benchmarks in this repo | +|-------|-------------------------| +| Agentic tool use (general) | `bfcl`, `tau_bench` (already in slm-evals) | +| End-to-end assistant | `gaia` (levels 1–2 for small models) | +| Code | `swe_bench` (keep `full_eval: false` unless Docker installed) | +| Lesson/education domain (finetune target) | **New** `education_qa` benchmark on [`research/data/benchmark-qa.jsonl`](research/data/benchmark-qa.jsonl) | +| Ensemble component value | Existing C1→C4 ablation ladder in JEPA harness | +| JEPA critic beats random | Existing selector comparison + `paired_bootstrap` | + +Run **in-distribution** (`education_qa`) + **out-of-distribution** (`bfcl`, `tau_bench`) to show generalization, not just lesson memorization. + +### 3. Statistical validation + +Reuse existing `paired_bootstrap` from [`metrics.py`](research/ensemble/src/ensemble/eval/metrics.py) in a new shared module. For each benchmark: + +- Align per-sample `passed` (or `score`) arrays by `samples[].id` +- Report: Δscore, win rate, `P(candidate > baseline)` from bootstrap, 95% CI via bootstrap percentiles +- Flag `p > 0.95` as "significant improvement" (same threshold as JEPA harness) + +For multi-seed training runs: run eval once per checkpoint seed, aggregate mean ± std in comparison report (optional `--runs` glob in compare CLI). + +--- + +## Implementation plan + +### Phase A — Shared eval infrastructure (`research/evals/`) + +**A1. Preset resolution in config loader** + +Extend [`config_loader.py`](research/evals/src/slm_evals/utils/config_loader.py) and CLI in [`run_benchmark.py`](research/evals/src/slm_evals/run_benchmark.py): + +- Add `--preset` flag and YAML field `preset:` (alternative to `model_path`) +- Resolve via existing [`inference.config.get_model_config`](libs/inference/src/inference/config.py): extract `model_id` + optional `adapter_path` +- Reject non-`transformers` / multimodal presets with clear error (same rule as finetune) + +**A2. PEFT adapter loading** + +Extend [`model_loader.py`](research/evals/src/slm_evals/utils/model_loader.py): + +- If `adapter_path` is set: load base from `model_id`, attach LoRA via `peft.PeftModel.from_pretrained` +- Support merged checkpoints (adapter_path absent) unchanged +- Record `base_model`, `adapter_path`, `param_count` in results metadata + +**A3. Reproducible sample subsets** + +Extend [`BaseBenchmark`](research/evals/src/slm_evals/benchmarks/base.py): + +- Accept `eval_seed` + `max_samples` from config +- After loading dataset: `rng = random.Random(eval_seed); indices = rng.sample(range(len(data)), min(max_samples, len(data)))` +- Persist `eval_seed`, `sample_ids` list in `results.json` so compare can verify identical subsets + +**A4. Lesson-domain benchmark** + +New file `research/evals/src/slm_evals/benchmarks/education_qa.py`: + +- Load [`research/data/benchmark-qa.jsonl`](research/data/benchmark-qa.jsonl) +- Prompt: `"Answer briefly.\nQ: {question}\nA:"` +- Score: token-overlap F1 + normalized substring EM (reuse logic from ensemble `metrics.py` — extract to shared `slm_evals/utils/scoring.py` or import from ensemble if dependency is acceptable) +- Register as `education_qa` in `BENCHMARK_REGISTRY` + +**A5. Comparison CLI — `slm-compare`** + +New module `research/evals/src/slm_evals/compare_runs.py` + console script in [`pyproject.toml`](research/evals/pyproject.toml): + +```bash +uv run --package slm-evals slm-compare \ + --baseline results/minicpm5-1b__baseline/results.json \ + --candidate results/minicpm5-1b-lora__v1/results.json \ + --output results/comparisons/minicpm5-lora-vs-base.md +``` + +Outputs: +- Per-benchmark delta table (score, passed/total, latency) +- Paired bootstrap p-value per benchmark +- Per-sample win/loss/tie counts (joined on sample `id`) +- Warnings if `eval_seed`, `max_samples`, or benchmark sets differ + +**A6. Experiment config templates** + +Add two configs under `research/evals/configs/`: + +- `baseline_minicpm5.yaml` — preset `minicpm5-1b`, benchmarks `[education_qa, bfcl, tau_bench]`, `max_samples: 100`, `eval_seed: 42` +- `compare_study.yaml` — documents baseline + candidate preset keys, shared eval settings, output naming convention + +Update [`research/evals/USAGE.md`](research/evals/USAGE.md) and [`research/USAGE.md`](research/USAGE.md) with the verification checklist from your research guide. + +--- + +### Phase B — Finetune integration (`research/finetune.py`) + +**B1. Post-finetune eval hook** + +Add optional flags to [`finetune.py`](research/finetune.py): + +- `--eval-after` — run slm-benchmark after training completes +- `--eval-config PATH` — YAML with benchmark settings (defaults to `baseline_minicpm5.yaml` structure) +- `--eval-baseline PRESET` — also eval base preset for side-by-side comparison + +On completion: +1. Write `training_results.json` (existing) +2. Run eval on output checkpoint +3. If `--eval-baseline` set, run baseline eval + invoke `slm-compare` +4. Append comparison summary path to `training_results.json` under `"post_eval": {...}` + +Implementation: subprocess call to `uv run --package slm-evals slm-benchmark` (avoids circular imports). + +--- + +### Phase C — Ensemble track unification (`research/ensemble/`) + +**C1. Persist harness results to JSON** + +Extend [`jepa_harness.py`](research/ensemble/src/ensemble/eval/jepa_harness.py) and [`world_harness.py`](research/ensemble/src/ensemble/eval/world_harness.py): + +- Add `--output-dir` flag +- Write `results.json` matching slm-evals schema where possible: + - `benchmarks.C4_full_jepa.samples[]` with per-question EM/F1 + - `benchmarks.C1_base` … `C4_full_jepa` aggregate scores + - `significance[]` with paired bootstrap between ladder steps + +**C2. Compare ensemble configs** + +`slm-compare` accepts ensemble JSON (detect by presence of ablation config keys) and renders ablation table + bootstrap block. + +**C3. Shared metrics module** + +Move or re-export `em_score`, `f1_score`, `paired_bootstrap` to `research/evals/src/slm_evals/utils/stats.py` (ensemble imports from there, or duplicate minimally to avoid cross-package coupling — prefer a tiny shared module under `research/` if both packages need it). + +--- + +### Phase D — End-to-end workflow documentation + +Add a **Verification Checklist** section to [`research/README.md`](research/README.md): + +```bash +# 1. Baseline (preset-aware, pinned seed) +uv run --package slm-evals slm-benchmark \ + --config research/evals/configs/baseline_minicpm5.yaml \ + --preset minicpm5-1b \ + --experiment-name minicpm5-1b__baseline + +# 2. Fine-tune +uv run python research/finetune.py --preset minicpm5-1b --mode lora --epochs 3 + +# 3. Candidate eval (same config, different preset/path) +uv run --package slm-evals slm-benchmark \ + --config research/evals/configs/baseline_minicpm5.yaml \ + --preset minicpm5-1b-lesson-lora \ + --experiment-name minicpm5-1b-lora__v1 + +# 4. Statistical comparison +uv run --package slm-evals slm-compare \ + --baseline results/minicpm5-1b__baseline/results.json \ + --candidate results/minicpm5-1b-lora__v1/results.json + +# 5. Ensemble ablation (domain QA) +uv run --package ensemble python -m ensemble.eval.jepa_harness \ + --llm openbmb/MiniCPM5-1B \ + --qa research/data/benchmark-qa.jsonl \ + --kb research/data/benchmark-kb.jsonl \ + --output-dir results/ensemble/jepa-ablation-v1 +``` + +**Claiming "better than baseline" requires:** +- Δscore > 0 on target benchmarks +- `P(candidate > baseline) > 0.95` on paired per-sample scores (or document inconclusive) +- No regression > 2pp on OOD benchmarks unless tradeoff is explicitly claimed +- Ablation shows which components contribute (ensemble) or which eval split improved (finetune) + +--- + +## File change summary + +| File | Change | +|------|--------| +| [`research/evals/src/slm_evals/utils/config_loader.py`](research/evals/src/slm_evals/utils/config_loader.py) | `preset`, `eval_seed`, adapter fields | +| [`research/evals/src/slm_evals/utils/model_loader.py`](research/evals/src/slm_evals/utils/model_loader.py) | PEFT adapter loading | +| [`research/evals/src/slm_evals/benchmarks/base.py`](research/evals/src/slm_evals/benchmarks/base.py) | Seeded sample selection | +| `research/evals/src/slm_evals/benchmarks/education_qa.py` | New lesson-domain benchmark | +| `research/evals/src/slm_evals/compare_runs.py` | New comparison CLI | +| `research/evals/src/slm_evals/utils/stats.py` | Bootstrap + scoring helpers | +| [`research/evals/pyproject.toml`](research/evals/pyproject.toml) | Register `slm-compare` script | +| [`research/finetune.py`](research/finetune.py) | `--eval-after`, `--eval-config`, `--eval-baseline` | +| [`research/ensemble/src/ensemble/eval/jepa_harness.py`](research/ensemble/src/ensemble/eval/jepa_harness.py) | `--output-dir`, JSON persistence | +| [`research/ensemble/src/ensemble/eval/world_harness.py`](research/ensemble/src/ensemble/eval/world_harness.py) | Same | +| [`research/evals/configs/baseline_minicpm5.yaml`](research/evals/configs/baseline_minicpm5.yaml) | New template | +| [`research/USAGE.md`](research/USAGE.md), [`research/evals/USAGE.md`](research/evals/USAGE.md) | Verification workflow docs | + +--- + +## Testing plan + +- **Unit**: `paired_bootstrap` on synthetic paired arrays; sample-id alignment in compare; preset resolution for `minicpm5-1b` and `minicpm5-1b-lesson-lora` +- **Smoke**: `slm-benchmark --preset minicpm5-1b --benchmarks education_qa --max-samples 5 --device cpu` +- **Integration**: baseline → finetune (1 epoch) → re-eval → `slm-compare` produces report with no subset mismatch warnings +- **Ensemble**: `jepa_harness --toy --output-dir /tmp/jepa-test` writes valid JSON readable by `slm-compare` + +--- + +## Out of scope (defer) + +- W&B / MLflow experiment tracking +- lm-evaluation-harness integration (GSM8K/MMLU — different from agentic suite; add later as separate benchmark group if needed) +- Running ensemble inference through BFCL (requires adapter wrapper, not just raw HF model) +- CI regression gates (can add after compare CLI stabilizes) diff --git a/.env.example b/.env.example index 81223738a3d486a26872294802f2cd83ec3aafac..e7e5bae7665a0b4466ba77d2867eeaf7a5498f05 100644 --- a/.env.example +++ b/.env.example @@ -28,3 +28,28 @@ ALLOW_MODEL_SWITCH=false # Optional: local fine-tuned merged weights # ACTIVE_MODEL=gemma-merged-local # MODEL_ID=./gemma_merged_model + +# --- Fine-tuning (research/finetune.py) --- +# FINETUNE_PRESET=minicpm5-1b +# FINETUNE_MODEL=openbmb/MiniCPM5-1B +# FINETUNE_DATASET=./research/data/education-lesson-chat.jsonl +# FINETUNE_DATASET=tatsu-lab/alpaca +# FINETUNE_DATASET_CONFIG= +# FINETUNE_DATASET_SPLIT=train +# FINETUNE_MAX_SAMPLES=500 +# FINETUNE_OUT=./models/finetuned/minicpm5-1b-lora +# FINETUNE_FORMAT=chat +# After training, point Gradio at the adapter preset: +# ACTIVE_MODEL=minicpm5-1b-lesson-lora + +# --- Ensemble research (research/ensemble/) --- +# Base LLM resolution (first match wins): ENSEMBLE_LLM, LLM_PATH, BASE, MODEL_ID, ACTIVE_MODEL +# LLM_PATH=./models/finetuned/minicpm5-1b-lora-merged +# ENSEMBLE_LLM=Qwen/Qwen2.5-0.5B-Instruct +# ENSEMBLE_PRESET=minicpm5-1b +# ENSEMBLE_OUT=./models/ensemble/minicpm5-1b-jepa-pretrain +# ENSEMBLE_QA=./research/data/benchmark-qa.jsonl +# ENSEMBLE_KB=./research/data/benchmark-kb.jsonl +# ENSEMBLE_CKPT=./models/ensemble/jepa-lesson-pretrain + +BASE=openbmb/MiniCPM5-1B \ No newline at end of file diff --git a/.gitignore b/.gitignore index 8d8aa2a44024a2589b49a1c224e242ec5c84729d..7344c2ce9b537468b10cecb38b75b1724f4f507c 100644 --- a/.gitignore +++ b/.gitignore @@ -10,4 +10,6 @@ models/ dist/ build/ -outputs/traces \ No newline at end of file +outputs/traces + +/results \ No newline at end of file diff --git a/README.md b/README.md index 91688d210631ae8f80797b1053ca04e82aad42da..9607d77a2ca1ddf74022d9d031771b09aacd1ddb 100644 --- a/README.md +++ b/README.md @@ -46,8 +46,11 @@ apps/gradio-space/ # Gradio tabs (Lesson slides + Chat debug) libs/agent/ # Skill agent runner, tools, trace recorder libs/inference/ # Transformers + llama.cpp backends skills/ # SKILL.md task definitions +research/ # Fine-tune, ensemble experiments, agentic evals (optional) ``` +Optional research tooling (not required for the Space): see [research/USAGE.md](research/USAGE.md). + ## Environment variables | Variable | Default | Description | diff --git a/libs/inference/pyproject.toml b/libs/inference/pyproject.toml index 41fec7e7bac0fd7414acee6f85958cf7cef628c8..923749d0a73b0d0fd6067ad56ad21785d6d67e20 100644 --- a/libs/inference/pyproject.toml +++ b/libs/inference/pyproject.toml @@ -14,6 +14,7 @@ dependencies = [ "pyyaml>=6.0.2", "torch>=2.5.0", "torchvision>=0.20.0", + "peft>=0.14.0", "transformers>=5.7.0", ] diff --git a/libs/inference/src/inference/config.py b/libs/inference/src/inference/config.py index b40b0750ae245372b2e16b53bab20bffbae5be2c..cd493e36998a71e07cf718a11c3c470c30719e5f 100644 --- a/libs/inference/src/inference/config.py +++ b/libs/inference/src/inference/config.py @@ -23,6 +23,7 @@ class ModelConfig: model_file: str | None = None model_path: str | None = None model_id: str | None = None + adapter_path: str | None = None trust_remote_code: bool = False multimodal: bool = False n_ctx: int = 4096 @@ -37,6 +38,7 @@ class ModelConfig: self.model_file, self.model_path, self.model_id, + self.adapter_path, self.trust_remote_code, self.multimodal, self.n_ctx, @@ -60,6 +62,9 @@ class ModelConfig: if self.model_id and self.model_id.startswith(("./", "../")): updates["model_id"] = str((base_dir / self.model_id).resolve()) + if self.adapter_path and self.adapter_path.startswith(("./", "../")): + updates["adapter_path"] = str((base_dir / self.adapter_path).resolve()) + return replace(self, **updates) if updates else self @@ -149,6 +154,7 @@ def _parse_model_entry(key: str, raw: dict[str, Any]) -> ModelConfig: model_file=raw.get("model_file"), model_path=raw.get("model_path"), model_id=raw.get("model_id"), + adapter_path=raw.get("adapter_path"), trust_remote_code=bool(raw.get("trust_remote_code", False)), multimodal=bool(raw.get("multimodal", False)), n_ctx=int(raw.get("n_ctx", 4096)), diff --git a/libs/inference/src/inference/transformers.py b/libs/inference/src/inference/transformers.py index d8ff2945d2c06b3932fde931f5d51a8133481872..704403863d5cc21a306b8db6b4db9ef54c91f43d 100644 --- a/libs/inference/src/inference/transformers.py +++ b/libs/inference/src/inference/transformers.py @@ -76,6 +76,18 @@ class TransformersBackend: **model_kwargs, ) + if self._config.adapter_path: + from pathlib import Path + + from peft import PeftModel + + adapter = Path(self._config.adapter_path) + if not adapter.is_dir(): + raise FileNotFoundError( + f"LoRA adapter not found for preset {self._config.key!r}: {adapter}" + ) + self._model = PeftModel.from_pretrained(self._model, str(adapter)) + if device == "cpu": self._model.to(device) diff --git a/models.yaml b/models.yaml index dc7e75b1ae0d1ac10689a6af09492d8623216d73..ecbffee1b73bf759088393cba4ea649acbaa91e4 100644 --- a/models.yaml +++ b/models.yaml @@ -2,9 +2,8 @@ # Select active preset with ACTIVE_MODEL; override any field via .env (see .env.example). defaults: - active_model: minicpm-v-4.6 - - # active_model: minicpm5-1b + # active_model: minicpm-v-4.6 + active_model: minicpm5-1b # Dev: set ALLOW_MODEL_SWITCH=true in .env to expose a dropdown in Gradio. # Space: keep false so visitors use one pinned model. @@ -55,3 +54,22 @@ models: label: Fine-tuned LoRA adapter (local path) backend: transformers model_id: ./gemma_finetuned_model + + minicpm5-1b-lesson-lora: + label: MiniCPM5 1B lesson LoRA (local) + backend: transformers + model_id: openbmb/MiniCPM5-1B + adapter_path: ./models/finetuned/minicpm5-1b-lora + trust_remote_code: true + + minicpm5-1b-lesson-merged: + label: MiniCPM5 1B lesson merged (local) + backend: transformers + model_id: ./models/finetuned/minicpm5-1b-lora-merged + trust_remote_code: true + + jepa-ensemble-lesson: + label: JEPA ensemble (LLM + emb + JEPA) lesson pretrain + backend: transformers + model_id: ./models/ensemble/jepa-lesson-pretrain + trust_remote_code: true diff --git a/pyproject.toml b/pyproject.toml index f4ab1144b986a8d86d1f9ac3f8018045a5ef57ea..e35a2013744d8b3c1801fb423f0ac1bfbb4430b5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -6,6 +6,7 @@ readme = "README.md" requires-python = ">=3.12" dependencies = [ "agent", + "ensemble", "gradio-space", "inference", ] @@ -15,14 +16,34 @@ dev = [ "ruff>=0.9.0", "pytest>=8.0.0", ] +finetune = [ + "bitsandbytes>=0.46.1", + "datasets>=3.0.0", + "peft>=0.14.0", +] +ensemble = [ + "accelerate>=1.2.0", + "peft>=0.14.0", + "transformers>=5.7.0", +] +evals = [ + "slm-evals", +] +lm-eval = [ + "slm-evals[lm-eval]", +] [tool.uv.workspace] members = [ "apps/*", "libs/*", + "research/ensemble", + "research/evals", ] [tool.uv.sources] agent = { workspace = true } +ensemble = { workspace = true } gradio-space = { workspace = true } inference = { workspace = true } +slm-evals = { workspace = true } diff --git a/research/README.md b/research/README.md new file mode 100644 index 0000000000000000000000000000000000000000..3e08d33310455424519c360dced42b000d3024d0 --- /dev/null +++ b/research/README.md @@ -0,0 +1,48 @@ +# Research + +Experimental code for **fine-tuning**, **ensemble architectures**, and **agentic benchmarks**. Nothing here is wired into the Gradio Lesson Agent by default — use it to train models, probe JEPA/world-model ideas, and score checkpoints against public benchmarks. + +| Path | Purpose | +| ---- | ------- | +| [`finetune.py`](finetune.py) | LoRA / QLoRA / full fine-tune on chat or instruction data | +| [`ensemble/`](ensemble/) | JEPA + world-model ensemble experiments (uv package `ensemble`) | +| [`evals/`](evals/) | SLM agentic benchmark suite — BFCL, τ-bench, GAIA, SWE-bench (uv package `slm-evals`) | +| [`data/`](data/) | Shared JSONL datasets for finetune and ensemble harnesses | + +## Quick links + +- **[USAGE.md](USAGE.md)** — install groups, commands, and typical workflows +- **[docs/overview.md](docs/overview.md)** — how the pieces fit together +- **[ensemble/README.md](ensemble/README.md)** — ensemble smoke tests and harnesses +- **[evals/USAGE.md](evals/USAGE.md)** — benchmark CLI, configs, and results +- **[evals/docs/benchmarks.md](evals/docs/benchmarks.md)** — what each benchmark measures + +## Install (from repo root) + +```bash +# Everything you need for research scripts +uv sync --group finetune --group ensemble --group evals +``` + +Individual groups: + +| Group | Command | Enables | +| ----- | ------- | ------- | +| `finetune` | `uv sync --group finetune` | `research/finetune.py` (LoRA, QLoRA, merge) | +| `ensemble` | `uv sync --group ensemble` | `research/ensemble/` package | +| `evals` | `uv sync --group evals` | `research/evals/` package (`slm-benchmark`) | + +## Typical workflow + +```text +research/data/education-lesson-chat.jsonl + │ + ▼ + research/finetune.py ──► models/finetuned/-lora/ + │ + ├──► research/evals/ (BFCL, τ-bench, GAIA, SWE-bench) + │ + └──► research/ensemble/ (JEPA / world-model ablations) +``` + +See [USAGE.md](USAGE.md) for copy-paste commands. diff --git a/research/USAGE.md b/research/USAGE.md new file mode 100644 index 0000000000000000000000000000000000000000..7793212277b32dc8a564b8647544eec96771118b --- /dev/null +++ b/research/USAGE.md @@ -0,0 +1,311 @@ +# Research usage + +How to run fine-tuning, ensemble experiments, and agentic benchmarks under `research/`. All commands assume the **repo root** as the working directory unless noted. + +The Lesson Agent app lives in `apps/gradio-space/` — see root [USAGE.md](../USAGE.md). Research code is optional and isolated here. + +## Prerequisites + +- [uv](https://docs.astral.sh/uv/) and Python 3.12 +- GPU recommended for real-model runs (CPU works for smoke tests and `tiny` backends) +- Hugging Face Hub access for model downloads and some benchmark datasets + +## Install dependency groups + +```bash +# All research tooling +uv sync --group finetune --group ensemble --group evals --group lm-eval + +# Or one at a time +uv sync --group finetune +uv sync --group ensemble +uv sync --group evals +uv sync --group lm-eval +``` + +| Group | Package / script | What it adds | +| ----- | ---------------- | ------------ | +| `finetune` | `research/finetune.py` | `peft`, `datasets`, `bitsandbytes` (QLoRA) | +| `ensemble` | `ensemble` workspace member | JEPA / world-model ensemble + harnesses | +| `evals` | `slm-evals` workspace member | `slm-benchmark` CLI | +| `lm-eval` | `slm-evals[lm-eval]` | `slm-lm-eval` CLI (GSM8K, ARC, HellaSwag, …) | + +--- + +## 1. Fine-tuning (`research/finetune.py`) + +Single script for **full**, **LoRA**, and **QLoRA** training. Defaults to the lesson-agent chat dataset at `research/data/education-lesson-chat.jsonl` and writes checkpoints under `models/finetuned/`. + +### Model resolution (first match wins) + +1. `--model ` +2. `--preset ` from root `models.yaml` +3. Env: `FINETUNE_MODEL`, `MODEL_ID`, or `BASE` +4. `ACTIVE_MODEL` preset from `.env` + +### Quick start + +```bash +# LoRA on default lesson chat data, 1 epoch +uv run python research/finetune.py --preset minicpm5-1b --mode lora --epochs 1 + +# Smoke run (50 steps) +uv run python research/finetune.py --mode lora --max_steps 50 + +# QLoRA on a Hub instruction dataset +uv run python research/finetune.py \ + --model Qwen/Qwen2.5-0.5B-Instruct \ + --dataset tatsu-lab/alpaca --format alpaca \ + --mode qlora --epochs 1 + +# Merge LoRA adapter into standalone weights +uv run python research/finetune.py \ + --merge ./models/finetuned/minicpm5-1b-lora \ + --out ./models/finetuned/minicpm5-1b-merged +``` + +### Dataset formats (`--format`) + +| Format | Expected columns | +| ------ | ---------------- | +| `chat` | `messages`: `[{"role": "...", "content": "..."}]` | +| `alpaca` | `instruction`, optional `input`, `output` | +| `prompt` | `prompt` / `completion` (or `response`) | +| `text` | `text`, or a plain `.txt` file | + +Local files: `.json`, `.jsonl`, `.csv`, `.txt`. Hub ids: any `datasets` repo id. + +### Outputs + +Training writes to `/` (default `./models/finetuned/-/`): + +- Adapter or full weights +- `training_results.json` — train/eval loss, perplexity, `result_score` (0–100) + +### Env vars + +| Variable | Description | +| -------- | ----------- | +| `FINETUNE_PRESET` | Preset key from `models.yaml` | +| `FINETUNE_DATASET` | Override dataset path or Hub id | +| `FINETUNE_DATASET_CONFIG` | Hub config name | +| `FINETUNE_DATASET_SPLIT` | Hub split (e.g. `train[:500]`) | +| `ACTIVE_MODEL` | Fallback preset when `--preset` omitted | + +--- + +## 2. Ensemble experiments (`research/ensemble/`) + +JEPA and world-model ensemble prototypes: small LLM + embedding memory + latent predictors + energy-based draft selection. **Not connected to the Gradio app.** + +Install: `uv sync --group ensemble` + +### Tier 1 — CPU smoke (no Hub download) + +```bash +uv run --package ensemble python -m ensemble.jepa_ensemble tiny +uv run --package ensemble python -m ensemble.world_ensemble tiny +bash research/ensemble/scripts/smoke.sh +``` + +### Tier 2 — Real small model + +```bash +uv run --package ensemble python -m ensemble.jepa_ensemble Qwen/Qwen2.5-0.5B-Instruct +uv run --package ensemble python -m ensemble.world_ensemble Qwen/Qwen2.5-0.5B-Instruct +``` + +### Pretrain + save (LLM + emb + JEPA) + +```bash +# Default LLM: ENSEMBLE_LLM → LLM_PATH → BASE → MODEL_ID → ACTIVE_MODEL (models.yaml) +uv run --package ensemble ensemble-pretrain --steps 200 + +# Or override +uv run --package ensemble ensemble-pretrain \ + --llm Qwen/Qwen2.5-0.5B-Instruct \ + --steps 200 + +# Benchmark saved ensemble with slm-evals (compare to base HF model) +uv run --package slm-evals slm-benchmark \ + --model ./models/ensemble/jepa-lesson-pretrain \ + --model-type ensemble \ + --benchmarks bfcl tau_bench --max-samples 20 +``` + +Checkpoint files: `manifest.json`, `aux.pt`, `llm/` (PEFT adapters), optional `store.pt`. + +### Tier 3 — Benchmark harnesses + +Uses `research/data/benchmark-qa.jsonl` (questions) and `benchmark-kb.jsonl` (retrieval snippets). + +```bash +# JEPA track — toy +uv run --package ensemble python -m ensemble.eval.jepa_harness \ + --llm tiny --toy --limit 20 --n_drafts 8 + +# JEPA track — education QA +uv run --package ensemble python -m ensemble.eval.jepa_harness \ + --llm Qwen/Qwen2.5-0.5B-Instruct \ + --qa research/data/benchmark-qa.jsonl \ + --kb research/data/benchmark-kb.jsonl \ + --limit 50 --n_drafts 8 + +# World-model track +uv run --package ensemble python -m ensemble.eval.world_harness \ + --llm tiny --toy --limit 20 --n_drafts 8 +``` + +More detail: [ensemble/README.md](ensemble/README.md), [docs/overview.md](docs/overview.md). + +### Legacy shims + +Top-level files re-export the package for old scripts: + +- `research/llm_emb_jepa_ensemble_pluggable.py` → `ensemble.jepa_ensemble` +- `research/world_model_ensemble.py` → `ensemble.world_ensemble` +- `research/eval_harness.py` → `ensemble.eval.jepa_harness` + +Prefer `uv run --package ensemble python -m ensemble.`. + +--- + +## 3. Agentic benchmarks (`research/evals/`) + +Evaluate local HuggingFace checkpoints on BFCL, τ-bench, GAIA, and SWE-bench Verified. + +Install: `uv sync --group evals` + +```bash +# Smoke test (20 samples, two benchmarks) +uv run --package slm-evals slm-benchmark \ + --model openbmb/MiniCPM5-1B \ + --benchmarks bfcl tau_bench \ + --max-samples 20 + +# Full config-driven run +uv run --package slm-evals slm-benchmark \ + --config research/evals/configs/experiment_001.yaml +``` + +Full reference: [evals/USAGE.md](evals/USAGE.md). + +--- + +## 4. Academic benchmarks (`slm-lm-eval`) + +Standard lm-evaluation-harness tasks (ARC, HellaSwag, GSM8K, …) for base presets, LoRA adapters, merged checkpoints, and ensemble manifests. + +Install: `uv sync --group lm-eval` + +Profile guide: [evals/docs/eval_profiles.md](evals/docs/eval_profiles.md) + +```bash +# List claim-matched profiles (reasoning, code, understanding, …) +uv run --package slm-evals slm-lm-eval --list-profiles + +# Run by profile name +uv run --package slm-evals slm-lm-eval \ + --profile reasoning \ + --preset minicpm5-1b \ + --experiment-name minicpm5-1b__reasoning-baseline + +# Smoke (25 samples, arc_easy + hellaswag) +uv run --package slm-evals slm-lm-eval \ + --profile smoke \ + --preset minicpm5-1b \ + --experiment-name minicpm5-1b__smoke + +# Full profile +uv run --package slm-evals slm-lm-eval \ + --config research/evals/configs/lm_eval_minicpm5.yaml \ + --preset minicpm5-1b-lesson-lora \ + --experiment-name minicpm5-1b-lora__v1 \ + --compare-to results/lm_eval/minicpm5-1b__baseline/results.json + +# Ensemble checkpoint +uv run --package slm-evals slm-lm-eval \ + --config research/evals/configs/lm_eval_smoke.yaml \ + --model ./models/ensemble/jepa-lesson-pretrain \ + --experiment-name ensemble-jepa__lm-eval +``` + +Post-training hook: + +```bash +uv run python research/finetune.py \ + --preset minicpm5-1b --mode lora --max_steps 50 \ + --lm-eval-after \ + --lm-eval-baseline minicpm5-1b +``` + +Full reference: [evals/USAGE.md](evals/USAGE.md#lm-evaluation-harness-slm-lm-eval). + +--- + +## Shared data (`research/data/`) + +| File | Used by | Format | +| ---- | ------- | ------ | +| `education-lesson-chat.jsonl` | `finetune.py` default | Chat messages for lesson agent | +| `benchmark-qa.jsonl` | Ensemble harnesses | `question`, `answer`, `domain` | +| `benchmark-kb.jsonl` | Ensemble harnesses | Retrieval snippets for memory routing | + +--- + +## Suggested end-to-end pipeline + +1. **Baseline lm-eval** — academic benchmarks on the base preset (pinned seed): + ```bash + uv run --package slm-evals slm-lm-eval \ + --config research/evals/configs/lm_eval_compare_study.yaml \ + --preset minicpm5-1b \ + --experiment-name minicpm5-1b__baseline + ``` + +2. **Baseline agentic eval** (optional): + ```bash + uv run --package slm-evals slm-benchmark \ + --model openbmb/MiniCPM5-1B --benchmarks bfcl --max-samples 50 + ``` + +3. **Fine-tune** on lesson data: + ```bash + uv run python research/finetune.py --preset minicpm5-1b --mode lora --epochs 1 + ``` + +4. **Re-eval candidate** with the same lm-eval config: + ```bash + uv run --package slm-evals slm-lm-eval \ + --config research/evals/configs/lm_eval_compare_study.yaml \ + --preset minicpm5-1b-lesson-lora \ + --experiment-name minicpm5-1b-lora__v1 \ + --compare-to results/lm_eval/minicpm5-1b__baseline/results.json + ``` + +5. **Optional** — probe ensemble ideas on the same QA/KB files: + ```bash + bash research/ensemble/scripts/smoke.sh + ``` + +### Verification checklist + +- Use the **same** lm-eval YAML (`tasks`, `num_fewshot`, `limit`, `seed`) for baseline and candidate runs. +- Compare lm-eval `results.json` files with `--compare-to`; do not compare `training_results.json` `result_score` to lm-eval accuracy. +- For LoRA checkpoints, prefer `--preset minicpm5-1b-lesson-lora` (base + adapter) over passing the adapter dir alone to `--model`. +- Report mean ± std only after multiple training seeds; single-seed deltas are indicative, not conclusive. +- Ensemble `loglikelihood` tasks score the underlying LLM head; generative tasks (`gsm8k`) use the full JEPA+RAG stack. + +--- + +## Troubleshooting + +| Symptom | Fix | +| ------- | --- | +| `No module named 'ensemble'` | `uv sync --group ensemble` | +| `slm-benchmark: command not found` | `uv sync --group evals` | +| `slm-lm-eval: command not found` | `uv sync --group lm-eval` | +| CUDA OOM during finetune | Use `--mode qlora` or reduce batch size in script args | +| BFCL / GAIA download slow | Set `max_samples` low first; cache HF datasets under `~/.cache/huggingface` | +| SWE-bench Docker errors | Keep `full_eval: false` in YAML unless `swebench` + Docker are installed | +| τ-bench API costs | Keep `use_llm_user: false` (rule-based user simulator) | diff --git a/research/data/benchmark-kb.jsonl b/research/data/benchmark-kb.jsonl new file mode 100644 index 0000000000000000000000000000000000000000..502ddc81c74dce22ae2ac57eeeaf138d40717c69 --- /dev/null +++ b/research/data/benchmark-kb.jsonl @@ -0,0 +1,10 @@ +{"text": "Photosynthesis: plants use sunlight, water, and CO2 to make glucose and release oxygen."} +{"text": "Fractions: the top number counts parts and the bottom number shows total equal parts."} +{"text": "Water cycle: evaporation turns liquid water into vapor; condensation forms clouds."} +{"text": "Precipitation returns water to land and bodies of water as rain, snow, sleet, or hail."} +{"text": "AI agents observe state, decide actions, and update plans using tools and memory."} +{"text": "American Revolution: colonists protested taxation without representation and trade restrictions."} +{"text": "Boston Tea Party: colonists dumped tea to protest the Tea Act in 1773."} +{"text": "Enlightenment ideas about rights and consent spread through pamphlets and speeches."} +{"text": "One-half means one of two equal parts; one-fourth means one of four equal parts."} +{"text": "Runoff carries water from land back toward rivers, lakes, and oceans."} diff --git a/research/data/benchmark-qa.jsonl b/research/data/benchmark-qa.jsonl new file mode 100644 index 0000000000000000000000000000000000000000..31e484a6e77a526b240f3cfc2d03271c632ab671 --- /dev/null +++ b/research/data/benchmark-qa.jsonl @@ -0,0 +1,10 @@ +{"question": "What is photosynthesis?", "answer": "Plants make food using sunlight, water, and carbon dioxide.", "domain": "science"} +{"question": "What do plants produce during photosynthesis?", "answer": "Glucose and oxygen.", "domain": "science"} +{"question": "What is a fraction?", "answer": "A fraction shows equal parts of a whole.", "domain": "math"} +{"question": "What does one-half mean?", "answer": "One of two equal parts.", "domain": "math"} +{"question": "What is evaporation in the water cycle?", "answer": "Heat turns liquid water into water vapor.", "domain": "science"} +{"question": "What is precipitation?", "answer": "Water falls as rain, snow, sleet, or hail.", "domain": "science"} +{"question": "What is an AI agent?", "answer": "Software that takes goals and acts to reach them.", "domain": "technology"} +{"question": "What is a colonial grievance before the American Revolution?", "answer": "Taxation without representation.", "domain": "history"} +{"question": "What happened at the Boston Tea Party?", "answer": "Colonists protested the Tea Act.", "domain": "history"} +{"question": "Why does photosynthesis matter?", "answer": "It feeds nearly all life on Earth and keeps the atmosphere balanced.", "domain": "science"} diff --git a/research/data/education-lesson-chat.jsonl b/research/data/education-lesson-chat.jsonl new file mode 100644 index 0000000000000000000000000000000000000000..7993862c0fa6838dae408bab6d6dbb66dcac7ea9 --- /dev/null +++ b/research/data/education-lesson-chat.jsonl @@ -0,0 +1,8 @@ +{"messages": [{"role": "system", "content": "You are a lesson-planning assistant for teachers.\nFollow the skill workflow below and output ONLY valid JSON (no markdown fences).\n\nSkill workflow:\n1. Ask for topic, audience grade, and slide count (3–8 content slides).\n2. Produce a JSON outline with `title` and `slides` (each slide has `title`, `bullets`, `speaker_note`).\n3. Call `create_pptx` with the validated outline.\n4. Return a download link and markdown preview for the teacher.\n\nJSON schema:\n{\n \"title\": \"string — presentation title\",\n \"slides\": [\n {\n \"title\": \"string — slide heading\",\n \"bullets\": [\"string\", \"...\"],\n \"speaker_note\": \"string — one sentence for the teacher\"\n }\n ]\n}\n\nRules:\n- Use exactly the requested number of content slides (title slide is added separately by the tool).\n- Bullets should be short, age-appropriate, and factual.\n- speaker_note is optional but helpful for each slide."}, {"role": "user", "content": "Topic: Photosynthesis\nGrade level: 6\nNumber of content slides: 4\nReturn JSON only."}, {"role": "assistant", "content": "{\"title\": \"Photosynthesis for 6th Grade\", \"slides\": [{\"title\": \"What Is Photosynthesis?\", \"bullets\": [\"Plants make their own food using sunlight\", \"The process happens mainly in leaves\"], \"speaker_note\": \"Ask students what they already know about how plants eat.\"}, {\"title\": \"Ingredients Plants Need\", \"bullets\": [\"Sunlight provides energy\", \"Water comes from roots and carbon dioxide from air\"], \"speaker_note\": \"Draw a simple leaf diagram while you explain inputs.\"}, {\"title\": \"What Plants Produce\", \"bullets\": [\"Glucose is the sugar plants use for energy\", \"Oxygen is released into the air\"], \"speaker_note\": \"Connect oxygen release to why forests matter for breathing.\"}, {\"title\": \"Why It Matters\", \"bullets\": [\"Photosynthesis feeds nearly all life on Earth\", \"It helps keep our atmosphere balanced\"], \"speaker_note\": \"End with a quick pair-share: one reason photosynthesis matters.\"}]}"}]} +{"messages": [{"role": "system", "content": "You are a lesson-planning assistant for teachers.\nFollow the skill workflow below and output ONLY valid JSON (no markdown fences).\n\nSkill workflow:\n1. Ask for topic, audience grade, and slide count (3–8 content slides).\n2. Produce a JSON outline with `title` and `slides` (each slide has `title`, `bullets`, `speaker_note`).\n3. Call `create_pptx` with the validated outline.\n4. Return a download link and markdown preview for the teacher.\n\nJSON schema:\n{\n \"title\": \"string — presentation title\",\n \"slides\": [\n {\n \"title\": \"string — slide heading\",\n \"bullets\": [\"string\", \"...\"],\n \"speaker_note\": \"string — one sentence for the teacher\"\n }\n ]\n}\n\nRules:\n- Use exactly the requested number of content slides (title slide is added separately by the tool).\n- Bullets should be short, age-appropriate, and factual.\n- speaker_note is optional but helpful for each slide."}, {"role": "user", "content": "Topic: Fractions\nGrade level: 4\nNumber of content slides: 3\nReturn JSON only."}, {"role": "assistant", "content": "{\"title\": \"Understanding Fractions\", \"slides\": [{\"title\": \"Parts of a Whole\", \"bullets\": [\"A fraction shows equal parts of a whole\", \"The top number counts parts; the bottom shows total equal parts\"], \"speaker_note\": \"Use a paper pizza cut into equal slices as a visual.\"}, {\"title\": \"Reading Common Fractions\", \"bullets\": [\"One-half means one of two equal parts\", \"One-fourth means one of four equal parts\"], \"speaker_note\": \"Have students name fractions they see in the room.\"}, {\"title\": \"Comparing Fractions\", \"bullets\": [\"Larger denominators can mean smaller pieces\", \"Use drawings to compare two simple fractions\"], \"speaker_note\": \"Give two fraction cards and ask which shows more pizza.\"}]}"}]} +{"messages": [{"role": "system", "content": "You are a lesson-planning assistant for teachers.\nFollow the skill workflow below and output ONLY valid JSON (no markdown fences).\n\nSkill workflow:\n1. Ask for topic, audience grade, and slide count (3–8 content slides).\n2. Produce a JSON outline with `title` and `slides` (each slide has `title`, `bullets`, `speaker_note`).\n3. Call `create_pptx` with the validated outline.\n4. Return a download link and markdown preview for the teacher.\n\nJSON schema:\n{\n \"title\": \"string — presentation title\",\n \"slides\": [\n {\n \"title\": \"string — slide heading\",\n \"bullets\": [\"string\", \"...\"],\n \"speaker_note\": \"string — one sentence for the teacher\"\n }\n ]\n}\n\nRules:\n- Use exactly the requested number of content slides (title slide is added separately by the tool).\n- Bullets should be short, age-appropriate, and factual.\n- speaker_note is optional but helpful for each slide."}, {"role": "user", "content": "Topic: The water cycle\nGrade level: 5\nNumber of content slides: 5\nReturn JSON only."}, {"role": "assistant", "content": "{\"title\": \"The Water Cycle\", \"slides\": [{\"title\": \"Water Moves Constantly\", \"bullets\": [\"Water travels between Earth, air, and living things\", \"The same water has been recycled for millions of years\"], \"speaker_note\": \"Show a simple cycle diagram before diving into each stage.\"}, {\"title\": \"Evaporation\", \"bullets\": [\"Heat turns liquid water into water vapor\", \"Oceans, lakes, and puddles lose water to the air\"], \"speaker_note\": \"Ask what happens to a wet sidewalk on a sunny day.\"}, {\"title\": \"Condensation\", \"bullets\": [\"Water vapor cools and forms tiny droplets\", \"Clouds form when droplets gather in the sky\"], \"speaker_note\": \"Hold a cold cup and point out the outside moisture.\"}, {\"title\": \"Precipitation\", \"bullets\": [\"Droplets combine and fall as rain, snow, sleet, or hail\", \"Precipitation returns water to land and water bodies\"], \"speaker_note\": \"Let students name types of precipitation they have seen.\"}, {\"title\": \"Collection and Runoff\", \"bullets\": [\"Water collects in rivers, lakes, and groundwater\", \"Runoff carries water back toward oceans\"], \"speaker_note\": \"Trace a raindrop's path on your diagram to close the loop.\"}]}"}]} +{"messages": [{"role": "system", "content": "You are a lesson-planning assistant for teachers.\nFollow the skill workflow below and output ONLY valid JSON (no markdown fences).\n\nSkill workflow:\n1. Ask for topic, audience grade, and slide count (3–8 content slides).\n2. Produce a JSON outline with `title` and `slides` (each slide has `title`, `bullets`, `speaker_note`).\n3. Call `create_pptx` with the validated outline.\n4. Return a download link and markdown preview for the teacher.\n\nJSON schema:\n{\n \"title\": \"string — presentation title\",\n \"slides\": [\n {\n \"title\": \"string — slide heading\",\n \"bullets\": [\"string\", \"...\"],\n \"speaker_note\": \"string — one sentence for the teacher\"\n }\n ]\n}\n\nRules:\n- Use exactly the requested number of content slides (title slide is added separately by the tool).\n- Bullets should be short, age-appropriate, and factual.\n- speaker_note is optional but helpful for each slide."}, {"role": "user", "content": "Topic: AI agents\nGrade level: 8\nNumber of content slides: 4\nReturn JSON only."}, {"role": "assistant", "content": "{\"title\": \"Introduction to AI Agents\", \"slides\": [{\"title\": \"What Is an AI Agent?\", \"bullets\": [\"An AI agent is software that takes goals and acts to reach them\", \"Agents can use tools, memory, and feedback loops\"], \"speaker_note\": \"Contrast a chatbot that only replies with an agent that plans steps.\"}, {\"title\": \"How Agents Work\", \"bullets\": [\"They observe input or environment state\", \"They decide actions and update their plan\"], \"speaker_note\": \"Walk through a simple example like booking a meeting.\"}, {\"title\": \"Everyday Examples\", \"bullets\": [\"Navigation apps reroute based on traffic\", \"Email filters can sort and draft responses\"], \"speaker_note\": \"Invite students to suggest other agent-like tools they use.\"}, {\"title\": \"Benefits and Risks\", \"bullets\": [\"Agents can save time on repetitive tasks\", \"They need clear rules and human oversight\"], \"speaker_note\": \"Close with a short discussion on when automation helps or hurts.\"}]}"}]} +{"messages": [{"role": "system", "content": "You are a lesson-planning assistant for teachers.\nFollow the skill workflow below and output ONLY valid JSON (no markdown fences).\n\nSkill workflow:\n1. Ask for topic, audience grade, and slide count (3–8 content slides).\n2. Produce a JSON outline with `title` and `slides` (each slide has `title`, `bullets`, `speaker_note`).\n3. Call `create_pptx` with the validated outline.\n4. Return a download link and markdown preview for the teacher.\n\nJSON schema:\n{\n \"title\": \"string — presentation title\",\n \"slides\": [\n {\n \"title\": \"string — slide heading\",\n \"bullets\": [\"string\", \"...\"],\n \"speaker_note\": \"string — one sentence for the teacher\"\n }\n ]\n}\n\nRules:\n- Use exactly the requested number of content slides (title slide is added separately by the tool).\n- Bullets should be short, age-appropriate, and factual.\n- speaker_note is optional but helpful for each slide."}, {"role": "user", "content": "Topic: The American Revolution\nGrade level: 8\nNumber of content slides: 5\nReturn JSON only."}, {"role": "assistant", "content": "{\"title\": \"Causes of the American Revolution\", \"slides\": [{\"title\": \"Colonial Grievances\", \"bullets\": [\"Colonists protested taxation without representation\", \"Trade restrictions limited colonial economies\"], \"speaker_note\": \"Connect taxes on everyday goods to student purchases today.\"}, {\"title\": \"Key Events\", \"bullets\": [\"The Boston Tea Party protested the Tea Act\", \"The Intolerable Acts tightened British control\"], \"speaker_note\": \"Place events on a timeline as you introduce each one.\"}, {\"title\": \"Ideas That Spread\", \"bullets\": [\"Enlightenment ideas emphasized rights and consent\", \"Pamphlets and speeches spread revolutionary arguments\"], \"speaker_note\": \"Read a short excerpt from a primary source if time allows.\"}, {\"title\": \"Leaders and Groups\", \"bullets\": [\"Committees and congresses organized resistance\", \"Leaders like Adams and Washington gained prominence\"], \"speaker_note\": \"Highlight diverse roles beyond famous military figures.\"}, {\"title\": \"Road to Independence\", \"bullets\": [\"The Declaration explained the break from Britain\", \"War followed when compromise failed\"], \"speaker_note\": \"End by asking what principle students would defend most strongly.\"}]}"}]} +{"messages": [{"role": "system", "content": "You are a lesson-planning assistant for teachers.\nFollow the skill workflow below and output ONLY valid JSON (no markdown fences).\n\nSkill workflow:\n1. Ask for topic, audience grade, and slide count (3–8 content slides).\n2. Produce a JSON outline with `title` and `slides` (each slide has `title`, `bullets`, `speaker_note`).\n3. Call `create_pptx` with the validated outline.\n4. Return a download link and markdown preview for the teacher.\n\nJSON schema:\n{\n \"title\": \"string — presentation title\",\n \"slides\": [\n {\n \"title\": \"string — slide heading\",\n \"bullets\": [\"string\", \"...\"],\n \"speaker_note\": \"string — one sentence for the teacher\"\n }\n ]\n}\n\nRules:\n- Use exactly the requested number of content slides (title slide is added separately by the tool).\n- Bullets should be short, age-appropriate, and factual.\n- speaker_note is optional but helpful for each slide."}, {"role": "user", "content": "Topic: Kindness and friendship\nGrade level: 2\nNumber of content slides: 3\nReturn JSON only."}, {"role": "assistant", "content": "{\"title\": \"Being a Good Friend\", \"slides\": [{\"title\": \"What Friends Do\", \"bullets\": [\"Friends listen and share\", \"Friends help when someone feels sad\"], \"speaker_note\": \"Ask students to share one kind thing a friend did for them.\"}, {\"title\": \"Using Kind Words\", \"bullets\": [\"Please and thank you show respect\", \"Compliments can make others feel happy\"], \"speaker_note\": \"Practice turning an unkind sentence into a kind one together.\"}, {\"title\": \"Solving Small Problems\", \"bullets\": [\"Take turns when you disagree\", \"Ask a grown-up if you need help\"], \"speaker_note\": \"Role-play a simple sharing conflict and a peaceful solution.\"}]}"}]} +{"messages": [{"role": "system", "content": "You are a lesson-planning assistant for teachers.\nFollow the skill workflow below and output ONLY valid JSON (no markdown fences).\n\nSkill workflow:\n1. Ask for topic, audience grade, and slide count (3–8 content slides).\n2. Produce a JSON outline with `title` and `slides` (each slide has `title`, `bullets`, `speaker_note`).\n3. Call `create_pptx` with the validated outline.\n4. Return a download link and markdown preview for the teacher.\n\nJSON schema:\n{\n \"title\": \"string — presentation title\",\n \"slides\": [\n {\n \"title\": \"string — slide heading\",\n \"bullets\": [\"string\", \"...\"],\n \"speaker_note\": \"string — one sentence for the teacher\"\n }\n ]\n}\n\nRules:\n- Use exactly the requested number of content slides (title slide is added separately by the tool).\n- Bullets should be short, age-appropriate, and factual.\n- speaker_note is optional but helpful for each slide."}, {"role": "user", "content": "Topic: Linear equations\nGrade level: 9\nNumber of content slides: 4\nReturn JSON only."}, {"role": "assistant", "content": "{\"title\": \"Introduction to Linear Equations\", \"slides\": [{\"title\": \"What Makes an Equation Linear?\", \"bullets\": [\"A linear equation graphs as a straight line\", \"The highest power of the variable is one\"], \"speaker_note\": \"Show y = 2x + 1 on a coordinate plane.\"}, {\"title\": \"Slope-Intercept Form\", \"bullets\": [\"y = mx + b uses slope m and y-intercept b\", \"m tells how steep the line is\"], \"speaker_note\": \"Have students identify m and b from one example.\"}, {\"title\": \"Graphing from an Equation\", \"bullets\": [\"Plot the y-intercept first\", \"Use the slope to find another point\"], \"speaker_note\": \"Work one example slowly on the board.\"}, {\"title\": \"Real-World Connections\", \"bullets\": [\"Phone plans with a flat fee plus per-minute cost are linear\", \"Distance at constant speed changes linearly over time\"], \"speaker_note\": \"Ask students to invent their own linear situation.\"}]}"}]} +{"messages": [{"role": "system", "content": "You are a lesson-planning assistant for teachers.\nFollow the skill workflow below and output ONLY valid JSON (no markdown fences).\n\nSkill workflow:\n1. Ask for topic, audience grade, and slide count (3–8 content slides).\n2. Produce a JSON outline with `title` and `slides` (each slide has `title`, `bullets`, `speaker_note`).\n3. Call `create_pptx` with the validated outline.\n4. Return a download link and markdown preview for the teacher.\n\nJSON schema:\n{\n \"title\": \"string — presentation title\",\n \"slides\": [\n {\n \"title\": \"string — slide heading\",\n \"bullets\": [\"string\", \"...\"],\n \"speaker_note\": \"string — one sentence for the teacher\"\n }\n ]\n}\n\nRules:\n- Use exactly the requested number of content slides (title slide is added separately by the tool).\n- Bullets should be short, age-appropriate, and factual.\n- speaker_note is optional but helpful for each slide."}, {"role": "user", "content": "Topic: The solar system\nGrade level: 3\nNumber of content slides: 4\nReturn JSON only."}, {"role": "assistant", "content": "{\"title\": \"Our Solar System\", \"slides\": [{\"title\": \"The Sun at the Center\", \"bullets\": [\"The Sun is a star that gives Earth light and heat\", \"Planets orbit, or travel around, the Sun\"], \"speaker_note\": \"Use a lamp and balls to model orbits if you have them.\"}, {\"title\": \"Inner Planets\", \"bullets\": [\"Mercury, Venus, Earth, and Mars are rocky planets\", \"Earth is the only one known to have life\"], \"speaker_note\": \"Compare sizes with classroom objects.\"}, {\"title\": \"Outer Planets\", \"bullets\": [\"Jupiter and Saturn are gas giants\", \"Uranus and Neptune are icy giants farther from the Sun\"], \"speaker_note\": \"Mention that Jupiter is the largest planet.\"}, {\"title\": \"Other Objects\", \"bullets\": [\"Asteroids and comets also orbit the Sun\", \"The Moon orbits Earth, not the Sun directly\"], \"speaker_note\": \"Clarify that moons are not planets.\"}]}"}]} diff --git a/research/docs/overview.md b/research/docs/overview.md new file mode 100644 index 0000000000000000000000000000000000000000..ad3672350a5e1d4ce8ec04cbd2813979cd6a1ea2 --- /dev/null +++ b/research/docs/overview.md @@ -0,0 +1,114 @@ +# Research overview + +How `research/` relates to the main hackathon repo and what each component does. + +## Position in the repo + +```text +small-model-hackathon/ +├── apps/gradio-space/ ← shipped Lesson Agent UI +├── libs/agent/ ← skill loop, tools, traces +├── libs/inference/ ← transformers + llama.cpp backends +├── models.yaml ← model presets (shared with finetune) +└── research/ ← experiments (this tree) + ├── finetune.py + ├── data/ + ├── ensemble/ ← uv workspace package + └── evals/ ← uv workspace package +``` + +Research code is a **uv workspace sibling** of `apps/*` and `libs/*`. Root `pyproject.toml` declares optional dependency groups (`finetune`, `ensemble`, `evals`) so the Docker Space image does not need to install torch-heavy extras unless you opt in locally. + +## Three tracks + +### Fine-tuning + +`research/finetune.py` adapts a small HF causal LM on instruction or chat data. It reuses root `models.yaml` presets and the shared inference config loader, so the same `minicpm5-1b` preset used in the Gradio app can be fine-tuned without duplicating model metadata. + +Outputs land in `models/finetuned/` — you can register a new preset in `models.yaml` pointing at merged weights for the **Well-Tuned** hackathon badge. + +### Ensemble (JEPA / world model) + +`research/ensemble/` explores a modular stack inspired by LeCun-style architectures: + +```text +Input ──► Embedder + VectorStore (retrieval memory) + │ + ▼ + JEPA encoder ──► latent state + │ + ├──► World model (multi-step latent rollout) + │ + └──► Energy model (scores LLM draft continuations) + │ + ▼ + Small LLM generates N drafts → pick lowest energy +``` + +Two entry ensembles: + +| Module | File | Critic | +| ------ | ---- | ------ | +| JEPA track | `ensemble.jepa_ensemble` | JEPA latent prediction | +| World track | `ensemble.world_ensemble` | Energy model over world-model rollouts | + +`TinyBackend` runs on CPU with random weights for smoke tests. `HFBackend` loads real Hub models via `transformers` + optional `peft` LoRA banks. + +Eval harnesses (`ensemble.eval.jepa_harness`, `ensemble.eval.world_harness`) measure draft-selection accuracy on `research/data/benchmark-qa.jsonl` with optional KB retrieval from `benchmark-kb.jsonl`. + +### Agentic evals + +`research/evals/` (`slm-evals` package) scores **whole models** on public agent benchmarks — function calling, multi-turn tool use, GAIA tasks, and SWE-bench patches. This complements ensemble harnesses: evals test end-to-end model behavior; ensemble harnesses test internal selection mechanisms on a small custom QA set. + +## Data flow + +```mermaid +flowchart LR + subgraph data [research/data] + lesson[education-lesson-chat.jsonl] + qa[benchmark-qa.jsonl] + kb[benchmark-kb.jsonl] + end + + subgraph train [finetune.py] + ckpt[models/finetuned/] + end + + subgraph evals [slm-evals] + bfcl[BFCL] + tau[tau-bench] + gaia[GAIA] + swe[SWE-bench] + end + + subgraph ens [ensemble] + jepa[JEPA harness] + world[World harness] + end + + lesson --> train + train --> ckpt + ckpt --> evals + qa --> jepa + kb --> jepa + qa --> world + kb --> world +``` + +## When to use which tool + +| Goal | Tool | +| ---- | ---- | +| Improve lesson slide quality on your data | `finetune.py` + optional eval before/after | +| Compare base vs LoRA on public agent tasks | `slm-benchmark` | +| Prototype latent draft selection | `ensemble` smoke → harness | +| Ship in Gradio Space | `apps/gradio-space` only — wire new weights via `models.yaml` | + +## Workspace packages + +Both subpackages are listed in root `[tool.uv.workspace] members`: + +- `research/ensemble` → import name `ensemble` +- `research/evals` → import name `slm_evals`, CLI `slm-benchmark` + +Run with `uv run --package ` from the repo root so uv resolves workspace paths and shared lockfile versions. diff --git a/research/ensemble/README.md b/research/ensemble/README.md new file mode 100644 index 0000000000000000000000000000000000000000..c6451074b4c82915e5246500ad5180f7636ec3b2 --- /dev/null +++ b/research/ensemble/README.md @@ -0,0 +1,113 @@ +# Ensemble research package + +JEPA and world-model ensemble experiments. Stays under `research/` — not wired into the Gradio agent. + +See also: [../USAGE.md](../USAGE.md) · [../docs/overview.md](../docs/overview.md) + +## Install + +```bash +uv sync --group ensemble +``` + +## Tier 1 — Smoke (CPU, no HF download) + +```bash +uv run --package ensemble python -m ensemble.jepa_ensemble tiny +uv run --package ensemble python -m ensemble.world_ensemble tiny +bash research/ensemble/scripts/smoke.sh +``` + +## Tier 2 — Micro demo (real small model) + +```bash +uv run --package ensemble python -m ensemble.jepa_ensemble Qwen/Qwen2.5-0.5B-Instruct +uv run --package ensemble python -m ensemble.world_ensemble Qwen/Qwen2.5-0.5B-Instruct +``` + +## Pretrain + save (LLM + emb + JEPA) + +Joint training writes a full checkpoint to `models/ensemble//`: + +```bash +# CPU smoke (tiny backend, no HF download) +uv run --package ensemble ensemble-pretrain \ + --llm tiny --steps 50 --no-kb \ + --out models/ensemble/jepa-smoke + +# Uses ACTIVE_MODEL / BASE / LLM_PATH from .env + models.yaml by default +uv run --package ensemble ensemble-pretrain \ + --data research/data/education-lesson-chat.jsonl \ + --kb research/data/benchmark-kb.jsonl \ + --steps 200 + +# Override base LLM explicitly +uv run --package ensemble ensemble-pretrain \ + --llm Qwen/Qwen2.5-0.5B-Instruct --steps 200 +``` + +Checkpoint layout: `manifest.json`, `aux.pt` (emb/jepa/bridge/router), `llm/` (PEFT adapters). + +Benchmark the saved ensemble with **slm-evals** (auto-detects `manifest.json`): + +```bash +uv run --package slm-evals slm-benchmark \ + --model ./models/ensemble/jepa-lesson-pretrain \ + --model-type ensemble \ + --benchmarks bfcl tau_bench --max-samples 20 + +# Or use the template config +uv run --package slm-evals slm-benchmark \ + --config research/evals/configs/ensemble_jepa_lesson.yaml +``` + +Compare against a base HF model by running the same config with `model_type: hf` and `model_path: openbmb/MiniCPM5-1B`. + +## Tier 3 — Benchmark + +### JEPA ablation ladder + +```bash +# Toy (no download) +uv run --package ensemble python -m ensemble.eval.jepa_harness \ + --llm tiny --toy --limit 20 --n_drafts 8 + +# Education QA set +uv run --package ensemble python -m ensemble.eval.jepa_harness \ + --llm Qwen/Qwen2.5-0.5B-Instruct \ + --qa research/data/benchmark-qa.jsonl \ + --kb research/data/benchmark-kb.jsonl \ + --limit 50 --n_drafts 8 +``` + +### World-model energy selector + +```bash +uv run --package ensemble python -m ensemble.eval.world_harness \ + --llm tiny --toy --limit 20 --n_drafts 8 + +uv run --package ensemble python -m ensemble.eval.world_harness \ + --llm Qwen/Qwen2.5-0.5B-Instruct \ + --qa research/data/benchmark-qa.jsonl \ + --kb research/data/benchmark-kb.jsonl \ + --limit 50 --n_drafts 8 +``` + +## Layout + +``` +research/ensemble/ + src/ensemble/ + backends.py # TinyBackend, HFBackend, TinyLLM, HFLLM + memory.py # Embedder, VectorStore, Router + jepa.py # JEPA latent predictor + bridge.py # LLM hidden -> latent alignment + world_model.py # Latent dynamics + rollout + energy.py # Energy-based critic + jepa_ensemble.py # Ensemble (JEPA track) + world_ensemble.py # WorldEnsemble + eval/ + metrics.py + jepa_harness.py + world_harness.py +``` diff --git a/research/ensemble/pyproject.toml b/research/ensemble/pyproject.toml new file mode 100644 index 0000000000000000000000000000000000000000..334dc1f6f3c6bf3ae9987aff00e1d0d668298a31 --- /dev/null +++ b/research/ensemble/pyproject.toml @@ -0,0 +1,16 @@ +[project] +name = "ensemble" +version = "0.1.0" +description = "JEPA and world-model ensemble research package" +readme = "README.md" +requires-python = ">=3.12" +dependencies = [ + "torch>=2.5.0", +] + +[project.scripts] +ensemble-pretrain = "ensemble.pretrain:main" + +[build-system] +requires = ["uv_build>=0.8.13,<0.9.0"] +build-backend = "uv_build" diff --git a/research/ensemble/scripts/smoke.sh b/research/ensemble/scripts/smoke.sh new file mode 100755 index 0000000000000000000000000000000000000000..7e21bce6d159ea810995bb9f14317d4ef0868571 --- /dev/null +++ b/research/ensemble/scripts/smoke.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT="$(cd "$(dirname "$0")/../../.." && pwd)" +cd "$ROOT" + +echo "== JEPA ensemble demo (tiny) ==" +uv run --package ensemble python -m ensemble.jepa_ensemble tiny + +echo "" +echo "== World ensemble demo (tiny) ==" +uv run --package ensemble python -m ensemble.world_ensemble tiny + +echo "" +echo "== JEPA harness (toy) ==" +uv run --package ensemble python -m ensemble.eval.jepa_harness \ + --llm tiny --toy --limit 10 --n_drafts 4 + +echo "== Pretrain smoke + checkpoint roundtrip ==" +uv run --package ensemble ensemble-pretrain \ + --llm tiny --steps 20 --no-kb \ + --out models/ensemble/jepa-smoke +uv run --package ensemble python -c " +from ensemble.checkpoint import load_checkpoint +ens = load_checkpoint('models/ensemble/jepa-smoke') +print('loaded ensemble, adapters:', ens.adapter_names) +" + +echo "" +echo "== World harness (toy) ==" +uv run --package ensemble python -m ensemble.eval.world_harness \ + --llm tiny --toy --limit 10 --n_drafts 4 + +echo "" +echo "All smoke checks passed." diff --git a/research/ensemble/src/ensemble/__init__.py b/research/ensemble/src/ensemble/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..36fb435baa63e9559ae38c7cb38b4c4d6164a39d --- /dev/null +++ b/research/ensemble/src/ensemble/__init__.py @@ -0,0 +1,15 @@ +"""Research ensemble package: JEPA and world-model tracks.""" + +__all__ = ["Ensemble", "WorldEnsemble"] + + +def __getattr__(name: str): + if name == "Ensemble": + from ensemble.jepa_ensemble import Ensemble + + return Ensemble + if name == "WorldEnsemble": + from ensemble.world_ensemble import WorldEnsemble + + return WorldEnsemble + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") diff --git a/research/ensemble/src/ensemble/backends.py b/research/ensemble/src/ensemble/backends.py new file mode 100644 index 0000000000000000000000000000000000000000..e4fbbf234a61852d1e26860c99b8fc191bbd72c1 --- /dev/null +++ b/research/ensemble/src/ensemble/backends.py @@ -0,0 +1,418 @@ +"""LLM backends: toy fallbacks and HuggingFace + LoRA loaders.""" + +from __future__ import annotations + +import torch +import torch.nn as nn +import torch.nn.functional as F + + +class LLMBackend(nn.Module): + """Contract for JEPA ensemble backends.""" + + vocab_size: int + hidden_size: int + + +class HFBackend(LLMBackend): + """HuggingFace causal LM with PEFT LoRA adapter bank.""" + + def __init__( + self, + model_path: str, + *, + load_in_4bit: bool = False, + lora_r: int = 16, + lora_alpha: int = 32, + target_modules=("q_proj", "v_proj"), + device: str | None = None, + torch_dtype=None, + ): + super().__init__() + from peft import LoraConfig, get_peft_model + from transformers import AutoModelForCausalLM, AutoTokenizer + + self.device_ = torch.device( + device or ("cuda" if torch.cuda.is_available() else "cpu") + ) + + kwargs = {} + if load_in_4bit: + from transformers import BitsAndBytesConfig + + kwargs["quantization_config"] = BitsAndBytesConfig( + load_in_4bit=True, + bnb_4bit_compute_dtype=torch.bfloat16, + bnb_4bit_quant_type="nf4", + ) + if torch_dtype is not None: + kwargs["torch_dtype"] = torch_dtype + + self.tokenizer = AutoTokenizer.from_pretrained(model_path) + if self.tokenizer.pad_token is None: + self.tokenizer.pad_token = self.tokenizer.eos_token + base = AutoModelForCausalLM.from_pretrained(model_path, **kwargs) + if not load_in_4bit: + base.to(self.device_) + + for p in base.parameters(): + p.requires_grad_(False) + + self._lora_cfg = LoraConfig( + r=lora_r, + lora_alpha=lora_alpha, + lora_dropout=0.05, + target_modules=list(target_modules), + task_type="CAUSAL_LM", + ) + self.model = get_peft_model(base, self._lora_cfg, adapter_name="general") + self._adapters = {"general"} + + self.vocab_size = self.model.config.vocab_size + self.hidden_size = self.model.config.hidden_size + + def add_adapter(self, name: str): + if name not in self._adapters: + self.model.add_adapter(name, self._lora_cfg) + self._adapters.add(name) + + def set_adapter(self, name: str): + self.model.set_adapter(name) + + def trainable_parameters(self): + return (p for p in self.model.parameters() if p.requires_grad) + + def forward(self, ids): + out = self.model( + input_ids=ids.to(self.device_), output_hidden_states=True + ) + return out.logits, out.hidden_states[-1] + + @torch.no_grad() + def generate(self, ids, n_new=64, temperature=0.8): + gen_kwargs: dict = dict( + input_ids=ids.to(self.device_), + max_new_tokens=n_new, + pad_token_id=self.tokenizer.pad_token_id, + ) + if temperature <= 0: + gen_kwargs["do_sample"] = False + else: + gen_kwargs.update(do_sample=True, temperature=temperature) + out = self.model.generate(**gen_kwargs) + return out + + def encode_text(self, text: str): + return self.tokenizer(text, return_tensors="pt").input_ids.to(self.device_) + + def decode(self, ids): + return self.tokenizer.decode(ids[0], skip_special_tokens=True) + + @property + def device(self): + return self.device_ + + +class TinyBackend(LLMBackend): + """Toy transformer with LoRA adapters (no transformers dependency).""" + + VOCAB, D_MODEL, N_LAYERS, N_HEADS, SEQ_LEN, LORA_R = 1000, 128, 2, 4, 32, 8 + + class _LoRALinear(nn.Module): + def __init__(self, d_in, d_out, r): + super().__init__() + self.base = nn.Linear(d_in, d_out) + self.base.weight.requires_grad_(False) + self.base.bias.requires_grad_(False) + self.adapters, self.active, self.r = nn.ModuleDict(), None, r + + def add_adapter(self, name): + A = nn.Linear(self.base.in_features, self.r, bias=False) + B = nn.Linear(self.r, self.base.out_features, bias=False) + nn.init.zeros_(B.weight) + self.adapters[name] = nn.Sequential(A, B) + + def forward(self, x): + y = self.base(x) + if self.active and self.active in self.adapters: + y = y + self.adapters[self.active](x) + return y + + class _Block(nn.Module): + def __init__(self, D, H, R): + super().__init__() + L = TinyBackend._LoRALinear + self.ln1 = nn.LayerNorm(D) + self.attn = nn.MultiheadAttention(D, H, batch_first=True) + self.ln2 = nn.LayerNorm(D) + self.up, self.down = L(D, 4 * D, R), L(4 * D, D, R) + + def forward(self, x, mask): + h = self.ln1(x) + a, _ = self.attn(h, h, h, attn_mask=mask, need_weights=False) + x = x + a + return x + self.down(F.gelu(self.up(self.ln2(x)))) + + def __init__(self): + super().__init__() + D, V = self.D_MODEL, self.VOCAB + self.tok = nn.Embedding(V, D) + self.pos = nn.Embedding(self.SEQ_LEN * 4, D) + self.blocks = nn.ModuleList( + [self._Block(D, self.N_HEADS, self.LORA_R) for _ in range(self.N_LAYERS)] + ) + self.ln_f, self.head = nn.LayerNorm(D), nn.Linear(D, V, bias=False) + self.vocab_size, self.hidden_size = V, D + self.add_adapter("general") + self.set_adapter("general") + + def add_adapter(self, name): + for b in self.blocks: + b.up.add_adapter(name) + b.down.add_adapter(name) + + def set_adapter(self, name): + for b in self.blocks: + b.up.active = name + b.down.active = name + + def trainable_parameters(self): + return (p for p in self.parameters() if p.requires_grad) + + def forward(self, ids): + B, T = ids.shape + x = self.tok(ids) + self.pos(torch.arange(T, device=ids.device)) + mask = torch.triu( + torch.full((T, T), float("-inf"), device=ids.device), 1 + ) + for b in self.blocks: + x = b(x, mask) + h = self.ln_f(x) + return self.head(h), h + + @torch.no_grad() + def generate(self, ids, n_new=16, temperature=1.0): + for _ in range(n_new): + logits, _ = self(ids[:, -self.SEQ_LEN :]) + if temperature <= 0: + nxt = logits[:, -1].argmax(dim=-1, keepdim=True) + else: + nxt = torch.multinomial( + F.softmax(logits[:, -1] / temperature, -1), 1 + ) + ids = torch.cat([ids, nxt], dim=1) + return ids + + def encode_text(self, text: str): + vals = [ord(c) % self.vocab_size for c in text[: self.SEQ_LEN]] + if not vals: + vals = [0] + return torch.tensor([vals], dtype=torch.long) + + def decode(self, ids): + return " ".join(str(int(t)) for t in ids[0].tolist()) + + @property + def device(self): + return next(self.parameters()).device + + +def make_backend(llm: str, **kw) -> LLMBackend: + """'tiny' -> toy model; anything else -> HF hub id or local path.""" + return TinyBackend() if llm == "tiny" else HFBackend(llm, **kw) + + +def load_hf_backend_from_checkpoint( + base_llm: str, + adapter_dir: str | None, + *, + adapter_names: tuple[str, ...] = ("general",), + device: str | None = None, + load_in_4bit: bool = False, + lora_r: int = 16, + lora_alpha: int = 32, +) -> HFBackend: + """Load a frozen base LM + saved PEFT adapters (ensemble checkpoint llm/).""" + from pathlib import Path + + from peft import LoraConfig, PeftModel, get_peft_model + from transformers import AutoModelForCausalLM, AutoTokenizer + + def _discover_adapter_dirs(root: Path) -> dict[str, Path]: + if (root / "adapter_config.json").is_file(): + return {"general": root} + discovered: dict[str, Path] = {} + for child in sorted(root.iterdir()): + if child.is_dir() and (child / "adapter_config.json").is_file(): + discovered[child.name] = child + return discovered + + resolved_device = device or ("cuda" if torch.cuda.is_available() else "cpu") + tokenizer = AutoTokenizer.from_pretrained(adapter_dir or base_llm) + if tokenizer.pad_token is None: + tokenizer.pad_token = tokenizer.eos_token + + kwargs: dict = {} + if load_in_4bit: + from transformers import BitsAndBytesConfig + + kwargs["quantization_config"] = BitsAndBytesConfig( + load_in_4bit=True, + bnb_4bit_compute_dtype=torch.bfloat16, + bnb_4bit_quant_type="nf4", + ) + elif resolved_device != "cpu": + kwargs["torch_dtype"] = torch.bfloat16 + + base = AutoModelForCausalLM.from_pretrained(base_llm, **kwargs) + if not load_in_4bit and resolved_device != "cpu": + base.to(resolved_device) + for p in base.parameters(): + p.requires_grad_(False) + + if adapter_dir: + adapter_dirs = _discover_adapter_dirs(Path(adapter_dir)) + if not adapter_dirs: + raise ValueError( + f"No PEFT adapters found under {adapter_dir} " + "(expected adapter_config.json or /adapter_config.json)" + ) + preferred = [name for name in adapter_names if name in adapter_dirs] + load_order = preferred + [ + name for name in adapter_dirs if name not in preferred + ] + first_name = load_order[0] + model = PeftModel.from_pretrained( + base, + str(adapter_dirs[first_name]), + adapter_name=first_name, + is_trainable=False, + ) + for name in load_order[1:]: + model.load_adapter(str(adapter_dirs[name]), adapter_name=name) + adapters = set(load_order) + else: + lora_cfg = LoraConfig( + r=lora_r, + lora_alpha=lora_alpha, + lora_dropout=0.05, + target_modules=["q_proj", "v_proj"], + task_type="CAUSAL_LM", + ) + model = get_peft_model(base, lora_cfg, adapter_name="general") + adapters = {"general"} + + backend = HFBackend.__new__(HFBackend) + nn.Module.__init__(backend) + backend.device_ = torch.device(resolved_device) + backend.tokenizer = tokenizer + backend.model = model + backend._lora_cfg = None + backend._adapters = adapters + backend.vocab_size = model.config.vocab_size + backend.hidden_size = model.config.hidden_size + if adapter_names: + backend.set_adapter(adapter_names[0]) + return backend + + +class TinyLLM(nn.Module): + """Simpler toy LLM for the world-model track (no adapter bank).""" + + VOCAB, D, L, H, T = 1000, 128, 2, 4, 32 + + def __init__(self): + super().__init__() + self.tok = nn.Embedding(self.VOCAB, self.D) + self.pos = nn.Embedding(self.T * 4, self.D) + layer = nn.TransformerEncoderLayer( + self.D, self.H, 4 * self.D, batch_first=True, norm_first=True + ) + self.blocks = nn.TransformerEncoder(layer, self.L) + self.head = nn.Linear(self.D, self.VOCAB, bias=False) + self.vocab_size, self.hidden_size = self.VOCAB, self.D + + def forward(self, ids): + Tn = ids.size(1) + x = self.tok(ids) + self.pos(torch.arange(Tn, device=ids.device)) + mask = torch.triu( + torch.full((Tn, Tn), float("-inf"), device=ids.device), 1 + ) + h = self.blocks(x, mask=mask) + return self.head(h), h + + @torch.no_grad() + def generate(self, ids, n_new=16, temperature=1.0): + for _ in range(n_new): + logits, _ = self(ids[:, -self.T :]) + nxt = torch.multinomial( + F.softmax(logits[:, -1] / temperature, -1), 1 + ) + ids = torch.cat([ids, nxt], 1) + return ids + + def trainable_parameters(self): + return self.parameters() + + @property + def device(self): + return next(self.parameters()).device + + +class HFLLM(nn.Module): + """Small HF model with single LoRA stack (world-model track).""" + + def __init__(self, path, lora_r=16): + super().__init__() + from peft import LoraConfig, get_peft_model + from transformers import AutoModelForCausalLM, AutoTokenizer + + self.tokenizer = AutoTokenizer.from_pretrained(path) + if self.tokenizer.pad_token is None: + self.tokenizer.pad_token = self.tokenizer.eos_token + base = AutoModelForCausalLM.from_pretrained( + path, + torch_dtype=torch.bfloat16 + if torch.cuda.is_available() + else torch.float32, + device_map="auto" if torch.cuda.is_available() else None, + ) + for p in base.parameters(): + p.requires_grad_(False) + cfg = LoraConfig( + r=lora_r, + lora_alpha=2 * lora_r, + lora_dropout=0.05, + target_modules=["q_proj", "v_proj"], + task_type="CAUSAL_LM", + ) + self.model = get_peft_model(base, cfg) + self.vocab_size = self.model.config.vocab_size + self.hidden_size = self.model.config.hidden_size + + def forward(self, ids): + out = self.model( + input_ids=ids.to(self.device), output_hidden_states=True + ) + return out.logits, out.hidden_states[-1] + + @torch.no_grad() + def generate(self, ids, n_new=32, temperature=0.8): + return self.model.generate( + input_ids=ids.to(self.device), + max_new_tokens=n_new, + do_sample=True, + temperature=temperature, + pad_token_id=self.tokenizer.pad_token_id, + ) + + def trainable_parameters(self): + return (p for p in self.model.parameters() if p.requires_grad) + + @property + def device(self): + return next(self.model.parameters()).device + + +def load_llm(spec: str): + return TinyLLM() if spec == "tiny" else HFLLM(spec) diff --git a/research/ensemble/src/ensemble/bridge.py b/research/ensemble/src/ensemble/bridge.py new file mode 100644 index 0000000000000000000000000000000000000000..0b8828c3ca62fafecd404aeb727a114fdc2807f1 --- /dev/null +++ b/research/ensemble/src/ensemble/bridge.py @@ -0,0 +1,28 @@ +"""Bridge: align LLM hidden states with JEPA latent space.""" + +from __future__ import annotations + +import torch +import torch.nn as nn +import torch.nn.functional as F + + +class Bridge(nn.Module): + def __init__(self, d_llm_hidden: int, d_latent: int): + super().__init__() + self.proj = nn.Sequential( + nn.Linear(d_llm_hidden, d_latent), + nn.GELU(), + nn.Linear(d_latent, d_latent), + ) + + def forward(self, llm_hidden): + return self.proj(llm_hidden.float().mean(dim=1)) + + def info_nce(self, z1, z2, tau=0.07): + z1, z2 = F.normalize(z1, dim=-1), F.normalize(z2, dim=-1) + logits = z1 @ z2.t() / tau + labels = torch.arange(z1.size(0), device=z1.device) + return 0.5 * ( + F.cross_entropy(logits, labels) + F.cross_entropy(logits.t(), labels) + ) diff --git a/research/ensemble/src/ensemble/checkpoint.py b/research/ensemble/src/ensemble/checkpoint.py new file mode 100644 index 0000000000000000000000000000000000000000..6684cd26e18eb56ece19fed3cdc576d8337eaede --- /dev/null +++ b/research/ensemble/src/ensemble/checkpoint.py @@ -0,0 +1,149 @@ +"""Save and load JEPA ensemble checkpoints under models/ensemble/.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +import torch + +from ensemble.backends import TinyBackend, load_hf_backend_from_checkpoint +from ensemble.jepa_ensemble import Ensemble + +MANIFEST_FILE = "manifest.json" +AUX_FILE = "aux.pt" +STORE_FILE = "store.pt" +LLM_DIR = "llm" +TINY_LLM_FILE = "tiny_llm.pt" + +CHECKPOINT_VERSION = 1 + + +def _aux_state_dict(ens: Ensemble) -> dict[str, torch.Tensor]: + return { + "emb": ens.emb.state_dict(), + "jepa": ens.jepa.state_dict(), + "bridge": ens.bridge.state_dict(), + "router": ens.router.state_dict(), + } + + +def _store_payload(ens: Ensemble) -> dict[str, Any]: + return { + "keys": [k for k in ens.store.keys], + "values": [v for v in ens.store.values], + } + + +def save_checkpoint( + ens: Ensemble, + out_dir: str | Path, + *, + base_llm: str, + training_meta: dict[str, Any] | None = None, +) -> Path: + """Persist ensemble (LLM adapters + emb + JEPA + bridge + router + store).""" + root = Path(out_dir).resolve() + root.mkdir(parents=True, exist_ok=True) + + backend = "tiny" if isinstance(ens.llm, TinyBackend) else "hf" + manifest: dict[str, Any] = { + "version": CHECKPOINT_VERSION, + "track": "jepa", + "backend": backend, + "base_llm": base_llm, + "adapter_names": list(ens.adapter_names), + "d_emb": ens.emb.d_emb, + "d_jepa": ens.jepa.d_latent, + "training": training_meta or {}, + } + + torch.save(_aux_state_dict(ens), root / AUX_FILE) + store = _store_payload(ens) + if store["keys"]: + torch.save(store, root / STORE_FILE) + + if backend == "hf": + llm_path = root / LLM_DIR + llm_path.mkdir(exist_ok=True) + ens.llm.model.save_pretrained(llm_path) + ens.llm.tokenizer.save_pretrained(llm_path) + else: + torch.save(ens.llm.state_dict(), root / TINY_LLM_FILE) + + with open(root / MANIFEST_FILE, "w") as f: + json.dump(manifest, f, indent=2) + + return root + + +def is_ensemble_checkpoint(path: str | Path) -> bool: + return (Path(path) / MANIFEST_FILE).is_file() + + +def load_checkpoint( + ckpt_dir: str | Path, + *, + device: str | None = None, + load_in_4bit: bool = False, +) -> Ensemble: + """Restore a saved JEPA ensemble from models/ensemble//.""" + root = Path(ckpt_dir).resolve() + manifest_path = root / MANIFEST_FILE + if not manifest_path.is_file(): + raise FileNotFoundError( + f"Not an ensemble checkpoint (missing {MANIFEST_FILE}): {root}" + ) + + with open(manifest_path) as f: + manifest = json.load(f) + + base_llm = manifest["base_llm"] + backend = manifest.get("backend", "hf") + adapter_names = tuple(manifest.get("adapter_names", ["general"])) + d_emb = manifest.get("d_emb", 64) + d_jepa = manifest.get("d_jepa", 64) + + if backend == "tiny": + ens = Ensemble( + llm="tiny", + adapter_names=adapter_names, + d_emb=d_emb, + d_jepa=d_jepa, + ) + tiny_state = torch.load( + root / TINY_LLM_FILE, map_location="cpu", weights_only=True + ) + ens.llm.load_state_dict(tiny_state) + else: + llm_dir = root / LLM_DIR + llm_backend = load_hf_backend_from_checkpoint( + base_llm, + str(llm_dir) if llm_dir.is_dir() else None, + adapter_names=adapter_names, + device=device, + load_in_4bit=load_in_4bit, + ) + ens = Ensemble( + llm=base_llm, + adapter_names=adapter_names, + d_emb=d_emb, + d_jepa=d_jepa, + llm_backend=llm_backend, + ) + + aux = torch.load(root / AUX_FILE, map_location="cpu", weights_only=True) + ens.emb.load_state_dict(aux["emb"]) + ens.jepa.load_state_dict(aux["jepa"]) + ens.bridge.load_state_dict(aux["bridge"]) + ens.router.load_state_dict(aux["router"]) + + store_path = root / STORE_FILE + if store_path.is_file(): + store = torch.load(store_path, map_location="cpu", weights_only=True) + ens.store.keys = list(store["keys"]) + ens.store.values = list(store["values"]) + + ens.eval() + return ens diff --git a/research/ensemble/src/ensemble/config.py b/research/ensemble/src/ensemble/config.py new file mode 100644 index 0000000000000000000000000000000000000000..a2533d4fdafc18a7a663500c9e30652fcfd133d9 --- /dev/null +++ b/research/ensemble/src/ensemble/config.py @@ -0,0 +1,163 @@ +"""Resolve base LLM for ensemble from .env and models.yaml (same order as finetune).""" + +from __future__ import annotations + +import os +import sys +from pathlib import Path + +_REPO_ROOT = Path(__file__).resolve().parents[4] +_FALLBACK_PRESET = "minicpm5-1b" + +_ENV_LLM_KEYS = ( + "ENSEMBLE_LLM", + "LLM_PATH", + "BASE", + "FINETUNE_MODEL", + "MODEL_ID", +) + + +def repo_root() -> Path: + return _REPO_ROOT + + +def load_dotenv() -> None: + """Load KEY=VALUE pairs from repo .env without overriding existing env vars.""" + path = _REPO_ROOT / ".env" + if not path.is_file(): + return + for line in path.read_text().splitlines(): + line = line.strip() + if not line or line.startswith("#") or "=" not in line: + continue + key, _, value = line.partition("=") + key = key.strip() + value = value.strip().strip('"').strip("'") + if key: + os.environ.setdefault(key, value) + + +def _ensure_inference_on_path() -> None: + libs = _REPO_ROOT / "libs" / "inference" / "src" + if str(libs) not in sys.path: + sys.path.insert(0, str(libs)) + + +def _is_ensemble_llm_preset(model) -> bool: + return model.backend == "transformers" and not model.multimodal and bool( + model.model_id + ) + + +def _llm_from_local_path(raw: str) -> str | None: + path = Path(raw) + if not path.is_absolute(): + path = (_REPO_ROOT / path).resolve() + if path.suffix == ".gguf": + return None + if path.is_dir() and (path / "config.json").is_file(): + return str(path) + if path.is_file(): + return None + return None + + +def _llm_from_env_paths() -> str | None: + for key in ("LLM_PATH", "MODEL_PATH"): + raw = os.environ.get(key) + if raw: + resolved = _llm_from_local_path(raw) + if resolved: + return resolved + return None + + +def resolve_llm( + *, + llm_arg: str | None = None, + preset_arg: str | None = None, +) -> tuple[str, str | None]: + """ + Return (hub_id_or_local_path, preset_key) for ensemble HF backends. + + Priority when llm_arg is None or ``auto``: + 1. ENSEMBLE_LLM, LLM_PATH (local HF dir), BASE, FINETUNE_MODEL, MODEL_ID + 2. MODEL_PATH if it points at a HuggingFace model directory (not .gguf) + 3. ENSEMBLE_PRESET, FINETUNE_PRESET, or ACTIVE_MODEL from models.yaml + 4. First fine-tunable transformers preset (default minicpm5-1b) + """ + if llm_arg and llm_arg not in ("auto",): + return llm_arg, preset_arg + + for env_name in _ENV_LLM_KEYS: + raw = os.environ.get(env_name) + if raw: + local = _llm_from_local_path(raw) + return local or raw, preset_arg + + local = _llm_from_env_paths() + if local: + return local, preset_arg + + _ensure_inference_on_path() + from inference.config import get_app_config, get_model_config + + app_config = get_app_config(reload=True) + preset_key = ( + preset_arg + or os.environ.get("ENSEMBLE_PRESET") + or os.environ.get("FINETUNE_PRESET") + or os.environ.get("ACTIVE_MODEL") + ) + + if preset_key and preset_key in app_config.models: + model = get_model_config(preset_key) + if not _is_ensemble_llm_preset(model): + preset_key = None + + if preset_key is None: + for candidate in (_FALLBACK_PRESET, *app_config.models): + if candidate not in app_config.models: + continue + model = get_model_config(candidate) + if _is_ensemble_llm_preset(model): + preset_key = candidate + break + + if not preset_key: + raise SystemExit( + "No transformers LLM found for ensemble. Pass --llm, set LLM_PATH/BASE/" + "MODEL_ID in .env, or ACTIVE_MODEL in models.yaml." + ) + + model = get_model_config(preset_key) + if not _is_ensemble_llm_preset(model): + raise SystemExit( + f"Preset {preset_key!r} cannot back an ensemble " + f"(backend={model.backend}, multimodal={model.multimodal})." + ) + return model.model_id, preset_key + + +def default_ensemble_out(preset_key: str | None) -> str: + label = preset_key or "custom" + return str((_REPO_ROOT / "models" / "ensemble" / f"{label}-jepa-pretrain").resolve()) + + +def resolve_llm_cli( + llm: str | None, + *, + toy: bool = False, + preset: str | None = None, +) -> str: + """CLI helper: explicit tiny, else .env / models.yaml unless --toy without --llm.""" + if llm == "tiny": + return "tiny" + if llm is None or llm == "auto": + if toy: + return "tiny" + load_dotenv() + resolved, _ = resolve_llm(preset_arg=preset) + return resolved + return llm diff --git a/research/ensemble/src/ensemble/energy.py b/research/ensemble/src/ensemble/energy.py new file mode 100644 index 0000000000000000000000000000000000000000..896eab28a477dbd21a30a2214ce4cad0da2137ec --- /dev/null +++ b/research/ensemble/src/ensemble/energy.py @@ -0,0 +1,45 @@ +"""Energy model: score candidate latents against world state.""" + +from __future__ import annotations + +import torch +import torch.nn as nn +import torch.nn.functional as F + + +class EnergyModel(nn.Module): + def __init__(self, d_latent: int): + super().__init__() + self.net = nn.Sequential( + nn.Linear(2 * d_latent, 2 * d_latent), + nn.GELU(), + nn.Linear(2 * d_latent, d_latent), + nn.GELU(), + nn.Linear(d_latent, 1), + ) + self.d_latent = d_latent + + def energy(self, s, z): + return self.net(torch.cat([s, z], -1)).squeeze(-1) + + def contrastive_loss(self, s, z_pos, z_negs=None, tau=0.5): + B = s.size(0) + s_rep = s.unsqueeze(1).expand(B, B, self.d_latent).reshape( + B * B, self.d_latent + ) + z_rep = z_pos.unsqueeze(0).expand(B, B, self.d_latent).reshape( + B * B, self.d_latent + ) + E = self.energy(s_rep, z_rep).view(B, B) + if z_negs is not None: + En = self.energy( + s.repeat_interleave(z_negs.size(1), 0), + z_negs.reshape(-1, self.d_latent), + ).view(B, -1) + E = torch.cat([E, En], dim=1) + labels = torch.arange(B, device=s.device) + return F.cross_entropy(-E / tau, labels) + + @torch.no_grad() + def rank(self, s, candidates): + return self.energy(s.expand(candidates.size(0), -1), candidates) diff --git a/research/ensemble/src/ensemble/eval/__init__.py b/research/ensemble/src/ensemble/eval/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..ece222c6d792a74ca7688c0b9f1ae6d71a216848 --- /dev/null +++ b/research/ensemble/src/ensemble/eval/__init__.py @@ -0,0 +1 @@ +"""Evaluation harnesses for JEPA and world-model ensembles.""" diff --git a/research/ensemble/src/ensemble/eval/jepa_harness.py b/research/ensemble/src/ensemble/eval/jepa_harness.py new file mode 100644 index 0000000000000000000000000000000000000000..912bb66fa42766aeb3855e7ffe2f27bb4cf0757c --- /dev/null +++ b/research/ensemble/src/ensemble/eval/jepa_harness.py @@ -0,0 +1,266 @@ +"""Ablation ladder + JEPA best-of-N benchmark for the ensemble.""" + +from __future__ import annotations + +import argparse +import json +import random +import time +from collections import defaultdict + +import torch +import torch.nn.functional as F + +from ensemble.eval.metrics import em_score, f1_score, paired_bootstrap +from ensemble.backends import TinyBackend +from ensemble.checkpoint import load_checkpoint +from ensemble.config import load_dotenv, resolve_llm_cli +from ensemble.jepa_ensemble import Ensemble + + +@torch.no_grad() +def generate_plain(ens, q_ids, n_new): + ens.llm.set_adapter(ens.adapter_names[0]) + t0 = time.time() + out = ens.llm.generate(q_ids.to(ens.llm.device), n_new=n_new, temperature=0.7) + return out[:, q_ids.size(1) :], time.time() - t0 + + +@torch.no_grad() +def generate_config( + ens, q_ids, n_new, *, use_rag, use_router, use_jepa, n_drafts=1, tau=0.0 +): + q_emb = ens.emb(q_ids.cpu()) + + if use_router: + a_idx = ens.router(q_emb).item() + ens.llm.set_adapter(ens.adapter_names[a_idx]) + else: + ens.llm.set_adapter(ens.adapter_names[0]) + + ctx = q_ids.cpu() + if use_rag: + mems = ens.store.search(q_emb, k=1) + if mems: + ctx = torch.cat([mems[0], ctx], dim=1) + + t0 = time.time() + if not use_jepa: + out = ens.llm.generate( + ctx.to(ens.llm.device), n_new=n_new, temperature=0.7 + ) + return out[:, ctx.size(1) :], time.time() - t0, None + + z_exp = ens.jepa.predict_next_latent(ctx) + drafts, scores = [], [] + for _ in range(n_drafts): + out = ens.llm.generate( + ctx.to(ens.llm.device), n_new=n_new, temperature=0.9 + ) + new = out[:, ctx.size(1) :].cpu() + drafts.append(new) + scores.append( + F.cosine_similarity(z_exp, ens.jepa.encode(new)).item() + ) + best = max(range(n_drafts), key=lambda i: scores[i]) + return drafts[best], time.time() - t0, (drafts, scores) + + +def selector_comparison(drafts_scores_gold, decode_fn, rng): + res = defaultdict(list) + for drafts, scores, gold in drafts_scores_gold: + texts = [decode_fn(d) for d in drafts] + ems = [em_score(t, gold) for t in texts] + res["first"].append(ems[0]) + res["random"].append(ems[rng.randrange(len(ems))]) + res["jepa"].append(ems[max(range(len(ems)), key=lambda i: scores[i])]) + res["oracle"].append(max(ems)) + return {k: sum(v) / len(v) for k, v in res.items()}, res + + +def load_jsonl(path): + with open(path) as f: + return [json.loads(line) for line in f if line.strip()] + + +def make_toy_data(ens, n_qa=20, vocab=None): + vocab = vocab or ens.llm.vocab_size + qa, kb = [], [] + for _ in range(n_qa): + key = torch.randint(0, vocab, (1, 6)) + ans = torch.randint(0, vocab, (1, 4)) + kb.append(torch.cat([key, ans], dim=1)) + qa.append({"q_ids": key, "answer_ids": ans}) + return qa, kb + + +def run(args): + torch.manual_seed(args.seed) + rng = random.Random(args.seed) + + if args.ckpt: + ens = load_checkpoint(args.ckpt) + print(f"loaded ensemble checkpoint: {args.ckpt}") + is_text = not isinstance(ens.llm, TinyBackend) + else: + load_dotenv() + args.llm = resolve_llm_cli( + args.llm, toy=args.toy, preset=getattr(args, "preset", None) + ) + print(f"Resolved LLM: {args.llm}") + ens = Ensemble(llm=args.llm) + is_text = args.llm != "tiny" + + if args.toy or not is_text: + qa, kb = make_toy_data(ens) + for mem in kb: + ens.memorize_ids(mem) + + def to_ids(item): + return item["q_ids"] + + def gold_text(item): + return " ".join(map(str, item["answer_ids"][0].tolist())) + + def decode(ids): + return " ".join(map(str, ids[0].tolist())) + else: + qa = load_jsonl(args.qa) + if args.kb: + for row in load_jsonl(args.kb): + ens.memorize_text(row["text"]) + + def to_ids(item): + return ens.llm.encode_text( + f"Answer briefly.\nQ: {item['question']}\nA:" + ) + + def gold_text(item): + return item["answer"] + + def decode(ids): + return ens.llm.decode(ids) + + qa = qa[: args.limit] + print( + f"eval set: {len(qa)} questions | store: {len(ens.store.keys)} memories\n" + ) + + configs = { + "C1_base": dict(use_rag=False, use_router=False, use_jepa=False), + "C2_rag": dict(use_rag=True, use_router=False, use_jepa=False), + "C3_rag_router": dict(use_rag=True, use_router=True, use_jepa=False), + "C4_full_jepa": dict( + use_rag=True, + use_router=True, + use_jepa=True, + n_drafts=args.n_drafts, + ), + } + + per_q = {} + summary = {} + jepa_material = [] + + for name, cfg in configs.items(): + ems, f1s, lats = [], [], [] + for item in qa: + ids = to_ids(item) + if name == "C1_base": + out, dt = generate_plain(ens, ids, args.n_new) + extra = None + else: + out, dt, extra = generate_config(ens, ids, args.n_new, **cfg) + pred, gold = decode(out), gold_text(item) + ems.append(em_score(pred, gold)) + f1s.append(f1_score(pred, gold)) + lats.append(dt) + if name == "C4_full_jepa" and extra is not None: + jepa_material.append((extra[0], extra[1], gold)) + per_q[name] = ems + summary[name] = ( + sum(ems) / len(ems), + sum(f1s) / len(f1s), + sum(lats) / len(lats), + ) + + print(f"{'config':<16}{'EM':>8}{'F1':>8}{'lat(s)':>9}") + for k, (em, f1, lat) in summary.items(): + print(f"{k:<16}{em:>8.3f}{f1:>8.3f}{lat:>9.3f}") + + print("\ncomponent contributions (paired bootstrap, P(B>A)):") + ladder = list(configs.keys()) + for a, b in zip(ladder, ladder[1:]): + d = summary[b][0] - summary[a][0] + p = paired_bootstrap(per_q[a], per_q[b]) + print(f" {b} - {a}: ΔEM={d:+.3f} P(better)={p:.2f}") + + if jepa_material: + sel, sel_per_q = selector_comparison(jepa_material, decode, rng) + print( + f"\nbest-of-N selector comparison (same drafts, N={args.n_drafts}):" + ) + for k in ("first", "random", "jepa", "oracle"): + print(f" {k:<8}EM={sel[k]:.3f}") + p = paired_bootstrap(sel_per_q["random"], sel_per_q["jepa"]) + verdict = ( + "JEPA critic WORKS" + if p > 0.95 + else "inconclusive — critic ~ random" + ) + print(f" P(jepa > random) = {p:.2f} {verdict}") + print(f" headroom to oracle: {sel['oracle'] - sel['jepa']:.3f}") + + if args.continual: + print( + "\ncontinual test: accuracy on task-A questions " + "before vs after adding adapters B and C" + ) + ems_before = per_q["C3_rag_router"] + ens.new_task_adapter("task_B") + ens.new_task_adapter("task_C") + ems_after = [] + for item in qa: + out, _, _ = generate_config( + ens, + to_ids(item), + args.n_new, + use_rag=True, + use_router=True, + use_jepa=False, + ) + ems_after.append(em_score(decode(out), gold_text(item))) + bt = sum(ems_after) / len(ems_after) - sum(ems_before) / len( + ems_before + ) + print(f" backward transfer (≈0 is ideal): {bt:+.3f}") + + return summary + + +def parse_args(): + p = argparse.ArgumentParser() + p.add_argument( + "--llm", + default=None, + help="HF id / path, 'tiny', or omit for LLM_PATH / ACTIVE_MODEL from .env", + ) + p.add_argument("--preset", default=None, help="models.yaml preset override") + p.add_argument("--qa", default=None, help="jsonl with question/answer") + p.add_argument("--kb", default=None, help="jsonl with text -> vector store") + p.add_argument( + "--ckpt", + default=None, + help="saved ensemble directory (models/ensemble/... with manifest.json)", + ) + p.add_argument("--toy", action="store_true", help="synthetic data smoke test") + p.add_argument("--limit", type=int, default=100) + p.add_argument("--n_new", type=int, default=24) + p.add_argument("--n_drafts", type=int, default=8) + p.add_argument("--continual", action="store_true") + p.add_argument("--seed", type=int, default=0) + return p.parse_args() + + +if __name__ == "__main__": + run(parse_args()) diff --git a/research/ensemble/src/ensemble/eval/metrics.py b/research/ensemble/src/ensemble/eval/metrics.py new file mode 100644 index 0000000000000000000000000000000000000000..4885e1dbd094aa777f351f7d495f275d9a577d9a --- /dev/null +++ b/research/ensemble/src/ensemble/eval/metrics.py @@ -0,0 +1,42 @@ +"""QA metrics and paired bootstrap significance.""" + +from __future__ import annotations + +import random +import re +import string +from collections import Counter + + +def normalize(s: str) -> str: + s = s.lower() + s = "".join(c for c in s if c not in string.punctuation) + s = re.sub(r"\b(a|an|the)\b", " ", s) + return " ".join(s.split()) + + +def em_score(pred: str, gold: str) -> float: + return float(normalize(gold) in normalize(pred)) + + +def f1_score(pred: str, gold: str) -> float: + p, g = normalize(pred).split(), normalize(gold).split() + if not p or not g: + return float(p == g) + common = Counter(p) & Counter(g) + overlap = sum(common.values()) + if overlap == 0: + return 0.0 + prec, rec = overlap / len(p), overlap / len(g) + return 2 * prec * rec / (prec + rec) + + +def paired_bootstrap(scores_a, scores_b, iters=2000, seed=0): + rng = random.Random(seed) + n, wins = len(scores_a), 0 + for _ in range(iters): + idx = [rng.randrange(n) for _ in range(n)] + da = sum(scores_a[i] for i in idx) / n + db = sum(scores_b[i] for i in idx) / n + wins += db > da + return wins / iters diff --git a/research/ensemble/src/ensemble/eval/world_harness.py b/research/ensemble/src/ensemble/eval/world_harness.py new file mode 100644 index 0000000000000000000000000000000000000000..2fb9c420574deac33e7407cc73c86e519d7d085d --- /dev/null +++ b/research/ensemble/src/ensemble/eval/world_harness.py @@ -0,0 +1,174 @@ +"""Energy-based draft selector benchmark for the world-model ensemble.""" + +from __future__ import annotations + +import argparse +import json +import random +import time +from collections import defaultdict + +import torch + +from ensemble.eval.metrics import em_score, f1_score, paired_bootstrap +from ensemble.world_ensemble import WorldEnsemble + + +@torch.no_grad() +def generate_drafts(ens, q_ids, n_new, n_drafts, use_rag=True): + q_emb = ens.emb(q_ids.cpu()) + mems = ens.store.search(q_emb, k=1) if use_rag else [] + segments = (mems + [q_ids.cpu()]) if mems else [q_ids.cpu()] + ctx = torch.cat(segments, dim=1) + + s = ens.world_state(segments) + ens.world.rollout(s, horizon=3) + + drafts, energies = [], [] + t0 = time.time() + for _ in range(n_drafts): + out = ens.llm.generate( + ctx.to(ens.llm.device), n_new=n_new, temperature=0.9 + ) + new = out[:, ctx.size(1) :].cpu() + drafts.append(new) + z = ens.jepa.encode(new) + energies.append(ens.energy.rank(s, z).item()) + return drafts, energies, time.time() - t0 + + +def selector_comparison(drafts_energy_gold, decode_fn, rng): + res = defaultdict(list) + for drafts, energies, gold in drafts_energy_gold: + texts = [decode_fn(d) for d in drafts] + ems = [em_score(t, gold) for t in texts] + res["first"].append(ems[0]) + res["random"].append(ems[rng.randrange(len(ems))]) + res["energy"].append( + ems[min(range(len(ems)), key=lambda i: energies[i])] + ) + res["oracle"].append(max(ems)) + return {k: sum(v) / len(v) for k, v in res.items()}, res + + +def load_jsonl(path): + with open(path) as f: + return [json.loads(line) for line in f if line.strip()] + + +def make_toy_data(ens, n_qa=20, vocab=None): + vocab = vocab or ens.llm.vocab_size + qa, kb = [], [] + for _ in range(n_qa): + key = torch.randint(0, vocab, (1, 6)) + ans = torch.randint(0, vocab, (1, 4)) + kb.append(torch.cat([key, ans], dim=1)) + qa.append({"q_ids": key, "answer_ids": ans}) + return qa, kb + + +def run(args): + from ensemble.config import load_dotenv, resolve_llm_cli + + torch.manual_seed(args.seed) + rng = random.Random(args.seed) + + load_dotenv() + args.llm = resolve_llm_cli( + args.llm, toy=args.toy, preset=getattr(args, "preset", None) + ) + print(f"Resolved LLM: {args.llm}") + ens = WorldEnsemble(args.llm) + if args.ckpt: + state = torch.load(args.ckpt, map_location="cpu") + ens.load_state_dict(state, strict=False) + print(f"loaded world ensemble checkpoint: {args.ckpt}") + + is_text = args.llm != "tiny" + + if args.toy or not is_text: + qa, kb = make_toy_data(ens) + for mem in kb: + ens.memorize(mem) + + def to_ids(item): + return item["q_ids"] + + def gold_text(item): + return " ".join(map(str, item["answer_ids"][0].tolist())) + + def decode(ids): + return " ".join(map(str, ids[0].tolist())) + else: + qa = load_jsonl(args.qa) + if args.kb: + for row in load_jsonl(args.kb): + ids = ens.llm.tokenizer( + row["text"], return_tensors="pt" + ).input_ids + ens.memorize(ids) + + def to_ids(item): + return ens.llm.tokenizer( + f"Answer briefly.\nQ: {item['question']}\nA:", + return_tensors="pt", + ).input_ids + + def gold_text(item): + return item["answer"] + + def decode(ids): + return ens.llm.tokenizer.decode(ids[0], skip_special_tokens=True) + + qa = qa[: args.limit] + print( + f"eval set: {len(qa)} questions | store: {len(ens.store.keys)} memories\n" + ) + + material = [] + lats = [] + for item in qa: + drafts, energies, dt = generate_drafts( + ens, to_ids(item), args.n_new, args.n_drafts + ) + material.append((drafts, energies, gold_text(item))) + lats.append(dt) + + sel, sel_per_q = selector_comparison(material, decode, rng) + print(f"best-of-N selector comparison (same drafts, N={args.n_drafts}):") + for k in ("first", "random", "energy", "oracle"): + print(f" {k:<8}EM={sel[k]:.3f}") + p = paired_bootstrap(sel_per_q["random"], sel_per_q["energy"]) + verdict = ( + "Energy critic WORKS" + if p > 0.95 + else "inconclusive — critic ~ random" + ) + print(f" P(energy > random) = {p:.2f} {verdict}") + print(f" headroom to oracle: {sel['oracle'] - sel['energy']:.3f}") + print(f" mean latency: {sum(lats) / len(lats):.3f}s") + + return sel + + +def parse_args(): + p = argparse.ArgumentParser() + p.add_argument( + "--llm", + default=None, + help="HF id / path, 'tiny', or omit for LLM_PATH / ACTIVE_MODEL from .env", + ) + p.add_argument("--preset", default=None, help="models.yaml preset override") + p.add_argument("--qa", default=None, help="jsonl with question/answer") + p.add_argument("--kb", default=None, help="jsonl with text -> vector store") + p.add_argument("--ckpt", default=None, help="trained world ensemble .pt") + p.add_argument("--toy", action="store_true", help="synthetic data smoke test") + p.add_argument("--limit", type=int, default=100) + p.add_argument("--n_new", type=int, default=24) + p.add_argument("--n_drafts", type=int, default=8) + p.add_argument("--seed", type=int, default=0) + return p.parse_args() + + +if __name__ == "__main__": + run(parse_args()) diff --git a/research/ensemble/src/ensemble/eval_harness.py b/research/ensemble/src/ensemble/eval_harness.py new file mode 100644 index 0000000000000000000000000000000000000000..901619e9d90de58fcb01aadbcf542f2ff19cae99 --- /dev/null +++ b/research/ensemble/src/ensemble/eval_harness.py @@ -0,0 +1,309 @@ +""" +eval_harness.py — Ablation ladder + JEPA best-of-N test for the ensemble +======================================================================== +Companion to `llm_emb_jepa_ensemble_pluggable.py` (must be importable, +i.e. in the same directory). + +What it runs +------------ +1. ABLATION LADDER on a QA set: + C1 base LLM alone + C2 C1 + RAG (embedding retrieval) + C3 C2 + router/adapters + C4 C3 + JEPA best-of-N critic + (C5 = C4 with a bridge-trained checkpoint — just pass --ckpt) + +2. BEST-OF-N SELECTOR comparison (the decisive JEPA experiment): + first-sample | random-pick | JEPA-score pick | oracle pick + All on the SAME N drafts per question, so differences are pure selection. + +3. CONTINUAL FORGETTING test (optional, --continual): + accuracy on task A before vs after training adapters for B and C. + +4. PAIRED BOOTSTRAP significance between any two configs. + +Usage +----- +# Smoke test, no GPU/deps beyond torch (toy backend, synthetic QA): +python eval_harness.py --llm tiny --toy + +# Real model + your QA file (jsonl: {"question": ..., "answer": ..., "context": optional}): +python eval_harness.py --llm Qwen/Qwen2.5-0.5B-Instruct \ + --qa ./domain_qa.jsonl --kb ./knowledge.jsonl --n_drafts 8 + +# With a bridge-trained ensemble checkpoint (C5): +python eval_harness.py --llm /models/llama-3.2-1b --qa ./qa.jsonl \ + --kb ./kb.jsonl --ckpt ./ensemble_bridge.pt + +QA file: {"question": str, "answer": str, "domain": optional str} +KB file: {"text": str} (each line becomes one memory in the vector store) +""" + +import argparse +import json +import random +import re +import string +import time +from collections import Counter, defaultdict + +import torch + +from llm_emb_jepa_ensemble_pluggable import Ensemble # same directory + +# ---------------------------------------------------------------------------- +# Metrics: normalized exact match + token F1 (SQuAD-style) +# ---------------------------------------------------------------------------- +def normalize(s: str) -> str: + s = s.lower() + s = "".join(c for c in s if c not in string.punctuation) + s = re.sub(r"\b(a|an|the)\b", " ", s) + return " ".join(s.split()) + + +def em_score(pred: str, gold: str) -> float: + return float(normalize(gold) in normalize(pred)) # containment EM + + +def f1_score(pred: str, gold: str) -> float: + p, g = normalize(pred).split(), normalize(gold).split() + if not p or not g: + return float(p == g) + common = Counter(p) & Counter(g) + overlap = sum(common.values()) + if overlap == 0: + return 0.0 + prec, rec = overlap / len(p), overlap / len(g) + return 2 * prec * rec / (prec + rec) + + +# ---------------------------------------------------------------------------- +# Paired bootstrap: P(config B beats config A) +# ---------------------------------------------------------------------------- +def paired_bootstrap(scores_a, scores_b, iters=2000, seed=0): + rng = random.Random(seed) + n, wins = len(scores_a), 0 + for _ in range(iters): + idx = [rng.randrange(n) for _ in range(n)] + da = sum(scores_a[i] for i in idx) / n + db = sum(scores_b[i] for i in idx) / n + wins += db > da + return wins / iters + + +# ---------------------------------------------------------------------------- +# Config runners — each returns per-question dicts +# ---------------------------------------------------------------------------- +@torch.no_grad() +def generate_plain(ens, q_ids, n_new): + """C1: base adapter, no retrieval, single sample.""" + ens.llm.set_adapter(ens.adapter_names[0]) + t0 = time.time() + out = ens.llm.generate(q_ids.to(ens.llm.device), n_new=n_new, temperature=0.7) + return out[:, q_ids.size(1):], time.time() - t0 + + +@torch.no_grad() +def generate_config(ens, q_ids, n_new, *, use_rag, use_router, use_jepa, + n_drafts=1, tau=0.0): + """Unified runner for C2/C3/C4.""" + q_emb = ens.emb(q_ids.cpu()) + + if use_router: + a_idx = ens.router(q_emb).item() + ens.llm.set_adapter(ens.adapter_names[a_idx]) + else: + ens.llm.set_adapter(ens.adapter_names[0]) + + ctx = q_ids.cpu() + if use_rag: + mems = ens.store.search(q_emb, k=1) + if mems: + ctx = torch.cat([mems[0], ctx], dim=1) + + t0 = time.time() + if not use_jepa: + out = ens.llm.generate(ctx.to(ens.llm.device), n_new=n_new, temperature=0.7) + return out[:, ctx.size(1):], time.time() - t0, None + + # JEPA best-of-N: sample drafts, keep the one closest to predicted latent + z_exp = ens.jepa.predict_next_latent(ctx) + drafts, scores = [], [] + for _ in range(n_drafts): + out = ens.llm.generate(ctx.to(ens.llm.device), n_new=n_new, temperature=0.9) + new = out[:, ctx.size(1):].cpu() + drafts.append(new) + scores.append(torch.nn.functional.cosine_similarity( + z_exp, ens.jepa.encode(new)).item()) + best = max(range(n_drafts), key=lambda i: scores[i]) + return drafts[best], time.time() - t0, (drafts, scores) + + +# ---------------------------------------------------------------------------- +# Best-of-N selector comparison on shared drafts +# ---------------------------------------------------------------------------- +def selector_comparison(drafts_scores_gold, decode_fn, rng): + """drafts_scores_gold: list of (drafts, jepa_scores, gold_answer). + Returns EM for: first | random | jepa | oracle — all on the SAME drafts.""" + res = defaultdict(list) + for drafts, scores, gold in drafts_scores_gold: + texts = [decode_fn(d) for d in drafts] + ems = [em_score(t, gold) for t in texts] + res["first"].append(ems[0]) + res["random"].append(ems[rng.randrange(len(ems))]) + res["jepa"].append(ems[max(range(len(ems)), key=lambda i: scores[i])]) + res["oracle"].append(max(ems)) # upper bound of selection + return {k: sum(v) / len(v) for k, v in res.items()}, res + + +# ---------------------------------------------------------------------------- +# Data loading +# ---------------------------------------------------------------------------- +def load_jsonl(path): + with open(path) as f: + return [json.loads(l) for l in f if l.strip()] + + +def make_toy_data(ens, n_qa=20, vocab=None): + """Synthetic QA for the tiny backend: 'answer' token sequence is planted + in the KB so RAG can genuinely help even with random weights.""" + vocab = vocab or ens.llm.vocab_size + qa, kb = [], [] + for i in range(n_qa): + key = torch.randint(0, vocab, (1, 6)) + ans = torch.randint(0, vocab, (1, 4)) + kb.append(torch.cat([key, ans], dim=1)) # memory = key+answer + qa.append({"q_ids": key, "answer_ids": ans}) + return qa, kb + + +# ---------------------------------------------------------------------------- +# Main evaluation +# ---------------------------------------------------------------------------- +def run(args): + torch.manual_seed(args.seed) + rng = random.Random(args.seed) + + ens = Ensemble(llm=args.llm) + if args.ckpt: + state = torch.load(args.ckpt, map_location="cpu") + ens.load_state_dict(state, strict=False) + print(f"loaded ensemble checkpoint: {args.ckpt}") + + is_text = args.llm != "tiny" + + # ---- load data and fill the vector store ------------------------------- + if args.toy or not is_text: + qa, kb = make_toy_data(ens) + for mem in kb: + ens.memorize_ids(mem) + def to_ids(item): return item["q_ids"] + def gold_of(item): return item["answer_ids"] + def decode(ids): return " ".join(map(str, ids[0].tolist())) + def gold_text(item): return decode(item["answer_ids"]) + else: + qa = load_jsonl(args.qa) + if args.kb: + for row in load_jsonl(args.kb): + ens.memorize_text(row["text"]) + def to_ids(item): return ens.llm.encode_text( + f"Answer briefly.\nQ: {item['question']}\nA:") + def gold_text(item): return item["answer"] + def decode(ids): return ens.llm.decode(ids) + + qa = qa[: args.limit] + print(f"eval set: {len(qa)} questions | store: {len(ens.store.keys)} memories\n") + + # ---- ablation ladder ---------------------------------------------------- + configs = { + "C1_base": dict(use_rag=False, use_router=False, use_jepa=False), + "C2_rag": dict(use_rag=True, use_router=False, use_jepa=False), + "C3_rag_router": dict(use_rag=True, use_router=True, use_jepa=False), + "C4_full_jepa": dict(use_rag=True, use_router=True, use_jepa=True, + n_drafts=args.n_drafts), + } + + per_q = {} # config -> list of EM scores (for bootstrap) + summary = {} + jepa_material = [] # (drafts, scores, gold) for selector comparison + + for name, cfg in configs.items(): + ems, f1s, lats = [], [], [] + for item in qa: + ids = to_ids(item) + if name == "C1_base": + out, dt = generate_plain(ens, ids, args.n_new) + extra = None + else: + out, dt, extra = generate_config(ens, ids, args.n_new, **cfg) + pred, gold = decode(out), gold_text(item) + ems.append(em_score(pred, gold)) + f1s.append(f1_score(pred, gold)) + lats.append(dt) + if name == "C4_full_jepa" and extra is not None: + jepa_material.append((extra[0], extra[1], gold)) + per_q[name] = ems + summary[name] = (sum(ems) / len(ems), sum(f1s) / len(f1s), + sum(lats) / len(lats)) + + print(f"{'config':<16}{'EM':>8}{'F1':>8}{'lat(s)':>9}") + for k, (em, f1, lat) in summary.items(): + print(f"{k:<16}{em:>8.3f}{f1:>8.3f}{lat:>9.3f}") + + # deltas + significance + print("\ncomponent contributions (paired bootstrap, P(B>A)):") + ladder = list(configs.keys()) + for a, b in zip(ladder, ladder[1:]): + d = summary[b][0] - summary[a][0] + p = paired_bootstrap(per_q[a], per_q[b]) + print(f" {b} - {a}: ΔEM={d:+.3f} P(better)={p:.2f}") + + # ---- decisive JEPA selector experiment ---------------------------------- + if jepa_material: + sel, sel_per_q = selector_comparison(jepa_material, decode, rng) + print("\nbest-of-N selector comparison (same drafts, N=" + f"{args.n_drafts}):") + for k in ("first", "random", "jepa", "oracle"): + print(f" {k:<8}EM={sel[k]:.3f}") + p = paired_bootstrap(sel_per_q["random"], sel_per_q["jepa"]) + print(f" P(jepa > random) = {p:.2f} " + f"{'JEPA critic WORKS' if p > 0.95 else 'inconclusive — critic ~ random'}") + gap = sel["oracle"] - sel["jepa"] + print(f" headroom to oracle: {gap:.3f}") + + # ---- continual forgetting (optional) ------------------------------------ + if args.continual: + print("\ncontinual test: accuracy on task-A questions " + "before vs after adding adapters B and C") + ems_before = per_q["C3_rag_router"] + ens.new_task_adapter("task_B") + ens.new_task_adapter("task_C") + ems_after = [] + for item in qa: + out, _, _ = generate_config(ens, to_ids(item), args.n_new, + use_rag=True, use_router=True, + use_jepa=False) + ems_after.append(em_score(decode(out), gold_text(item))) + bt = sum(ems_after) / len(ems_after) - sum(ems_before) / len(ems_before) + print(f" backward transfer (≈0 is ideal): {bt:+.3f}") + + return summary + + +def parse_args(): + p = argparse.ArgumentParser() + p.add_argument("--llm", default="tiny", help="'tiny' | HF id | local path") + p.add_argument("--qa", default=None, help="jsonl with question/answer") + p.add_argument("--kb", default=None, help="jsonl with text -> vector store") + p.add_argument("--ckpt", default=None, help="bridge-trained ensemble .pt (C5)") + p.add_argument("--toy", action="store_true", help="synthetic data smoke test") + p.add_argument("--limit", type=int, default=100) + p.add_argument("--n_new", type=int, default=24) + p.add_argument("--n_drafts", type=int, default=8) + p.add_argument("--continual", action="store_true") + p.add_argument("--seed", type=int, default=0) + return p.parse_args() + + +if __name__ == "__main__": + run(parse_args()) diff --git a/research/ensemble/src/ensemble/jepa.py b/research/ensemble/src/ensemble/jepa.py new file mode 100644 index 0000000000000000000000000000000000000000..5bf433a8c035c3296909abad7ceaaa2f78816702 --- /dev/null +++ b/research/ensemble/src/ensemble/jepa.py @@ -0,0 +1,75 @@ +"""JEPA latent predictor with EMA target encoder.""" + +from __future__ import annotations + +import copy + +import torch +import torch.nn as nn +import torch.nn.functional as F + + +class _SegEncoder(nn.Module): + def __init__(self, vocab_size, d): + super().__init__() + self.tok = nn.Embedding(vocab_size, d) + self.enc = nn.GRU(d, d, batch_first=True) + self.out = nn.Linear(d, d) + + def forward(self, ids): + h, _ = self.enc(self.tok(ids)) + return self.out(h.mean(dim=1)) + + +class JEPA(nn.Module): + def __init__(self, vocab_size: int, d_latent: int = 64, ema_m: float = 0.996): + super().__init__() + self.ctx_enc = _SegEncoder(vocab_size, d_latent) + self.tgt_enc = copy.deepcopy(self.ctx_enc) + for p in self.tgt_enc.parameters(): + p.requires_grad_(False) + self.predictor = nn.Sequential( + nn.Linear(d_latent, 2 * d_latent), + nn.GELU(), + nn.Linear(2 * d_latent, d_latent), + ) + self.m = ema_m + self.d_latent = d_latent + + @property + def enc(self): + """Alias used by world-model track.""" + return self.ctx_enc + + @property + def tgt(self): + return self.tgt_enc + + @property + def pred(self): + return self.predictor + + @torch.no_grad() + def ema_update(self): + for p_t, p_c in zip(self.tgt_enc.parameters(), self.ctx_enc.parameters()): + p_t.mul_(self.m).add_(p_c.detach(), alpha=1 - self.m) + + def ema(self): + """Alias used by world-model track.""" + self.ema_update() + + def loss(self, seg_ctx, seg_tgt): + z_hat = self.predictor(self.ctx_enc(seg_ctx)) + with torch.no_grad(): + z_tgt = self.tgt_enc(seg_tgt) + pred = F.mse_loss(z_hat, z_tgt) + var_reg = F.relu(1.0 - z_hat.std(dim=0)).mean() + return pred + 0.5 * var_reg + + @torch.no_grad() + def predict_next_latent(self, seg_ctx): + return self.predictor(self.ctx_enc(seg_ctx)) + + @torch.no_grad() + def encode(self, seg): + return self.tgt_enc(seg) diff --git a/research/ensemble/src/ensemble/jepa_ensemble.py b/research/ensemble/src/ensemble/jepa_ensemble.py new file mode 100644 index 0000000000000000000000000000000000000000..d3f26d1b5adfe107e7aaf43e3781efaf38dc5021 --- /dev/null +++ b/research/ensemble/src/ensemble/jepa_ensemble.py @@ -0,0 +1,232 @@ +"""JEPA ensemble: route -> retrieve -> generate -> JEPA-verify.""" + +from __future__ import annotations + +import torch +import torch.nn as nn +import torch.nn.functional as F + +from ensemble.backends import HFBackend, make_backend +from ensemble.bridge import Bridge +from ensemble.jepa import JEPA +from ensemble.memory import Embedder, Router, VectorStore + +torch.manual_seed(0) + + +class Ensemble(nn.Module): + def __init__( + self, + llm: str = "tiny", + adapter_names=("general",), + d_emb: int = 64, + d_jepa: int = 64, + llm_backend: HFBackend | None = None, + **backend_kw, + ): + super().__init__() + self.llm = llm_backend if llm_backend is not None else make_backend(llm, **backend_kw) + V, H = self.llm.vocab_size, self.llm.hidden_size + + self.emb = Embedder(V, d_emb) + self.jepa = JEPA(V, d_jepa) + self.bridge = Bridge(H, d_jepa) + self.store = VectorStore() + + self.adapter_names = list(adapter_names) + for n in self.adapter_names: + self.llm.add_adapter(n) + self.llm.set_adapter(self.adapter_names[0]) + self.router = Router(d_emb, len(self.adapter_names)) + + @torch.no_grad() + def answer_ids( + self, + query_ids, + n_new=32, + tau_consistency=0.0, + max_retries=2, + temperature: float = 0.7, + ): + q_emb = self.emb(query_ids.cpu()) + a_idx = self.router(q_emb).item() + self.llm.set_adapter(self.adapter_names[a_idx]) + + mems = self.store.search(q_emb, k=1) + ctx = ( + torch.cat([mems[0], query_ids.cpu()], dim=1) + if mems + else query_ids.cpu() + ) + + z_expected = self.jepa.predict_next_latent(ctx) + + best = None + for attempt in range(max_retries + 1): + temp = temperature if attempt == 0 else max(temperature, 0.8 + 0.3 * attempt) + draft = self.llm.generate( + ctx.to(self.llm.device), + n_new=n_new, + temperature=temp, + ) + new_part = draft[:, ctx.size(1) :].cpu() + score = F.cosine_similarity( + z_expected, self.jepa.encode(new_part) + ).item() + if best is None or score > best[1]: + best = (draft, score, attempt) + if score >= tau_consistency: + break + draft, score, attempt = best + return draft, score, self.adapter_names[a_idx], attempt + + def answer_text(self, prompt: str, **kw): + ids = self.llm.encode_text(prompt) + out, score, adapter, retries = self.answer_ids(ids, **kw) + return self.llm.decode(out), score, adapter, retries + + def generate_text( + self, + prompt: str, + *, + max_new_tokens: int = 512, + temperature: float = 0.0, + ) -> str: + """Greedy or sampled generation through the full ensemble stack.""" + ids = self.llm.encode_text(prompt) + out, _, _, _ = self.answer_ids( + ids, + n_new=max_new_tokens, + tau_consistency=-1.0, + max_retries=0 if temperature <= 0 else 1, + temperature=temperature, + ) + return self.llm.decode(out) + + def memorize_ids(self, ids): + self.store.add(self.emb(ids.cpu()), ids.cpu()) + + def memorize_text(self, text: str): + self.memorize_ids(self.llm.encode_text(text)) + + def new_task_adapter(self, name: str): + self.adapter_names.append(name) + self.llm.add_adapter(name) + old = self.router + self.router = Router(self.emb.d_emb, len(self.adapter_names)) + with torch.no_grad(): + self.router.fc.weight[: old.fc.out_features] = old.fc.weight + self.router.fc.bias[: old.fc.out_features] = old.fc.bias + + def train_step(self, seg_a, seg_b, opt, w_bridge=0.1): + logits, hidden = self.llm(seg_a.to(self.llm.device)) + lm_loss = F.cross_entropy( + logits[:, :-1].reshape(-1, self.llm.vocab_size).float(), + seg_a[:, 1:].reshape(-1).to(logits.device), + ) + + jepa_loss = self.jepa.loss(seg_a.cpu(), seg_b.cpu()) + + z_llm = self.bridge( + hidden.cpu() if hidden.device.type != "cpu" else hidden + ) + z_jepa = self.jepa.ctx_enc(seg_a.cpu()).detach() + bridge_loss = self.bridge.info_nce(z_llm, z_jepa.to(z_llm.device)) + + loss = lm_loss.cpu() + jepa_loss + w_bridge * bridge_loss + opt.zero_grad() + loss.backward() + opt.step() + self.jepa.ema_update() + return { + "lm": lm_loss.item(), + "jepa": jepa_loss.item(), + "bridge": bridge_loss.item(), + } + + def make_optimizer(self, lr_lora=2e-4, lr_aux=1e-3): + return torch.optim.AdamW( + [ + {"params": list(self.llm.trainable_parameters()), "lr": lr_lora}, + { + "params": list(self.jepa.ctx_enc.parameters()) + + list(self.jepa.predictor.parameters()), + "lr": lr_aux, + }, + { + "params": list(self.bridge.parameters()) + + list(self.emb.parameters()) + + list(self.router.parameters()), + "lr": lr_aux, + }, + ] + ) + + +def segment_pairs_from_texts(backend: HFBackend, texts, seg_len=64): + a_list, b_list = [], [] + for t in texts: + ids = backend.tokenizer(t, return_tensors="pt").input_ids[0] + for i in range(0, len(ids) - 2 * seg_len, seg_len): + a_list.append(ids[i : i + seg_len]) + b_list.append(ids[i + seg_len : i + 2 * seg_len]) + if not a_list: + raise ValueError("texts too short for the chosen seg_len") + return torch.stack(a_list), torch.stack(b_list) + + +def demo_tiny(steps=50): + ens = Ensemble(llm="tiny") + opt = ens.make_optimizer() + for s in range(steps): + seg_a = torch.randint(0, ens.llm.vocab_size, (8, 32)) + seg_b = torch.randint(0, ens.llm.vocab_size, (8, 32)) + logs = ens.train_step(seg_a, seg_b, opt) + if s % 10 == 0: + print( + f"step {s:3d} | " + + " | ".join(f"{k} {v:.3f}" for k, v in logs.items()) + ) + + for _ in range(5): + ens.memorize_ids(torch.randint(0, ens.llm.vocab_size, (1, 32))) + ens.new_task_adapter("medical") + + q = torch.randint(0, ens.llm.vocab_size, (1, 8)) + out, score, adapter, retries = ens.answer_ids(q, tau_consistency=-1.0) + print(f"\nadapter={adapter} jepa_consistency={score:.3f} retries={retries}") + + +def demo_hf(model_path="Qwen/Qwen2.5-0.5B-Instruct"): + ens = Ensemble(llm=model_path, load_in_4bit=False) + opt = ens.make_optimizer() + + texts = ["Replace this with your real corpus. " * 50] + seg_a, seg_b = segment_pairs_from_texts(ens.llm, texts, seg_len=32) + for s in range(10): + logs = ens.train_step(seg_a[:4], seg_b[:4], opt) + print(f"step {s} | " + " | ".join(f"{k} {v:.3f}" for k, v in logs.items())) + + ens.memorize_text("The project codename is AURORA and it ships in Q3.") + ens.new_task_adapter("project_aurora") + + text, score, adapter, retries = ens.answer_text( + "What is the project codename?", n_new=24, tau_consistency=-1.0 + ) + print(f"\n[{adapter} | jepa={score:.3f} | retries={retries}]\n{text}") + + +if __name__ == "__main__": + import sys + + from ensemble.config import load_dotenv, resolve_llm + + load_dotenv() + arg = sys.argv[1] if len(sys.argv) > 1 else None + if arg is None or arg == "auto": + arg, preset = resolve_llm() + print(f"Resolved LLM: {arg} (preset {preset})") + if arg == "tiny": + demo_tiny() + else: + demo_hf(arg) diff --git a/research/ensemble/src/ensemble/llm_emb_jepa_ensemble_pluggable.py b/research/ensemble/src/ensemble/llm_emb_jepa_ensemble_pluggable.py new file mode 100644 index 0000000000000000000000000000000000000000..09f2b6919779d798efdcdc0f632e84ca7540e348 --- /dev/null +++ b/research/ensemble/src/ensemble/llm_emb_jepa_ensemble_pluggable.py @@ -0,0 +1,507 @@ +""" +LLM + Embedding + JEPA Ensemble — pluggable base-model edition +============================================================== +Now the LLM is a swappable BACKEND. Three ways to load it: + + # 1. HuggingFace Hub id + ens = Ensemble(llm="Qwen/Qwen2.5-0.5B-Instruct") + + # 2. Local path (e.g. downloaded Llama / converted checkpoint) + ens = Ensemble(llm="/models/llama-3.2-1b") + + # 3. Toy fallback (no transformers needed, runs on CPU in seconds) + ens = Ensemble(llm="tiny") + +Requirements for real models: + pip install torch transformers peft accelerate + (optional 4-bit: pip install bitsandbytes -> load_in_4bit=True) + +Everything else (Embedder, JEPA, Bridge, VectorStore, Router, the +JEPA-critic inference loop, continual-learning hooks) only touches +token ids / hidden states / latents, so it works with ANY backend. +""" + +from __future__ import annotations +import copy +import torch +import torch.nn as nn +import torch.nn.functional as F + +torch.manual_seed(0) + +# ---------------------------------------------------------------------------- +# 0. Backend interface — everything the ensemble needs from "an LLM" +# ---------------------------------------------------------------------------- +class LLMBackend(nn.Module): + """Contract: + vocab_size : int + hidden_size: int + device : torch.device + forward(ids) -> (logits [B,T,V], hidden [B,T,H]) + generate(ids, n_new) -> ids [B, T+n_new] + add_adapter(name) / set_adapter(name) + trainable_parameters() -> iterable of params to optimize + encode_text(str) / decode(ids) (real backends only) + """ + vocab_size: int + hidden_size: int + + +# ---------------------------------------------------------------------------- +# 0a. HuggingFace backend (local path OR hub id) with PEFT LoRA adapters +# ---------------------------------------------------------------------------- +class HFBackend(LLMBackend): + def __init__(self, model_path: str, *, load_in_4bit: bool = False, + lora_r: int = 16, lora_alpha: int = 32, + target_modules=("q_proj", "v_proj"), + device: str | None = None, torch_dtype=None): + super().__init__() + from transformers import AutoModelForCausalLM, AutoTokenizer + from peft import LoraConfig, get_peft_model + + self.device_ = torch.device( + device or ("cuda" if torch.cuda.is_available() else "cpu")) + + kwargs = {} + if load_in_4bit: + from transformers import BitsAndBytesConfig + kwargs["quantization_config"] = BitsAndBytesConfig( + load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16, + bnb_4bit_quant_type="nf4") + if torch_dtype is not None: + kwargs["torch_dtype"] = torch_dtype + + # `model_path` may be "Qwen/Qwen2.5-0.5B-Instruct", "meta-llama/...", + # or a local directory like "/models/llama-3.2-1b". + self.tokenizer = AutoTokenizer.from_pretrained(model_path) + if self.tokenizer.pad_token is None: + self.tokenizer.pad_token = self.tokenizer.eos_token + base = AutoModelForCausalLM.from_pretrained(model_path, **kwargs) + if not load_in_4bit: + base.to(self.device_) + + # Freeze the base; all learning happens in LoRA adapters. + for p in base.parameters(): + p.requires_grad_(False) + + self._lora_cfg = LoraConfig( + r=lora_r, lora_alpha=lora_alpha, lora_dropout=0.05, + target_modules=list(target_modules), task_type="CAUSAL_LM") + self.model = get_peft_model(base, self._lora_cfg, adapter_name="general") + self._adapters = {"general"} + + self.vocab_size = self.model.config.vocab_size + self.hidden_size = self.model.config.hidden_size + + # ---- adapters ----------------------------------------------------------- + def add_adapter(self, name: str): + if name not in self._adapters: + self.model.add_adapter(name, self._lora_cfg) + self._adapters.add(name) + + def set_adapter(self, name: str): + self.model.set_adapter(name) + + def trainable_parameters(self): + return (p for p in self.model.parameters() if p.requires_grad) + + # ---- core ops ----------------------------------------------------------- + def forward(self, ids): + out = self.model(input_ids=ids.to(self.device_), + output_hidden_states=True) + return out.logits, out.hidden_states[-1] # last layer hidden + + @torch.no_grad() + def generate(self, ids, n_new=64, temperature=0.8): + out = self.model.generate( + input_ids=ids.to(self.device_), + max_new_tokens=n_new, do_sample=True, temperature=temperature, + pad_token_id=self.tokenizer.pad_token_id) + return out + + # ---- text helpers ------------------------------------------------------- + def encode_text(self, text: str): + return self.tokenizer(text, return_tensors="pt").input_ids.to(self.device_) + + def decode(self, ids): + return self.tokenizer.decode(ids[0], skip_special_tokens=True) + + @property + def device(self): + return self.device_ + + +# ---------------------------------------------------------------------------- +# 0b. Tiny fallback backend (no transformers; same toy model as before) +# ---------------------------------------------------------------------------- +class TinyBackend(LLMBackend): + VOCAB, D_MODEL, N_LAYERS, N_HEADS, SEQ_LEN, LORA_R = 1000, 128, 2, 4, 32, 8 + + class _LoRALinear(nn.Module): + def __init__(self, d_in, d_out, r): + super().__init__() + self.base = nn.Linear(d_in, d_out) + self.base.weight.requires_grad_(False) + self.base.bias.requires_grad_(False) + self.adapters, self.active, self.r = nn.ModuleDict(), None, r + + def add_adapter(self, name): + A = nn.Linear(self.base.in_features, self.r, bias=False) + B = nn.Linear(self.r, self.base.out_features, bias=False) + nn.init.zeros_(B.weight) + self.adapters[name] = nn.Sequential(A, B) + + def forward(self, x): + y = self.base(x) + if self.active and self.active in self.adapters: + y = y + self.adapters[self.active](x) + return y + + class _Block(nn.Module): + def __init__(self, D, H, R): + super().__init__() + L = TinyBackend._LoRALinear + self.ln1 = nn.LayerNorm(D) + self.attn = nn.MultiheadAttention(D, H, batch_first=True) + self.ln2 = nn.LayerNorm(D) + self.up, self.down = L(D, 4 * D, R), L(4 * D, D, R) + + def forward(self, x, mask): + h = self.ln1(x) + a, _ = self.attn(h, h, h, attn_mask=mask, need_weights=False) + x = x + a + return x + self.down(F.gelu(self.up(self.ln2(x)))) + + def __init__(self): + super().__init__() + D, V = self.D_MODEL, self.VOCAB + self.tok = nn.Embedding(V, D) + self.pos = nn.Embedding(self.SEQ_LEN * 4, D) + self.blocks = nn.ModuleList( + [self._Block(D, self.N_HEADS, self.LORA_R) for _ in range(self.N_LAYERS)]) + self.ln_f, self.head = nn.LayerNorm(D), nn.Linear(D, V, bias=False) + self.vocab_size, self.hidden_size = V, D + self.add_adapter("general") + self.set_adapter("general") + + def add_adapter(self, name): + for b in self.blocks: + b.up.add_adapter(name); b.down.add_adapter(name) + + def set_adapter(self, name): + for b in self.blocks: + b.up.active = name; b.down.active = name + + def trainable_parameters(self): + return (p for p in self.parameters() if p.requires_grad) + + def forward(self, ids): + B, T = ids.shape + x = self.tok(ids) + self.pos(torch.arange(T, device=ids.device)) + mask = torch.triu(torch.full((T, T), float("-inf"), device=ids.device), 1) + for b in self.blocks: + x = b(x, mask) + h = self.ln_f(x) + return self.head(h), h + + @torch.no_grad() + def generate(self, ids, n_new=16, temperature=1.0): + for _ in range(n_new): + logits, _ = self(ids[:, -self.SEQ_LEN:]) + nxt = torch.multinomial(F.softmax(logits[:, -1] / temperature, -1), 1) + ids = torch.cat([ids, nxt], dim=1) + return ids + + @property + def device(self): + return next(self.parameters()).device + + +def make_backend(llm: str, **kw) -> LLMBackend: + """'tiny' -> toy model; anything else -> HF hub id or local path.""" + return TinyBackend() if llm == "tiny" else HFBackend(llm, **kw) + + +# ---------------------------------------------------------------------------- +# 1. Embedder — vocab-agnostic (sized from the backend's tokenizer) +# Swap for a real model: pass embed_fn=lambda txt: sbert.encode(...) +# ---------------------------------------------------------------------------- +class Embedder(nn.Module): + def __init__(self, vocab_size: int, d_emb: int = 64): + super().__init__() + self.tok = nn.Embedding(vocab_size, d_emb) + self.enc = nn.GRU(d_emb, d_emb, batch_first=True, bidirectional=True) + self.proj = nn.Linear(2 * d_emb, d_emb) + self.d_emb = d_emb + + def forward(self, ids): + h, _ = self.enc(self.tok(ids)) + return F.normalize(self.proj(h.mean(dim=1)), dim=-1) + + +# ---------------------------------------------------------------------------- +# 2. JEPA — vocab-agnostic latent predictor with EMA target encoder +# ---------------------------------------------------------------------------- +class _JEPAEncoder(nn.Module): + def __init__(self, vocab_size, d): + super().__init__() + self.tok = nn.Embedding(vocab_size, d) + self.enc = nn.GRU(d, d, batch_first=True) + self.out = nn.Linear(d, d) + + def forward(self, ids): + h, _ = self.enc(self.tok(ids)) + return self.out(h.mean(dim=1)) + + +class JEPA(nn.Module): + def __init__(self, vocab_size: int, d_jepa: int = 64, ema_m: float = 0.996): + super().__init__() + self.ctx_enc = _JEPAEncoder(vocab_size, d_jepa) + self.tgt_enc = copy.deepcopy(self.ctx_enc) + for p in self.tgt_enc.parameters(): + p.requires_grad_(False) + self.predictor = nn.Sequential( + nn.Linear(d_jepa, 2 * d_jepa), nn.GELU(), nn.Linear(2 * d_jepa, d_jepa)) + self.m, self.d_jepa = ema_m, d_jepa + + @torch.no_grad() + def ema_update(self): + for p_t, p_c in zip(self.tgt_enc.parameters(), self.ctx_enc.parameters()): + p_t.mul_(self.m).add_(p_c.detach(), alpha=1 - self.m) + + def loss(self, seg_ctx, seg_tgt): + z_hat = self.predictor(self.ctx_enc(seg_ctx)) + with torch.no_grad(): + z_tgt = self.tgt_enc(seg_tgt) + pred = F.mse_loss(z_hat, z_tgt) + var_reg = F.relu(1.0 - z_hat.std(dim=0)).mean() # anti-collapse + return pred + 0.5 * var_reg + + @torch.no_grad() + def predict_next_latent(self, seg_ctx): + return self.predictor(self.ctx_enc(seg_ctx)) + + @torch.no_grad() + def encode(self, seg): + return self.tgt_enc(seg) + + +# ---------------------------------------------------------------------------- +# 3. Bridge — sized from backend.hidden_size at construction +# ---------------------------------------------------------------------------- +class Bridge(nn.Module): + def __init__(self, d_llm_hidden: int, d_jepa: int): + super().__init__() + self.proj = nn.Sequential( + nn.Linear(d_llm_hidden, d_jepa), nn.GELU(), nn.Linear(d_jepa, d_jepa)) + + def forward(self, llm_hidden): # [B,T,H] -> [B,d_jepa] + return self.proj(llm_hidden.float().mean(dim=1)) + + def info_nce(self, z1, z2, tau=0.07): + z1, z2 = F.normalize(z1, dim=-1), F.normalize(z2, dim=-1) + logits = z1 @ z2.t() / tau + labels = torch.arange(z1.size(0), device=z1.device) + return 0.5 * (F.cross_entropy(logits, labels) + + F.cross_entropy(logits.t(), labels)) + + +# ---------------------------------------------------------------------------- +# 4. Memory + Router +# ---------------------------------------------------------------------------- +class VectorStore: + def __init__(self): + self.keys, self.values = [], [] + + def add(self, emb, payload): + self.keys.append(emb.squeeze(0).detach().cpu()) + self.values.append(payload) + + def search(self, q, k=2): + if not self.keys: + return [] + K = torch.stack(self.keys) + sims = (q.detach().cpu() @ K.t()).squeeze(0) + top = sims.topk(min(k, len(self.keys))).indices + return [self.values[i] for i in top] + + +class Router(nn.Module): + def __init__(self, d_emb, n_adapters): + super().__init__() + self.fc = nn.Linear(d_emb, n_adapters) + + def forward(self, emb): + return self.fc(emb).argmax(dim=-1) + + +# ---------------------------------------------------------------------------- +# 5. Ensemble — backend-agnostic +# ---------------------------------------------------------------------------- +class Ensemble(nn.Module): + def __init__(self, llm: str = "tiny", adapter_names=("general",), + d_emb: int = 64, d_jepa: int = 64, **backend_kw): + super().__init__() + self.llm = make_backend(llm, **backend_kw) + V, H = self.llm.vocab_size, self.llm.hidden_size + + self.emb = Embedder(V, d_emb) + self.jepa = JEPA(V, d_jepa) + self.bridge = Bridge(H, d_jepa) + self.store = VectorStore() + + self.adapter_names = list(adapter_names) + for n in self.adapter_names: + self.llm.add_adapter(n) + self.llm.set_adapter(self.adapter_names[0]) + self.router = Router(d_emb, len(self.adapter_names)) + + # -------- inference: route -> retrieve -> generate -> JEPA-verify ------- + @torch.no_grad() + def answer_ids(self, query_ids, n_new=32, tau_consistency=0.0, max_retries=2): + q_emb = self.emb(query_ids.cpu()) + a_idx = self.router(q_emb).item() + self.llm.set_adapter(self.adapter_names[a_idx]) + + mems = self.store.search(q_emb, k=1) + ctx = (torch.cat([mems[0], query_ids.cpu()], dim=1) + if mems else query_ids.cpu()) + + z_expected = self.jepa.predict_next_latent(ctx) + + best = None + for attempt in range(max_retries + 1): + draft = self.llm.generate(ctx.to(self.llm.device), n_new=n_new, + temperature=0.8 + 0.3 * attempt) + new_part = draft[:, ctx.size(1):].cpu() + score = F.cosine_similarity( + z_expected, self.jepa.encode(new_part)).item() + if best is None or score > best[1]: + best = (draft, score, attempt) + if score >= tau_consistency: + break + draft, score, attempt = best + return draft, score, self.adapter_names[a_idx], attempt + + def answer_text(self, prompt: str, **kw): + """Convenience wrapper for HF backends (uses the real tokenizer).""" + ids = self.llm.encode_text(prompt) + out, score, adapter, retries = self.answer_ids(ids, **kw) + return self.llm.decode(out), score, adapter, retries + + # -------- continual learning hooks --------------------------------------- + def memorize_ids(self, ids): + self.store.add(self.emb(ids.cpu()), ids.cpu()) + + def memorize_text(self, text: str): + self.memorize_ids(self.llm.encode_text(text)) + + def new_task_adapter(self, name: str): + self.adapter_names.append(name) + self.llm.add_adapter(name) + old = self.router + self.router = Router(self.emb.d_emb, len(self.adapter_names)) + with torch.no_grad(): + self.router.fc.weight[: old.fc.out_features] = old.fc.weight + self.router.fc.bias[: old.fc.out_features] = old.fc.bias + + # -------- one joint training step (LM + JEPA + Bridge) ------------------- + def train_step(self, seg_a, seg_b, opt, w_bridge=0.1): + """seg_a, seg_b: consecutive token-id segments [B, T] (same tokenizer + as the backend!). For HF backends build them with backend.tokenizer.""" + logits, hidden = self.llm(seg_a.to(self.llm.device)) + lm_loss = F.cross_entropy( + logits[:, :-1].reshape(-1, self.llm.vocab_size).float(), + seg_a[:, 1:].reshape(-1).to(logits.device)) + + jepa_loss = self.jepa.loss(seg_a.cpu(), seg_b.cpu()) + + z_llm = self.bridge(hidden.cpu() if hidden.device.type != "cpu" else hidden) + z_jepa = self.jepa.ctx_enc(seg_a.cpu()).detach() + bridge_loss = self.bridge.info_nce(z_llm, z_jepa.to(z_llm.device)) + + loss = lm_loss.cpu() + jepa_loss + w_bridge * bridge_loss + opt.zero_grad(); loss.backward(); opt.step() + self.jepa.ema_update() + return {"lm": lm_loss.item(), "jepa": jepa_loss.item(), + "bridge": bridge_loss.item()} + + def make_optimizer(self, lr_lora=2e-4, lr_aux=1e-3): + return torch.optim.AdamW([ + {"params": list(self.llm.trainable_parameters()), "lr": lr_lora}, + {"params": list(self.jepa.ctx_enc.parameters()) + + list(self.jepa.predictor.parameters()), "lr": lr_aux}, + {"params": list(self.bridge.parameters()) + + list(self.emb.parameters()) + + list(self.router.parameters()), "lr": lr_aux}, + ]) + + +# ---------------------------------------------------------------------------- +# 6. Helpers: turn raw text into (seg_a, seg_b) pairs with the HF tokenizer +# ---------------------------------------------------------------------------- +def segment_pairs_from_texts(backend: HFBackend, texts, seg_len=64): + """Yields consecutive-segment id pairs for the JEPA + LM losses.""" + a_list, b_list = [], [] + for t in texts: + ids = backend.tokenizer(t, return_tensors="pt").input_ids[0] + for i in range(0, len(ids) - 2 * seg_len, seg_len): + a_list.append(ids[i:i + seg_len]) + b_list.append(ids[i + seg_len:i + 2 * seg_len]) + if not a_list: + raise ValueError("texts too short for the chosen seg_len") + return torch.stack(a_list), torch.stack(b_list) + + +# ---------------------------------------------------------------------------- +# 7. Demos +# ---------------------------------------------------------------------------- +def demo_tiny(steps=50): + """No-dependency smoke test.""" + ens = Ensemble(llm="tiny") + opt = ens.make_optimizer() + for s in range(steps): + seg_a = torch.randint(0, ens.llm.vocab_size, (8, 32)) + seg_b = torch.randint(0, ens.llm.vocab_size, (8, 32)) + logs = ens.train_step(seg_a, seg_b, opt) + if s % 10 == 0: + print(f"step {s:3d} | " + " | ".join(f"{k} {v:.3f}" for k, v in logs.items())) + + for _ in range(5): + ens.memorize_ids(torch.randint(0, ens.llm.vocab_size, (1, 32))) + ens.new_task_adapter("medical") + + q = torch.randint(0, ens.llm.vocab_size, (1, 8)) + out, score, adapter, retries = ens.answer_ids(q, tau_consistency=-1.0) + print(f"\nadapter={adapter} jepa_consistency={score:.3f} retries={retries}") + + +def demo_hf(model_path="Qwen/Qwen2.5-0.5B-Instruct"): + """Real model from hub id OR local path, e.g. '/models/llama-3.2-1b'. + For gated Llama repos: huggingface-cli login first.""" + ens = Ensemble(llm=model_path, load_in_4bit=False) # 4bit needs bitsandbytes + opt = ens.make_optimizer() + + texts = ["Replace this with your real corpus. " * 50] + seg_a, seg_b = segment_pairs_from_texts(ens.llm, texts, seg_len=32) + for s in range(10): # tiny demo run + logs = ens.train_step(seg_a[:4], seg_b[:4], opt) + print(f"step {s} | " + " | ".join(f"{k} {v:.3f}" for k, v in logs.items())) + + ens.memorize_text("The project codename is AURORA and it ships in Q3.") + ens.new_task_adapter("project_aurora") + + text, score, adapter, retries = ens.answer_text( + "What is the project codename?", n_new=24, tau_consistency=-1.0) + print(f"\n[{adapter} | jepa={score:.3f} | retries={retries}]\n{text}") + + +if __name__ == "__main__": + import sys + arg = sys.argv[1] if len(sys.argv) > 1 else "tiny" + if arg == "tiny": + demo_tiny() + else: + demo_hf(arg) # python ensemble.py /models/llama-3.2-1b + # python ensemble.py Qwen/Qwen2.5-0.5B-Instruct diff --git a/research/ensemble/src/ensemble/memory.py b/research/ensemble/src/ensemble/memory.py new file mode 100644 index 0000000000000000000000000000000000000000..63c0c821034dfe1770a38196162342bf7be4b081 --- /dev/null +++ b/research/ensemble/src/ensemble/memory.py @@ -0,0 +1,46 @@ +"""Retrieval memory: embedder, vector store, and adapter router.""" + +from __future__ import annotations + +import torch +import torch.nn as nn +import torch.nn.functional as F + + +class Embedder(nn.Module): + def __init__(self, vocab_size: int, d_emb: int = 64): + super().__init__() + self.tok = nn.Embedding(vocab_size, d_emb) + self.enc = nn.GRU(d_emb, d_emb, batch_first=True, bidirectional=True) + self.proj = nn.Linear(2 * d_emb, d_emb) + self.d_emb = d_emb + + def forward(self, ids): + h, _ = self.enc(self.tok(ids)) + return F.normalize(self.proj(h.mean(dim=1)), dim=-1) + + +class VectorStore: + def __init__(self): + self.keys, self.values = [], [] + + def add(self, emb, payload): + self.keys.append(emb.squeeze(0).detach().cpu()) + self.values.append(payload) + + def search(self, q, k=2): + if not self.keys: + return [] + K = torch.stack(self.keys) + sims = (q.detach().cpu() @ K.t()).squeeze(0) + top = sims.topk(min(k, len(self.keys))).indices + return [self.values[i] for i in top] + + +class Router(nn.Module): + def __init__(self, d_emb, n_adapters): + super().__init__() + self.fc = nn.Linear(d_emb, n_adapters) + + def forward(self, emb): + return self.fc(emb).argmax(dim=-1) diff --git a/research/ensemble/src/ensemble/pretrain.py b/research/ensemble/src/ensemble/pretrain.py new file mode 100644 index 0000000000000000000000000000000000000000..bfa5968c2e06eddf5297d770c2f95a6526343dac --- /dev/null +++ b/research/ensemble/src/ensemble/pretrain.py @@ -0,0 +1,198 @@ +"""Joint pretrain: LLM (LoRA) + embedder + JEPA + bridge, saved to models/ensemble/.""" + +from __future__ import annotations + +import argparse +import json +import os +import random +import time +from pathlib import Path + +import torch + +from ensemble.checkpoint import save_checkpoint +from ensemble.config import default_ensemble_out, load_dotenv, resolve_llm +from ensemble.jepa_ensemble import Ensemble, segment_pairs_from_texts + +_REPO_ROOT = Path(__file__).resolve().parents[4] +_DEFAULT_DATA = _REPO_ROOT / "research/data/education-lesson-chat.jsonl" +_DEFAULT_KB = _REPO_ROOT / "research/data/benchmark-kb.jsonl" + + +def _load_jsonl(path: Path) -> list[dict]: + rows = [] + with open(path) as f: + for line in f: + line = line.strip() + if line: + rows.append(json.loads(line)) + return rows + + +def _chat_to_text(row: dict) -> str: + messages = row.get("messages", []) + parts = [f"{m.get('role', 'user')}: {m.get('content', '')}" for m in messages] + return "\n".join(parts) + + +def _collect_texts(data_path: Path, max_samples: int | None) -> list[str]: + rows = _load_jsonl(data_path) + if max_samples is not None: + rows = rows[:max_samples] + return [_chat_to_text(r) for r in rows if _chat_to_text(r).strip()] + + +def _seed_memory(ens: Ensemble, kb_path: Path | None) -> int: + if kb_path is None or not kb_path.is_file(): + return 0 + count = 0 + for row in _load_jsonl(kb_path): + text = row.get("text", "").strip() + if text: + ens.memorize_text(text) + count += 1 + return count + + +def pretrain(args) -> Path: + torch.manual_seed(args.seed) + random.seed(args.seed) + + data_path = Path(args.data).resolve() + out_dir = Path(args.out).resolve() + kb_path = Path(args.kb).resolve() if args.kb else None + + print(f"Loading ensemble backend: {args.llm}") + ens = Ensemble(llm=args.llm, load_in_4bit=args.load_in_4bit) + opt = ens.make_optimizer(lr_lora=args.lr_lora, lr_aux=args.lr_aux) + + texts = _collect_texts(data_path, args.max_samples) + if not texts and args.llm != "tiny": + raise SystemExit(f"No training texts found in {data_path}") + + mem_count = _seed_memory(ens, kb_path) + print(f"Training texts: {len(texts)} | memory snippets: {mem_count}") + + if args.llm == "tiny": + n_pairs = max(args.steps * args.batch_size, args.batch_size) + v = ens.llm.vocab_size + seg_a = torch.randint(0, v, (n_pairs, args.seg_len)) + seg_b = torch.randint(0, v, (n_pairs, args.seg_len)) + else: + seg_a, seg_b = segment_pairs_from_texts( + ens.llm, texts, seg_len=args.seg_len + ) + n_pairs = seg_a.size(0) + batch = min(args.batch_size, n_pairs) + print(f"Segment pairs: {n_pairs} | batch={batch} | steps={args.steps}") + + t0 = time.time() + for step in range(args.steps): + idx = torch.randint(0, n_pairs, (batch,)) + logs = ens.train_step(seg_a[idx], seg_b[idx], opt, w_bridge=args.w_bridge) + if step % max(1, args.log_every) == 0 or step == args.steps - 1: + parts = " | ".join(f"{k} {v:.4f}" for k, v in logs.items()) + print(f"step {step:4d}/{args.steps} | {parts}") + + elapsed = time.time() - t0 + meta = { + "steps": args.steps, + "batch_size": batch, + "seg_len": args.seg_len, + "data": str(data_path), + "kb": str(kb_path) if kb_path else None, + "memory_count": mem_count, + "text_count": len(texts), + "elapsed_s": round(elapsed, 1), + "lr_lora": args.lr_lora, + "lr_aux": args.lr_aux, + "w_bridge": args.w_bridge, + "seed": args.seed, + "preset": getattr(args, "preset", None), + } + + saved = save_checkpoint( + ens, + out_dir, + base_llm=args.llm, + training_meta=meta, + ) + print(f"\nSaved ensemble checkpoint → {saved}") + print("Benchmark with slm-evals:") + print( + f" uv run --package slm-evals slm-benchmark " + f"--model {saved} --model-type ensemble " + f"--benchmarks bfcl --max-samples 5" + ) + return saved + + +def parse_args(): + p = argparse.ArgumentParser( + description="Pretrain JEPA ensemble (LLM+emb+JEPA) and save to models/ensemble/" + ) + p.add_argument( + "--llm", + default=None, + help=( + "HF hub id / local path, 'tiny' for CPU smoke, or omit to use " + "LLM_PATH / BASE / MODEL_ID / ACTIVE_MODEL from .env + models.yaml" + ), + ) + p.add_argument( + "--preset", + default=None, + help="models.yaml preset key (default: ENSEMBLE_PRESET or ACTIVE_MODEL)", + ) + p.add_argument( + "--data", + default=str(_DEFAULT_DATA), + help="Chat JSONL (messages[]) for segment-pair training", + ) + p.add_argument( + "--kb", + default=str(_DEFAULT_KB), + help="Optional KB JSONL (text field) loaded into vector store", + ) + p.add_argument( + "--out", + default=None, + help="Output dir (default: ENSEMBLE_OUT or models/ensemble/-jepa-pretrain)", + ) + p.add_argument("--steps", type=int, default=100) + p.add_argument("--batch-size", type=int, default=4) + p.add_argument("--seg-len", type=int, default=32) + p.add_argument("--max-samples", type=int, default=None) + p.add_argument("--lr-lora", type=float, default=2e-4) + p.add_argument("--lr-aux", type=float, default=1e-3) + p.add_argument("--w-bridge", type=float, default=0.1) + p.add_argument("--log-every", type=int, default=10) + p.add_argument("--seed", type=int, default=0) + p.add_argument("--load-in-4bit", action="store_true") + p.add_argument("--no-kb", action="store_true", help="Skip loading KB into memory") + return p.parse_args() + + +def main(): + load_dotenv() + args = parse_args() + if args.no_kb: + args.kb = None + + preset_key = args.preset + if args.llm is None or args.llm == "auto": + args.llm, preset_key = resolve_llm(preset_arg=args.preset) + elif args.llm != "tiny" and not args.preset: + _, preset_key = resolve_llm(llm_arg=args.llm) + + if not args.out: + args.out = os.environ.get("ENSEMBLE_OUT") or default_ensemble_out(preset_key) + + args.preset = preset_key + print(f"Resolved LLM: {args.llm}" + (f" (preset {preset_key})" if preset_key else "")) + pretrain(args) + + +if __name__ == "__main__": + main() diff --git a/research/ensemble/src/ensemble/world_ensemble.py b/research/ensemble/src/ensemble/world_ensemble.py new file mode 100644 index 0000000000000000000000000000000000000000..7627cad67000e48514652c4866174b8bb5428ebd --- /dev/null +++ b/research/ensemble/src/ensemble/world_ensemble.py @@ -0,0 +1,228 @@ +"""World-model ensemble: plan -> generate -> energy-rank.""" + +from __future__ import annotations + +import sys +import time + +import torch +import torch.nn as nn +import torch.nn.functional as F + +from ensemble.backends import HFLLM, load_llm +from ensemble.bridge import Bridge +from ensemble.energy import EnergyModel +from ensemble.jepa import JEPA +from ensemble.memory import Embedder, VectorStore +from ensemble.world_model import WorldModel + +torch.manual_seed(0) + +D_LAT = 96 +D_EMB = 64 + + +class WorldEnsemble(nn.Module): + def __init__(self, llm_spec="tiny"): + super().__init__() + self.llm = load_llm(llm_spec) + V, H = self.llm.vocab_size, self.llm.hidden_size + self.emb = Embedder(V, D_EMB) + self.jepa = JEPA(V, D_LAT) + self.world = WorldModel(D_LAT) + self.energy = EnergyModel(D_LAT) + self.bridge = Bridge(H, D_LAT) + self.store = VectorStore() + + @torch.no_grad() + def world_state(self, segments): + s = self.world.init_state(1, "cpu") + for seg in segments: + z = self.jepa.encode(seg.cpu()) + s, _ = self.world.step(s, z) + return s + + @torch.no_grad() + def answer(self, query_ids, n_new=24, n_drafts=6, horizon=3): + q_emb = self.emb(query_ids.cpu()) + mems = self.store.search(q_emb, k=1) + segments = (mems + [query_ids.cpu()]) if mems else [query_ids.cpu()] + ctx = torch.cat(segments, dim=1) + + s = self.world_state(segments) + plan, _ = self.world.rollout(s, horizon) + + drafts, lat = [], [] + for _ in range(n_drafts): + out = self.llm.generate( + ctx.to(self.llm.device), n_new=n_new, temperature=0.9 + ) + new = out[:, ctx.size(1) :].cpu() + drafts.append(new) + lat.append(self.jepa.encode(new)) + Z = torch.cat(lat, 0) + E = self.energy.rank(s, Z) + best = E.argmin().item() + return { + "output": drafts[best], + "energy": E[best].item(), + "all_energies": E.tolist(), + "plan_alignment": F.cosine_similarity( + plan[:, 0], Z[best : best + 1] + ).item(), + } + + def memorize(self, ids): + self.store.add(self.emb(ids.cpu()), ids.cpu()) + + def train_step( + self, + seg_seq, + opt, + w=None, + hard_negs=True, + ): + if w is None: + w = dict(lm=1.0, jepa=1.0, world=1.0, ebm=1.0, bridge=0.1) + + B, T, L = seg_seq.shape + dev = self.llm.device + + flat = seg_seq[:, 0].to(dev) + logits, hidden = self.llm(flat) + lm = F.cross_entropy( + logits[:, :-1].reshape(-1, self.llm.vocab_size).float(), + flat[:, 1:].reshape(-1), + ) + + jepa = self.jepa.loss(seg_seq[:, 0], seg_seq[:, 1]) + + z_seq = torch.stack( + [self.jepa.enc(seg_seq[:, t]) for t in range(T)], 1 + ) + world = self.world.sequence_loss(z_seq) + + s = self.world.init_state(B, z_seq.device) + s, _ = self.world.step(s, z_seq[:, 0].detach()) + z_pos = z_seq[:, 1].detach() + z_negs = None + if hard_negs: + with torch.no_grad(): + gen = self.llm.generate(seg_seq[:, 0].to(dev), n_new=L) + gen_new = gen[:, seg_seq.size(2) :].cpu() + z_negs = self.jepa.encode(gen_new).unsqueeze(1) + ebm = self.energy.contrastive_loss(s, z_pos, z_negs) + + bridge = self.bridge.info_nce( + self.bridge( + hidden.cpu() if hidden.device.type != "cpu" else hidden + ), + self.jepa.enc(seg_seq[:, 0]).detach(), + ) + + loss = ( + w["lm"] * lm.cpu() + + w["jepa"] * jepa + + w["world"] * world + + w["ebm"] * ebm + + w["bridge"] * bridge + ) + opt.zero_grad() + loss.backward() + opt.step() + self.jepa.ema() + return dict( + lm=lm.item(), + jepa=jepa.item(), + world=world.item(), + ebm=ebm.item(), + bridge=bridge.item(), + ) + + def make_optimizer(self, lr_lora=2e-4, lr_aux=1e-3): + return torch.optim.AdamW( + [ + {"params": list(self.llm.trainable_parameters()), "lr": lr_lora}, + { + "params": list(self.jepa.enc.parameters()) + + list(self.jepa.pred.parameters()), + "lr": lr_aux, + }, + {"params": list(self.world.parameters()), "lr": lr_aux}, + {"params": list(self.energy.parameters()), "lr": lr_aux}, + { + "params": list(self.bridge.parameters()) + + list(self.emb.parameters()), + "lr": lr_aux, + }, + ] + ) + + +def toy_segment_sequences(B=8, T=4, L=24, vocab=1000): + return torch.randint(0, vocab, (B, T, L)) + + +def hf_segment_sequences(llm: HFLLM, texts, T=4, L=64): + seqs = [] + for t in texts: + ids = llm.tokenizer(t, return_tensors="pt").input_ids[0] + n = (len(ids) // (T * L)) * T * L + if n: + seqs.append(ids[:n].view(-1, T, L)) + if not seqs: + raise ValueError("corpus too short for T*L window") + return torch.cat(seqs, 0) + + +def demo(spec="tiny", steps=60): + ens = WorldEnsemble(spec) + opt = ens.make_optimizer() + + if spec == "tiny": + get_batch = lambda: toy_segment_sequences(vocab=ens.llm.vocab_size) + else: + corpus = ["Replace with your real documents. " * 200] + data = hf_segment_sequences(ens.llm, corpus, T=4, L=32) + get_batch = lambda: data[torch.randperm(len(data))[:4]] + steps = min(steps, 10) + + t0 = time.time() + for s in range(steps): + logs = ens.train_step( + get_batch(), opt, hard_negs=(s > steps // 2) + ) + if s % 10 == 0: + print( + f"step {s:3d} | " + + " | ".join(f"{k} {v:.3f}" for k, v in logs.items()) + ) + print(f"trained {steps} steps in {time.time() - t0:.1f}s") + + for _ in range(4): + if spec == "tiny": + ens.memorize(torch.randint(0, ens.llm.vocab_size, (1, 24))) + q = ( + torch.randint(0, ens.llm.vocab_size, (1, 12)) + if spec == "tiny" + else ens.llm.tokenizer( + "What is this document about?", return_tensors="pt" + ).input_ids + ) + res = ens.answer(q, n_drafts=6, horizon=3) + print( + f"\nselected draft energy={res['energy']:.3f} " + f"(all: {[f'{e:.2f}' for e in res['all_energies']]})" + ) + print(f"plan↔output alignment: {res['plan_alignment']:.3f}") + + +if __name__ == "__main__": + from ensemble.config import load_dotenv, resolve_llm + + load_dotenv() + spec = sys.argv[1] if len(sys.argv) > 1 else None + if spec is None or spec == "auto": + spec, preset = resolve_llm() + print(f"Resolved LLM: {spec} (preset {preset})") + demo(spec or "tiny") diff --git a/research/ensemble/src/ensemble/world_model.py b/research/ensemble/src/ensemble/world_model.py new file mode 100644 index 0000000000000000000000000000000000000000..bb16308a423caa77db22c0de15d032176ea7c075 --- /dev/null +++ b/research/ensemble/src/ensemble/world_model.py @@ -0,0 +1,40 @@ +"""Latent world model: multi-step rollout in JEPA space.""" + +from __future__ import annotations + +import torch +import torch.nn as nn +import torch.nn.functional as F + + +class WorldModel(nn.Module): + def __init__(self, d_latent: int): + super().__init__() + self.cell = nn.GRUCell(d_latent, d_latent) + self.head = nn.Linear(d_latent, d_latent) + self.s0 = nn.Parameter(torch.zeros(d_latent)) + self.d_latent = d_latent + + def init_state(self, B, device): + return self.s0.unsqueeze(0).expand(B, -1).contiguous().to(device) + + def step(self, s, z): + s = self.cell(z, s) + return s, self.head(s) + + def rollout(self, s, horizon): + preds = [] + for _ in range(horizon): + z_hat = self.head(s) + preds.append(z_hat) + s = self.cell(z_hat, s) + return torch.stack(preds, 1), s + + def sequence_loss(self, z_seq): + B, T, _ = z_seq.shape + s = self.init_state(B, z_seq.device) + loss = 0.0 + for t in range(T - 1): + s, z_hat = self.step(s, z_seq[:, t]) + loss = loss + F.mse_loss(z_hat, z_seq[:, t + 1]) + return loss / (T - 1) diff --git a/research/ensemble/src/ensemble/world_model_ensemble.py b/research/ensemble/src/ensemble/world_model_ensemble.py new file mode 100644 index 0000000000000000000000000000000000000000..bf54bc670a4ccdbc6c061b73f0584a8a13c78279 --- /dev/null +++ b/research/ensemble/src/ensemble/world_model_ensemble.py @@ -0,0 +1,499 @@ +""" +World-Model Ensemble: EMB + EBM + JEPA + World Model + small LLM (from path) +============================================================================= +A LeCun-style modular agent built around a small language model. + +ARCHITECTURE +------------ + ┌────────────────────────────┐ + input tokens ──► EMB ──┤ VectorStore (retrieval/CL) │──► context + │ └────────────────────────────┘ │ + │ │ + ▼ ▼ + JEPA encoder ──► latent state s_t ──► WORLD MODEL ──► ŝ_{t+1..t+H} + │ (GRU dynamics, multi-step rollout) │ + │ │ + │ ┌────────────────────────────────────┐ │ + └──────────► │ ENERGY MODEL E(s_ctx, z_candidate)│ ◄─┘ + │ low energy = compatible/plausible │ + └────────────────┬───────────────────┘ + │ scores drafts / plans + ▼ + LLM (small, loaded from path, LoRA bank) ──► N drafts ──► pick argmin E + +ROLES +----- +EMB perception for retrieval + routing (non-parametric memory) +JEPA learns the latent space: predict z(next segment) from z(context) + (EMA target encoder + variance reg, no token reconstruction) +WORLD MODEL deterministic latent dynamics s_{t+1} = f(s_t, z_t): + rolls the conversation/document state forward H steps in + LATENT space — cheap lookahead without decoding tokens +ENERGY E(s, z) ∈ R, trained so true continuations have LOW energy and + negatives (shuffled / model-generated) have HIGH energy. + At inference it is the critic: rank LLM drafts, reject bad plans. +LLM the only token-level generator. Loaded from a local path or HF id; + frozen base + LoRA adapters (continual learning by isolation). + +WHY EBM *and* JEPA? JEPA gives a point prediction ẑ of the future latent; +the EBM gives a *compatibility landscape* E(s, z) — it can say "both A and B +are plausible" where a point predictor must average them. JEPA trains the +representation; the EBM scores hypotheses in it. World model chains JEPA +one-step predictions into multi-step rollouts that the EBM can evaluate. + +USAGE +----- + pip install torch # toy mode + pip install transformers peft accelerate # real LLM mode + + python world_model_ensemble.py tiny # smoke test + python world_model_ensemble.py /models/llama-3.2-1b # local weights + python world_model_ensemble.py Qwen/Qwen2.5-0.5B-Instruct +""" + +from __future__ import annotations +import copy +import math +import sys +import time + +import torch +import torch.nn as nn +import torch.nn.functional as F + +torch.manual_seed(0) + +D_LAT = 96 # shared latent dimension (JEPA / world / energy) +D_EMB = 64 # retrieval embedding dim + + +# ============================================================================ +# 1. LLM backend — load small model from path / hub, or toy fallback +# (same contract as before: forward -> (logits, hidden), generate, adapters) +# ============================================================================ +class TinyLLM(nn.Module): + VOCAB, D, L, H, T = 1000, 128, 2, 4, 32 + + def __init__(self): + super().__init__() + self.tok = nn.Embedding(self.VOCAB, self.D) + self.pos = nn.Embedding(self.T * 4, self.D) + layer = nn.TransformerEncoderLayer(self.D, self.H, 4 * self.D, + batch_first=True, norm_first=True) + self.blocks = nn.TransformerEncoder(layer, self.L) + self.head = nn.Linear(self.D, self.VOCAB, bias=False) + self.vocab_size, self.hidden_size = self.VOCAB, self.D + + def forward(self, ids): + Tn = ids.size(1) + x = self.tok(ids) + self.pos(torch.arange(Tn, device=ids.device)) + mask = torch.triu(torch.full((Tn, Tn), float("-inf"), + device=ids.device), 1) + h = self.blocks(x, mask=mask) + return self.head(h), h + + @torch.no_grad() + def generate(self, ids, n_new=16, temperature=1.0): + for _ in range(n_new): + logits, _ = self(ids[:, -self.T:]) + nxt = torch.multinomial( + F.softmax(logits[:, -1] / temperature, -1), 1) + ids = torch.cat([ids, nxt], 1) + return ids + + def trainable_parameters(self): + return self.parameters() + + @property + def device(self): + return next(self.parameters()).device + + +class HFLLM(nn.Module): + """Small model from a local path or HF id, frozen base + LoRA.""" + def __init__(self, path, lora_r=16): + super().__init__() + from transformers import AutoModelForCausalLM, AutoTokenizer + from peft import LoraConfig, get_peft_model + self.tokenizer = AutoTokenizer.from_pretrained(path) + if self.tokenizer.pad_token is None: + self.tokenizer.pad_token = self.tokenizer.eos_token + base = AutoModelForCausalLM.from_pretrained( + path, torch_dtype=torch.bfloat16 + if torch.cuda.is_available() else torch.float32, + device_map="auto" if torch.cuda.is_available() else None) + for p in base.parameters(): + p.requires_grad_(False) + cfg = LoraConfig(r=lora_r, lora_alpha=2 * lora_r, lora_dropout=0.05, + target_modules=["q_proj", "v_proj"], + task_type="CAUSAL_LM") + self.model = get_peft_model(base, cfg) + self.vocab_size = self.model.config.vocab_size + self.hidden_size = self.model.config.hidden_size + + def forward(self, ids): + out = self.model(input_ids=ids.to(self.device), + output_hidden_states=True) + return out.logits, out.hidden_states[-1] + + @torch.no_grad() + def generate(self, ids, n_new=32, temperature=0.8): + return self.model.generate( + input_ids=ids.to(self.device), max_new_tokens=n_new, + do_sample=True, temperature=temperature, + pad_token_id=self.tokenizer.pad_token_id) + + def trainable_parameters(self): + return (p for p in self.model.parameters() if p.requires_grad) + + @property + def device(self): + return next(self.model.parameters()).device + + +def load_llm(spec: str): + return TinyLLM() if spec == "tiny" else HFLLM(spec) + + +# ============================================================================ +# 2. Embedder (retrieval) — vocab-agnostic +# ============================================================================ +class Embedder(nn.Module): + def __init__(self, vocab): + super().__init__() + self.tok = nn.Embedding(vocab, D_EMB) + self.gru = nn.GRU(D_EMB, D_EMB, batch_first=True, bidirectional=True) + self.out = nn.Linear(2 * D_EMB, D_EMB) + + def forward(self, ids): + h, _ = self.gru(self.tok(ids)) + return F.normalize(self.out(h.mean(1)), dim=-1) + + +class VectorStore: + def __init__(self): + self.K, self.V = [], [] + + def add(self, k, v): + self.K.append(k.squeeze(0).detach().cpu()); self.V.append(v) + + def search(self, q, k=1): + if not self.K: + return [] + sims = (q.detach().cpu() @ torch.stack(self.K).t()).squeeze(0) + return [self.V[i] for i in sims.topk(min(k, len(self.K))).indices] + + +# ============================================================================ +# 3. JEPA — owns the latent space (EMA target encoder, variance-regularized) +# ============================================================================ +class SegEncoder(nn.Module): + def __init__(self, vocab): + super().__init__() + self.tok = nn.Embedding(vocab, D_LAT) + self.gru = nn.GRU(D_LAT, D_LAT, batch_first=True) + self.out = nn.Linear(D_LAT, D_LAT) + + def forward(self, ids): + h, _ = self.gru(self.tok(ids)) + return self.out(h.mean(1)) # [B, D_LAT] + + +class JEPA(nn.Module): + def __init__(self, vocab, m=0.996): + super().__init__() + self.enc = SegEncoder(vocab) # context/online enc + self.tgt = copy.deepcopy(self.enc) # EMA target + for p in self.tgt.parameters(): + p.requires_grad_(False) + self.pred = nn.Sequential(nn.Linear(D_LAT, 2 * D_LAT), nn.GELU(), + nn.Linear(2 * D_LAT, D_LAT)) + self.m = m + + @torch.no_grad() + def ema(self): + for pt, pc in zip(self.tgt.parameters(), self.enc.parameters()): + pt.mul_(self.m).add_(pc.detach(), alpha=1 - self.m) + + def loss(self, seg_a, seg_b): + z_hat = self.pred(self.enc(seg_a)) + with torch.no_grad(): + z_tgt = self.tgt(seg_b) + var = F.relu(1.0 - z_hat.std(0)).mean() # anti-collapse + return F.mse_loss(z_hat, z_tgt) + 0.5 * var + + @torch.no_grad() + def encode(self, seg): # target space + return self.tgt(seg) + + +# ============================================================================ +# 4. WORLD MODEL — latent dynamics s_{t+1} = f(s_t, z_t), multi-step rollout +# Trained on SEQUENCES of segments: predict each next latent from state. +# ============================================================================ +class WorldModel(nn.Module): + def __init__(self): + super().__init__() + self.cell = nn.GRUCell(D_LAT, D_LAT) # state update + self.head = nn.Linear(D_LAT, D_LAT) # state -> ẑ_{t+1} + self.s0 = nn.Parameter(torch.zeros(D_LAT)) + + def init_state(self, B, device): + return self.s0.unsqueeze(0).expand(B, -1).contiguous().to(device) + + def step(self, s, z): + """Consume observed latent z_t, return (new state, prediction ẑ_{t+1}).""" + s = self.cell(z, s) + return s, self.head(s) + + def rollout(self, s, horizon): + """Imagine H future latents feeding its own predictions back in.""" + preds = [] + for _ in range(horizon): + z_hat = self.head(s) + preds.append(z_hat) + s = self.cell(z_hat, s) + return torch.stack(preds, 1), s # [B, H, D_LAT] + + def sequence_loss(self, z_seq): + """z_seq: [B, T, D_LAT] observed segment latents (teacher forcing).""" + B, T, _ = z_seq.shape + s = self.init_state(B, z_seq.device) + loss = 0.0 + for t in range(T - 1): + s, z_hat = self.step(s, z_seq[:, t]) + loss = loss + F.mse_loss(z_hat, z_seq[:, t + 1]) + return loss / (T - 1) + + +# ============================================================================ +# 5. ENERGY MODEL — E(state, candidate latent) ∈ R, low = plausible +# Trained with InfoNCE-style contrastive: positives = true next latent, +# negatives = (a) other batch items, (b) LLM-generated drafts (optional). +# ============================================================================ +class EnergyModel(nn.Module): + def __init__(self): + super().__init__() + self.net = nn.Sequential( + nn.Linear(2 * D_LAT, 2 * D_LAT), nn.GELU(), + nn.Linear(2 * D_LAT, D_LAT), nn.GELU(), + nn.Linear(D_LAT, 1)) + + def energy(self, s, z): + """s: [B, D_LAT] context state; z: [B, D_LAT] candidate. -> [B]""" + return self.net(torch.cat([s, z], -1)).squeeze(-1) + + def contrastive_loss(self, s, z_pos, z_negs=None, tau=0.5): + """Softmax over energies: true continuation must be the argmin. + In-batch negatives: every other item's z_pos is a negative for s_i.""" + B = s.size(0) + # pairwise energies: E(s_i, z_j) for all i, j + s_rep = s.unsqueeze(1).expand(B, B, D_LAT).reshape(B * B, D_LAT) + z_rep = z_pos.unsqueeze(0).expand(B, B, D_LAT).reshape(B * B, D_LAT) + E = self.energy(s_rep, z_rep).view(B, B) # [B, B] + if z_negs is not None: # extra hard negatives + En = self.energy( + s.repeat_interleave(z_negs.size(1), 0), + z_negs.reshape(-1, D_LAT)).view(B, -1) + E = torch.cat([E, En], dim=1) + labels = torch.arange(B, device=s.device) + return F.cross_entropy(-E / tau, labels) # low E ⇒ high logit + + @torch.no_grad() + def rank(self, s, candidates): + """candidates: [N, D_LAT]; returns energies [N] (lower = better).""" + return self.energy(s.expand(candidates.size(0), -1), candidates) + + +# ============================================================================ +# 6. Bridge — LLM hidden states -> shared latent space (alignment) +# ============================================================================ +class Bridge(nn.Module): + def __init__(self, d_hidden): + super().__init__() + self.proj = nn.Sequential(nn.Linear(d_hidden, D_LAT), nn.GELU(), + nn.Linear(D_LAT, D_LAT)) + + def forward(self, h): # [B,T,H] -> [B,D_LAT] + return self.proj(h.float().mean(1)) + + def info_nce(self, a, b, tau=0.07): + a, b = F.normalize(a, -1), F.normalize(b, -1) + logits = a @ b.t() / tau + y = torch.arange(a.size(0), device=a.device) + return 0.5 * (F.cross_entropy(logits, y) + + F.cross_entropy(logits.t(), y)) + + +# ============================================================================ +# 7. THE ENSEMBLE — wiring + inference (plan -> generate -> energy-rank) +# ============================================================================ +class WorldEnsemble(nn.Module): + def __init__(self, llm_spec="tiny"): + super().__init__() + self.llm = load_llm(llm_spec) + V, H = self.llm.vocab_size, self.llm.hidden_size + self.emb = Embedder(V) + self.jepa = JEPA(V) + self.world = WorldModel() + self.energy = EnergyModel() + self.bridge = Bridge(H) + self.store = VectorStore() + + # ------------------------- inference --------------------------------- + @torch.no_grad() + def world_state(self, segments): + """Fold a list of [1,T] segment tensors into a latent state.""" + s = self.world.init_state(1, "cpu") + for seg in segments: + z = self.jepa.encode(seg.cpu()) + s, _ = self.world.step(s, z) + return s + + @torch.no_grad() + def answer(self, query_ids, n_new=24, n_drafts=6, horizon=3): + """retrieve -> build world state -> imagine -> generate N -> argmin E.""" + q_emb = self.emb(query_ids.cpu()) + mems = self.store.search(q_emb, k=1) + segments = (mems + [query_ids.cpu()]) if mems else [query_ids.cpu()] + ctx = torch.cat(segments, dim=1) + + s = self.world_state(segments) # latent context state + plan, _ = self.world.rollout(s, horizon) # imagined future + # (plan is available for planning losses / steering; logged here) + + drafts, lat = [], [] + for _ in range(n_drafts): + out = self.llm.generate(ctx.to(self.llm.device), n_new=n_new, + temperature=0.9) + new = out[:, ctx.size(1):].cpu() + drafts.append(new) + lat.append(self.jepa.encode(new)) + Z = torch.cat(lat, 0) # [N, D_LAT] + E = self.energy.rank(s, Z) # lower = better + best = E.argmin().item() + return {"output": drafts[best], "energy": E[best].item(), + "all_energies": E.tolist(), + "plan_alignment": F.cosine_similarity( + plan[:, 0], Z[best:best + 1]).item()} + + def memorize(self, ids): + self.store.add(self.emb(ids.cpu()), ids.cpu()) + + # ------------------------- training ---------------------------------- + def train_step(self, seg_seq, opt, w=dict(lm=1.0, jepa=1.0, world=1.0, + ebm=1.0, bridge=0.1), + hard_negs=True): + """seg_seq: [B, T_seg, L] — B documents, each split into T_seg + consecutive segments of length L (same tokenizer as the LLM).""" + B, T, L = seg_seq.shape + dev = self.llm.device + + # (1) LM loss on the first segment (or all, batched, if budget allows) + flat = seg_seq[:, 0].to(dev) + logits, hidden = self.llm(flat) + lm = F.cross_entropy( + logits[:, :-1].reshape(-1, self.llm.vocab_size).float(), + flat[:, 1:].reshape(-1)) + + # (2) JEPA: adjacent segment pairs + jepa = self.jepa.loss(seg_seq[:, 0], seg_seq[:, 1]) + + # (3) World model: sequence of latents (online encoder, grads flow) + z_seq = torch.stack([self.jepa.enc(seg_seq[:, t]) + for t in range(T)], 1) # [B, T, D_LAT] + world = self.world.sequence_loss(z_seq) + + # (4) Energy: state after t=0 must give low E to true z_1, + # high E to in-batch + (optionally) LLM-generated negatives + s = self.world.init_state(B, z_seq.device) + s, _ = self.world.step(s, z_seq[:, 0].detach()) + z_pos = z_seq[:, 1].detach() + z_negs = None + if hard_negs: + with torch.no_grad(): # model drafts as negs + gen = self.llm.generate(seg_seq[:, 0].to(dev), n_new=L) + gen_new = gen[:, seg_seq.size(2):].cpu() + z_negs = self.jepa.encode(gen_new).unsqueeze(1) # [B,1,D] + ebm = self.energy.contrastive_loss(s, z_pos, z_negs) + + # (5) Bridge: align LLM hidden(seg0) with JEPA latent(seg0) + bridge = self.bridge.info_nce( + self.bridge(hidden.cpu() if hidden.device.type != "cpu" else hidden), + self.jepa.enc(seg_seq[:, 0]).detach()) + + loss = (w["lm"] * lm.cpu() + w["jepa"] * jepa + w["world"] * world + + w["ebm"] * ebm + w["bridge"] * bridge) + opt.zero_grad(); loss.backward(); opt.step() + self.jepa.ema() + return dict(lm=lm.item(), jepa=jepa.item(), world=world.item(), + ebm=ebm.item(), bridge=bridge.item()) + + def make_optimizer(self, lr_lora=2e-4, lr_aux=1e-3): + return torch.optim.AdamW([ + {"params": list(self.llm.trainable_parameters()), "lr": lr_lora}, + {"params": list(self.jepa.enc.parameters()) + + list(self.jepa.pred.parameters()), "lr": lr_aux}, + {"params": list(self.world.parameters()), "lr": lr_aux}, + {"params": list(self.energy.parameters()), "lr": lr_aux}, + {"params": list(self.bridge.parameters()) + + list(self.emb.parameters()), "lr": lr_aux}]) + + +# ============================================================================ +# 8. Data helpers + demo +# ============================================================================ +def toy_segment_sequences(B=8, T=4, L=24, vocab=1000): + """Random docs split into T consecutive segments. Replace with real + corpus: tokenize each document, reshape into [T, L] windows.""" + return torch.randint(0, vocab, (B, T, L)) + + +def hf_segment_sequences(llm: HFLLM, texts, T=4, L=64): + seqs = [] + for t in texts: + ids = llm.tokenizer(t, return_tensors="pt").input_ids[0] + n = (len(ids) // (T * L)) * T * L + if n: + seqs.append(ids[:n].view(-1, T, L)) + if not seqs: + raise ValueError("corpus too short for T*L window") + return torch.cat(seqs, 0) + + +def demo(spec="tiny", steps=60): + ens = WorldEnsemble(spec) + opt = ens.make_optimizer() + + if spec == "tiny": + get_batch = lambda: toy_segment_sequences(vocab=ens.llm.vocab_size) + else: + corpus = ["Replace with your real documents. " * 200] + data = hf_segment_sequences(ens.llm, corpus, T=4, L=32) + get_batch = lambda: data[torch.randperm(len(data))[:4]] + steps = min(steps, 10) + + t0 = time.time() + for s in range(steps): + logs = ens.train_step(get_batch(), opt, + hard_negs=(s > steps // 2)) # warmup w/o negs + if s % 10 == 0: + print(f"step {s:3d} | " + + " | ".join(f"{k} {v:.3f}" for k, v in logs.items())) + print(f"trained {steps} steps in {time.time()-t0:.1f}s") + + # memory + inference + for _ in range(4): + if spec == "tiny": + ens.memorize(torch.randint(0, ens.llm.vocab_size, (1, 24))) + q = (torch.randint(0, ens.llm.vocab_size, (1, 12)) if spec == "tiny" + else ens.llm.tokenizer("What is this document about?", + return_tensors="pt").input_ids) + res = ens.answer(q, n_drafts=6, horizon=3) + print(f"\nselected draft energy={res['energy']:.3f} " + f"(all: {[f'{e:.2f}' for e in res['all_energies']]})") + print(f"plan↔output alignment: {res['plan_alignment']:.3f}") + + +if __name__ == "__main__": + demo(sys.argv[1] if len(sys.argv) > 1 else "tiny") diff --git a/research/eval_harness.py b/research/eval_harness.py new file mode 100644 index 0000000000000000000000000000000000000000..1db04110a87f9aba0ec53b7b76fac58c7e887baa --- /dev/null +++ b/research/eval_harness.py @@ -0,0 +1,6 @@ +"""Deprecated shim — use `ensemble.eval.jepa_harness` instead.""" + +from ensemble.eval.jepa_harness import run, parse_args + +if __name__ == "__main__": + run(parse_args()) diff --git a/research/evals/README.md b/research/evals/README.md new file mode 100644 index 0000000000000000000000000000000000000000..93f4475b1395c0d51bbb4273dfcfa093efbe3214 --- /dev/null +++ b/research/evals/README.md @@ -0,0 +1,98 @@ +# SLM Agentic Benchmark Suite + +A uv workspace package to evaluate **local HuggingFace models** against agentic and academic benchmarks. + +**Docs:** [USAGE.md](USAGE.md) (commands and workflows) · [docs/benchmarks.md](docs/benchmarks.md) (per-benchmark reference) · [../USAGE.md](../USAGE.md) (full research tree) + +| Suite | CLI | What it measures | +|---|---|---| +| **Agentic** | `slm-benchmark` | BFCL, τ-bench, GAIA, SWE-bench | +| **Academic** | `slm-lm-eval` | ARC, HellaSwag, GSM8K, … (lm-evaluation-harness) | + +## Install + +From the repo root: + +```bash +uv sync --group evals +uv sync --group lm-eval # optional: slm-lm-eval academic benchmarks +``` + +## Quickstart + +```bash +# From repo root (recommended) +uv run --package slm-evals slm-benchmark \ + --model openbmb/MiniCPM5-1B \ + --benchmarks bfcl tau_bench \ + --max-samples 20 + +# Or as a module +uv run --package slm-evals python -m slm_evals.run_benchmark \ + --model openbmb/MiniCPM5-1B \ + --benchmarks bfcl tau_bench \ + --max-samples 20 + +# YAML config +uv run --package slm-evals slm-benchmark \ + --config research/evals/configs/experiment_001.yaml +``` + +## Project structure + +``` +research/evals/ +├── pyproject.toml +├── configs/ +│ └── experiment_001.yaml +├── src/slm_evals/ +│ ├── run_benchmark.py +│ ├── benchmarks/ +│ │ ├── base.py +│ │ ├── bfcl.py +│ │ ├── tau_bench.py +│ │ ├── gaia.py +│ │ └── swe_bench.py +│ └── utils/ +│ ├── model_loader.py +│ ├── reporter.py +│ └── config_loader.py +└── results/ # created at runtime (relative to cwd) +``` + +## CLI reference + +``` +--model Path to local HF model dir (or Hub ID) +--benchmarks Space-separated: bfcl tau_bench gaia swe_bench all +--config YAML config file (overrides CLI flags) +--max-samples Cap samples per benchmark +--output-dir Results directory (default: ./results) +--experiment-name Tag for this run +--device auto | cpu | cuda | cuda:0 +--dtype float32 | float16 | bfloat16 | int8 | int4 +--max-new-tokens Max tokens per generation (default: 512) +--temperature Sampling temp (default: 0.0 = greedy) +``` + +## Adding a custom benchmark + +1. Create `src/slm_evals/benchmarks/my_bench.py` and subclass `BaseBenchmark`. +2. Register it in `src/slm_evals/run_benchmark.py` → `BENCHMARK_REGISTRY`. +3. Run: `uv run --package slm-evals slm-benchmark --model ./my-model --benchmarks my_bench` + +## Output formats + +Results are written under `//`: + +- `results.json` — full structured dump +- `results.csv` — one row per benchmark +- `report.md` — human-readable summary + +## Notes + +**τ-bench user simulator**: Default is a lightweight rule-based simulator. Set `use_llm_user: true` in config for the GPT-4o user agent (API cost). + +**SWE-bench full eval**: Set `full_eval: true` to run the official Docker harness (`pip install swebench docker`). + +**GAIA tools**: Offline by default (`tool_mode: describe`). Wire real tools in `gaia.py` for live eval. diff --git a/research/evals/USAGE.md b/research/evals/USAGE.md new file mode 100644 index 0000000000000000000000000000000000000000..94f8bc919196610b9ae8e5186b402f7efe2d2108 --- /dev/null +++ b/research/evals/USAGE.md @@ -0,0 +1,376 @@ +# Evals usage + +Run the **SLM Agentic Benchmark Suite** (`slm-evals`) against a local HuggingFace model directory or Hub id. + +Benchmark details: [docs/benchmarks.md](docs/benchmarks.md). Package overview: [README.md](README.md). + +## Install + +From the repo root: + +```bash +uv sync --group evals +``` + +For academic benchmarks (lm-evaluation-harness): + +```bash +uv sync --group lm-eval +``` + +This installs the `slm-evals` workspace package and registers the `slm-benchmark` and `slm-lm-eval` console scripts. + +## Quick start + +```bash +# Two benchmarks, capped samples (good first run) +uv run --package slm-evals slm-benchmark \ + --model openbmb/MiniCPM5-1B \ + --benchmarks bfcl tau_bench \ + --max-samples 20 + +# All four benchmarks +uv run --package slm-evals slm-benchmark \ + --model ./models/finetuned/minicpm5-1b-lora \ + --benchmarks all \ + --max-samples 50 + +# Equivalent module invocation +uv run --package slm-evals python -m slm_evals.run_benchmark \ + --model openbmb/MiniCPM5-1B \ + --benchmarks bfcl \ + --max-samples 10 +``` + +## Config-driven runs + +Copy and edit the template, then pass `--config`: + +```bash +cp research/evals/configs/experiment_001.yaml research/evals/configs/my_run.yaml +# edit model_path, benchmarks, max_samples, overrides + +uv run --package slm-evals slm-benchmark \ + --config research/evals/configs/my_run.yaml +``` + +When `--config` is set, **YAML values override CLI flags**. Use configs for reproducible experiment names and per-benchmark settings. + +### Template fields + +| Key | Description | +| --- | ----------- | +| `model_path` | Local directory or HF Hub id | +| `device` | `auto`, `cpu`, `cuda`, `cuda:0`, … | +| `dtype` | `float32`, `float16`, `bfloat16`, `int8`, `int4` | +| `max_new_tokens` | Cap per generation (default 512) | +| `temperature` | `0.0` = greedy (recommended for evals) | +| `experiment_name` | Folder name under `output_dir` | +| `output_dir` | Root for results (default `results`) | +| `benchmarks` | List: `bfcl`, `tau_bench`, `gaia`, `swe_bench` | +| `max_samples` | Cap per benchmark; omit or `null` for full split | +| `benchmark_overrides` | Per-benchmark dict (see [docs/benchmarks.md](docs/benchmarks.md)) | + +--- + +## CLI reference + +``` +slm-benchmark [OPTIONS] + +--list-benchmarks Show agentic benchmark keys and preset suites +--model PATH Local HF dir or Hub id (required unless --config) +--benchmarks NAMES bfcl tau_bench gaia swe_bench all (default: all) +--config PATH YAML config (overrides other flags) +--max-samples N Cap samples per benchmark +--output-dir DIR Results root (default: ./results) +--experiment-name TAG Run folder name (auto timestamp if omitted) +--device MAP auto | cpu | cuda | cuda:0 +--dtype TYPE float32 | float16 | bfloat16 | int8 | int4 +--max-new-tokens N Default 512 +--temperature T Default 0.0 +``` + +--- + +## Results + +Each run writes to `//`: + +| File | Contents | +| ---- | -------- | +| `results.json` | Full structured payload (per-sample + aggregates) | +| `results.csv` | One row per benchmark | +| `report.md` | Human-readable summary | + +Example layout: + +```text +results/ +└── minicpm5-1b__bfcl-tau__v1/ + ├── results.json + ├── results.csv + └── report.md +``` + +`output_dir` is relative to **current working directory**. Run from repo root so paths stay predictable, or set an absolute `output_dir` in YAML. + +--- + +## Per-benchmark tips + +### BFCL (function calling) + +- Default: downloads from `gorilla-llm/Berkeley-Function-Calling-Leaderboard` +- `strict: false` in YAML — fuzzy argument matching (better for small models) +- Local JSONL: set `benchmark_overrides.bfcl.data_path` + +### τ-bench (multi-turn tools) + +- Domains: `retail`, `airline`, or `both` +- `use_llm_user: false` — free rule-based user simulator (default) +- `use_llm_user: true` — GPT-4o user agent (**API cost**) + +### GAIA + +- Default split: `validation` (public) +- `tool_mode: describe` — offline tool descriptions (no live web) +- Level filter: `levels: [1, 2]` or `[1, 2, 3]` + +### SWE-bench Verified + +- Default: lightweight patch-generation scoring (no Docker) +- `full_eval: true` — official harness (`pip install swebench docker`) + +See [docs/benchmarks.md](docs/benchmarks.md) for scoring semantics. + +--- + +## lm-evaluation-harness (`slm-lm-eval`) + +Run standard academic benchmarks (ARC, HellaSwag, PIQA, BoolQ, GSM8K) via [EleutherAI lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness). + +Install: `uv sync --group lm-eval` + +Full profile guide: [docs/eval_profiles.md](docs/eval_profiles.md) + +### Discover profiles and tasks + +```bash +# Claim-matched lm-eval profiles (reasoning, code, smoke, …) +uv run --package slm-evals slm-lm-eval --list-profiles + +# Also show agentic suites + external benchmark notes +uv run --package slm-evals slm-lm-eval --list-profiles-all + +# lm-eval task names +uv run --package slm-evals slm-lm-eval --list-tasks + +# Agentic benchmarks (BFCL, τ-bench, GAIA, SWE) +uv run --package slm-evals slm-benchmark --list-benchmarks +``` + +### Quick start + +```bash +# By profile name (recommended) +uv run --package slm-evals slm-lm-eval \ + --profile reasoning \ + --preset minicpm5-1b \ + --experiment-name minicpm5-1b__reasoning-baseline + +# Smoke profile (25 samples) +uv run --package slm-evals slm-lm-eval \ + --profile smoke \ + --preset minicpm5-1b \ + --experiment-name minicpm5-1b__smoke + +# LoRA adapter via preset (base + peft resolved automatically) +uv run --package slm-evals slm-lm-eval \ + --config research/evals/configs/lm_eval_minicpm5.yaml \ + --preset minicpm5-1b-lesson-lora \ + --experiment-name minicpm5-1b-lora__v1 + +# Explicit base + adapter +uv run --package slm-evals slm-lm-eval \ + --config research/evals/configs/lm_eval_smoke.yaml \ + --model openbmb/MiniCPM5-1B \ + --adapter ./models/finetuned/minicpm5-1b-lora \ + --experiment-name minicpm5-1b-lora__manual + +# Ensemble checkpoint (manifest.json auto-detected) +uv run --package slm-evals slm-lm-eval \ + --config research/evals/configs/lm_eval_smoke.yaml \ + --model ./models/ensemble/jepa-lesson-pretrain \ + --experiment-name ensemble-jepa__lm-eval +``` + +### Compare baseline vs candidate + +Use the **same config** for both runs; only change `--preset` / `--experiment-name`: + +```bash +uv run --package slm-evals slm-lm-eval \ + --config research/evals/configs/lm_eval_compare_study.yaml \ + --preset minicpm5-1b \ + --experiment-name minicpm5-1b__baseline + +uv run --package slm-evals slm-lm-eval \ + --config research/evals/configs/lm_eval_compare_study.yaml \ + --preset minicpm5-1b-lesson-lora \ + --experiment-name minicpm5-1b-lora__v1 \ + --compare-to results/lm_eval/minicpm5-1b__baseline/results.json +``` + +### Config templates + +Catalog: `configs/eval_profiles.yaml` — maps **claim → profile → tasks**. + +| Profile (`--profile`) | Config file | Purpose | +| --------------------- | ----------- | ------- | +| `smoke` | `lm_eval_smoke.yaml` | Fast validation (`limit: 25`, 2 tasks) | +| `reasoning` | `lm_eval_reasoning.yaml` | Math + commonsense (GSM8K, ARC, HellaSwag) | +| `understanding` | `lm_eval_understanding.yaml` | NLU (BoolQ, PIQA, COPA, RTE) | +| `code` | `lm_eval_code.yaml` | HumanEval + MBPP | +| `instructions` | `lm_eval_instructions.yaml` | IFEval instruction following | +| `general_slm` | `lm_eval_minicpm5.yaml` | Full ~1B SLM profile (6 tasks) | +| `compare_study` | `lm_eval_compare_study.yaml` | Baseline vs finetune comparison defaults | + +| Key | Description | +| --- | ----------- | +| `tasks` | lm-eval task names (e.g. `arc_easy`, `gsm8k`) | +| `num_fewshot` | Few-shot count (gsm8k may use task default 8) | +| `limit` | Max samples per task; `null` = full split | +| `seed` | Random seed (applied to all lm-eval RNGs) | +| `batch_size` | `auto` or integer | +| `device` | `auto`, `cpu`, `cuda`, … | +| `dtype` | `bfloat16`, `float16`, `int4`, … | +| `trust_remote_code` | Required for MiniCPM / Gemma presets | +| `output_dir` | Root for runs (default `results/lm_eval`) | + +### CLI reference + +``` +slm-lm-eval [OPTIONS] + +--list-profiles Show claim-matched profiles and example commands +--list-profiles-all Include agentic suites and external benchmark notes +--list-tasks List lm-eval task names (catalog fallback if not installed) +--list-tasks-all Full lm-eval task list +--profile NAME Shorthand for --config (reasoning, code, smoke, …) +--config PATH YAML config (tasks, seed, limit, …) +--preset KEY models.yaml preset (base, LoRA, merged, ensemble) +--model PATH HF Hub id, merged dir, or ensemble checkpoint +--adapter PATH LoRA adapter (alternative to preset adapter_path) +--tasks NAMES Override task list +--num-fewshot N +--limit N Cap samples per task +--seed N +--batch-size VALUE +--device MAP +--dtype TYPE +--output-dir DIR Default: results/lm_eval +--experiment-name TAG Run folder name +--compare-to PATH Baseline results.json for delta table +``` + +### Results + +Each run writes to `//`: + +| File | Contents | +| ---- | -------- | +| `results.json` | lm-eval native payload + `run_meta` | +| `summary.md` | Task → metric table | +| `run_meta.json` | Preset, base model, adapter, tasks, seed | +| `comparison.md` | Delta table (when `--compare-to` set) | + +### Ensemble backend notes + +- **`ensemble-lm`** loads JEPA checkpoints via `manifest.json`. +- **`generate_until`** tasks (e.g. `gsm8k`) use the full ensemble stack (`generate_text`). +- **`loglikelihood`** tasks (e.g. `arc_easy`, `hellaswag`) score the underlying HF LLM head (adapter 0), not the JEPA selector. Use [`jepa_harness`](../ensemble/README.md) to measure selector value on domain QA. + +### PEFT / LoRA + +lm-eval expects `pretrained=,peft=`. The preset resolver handles this for keys like `minicpm5-1b-lesson-lora`. Merged checkpoints use `--preset minicpm5-1b-lesson-merged` or `--model ./models/finetuned/...-merged`. + +--- + +## Adding a custom benchmark + +1. Create `src/slm_evals/benchmarks/my_bench.py` subclassing `BaseBenchmark`: + - `load_dataset()` → list of sample dicts + - `build_prompt(sample)` → prompt string + - `evaluate_sample(sample, prediction)` → `{passed, score, note}` + +2. Register in `src/slm_evals/run_benchmark.py` → `BENCHMARK_REGISTRY`. + +3. Run: + ```bash + uv run --package slm-evals slm-benchmark \ + --model ./my-model --benchmarks my_bench --max-samples 10 + ``` + +--- + +## Suggested workflows + +### Smoke (CPU/GPU, ~5 min) + +```bash +uv run --package slm-evals slm-benchmark \ + --model openbmb/MiniCPM5-1B \ + --benchmarks bfcl \ + --max-samples 5 \ + --device cpu +``` + +### Before / after fine-tune + +```bash +BASE=openbmb/MiniCPM5-1B +ADAPTER=./models/finetuned/minicpm5-1b-lora + +for M in "$BASE" "$ADAPTER"; do + uv run --package slm-evals slm-benchmark \ + --model "$M" \ + --benchmarks bfcl tau_bench \ + --max-samples 100 \ + --experiment-name "$(basename "$M")__bfcl-tau" +done +``` + +### Full experiment (YAML) + +Edit `configs/experiment_001.yaml` with your `model_path` and `experiment_name`, then: + +```bash +uv run --package slm-evals slm-benchmark \ + --config research/evals/configs/experiment_001.yaml +``` + +--- + +## Troubleshooting + +| Symptom | Likely cause | Fix | +| ------- | ------------ | --- | +| `error: --model is required` | No `--config` and no `--model` | Pass one of them | +| CUDA OOM | Model too large for VRAM | `--dtype int4` or `--device cpu` | +| HF dataset 401 on GAIA test | Gated split | Use `split: validation` | +| τ-bench hangs / costs | LLM user enabled | Set `use_llm_user: false` | +| Empty `results/` | Wrong cwd | Run from repo root or use absolute `output_dir` | +| Import errors | Evals group not synced | `uv sync --group evals` | + +--- + +## Entry points + +| Path | Role | +| ---- | ---- | +| `slm-benchmark` | Agentic benchmarks (BFCL, τ-bench, GAIA, SWE) | +| `slm-lm-eval` | Academic benchmarks via lm-evaluation-harness | +| `python -m slm_evals.run_benchmark` | Same as `slm-benchmark` | +| `python -m slm_evals.run_lm_eval` | Same as `slm-lm-eval` | +| `research/evals/run_benchmark.py` | Thin shim for backward compatibility | diff --git a/research/evals/configs/ensemble_jepa_lesson.yaml b/research/evals/configs/ensemble_jepa_lesson.yaml new file mode 100644 index 0000000000000000000000000000000000000000..72575f199580c57a7860d316e2968042c7505b9d --- /dev/null +++ b/research/evals/configs/ensemble_jepa_lesson.yaml @@ -0,0 +1,24 @@ +# JEPA ensemble checkpoint (models/ensemble/jepa-lesson-pretrain) +# Pretrain: uv run --package ensemble ensemble-pretrain --llm Qwen/Qwen2.5-0.5B-Instruct +# Compare baseline: copy this file, set model_path to the base Hub id and model_type: hf + +model_path: "./models/ensemble/jepa-lesson-pretrain" +model_type: "ensemble" +device: "auto" +dtype: "bfloat16" + +max_new_tokens: 512 +temperature: 0.0 + +experiment_name: "jepa-ensemble-lesson__bfcl-tau__v1" +output_dir: "results" + +benchmarks: + - bfcl + - tau_bench + +max_samples: 20 + +benchmark_overrides: + tau_bench: + use_llm_user: false diff --git a/research/evals/configs/eval_profiles.yaml b/research/evals/configs/eval_profiles.yaml new file mode 100644 index 0000000000000000000000000000000000000000..73b21bba5020dc0a2ae5c0b6e03f4f1b340a7b12 --- /dev/null +++ b/research/evals/configs/eval_profiles.yaml @@ -0,0 +1,160 @@ +# Eval profile catalog — match your model claim to a profile, then run with: +# slm-lm-eval --list-profiles +# slm-lm-eval --profile reasoning --preset minicpm5-1b --experiment-name my-run +# +# See research/evals/docs/eval_profiles.md for full guide. + +profiles: + smoke: + tool: slm-lm-eval + claim: Quick sanity check + description: Fast validation before a long run (2 tasks, capped samples). + config: lm_eval_smoke.yaml + tasks: + - arc_easy + - hellaswag + + reasoning: + tool: slm-lm-eval + claim: Better reasoning + description: Math + commonsense + science QA (generation + multiple-choice). + config: lm_eval_reasoning.yaml + tasks: + - gsm8k + - arc_easy + - arc_challenge + - hellaswag + + understanding: + tool: slm-lm-eval + claim: Better language understanding + description: NLU / reading comprehension (SuperGLUE-style multiple-choice). + config: lm_eval_understanding.yaml + tasks: + - boolq + - piqa + - copa + - rte + + code: + tool: slm-lm-eval + claim: Better code generation + description: Python function synthesis (HumanEval + MBPP via lm-eval). + config: lm_eval_code.yaml + tasks: + - humaneval + - mbpp + + instructions: + tool: slm-lm-eval + claim: Better instruction following + description: Verifiable instruction constraints (IFEval). + config: lm_eval_instructions.yaml + tasks: + - ifeval + + general_slm: + tool: slm-lm-eval + claim: General ~1B SLM baseline + description: Balanced academic mix for before/after fine-tune on chat data. + config: lm_eval_minicpm5.yaml + tasks: + - arc_easy + - arc_challenge + - hellaswag + - piqa + - boolq + - gsm8k + + compare_study: + tool: slm-lm-eval + claim: Baseline vs finetune comparison + description: Same tasks as general_slm with limit 100 for paired studies. + config: lm_eval_compare_study.yaml + tasks: + - arc_easy + - arc_challenge + - hellaswag + - piqa + - boolq + - gsm8k + +suites: + agentic_tool_use: + tool: slm-benchmark + claim: Tool use and function calling + description: BFCL + tau-bench for agents that call tools. + command: >- + uv run --package slm-evals slm-benchmark + --model --benchmarks bfcl tau_bench --max-samples 50 + benchmarks: + - bfcl + - tau_bench + + agentic_gaia: + tool: slm-benchmark + claim: End-to-end assistant tasks + description: GAIA — multi-step reasoning with optional tools. + command: >- + uv run --package slm-evals slm-benchmark + --model --benchmarks gaia --max-samples 20 + benchmarks: + - gaia + + agentic_code: + tool: slm-benchmark + claim: Real-world code repair + description: SWE-bench Verified patch generation (lightweight mode by default). + command: >- + uv run --package slm-evals slm-benchmark + --model --benchmarks swe_bench --max-samples 10 + benchmarks: + - swe_bench + + agentic_all: + tool: slm-benchmark + claim: Full agentic suite + description: All four slm-benchmark benchmarks. + command: >- + uv run --package slm-evals slm-benchmark + --model --benchmarks all --max-samples 50 + benchmarks: + - bfcl + - tau_bench + - gaia + - swe_bench + + jepa_selector: + tool: jepa_harness + claim: JEPA / draft selector quality + description: >- + Domain QA on benchmark-qa.jsonl; compares first vs random vs JEPA vs oracle + draft selection. Use for ensemble checkpoints, not base LLM alone. + command: >- + uv run --package ensemble python -m ensemble.eval.jepa_harness + --checkpoint ./models/ensemble/ --data research/data/benchmark-qa.jsonl + + world_model: + tool: world_harness + claim: World model / energy-based planning + description: Energy-ranked draft selection on domain QA (world-model track). + command: >- + uv run --package ensemble python -m ensemble.eval.world_harness + --checkpoint ./models/ensemble/ --data research/data/benchmark-qa.jsonl + +external: + embeddings_mteb: + tool: mteb + claim: Better embeddings + description: MTEB, BEIR, STS — not wired in this repo; use embeddings-benchmark/mteb. + tasks_note: 56+ embedding tasks; requires a dedicated encoder model. + + chat_judge: + tool: mt_bench_alpacaeval + claim: Chat / instruction quality (judge-based) + description: MT-Bench, AlpacaEval — require LLM-as-judge APIs; not in slm-lm-eval. + + rl_envs: + tool: babyai_minigrid + claim: Embodied planning + description: BabyAI, MiniGrid — separate RL env evals; not integrated here. diff --git a/research/evals/configs/experiment_001.yaml b/research/evals/configs/experiment_001.yaml new file mode 100644 index 0000000000000000000000000000000000000000..7138e7dc17aeeece637b340f3e0c878820f41788 --- /dev/null +++ b/research/evals/configs/experiment_001.yaml @@ -0,0 +1,53 @@ +# configs/experiment_001.yaml +# ───────────────────────────────────────────────────────────────────────────── +# Template config for a local SLM agentic benchmark run. +# Copy and edit this file for each new experiment. +# Run with: uv run --package slm-evals slm-benchmark --config research/evals/configs/experiment_001.yaml +# ───────────────────────────────────────────────────────────────────────────── + +# ── Model ───────────────────────────────────────────────────────────────────── +model_path: "./models/my-small-model-7b" # local directory or HF Hub ID +device: "auto" # auto | cpu | cuda | cuda:0 +dtype: "bfloat16" # float32 | float16 | bfloat16 | int8 | int4 + +# ── Inference ───────────────────────────────────────────────────────────────── +max_new_tokens: 512 +temperature: 0.0 # 0 = greedy (recommended for deterministic evals) + +# ── Experiment ──────────────────────────────────────────────────────────────── +experiment_name: "my-model-7b__bfcl-tau-gaia-swe__v1" +output_dir: "results" + +# ── Benchmarks ──────────────────────────────────────────────────────────────── +benchmarks: + - bfcl + - tau_bench + - gaia + - swe_bench + +max_samples: 100 # null = run all; set a number for quick smoke tests + +# ── Per-benchmark overrides ─────────────────────────────────────────────────── +benchmark_overrides: + + bfcl: + # data_path: "./data/bfcl_v4_local.jsonl" # uncomment to use local copy + strict: false # false = fuzzy arg matching (recommended for SLMs) + categories: [] # [] = all categories + + tau_bench: + # data_path: "./data/tau_bench_retail.jsonl" + domain: "retail" # retail | airline | both + max_turns: 15 + use_llm_user: false # true = use GPT-4o as user simulator (costs $) + + gaia: + # data_path: "./data/gaia_validation.jsonl" + split: "validation" # validation (public) | test (needs HF auth) + levels: [1, 2] # [1] easy only, [1,2,3] all levels + tool_mode: "describe" # describe (offline) | none + + swe_bench: + # data_path: "./data/swe_bench_verified.jsonl" + full_eval: false # true = run Docker harness (requires swebench pkg) + context_lines: 80 diff --git a/research/evals/configs/lm_eval_code.yaml b/research/evals/configs/lm_eval_code.yaml new file mode 100644 index 0000000000000000000000000000000000000000..ac353b826217baf681a9432c2f9a0e1557a15a42 --- /dev/null +++ b/research/evals/configs/lm_eval_code.yaml @@ -0,0 +1,19 @@ +# Code generation profile — HumanEval + MBPP +# Run: slm-lm-eval --profile code --preset minicpm5-1b-lesson-lora +# Note: small models often score low; use --limit 25 for smoke tests. + +profile: code +claim: Better code generation + +tasks: + - humaneval + - mbpp + +num_fewshot: 0 +limit: 50 +seed: 42 +batch_size: auto +device: auto +dtype: bfloat16 +trust_remote_code: true +output_dir: results/lm_eval diff --git a/research/evals/configs/lm_eval_compare_study.yaml b/research/evals/configs/lm_eval_compare_study.yaml new file mode 100644 index 0000000000000000000000000000000000000000..f65ed93a072785e3195aa53e07819b2b0d02598f --- /dev/null +++ b/research/evals/configs/lm_eval_compare_study.yaml @@ -0,0 +1,42 @@ +# configs/lm_eval_compare_study.yaml +# ───────────────────────────────────────────────────────────────────────────── +# Shared settings for baseline vs finetuned comparisons. +# Use the SAME config for both runs; only change --preset / --experiment-name. +# +# Baseline: +# uv run --package slm-evals slm-lm-eval \ +# --config research/evals/configs/lm_eval_compare_study.yaml \ +# --preset minicpm5-1b \ +# --experiment-name minicpm5-1b__baseline +# +# Candidate (after finetune): +# uv run --package slm-evals slm-lm-eval \ +# --config research/evals/configs/lm_eval_compare_study.yaml \ +# --preset minicpm5-1b-lesson-lora \ +# --experiment-name minicpm5-1b-lora__v1 \ +# --compare-to results/lm_eval/minicpm5-1b__baseline/results.json +# ───────────────────────────────────────────────────────────────────────────── + +study: + baseline_preset: minicpm5-1b + candidate_preset: minicpm5-1b-lesson-lora + notes: > + Keep tasks, num_fewshot, limit, and seed identical across runs. + Do not compare training_results.json result_score to lm-eval accuracy. + +tasks: + - arc_easy + - arc_challenge + - hellaswag + - piqa + - boolq + - gsm8k + +num_fewshot: 5 +limit: 100 +seed: 42 +batch_size: auto +device: auto +dtype: bfloat16 +trust_remote_code: true +output_dir: results/lm_eval diff --git a/research/evals/configs/lm_eval_instructions.yaml b/research/evals/configs/lm_eval_instructions.yaml new file mode 100644 index 0000000000000000000000000000000000000000..5a43e57cfcfa0528c1cf5bd6687bf8ccc6ea41cc --- /dev/null +++ b/research/evals/configs/lm_eval_instructions.yaml @@ -0,0 +1,18 @@ +# Instruction following profile — IFEval (verifiable constraints) +# Run: slm-lm-eval --profile instructions --preset minicpm5-1b +# Requires lm-eval extras; install with: uv sync --group lm-eval + +profile: instructions +claim: Better instruction following + +tasks: + - ifeval + +num_fewshot: 0 +limit: 100 +seed: 42 +batch_size: auto +device: auto +dtype: bfloat16 +trust_remote_code: true +output_dir: results/lm_eval diff --git a/research/evals/configs/lm_eval_minicpm5.yaml b/research/evals/configs/lm_eval_minicpm5.yaml new file mode 100644 index 0000000000000000000000000000000000000000..a3b360f83f40c81b90b42d63c206aebe0bf3828e --- /dev/null +++ b/research/evals/configs/lm_eval_minicpm5.yaml @@ -0,0 +1,26 @@ +# configs/lm_eval_minicpm5.yaml +# ───────────────────────────────────────────────────────────────────────────── +# Full academic benchmark profile for ~1B SLMs (MiniCPM5, Gemma E2B, etc.). +# Run with: +# uv run --package slm-evals slm-lm-eval \ +# --config research/evals/configs/lm_eval_minicpm5.yaml \ +# --preset minicpm5-1b \ +# --experiment-name minicpm5-1b__baseline +# ───────────────────────────────────────────────────────────────────────────── + +tasks: + - arc_easy + - arc_challenge + - hellaswag + - piqa + - boolq + - gsm8k + +num_fewshot: 5 +limit: null +seed: 42 +batch_size: auto +device: auto +dtype: bfloat16 +trust_remote_code: true +output_dir: results/lm_eval diff --git a/research/evals/configs/lm_eval_reasoning.yaml b/research/evals/configs/lm_eval_reasoning.yaml new file mode 100644 index 0000000000000000000000000000000000000000..fc2ba45198cf0136ceb9c1ef2d20465009c958bc --- /dev/null +++ b/research/evals/configs/lm_eval_reasoning.yaml @@ -0,0 +1,20 @@ +# Reasoning profile — math + commonsense + science QA +# Run: slm-lm-eval --profile reasoning --preset minicpm5-1b --experiment-name reasoning-baseline + +profile: reasoning +claim: Better reasoning + +tasks: + - gsm8k + - arc_easy + - arc_challenge + - hellaswag + +num_fewshot: 5 +limit: 100 +seed: 42 +batch_size: auto +device: auto +dtype: bfloat16 +trust_remote_code: true +output_dir: results/lm_eval diff --git a/research/evals/configs/lm_eval_smoke.yaml b/research/evals/configs/lm_eval_smoke.yaml new file mode 100644 index 0000000000000000000000000000000000000000..43167904a7c338dd0212fe7d638dde98e67182cb --- /dev/null +++ b/research/evals/configs/lm_eval_smoke.yaml @@ -0,0 +1,20 @@ +# configs/lm_eval_smoke.yaml +# Quick smoke profile — use for CI or first-run validation. +# Run with: +# uv run --package slm-evals slm-lm-eval \ +# --config research/evals/configs/lm_eval_smoke.yaml \ +# --preset minicpm5-1b \ +# --experiment-name minicpm5-1b__smoke + +tasks: + - arc_easy + - hellaswag + +num_fewshot: 0 +limit: 25 +seed: 42 +batch_size: auto +device: auto +dtype: bfloat16 +trust_remote_code: true +output_dir: results/lm_eval diff --git a/research/evals/configs/lm_eval_understanding.yaml b/research/evals/configs/lm_eval_understanding.yaml new file mode 100644 index 0000000000000000000000000000000000000000..f0a4515fdbe0637cc40714a886145ad6b325ed2d --- /dev/null +++ b/research/evals/configs/lm_eval_understanding.yaml @@ -0,0 +1,20 @@ +# Language understanding profile — NLU / reading comprehension +# Run: slm-lm-eval --profile understanding --preset minicpm5-1b + +profile: understanding +claim: Better language understanding + +tasks: + - boolq + - piqa + - copa + - rte + +num_fewshot: 0 +limit: 100 +seed: 42 +batch_size: auto +device: auto +dtype: bfloat16 +trust_remote_code: true +output_dir: results/lm_eval diff --git a/research/evals/docs/benchmarks.md b/research/evals/docs/benchmarks.md new file mode 100644 index 0000000000000000000000000000000000000000..35ff4c06669dc637fda9e38e56f1ac27137b1a67 --- /dev/null +++ b/research/evals/docs/benchmarks.md @@ -0,0 +1,165 @@ +# Benchmark reference + +What each benchmark in `slm-evals` measures, where data comes from, and how to configure overrides. + +All benchmarks extend `BaseBenchmark` (`src/slm_evals/benchmarks/base.py`): + +1. `load_dataset()` — fetch samples (Hub or local JSONL) +2. `build_prompt(sample)` — format the model input +3. `evaluate_sample(sample, prediction)` — return `{passed, score, note}` +4. `run()` — iterate, call `generate_fn`, aggregate scores (inherited) + +--- + +## Summary table + +| Key | Benchmark | Measures | Default dataset | +| --- | --------- | -------- | --------------- | +| `bfcl` | Berkeley Function-Calling Leaderboard v4 | Single-turn function call accuracy | `gorilla-llm/Berkeley-Function-Calling-Leaderboard` | +| `tau_bench` | τ-bench | Multi-turn tool + user simulation | `ShishirPatil/tau-bench` | +| `gaia` | GAIA | End-to-end agent tasks (reasoning + tools) | `gaia-benchmark/GAIA` | +| `swe_bench` | SWE-bench Verified | Code patch generation for real issues | `princeton-nlp/SWE-bench_Verified` | + +--- + +## BFCL (`bfcl`) + +**Goal:** Given a user request and a function schema, does the model emit a valid JSON tool call with the correct name and arguments? + +**Prompt style:** System message lists available functions; model must reply with only: + +```json +{"name": "", "arguments": {: }} +``` + +**Scoring:** + +- Function name must match exactly +- Arguments: exact match if `strict: true`, fuzzy match if `strict: false` (recommended for SLMs) + +**Config overrides** (`benchmark_overrides.bfcl`): + +| Key | Default | Description | +| --- | ------- | ----------- | +| `data_path` | Hub | Local JSONL instead of Hub download | +| `categories` | `[]` (all) | Filter BFCL categories | +| `strict` | `false` | Require perfect argument match | + +**Implementation:** `src/slm_evals/benchmarks/bfcl.py` + +--- + +## τ-bench (`tau_bench`) + +**Goal:** Multi-turn dialogue where the model acts as a tool-using agent while a simulated user drives the conversation toward a goal (e.g. retail order change). + +**Scoring:** Task success after up to `max_turns` exchanges — did the agent satisfy the user's underlying intent using the right tools? + +**Config overrides** (`benchmark_overrides.tau_bench`): + +| Key | Default | Description | +| --- | ------- | ----------- | +| `data_path` | Hub | Local JSONL | +| `domain` | `retail` | `retail`, `airline`, or `both` | +| `max_turns` | `15` | Dialogue cap | +| `use_llm_user` | `false` | `true` → GPT-4o user simulator (paid API) | + +**Notes:** + +- Default user simulator is rule-based — no API key required +- Small models often struggle on long horizons; start with `--max-samples 10` + +**Implementation:** `src/slm_evals/benchmarks/tau_bench.py` + +--- + +## GAIA (`gaia`) + +**Goal:** Real-world assistant tasks requiring reasoning, optional tool use, and concise final answers (web search, files, calculation, etc.). + +**Prompt style:** Question + level metadata; tool availability depends on `tool_mode`. + +**Scoring:** Normalized answer match against GAIA reference (with level breakdown in aggregates). + +**Config overrides** (`benchmark_overrides.gaia`): + +| Key | Default | Description | +| --- | ------- | ----------- | +| `data_path` | Hub | Local JSONL | +| `split` | `validation` | Public `validation`; `test` may need HF auth | +| `levels` | `[1, 2]` | Difficulty levels 1–3 | +| `tool_mode` | `describe` | `describe` = offline tool docs; `none` = no tools | + +**Notes:** + +- `tool_mode: describe` does not execute live tools — suitable for offline SLM scoring +- For live tool eval, extend `gaia.py` with real tool backends + +**Implementation:** `src/slm_evals/benchmarks/gaia.py` + +--- + +## SWE-bench Verified (`swe_bench`) + +**Goal:** Given a GitHub issue and codebase context, produce a unified diff that fixes the bug. + +**Modes:** + +| `full_eval` | Behavior | +| ----------- | -------- | +| `false` (default) | Generate patch text; score with lightweight heuristics / match checks — no Docker | +| `true` | Official SWE-bench harness — runs tests in containers (`swebench` + Docker) | + +**Config overrides** (`benchmark_overrides.swe_bench`): + +| Key | Default | Description | +| --- | ------- | ----------- | +| `data_path` | Hub | Local JSONL | +| `full_eval` | `false` | Enable Docker harness | +| `context_lines` | `80` | Surrounding code context in prompt | + +**Notes:** + +- Full eval is slow and resource-heavy — use for final validation only +- SLMs typically score low; use `--max-samples` for iterative prompt tuning + +**Implementation:** `src/slm_evals/benchmarks/swe_bench.py` + +--- + +## Model loading + +Shared loader: `src/slm_evals/utils/model_loader.py` + +Returns a `model_bundle` dict passed to each benchmark: + +- `generate_fn(prompt, max_new_tokens, temperature)` — unified generation interface +- `param_count` — billions of parameters (for reporting) +- Underlying `model` / `tokenizer` handles + +Quantization (`int8`, `int4`) uses `bitsandbytes` when available. + +--- + +## Reporter output schema + +`Reporter.save()` (`src/slm_evals/utils/reporter.py`) writes: + +**Per benchmark in JSON:** + +```json +{ + "name": "bfcl", + "total": 100, + "passed": 42, + "score": 0.42, + "samples": [...] +} +``` + +**Aggregate fields:** + +- `experiment_name`, `model_path`, `timestamp` +- `aggregate_score` — mean of benchmark scores + +CSV columns: `benchmark`, `total`, `passed`, `score`. diff --git a/research/evals/docs/eval_profiles.md b/research/evals/docs/eval_profiles.md new file mode 100644 index 0000000000000000000000000000000000000000..a5c0c76c5164fb824183e5636b2b537658c74bf4 --- /dev/null +++ b/research/evals/docs/eval_profiles.md @@ -0,0 +1,167 @@ +# Eval profiles guide + +Match what your model is supposed to improve to the right benchmark profile, then run it with one command. + +Catalog file: [`configs/eval_profiles.yaml`](../configs/eval_profiles.yaml) + +## Quick commands + +```bash +# See all lm-eval profiles (reasoning, code, smoke, …) +uv run --package slm-evals slm-lm-eval --list-profiles + +# Include agentic suites (slm-benchmark) and external notes +uv run --package slm-evals slm-lm-eval --list-profiles-all + +# List lm-eval task names available in the harness +uv run --package slm-evals slm-lm-eval --list-tasks + +# Agentic benchmark keys (BFCL, τ-bench, GAIA, SWE) +uv run --package slm-evals slm-benchmark --list-benchmarks + +# Run a profile by name +uv run --package slm-evals slm-lm-eval \ + --profile reasoning \ + --preset minicpm5-1b \ + --experiment-name minicpm5-1b__reasoning-baseline +``` + +Install lm-eval extras first: `uv sync --group lm-eval` + +--- + +## Three eval systems in this repo + +| System | CLI | What it measures | +| ------ | --- | ---------------- | +| Academic (lm-eval harness) | `slm-lm-eval` | ARC, GSM8K, HumanEval, IFEval, … | +| Agentic | `slm-benchmark` | Function calling, multi-turn tools, GAIA, SWE | +| Ensemble-specific | `jepa_harness`, `world_harness` | JEPA draft selection, world-model energy ranking | + +Use **one profile per claim**. Do not compare training loss to lm-eval accuracy. + +--- + +## Match your claim → profile + +| If you claim… | Profile / suite | Tool | Tasks or benchmarks | +| ------------- | ----------------- | ---- | ------------------- | +| Quick sanity check | `smoke` | `slm-lm-eval` | `arc_easy`, `hellaswag` (limit 25) | +| Better reasoning | `reasoning` | `slm-lm-eval` | `gsm8k`, `arc_easy`, `arc_challenge`, `hellaswag` | +| Better language understanding | `understanding` | `slm-lm-eval` | `boolq`, `piqa`, `copa`, `rte` | +| Better code generation | `code` | `slm-lm-eval` | `humaneval`, `mbpp` | +| Better instruction following | `instructions` | `slm-lm-eval` | `ifeval` | +| General ~1B SLM baseline | `general_slm` | `slm-lm-eval` | 6-task mix (full splits) | +| Baseline vs finetune study | `compare_study` | `slm-lm-eval` | Same 6 tasks, limit 100 | +| Tool use / function calling | `agentic_tool_use` | `slm-benchmark` | `bfcl`, `tau_bench` | +| End-to-end assistant tasks | `agentic_gaia` | `slm-benchmark` | `gaia` | +| Real-world code repair | `agentic_code` | `slm-benchmark` | `swe_bench` | +| JEPA / selector quality | `jepa_selector` | `jepa_harness` | Domain QA + draft ablations | +| World model / planning | `world_model` | `world_harness` | Energy-ranked drafts on QA | +| Better embeddings | `embeddings_mteb` | external (MTEB) | Not in this repo | +| Chat quality (judge-based) | `chat_judge` | external | MT-Bench, AlpacaEval | + +--- + +## Profile YAML files + +| Profile key | Config file | +| ----------- | ----------- | +| `smoke` | `lm_eval_smoke.yaml` | +| `reasoning` | `lm_eval_reasoning.yaml` | +| `understanding` | `lm_eval_understanding.yaml` | +| `code` | `lm_eval_code.yaml` | +| `instructions` | `lm_eval_instructions.yaml` | +| `general_slm` | `lm_eval_minicpm5.yaml` | +| `compare_study` | `lm_eval_compare_study.yaml` | + +Equivalent to `--profile reasoning`: + +```bash +uv run --package slm-evals slm-lm-eval \ + --config research/evals/configs/lm_eval_reasoning.yaml \ + --preset minicpm5-1b +``` + +--- + +## Baseline vs candidate workflow + +Use the **same profile** for both runs; only change preset and experiment name: + +```bash +PROFILE=reasoning +BASE=minicpm5-1b__reasoning-baseline +CAND=minicpm5-1b-lora__reasoning + +uv run --package slm-evals slm-lm-eval \ + --profile "$PROFILE" --preset minicpm5-1b --experiment-name "$BASE" + +uv run --package slm-evals slm-lm-eval \ + --profile "$PROFILE" --preset minicpm5-1b-lesson-lora \ + --experiment-name "$CAND" \ + --compare-to "results/lm_eval/${BASE}/results.json" +``` + +Or after finetune: + +```bash +uv run python research/finetune.py --preset minicpm5-1b --mode lora \ + --lm-eval-after \ + --lm-eval-config research/evals/configs/lm_eval_reasoning.yaml \ + --lm-eval-baseline minicpm5-1b +``` + +--- + +## Ensemble models + +For `slm-lm-eval` on ensemble checkpoints: + +- **`generate_until` tasks** (e.g. `gsm8k`) use the full ensemble stack. +- **`loglikelihood` tasks** (e.g. `arc_easy`, `hellaswag`) score the underlying LLM head only. + +To prove the JEPA selector helps, run **`jepa_harness`** — not ARC/HellaSwag alone. + +```bash +uv run --package slm-evals slm-lm-eval \ + --profile reasoning \ + --model ./models/ensemble/custom-jepa-pretrain \ + --experiment-name ensemble__reasoning + +uv run --package ensemble python -m ensemble.eval.jepa_harness \ + --checkpoint ./models/ensemble/custom-jepa-pretrain \ + --data research/data/benchmark-qa.jsonl +``` + +--- + +## Results layout + +**slm-lm-eval** → `results/lm_eval//` + +| File | Contents | +| ---- | -------- | +| `results.json` | Full lm-eval payload + `run_meta` | +| `summary.md` | Task → metric table | +| `run_meta.json` | Profile tasks, preset, seed | +| `comparison.md` | Delta vs baseline (with `--compare-to`) | + +**slm-benchmark** → `results//` (`results.json`, `results.csv`, `report.md`) + +--- + +## Custom tasks + +Override tasks on any profile: + +```bash +uv run --package slm-evals slm-lm-eval \ + --profile smoke \ + --tasks gsm8k arc_easy \ + --preset minicpm5-1b +``` + +Browse all harness tasks: `slm-lm-eval --list-tasks-all` + +See also: [USAGE.md](../USAGE.md), [benchmarks.md](benchmarks.md) diff --git a/research/evals/pyproject.toml b/research/evals/pyproject.toml new file mode 100644 index 0000000000000000000000000000000000000000..62693b819c6a987a1bc4d878e879c23e99c57489 --- /dev/null +++ b/research/evals/pyproject.toml @@ -0,0 +1,35 @@ +[project] +name = "slm-evals" +version = "0.1.0" +description = "SLM agentic benchmark suite (BFCL, tau-bench, GAIA, SWE-bench)" +readme = "README.md" +requires-python = ">=3.12" +dependencies = [ + "accelerate>=0.30.0", + "bitsandbytes>=0.43.0", + "datasets>=2.19.0", + "ensemble", + "huggingface-hub>=0.22.0", + "pandas>=2.0.0", + "peft>=0.14.0", + "pyyaml>=6.0", + "torch>=2.2.0", + "tqdm>=4.66.0", + "transformers>=4.41.0", +] + +[tool.uv.sources] +ensemble = { workspace = true } + +[project.optional-dependencies] +lm-eval = [ + "lm-eval[hf]>=0.4.9", +] + +[project.scripts] +slm-benchmark = "slm_evals.run_benchmark:main" +slm-lm-eval = "slm_evals.run_lm_eval:main" + +[build-system] +requires = ["uv_build>=0.8.13,<0.9.0"] +build-backend = "uv_build" diff --git a/research/evals/run_benchmark.py b/research/evals/run_benchmark.py new file mode 100644 index 0000000000000000000000000000000000000000..facfb4cbf4a37af32f6ba596d4c83cd250d89b7a --- /dev/null +++ b/research/evals/run_benchmark.py @@ -0,0 +1,6 @@ +"""Backward-compatible entrypoint. Prefer: uv run --package slm-evals slm-benchmark""" + +from slm_evals.run_benchmark import main + +if __name__ == "__main__": + main() diff --git a/research/evals/src/slm_evals/__init__.py b/research/evals/src/slm_evals/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..7cff7f5716e777509e511f2caf84626c29766b0c --- /dev/null +++ b/research/evals/src/slm_evals/__init__.py @@ -0,0 +1 @@ +"""SLM agentic benchmark suite.""" diff --git a/research/evals/src/slm_evals/benchmarks/__init__.py b/research/evals/src/slm_evals/benchmarks/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..54639cc95df46f2c00160aa64e183617e9216a6a --- /dev/null +++ b/research/evals/src/slm_evals/benchmarks/__init__.py @@ -0,0 +1 @@ +"""Benchmark adapters.""" diff --git a/research/evals/src/slm_evals/benchmarks/base.py b/research/evals/src/slm_evals/benchmarks/base.py new file mode 100644 index 0000000000000000000000000000000000000000..283c113ebbacc1374a9f2ace1b2b976b5b3fe63d --- /dev/null +++ b/research/evals/src/slm_evals/benchmarks/base.py @@ -0,0 +1,118 @@ +""" +benchmarks/base.py +─────────────────── +Abstract base class every benchmark extends. +""" + +from __future__ import annotations +import time +from abc import ABC, abstractmethod +from typing import Any + + +class BaseBenchmark(ABC): + """ + Subclass this for each benchmark. + + Concrete subclasses must implement: + load_dataset() → list of sample dicts + evaluate_sample(sample, prediction) → dict with keys: + passed (bool), score (float 0-1), note (str) + build_prompt(sample) → str + """ + + name: str = "base" + + def __init__( + self, + model_bundle: dict[str, Any], + max_samples: int | None = None, + max_new_tokens: int = 512, + temperature: float = 0.0, + benchmark_cfg: dict | None = None, + ): + self.model_bundle = model_bundle + self.generate = model_bundle["generate_fn"] + self.max_samples = max_samples + self.max_new_tokens = max_new_tokens + self.temperature = temperature + self.cfg = benchmark_cfg or {} + + # ── Must implement ──────────────────────────────────────────────────────── + + @abstractmethod + def load_dataset(self) -> list[dict]: + """Return a list of sample dicts.""" + + @abstractmethod + def build_prompt(self, sample: dict) -> str: + """Convert a sample dict into the prompt string sent to the model.""" + + @abstractmethod + def evaluate_sample(self, sample: dict, prediction: str) -> dict: + """ + Score one prediction. + + Returns dict: + passed (bool) + score (float, 0–1) + note (str, optional explanation) + """ + + # ── Orchestration — override if needed ─────────────────────────────────── + + def run(self) -> dict[str, Any]: + """Run all samples and aggregate results.""" + dataset = self.load_dataset() + if self.max_samples: + dataset = dataset[: self.max_samples] + + samples_out = [] + total_latency = 0.0 + errors = 0 + + for sample in dataset: + prompt = self.build_prompt(sample) + t0 = time.perf_counter() + try: + prediction = self.generate( + prompt, + max_new_tokens=self.max_new_tokens, + temperature=self.temperature, + ) + latency = time.perf_counter() - t0 + eval_result = self.evaluate_sample(sample, prediction) + except Exception as exc: + latency = time.perf_counter() - t0 + errors += 1 + eval_result = { + "passed": False, + "score": 0.0, + "note": f"ERROR: {exc}", + } + prediction = "" + + samples_out.append( + { + "id": sample.get("id", ""), + "prediction": prediction, + "latency_s": round(latency, 3), + **eval_result, + } + ) + total_latency += latency + + passed = sum(1 for s in samples_out if s["passed"]) + total = len(samples_out) + score = (passed / total) if total else 0.0 + avg_lat = round(total_latency / total, 3) if total else 0.0 + + return { + "benchmark": self.name, + "passed": passed, + "total": total, + "score": score, + "error_count": errors, + "avg_latency_s": avg_lat, + "samples": samples_out, + } diff --git a/research/evals/src/slm_evals/benchmarks/bfcl.py b/research/evals/src/slm_evals/benchmarks/bfcl.py new file mode 100644 index 0000000000000000000000000000000000000000..bc952a19d2875034aaf40781c6152fdef99e8c31 --- /dev/null +++ b/research/evals/src/slm_evals/benchmarks/bfcl.py @@ -0,0 +1,189 @@ +""" +benchmarks/bfcl.py +─────────────────── +Berkeley Function-Calling Leaderboard (BFCL) — local eval adapter. + +What it tests: Can the model call functions with correct names, +argument names, and argument values given a user request + schema? + +Dataset: gorilla-llm/Berkeley-Function-Calling-Leaderboard on HF Hub + OR local JSONL at cfg["data_path"] + +Scoring: Exact-match on function name + fuzzy-match on arguments. +""" + +from __future__ import annotations +import json +import re +from typing import Any + +from slm_evals.benchmarks.base import BaseBenchmark + + +SYSTEM_PROMPT = """\ +You are a helpful assistant with access to the following functions. +To call a function, respond ONLY with a JSON object in this exact format: +{"name": "", "arguments": {: , ...}} +Do not add any explanation or text outside the JSON. +""" + + +class BFCLBenchmark(BaseBenchmark): + """ + BFCL function-calling benchmark. + + Config keys (in benchmark_overrides.bfcl): + data_path – local JSONL file (optional; falls back to HF Hub) + categories – list of categories to filter on (optional) + strict – bool, require perfect arg match (default: False) + """ + + name = "bfcl" + + # ── Dataset ─────────────────────────────────────────────────────────────── + + def load_dataset(self) -> list[dict]: + data_path = self.cfg.get("data_path") + + if data_path: + return self._load_local(data_path) + else: + return self._load_from_hub() + + def _load_local(self, path: str) -> list[dict]: + samples = [] + with open(path) as f: + for line in f: + line = line.strip() + if line: + samples.append(json.loads(line)) + return samples + + def _load_from_hub(self) -> list[dict]: + """ + Pulls the BFCL dataset from HF Hub. + Requires: pip install datasets + """ + try: + from datasets import load_dataset + except ImportError: + raise ImportError("pip install datasets (required to fetch BFCL from Hub)") + + ds = load_dataset( + "gorilla-llm/Berkeley-Function-Calling-Leaderboard", + split="train", + trust_remote_code=True, + ) + return list(ds) + + # ── Prompt ──────────────────────────────────────────────────────────────── + + def build_prompt(self, sample: dict) -> str: + functions_block = json.dumps(sample.get("function", []), indent=2) + user_query = sample.get("question", sample.get("input", "")) + + return ( + f"{SYSTEM_PROMPT}\n" + f"Available functions:\n{functions_block}\n\n" + f"User: {user_query}\n" + f"Assistant:" + ) + + # ── Evaluation ──────────────────────────────────────────────────────────── + + def evaluate_sample(self, sample: dict, prediction: str) -> dict: + """ + 1. Parse model output as JSON. + 2. Check function name matches ground truth. + 3. Check arguments (strict = exact keys+values, else fuzzy). + """ + ground_truth = sample.get("ground_truth", {}) + if isinstance(ground_truth, str): + try: + ground_truth = json.loads(ground_truth) + except json.JSONDecodeError: + ground_truth = {} + + # ── Parse prediction ────────────────────────────────────────────────── + parsed = self._extract_json(prediction) + if parsed is None: + return { + "passed": False, + "score": 0.0, + "note": "Could not parse JSON from model output", + } + + # ── Check function name ─────────────────────────────────────────────── + expected_name = ( + ground_truth.get("name") + or ground_truth.get("function_name") + or "" + ) + predicted_name = parsed.get("name", "") + name_ok = predicted_name.strip() == expected_name.strip() + + if not name_ok: + return { + "passed": False, + "score": 0.2, # partial credit: valid JSON produced + "note": f"Wrong fn name: got '{predicted_name}', expected '{expected_name}'", + } + + # ── Check arguments ─────────────────────────────────────────────────── + expected_args = ground_truth.get("arguments", ground_truth.get("args", {})) + predicted_args = parsed.get("arguments", parsed.get("args", {})) + strict = self.cfg.get("strict", False) + + arg_score = self._score_args(expected_args, predicted_args, strict=strict) + passed = arg_score >= (1.0 if strict else 0.8) + + return { + "passed": passed, + "score": round((0.5 + 0.5 * arg_score), 3), # name correct = 0.5, args = 0.5 + "note": f"arg_match={arg_score:.2f}", + } + + # ── Helpers ─────────────────────────────────────────────────────────────── + + @staticmethod + def _extract_json(text: str) -> dict | None: + """Extract the first JSON object from free-form model output.""" + # Try direct parse first + try: + return json.loads(text.strip()) + except json.JSONDecodeError: + pass + # Regex fallback: find first {...} block + match = re.search(r"\{.*\}", text, re.DOTALL) + if match: + try: + return json.loads(match.group()) + except json.JSONDecodeError: + pass + return None + + @staticmethod + def _score_args( + expected: dict, + predicted: dict, + strict: bool = False, + ) -> float: + """Return 0–1 argument match score.""" + if not expected: + return 1.0 if not predicted else 0.8 + + if strict: + return 1.0 if expected == predicted else 0.0 + + # Fuzzy: what fraction of expected keys are correctly predicted? + hits = 0 + for key, exp_val in expected.items(): + pred_val = predicted.get(key) + if pred_val is None: + continue + if str(pred_val).strip().lower() == str(exp_val).strip().lower(): + hits += 1 + elif str(exp_val).strip().lower() in str(pred_val).strip().lower(): + hits += 0.5 # partial credit + + return hits / len(expected) diff --git a/research/evals/src/slm_evals/benchmarks/gaia.py b/research/evals/src/slm_evals/benchmarks/gaia.py new file mode 100644 index 0000000000000000000000000000000000000000..e69f666916bbaffe52da7e2702c537ad062755d9 --- /dev/null +++ b/research/evals/src/slm_evals/benchmarks/gaia.py @@ -0,0 +1,156 @@ +""" +benchmarks/gaia.py +─────────────────── +GAIA — General AI Assistants benchmark. + +What it tests: Multi-step reasoning with tool use (web search, +code execution, file reading) toward a single final answer. + +Dataset: gaia-benchmark/GAIA on HF Hub (requires auth for test split). + We use the validation split by default, which is public. + +Scoring: Exact-match on final answer string (case-insensitive, stripped). + GAIA also supports a normalised-string partial match — implemented here. + +Levels: + Level 1 – single-step, no tools needed + Level 2 – multi-step, 1-2 tools + Level 3 – complex, multi-tool chains (hardest) +""" + +from __future__ import annotations +import re +import unicodedata +from typing import Any + +from slm_evals.benchmarks.base import BaseBenchmark + + +SYSTEM_PROMPT = """\ +You are a highly capable AI assistant. +Answer the user's question as precisely as possible. +If you need to reason step-by-step, do so in ... tags. +Your final answer must appear on its own line starting with: ANSWER: +""" + +# Tools the agent could use in a real setup — logged in prompt as context. +AVAILABLE_TOOLS_DESCRIPTION = """ +Available tools (describe their use in your reasoning, then give ANSWER): +- web_search(query: str) → search results +- code_exec(code: str) → run Python code, returns stdout +- read_file(path: str) → read a local file +""" + + +class GAIABenchmark(BaseBenchmark): + """ + GAIA benchmark adapter. + + Config keys (benchmark_overrides.gaia): + data_path – local JSONL (optional) + split – "validation" | "test" (default: "validation") + levels – list[int], e.g. [1, 2] to run only levels 1 & 2 + tool_mode – "describe" | "none" (default: "describe") + In "describe" mode tools are mentioned in prompt but + not actually executed (offline eval). Set up real tools + in your own subclass for live eval. + """ + + name = "gaia" + + def load_dataset(self) -> list[dict]: + data_path = self.cfg.get("data_path") + if data_path: + return self._load_local(data_path) + return self._load_from_hub() + + def _load_local(self, path: str) -> list[dict]: + import json + samples = [] + with open(path) as f: + for line in f: + line = line.strip() + if line: + samples.append(json.loads(line)) + return self._filter_levels(samples) + + def _load_from_hub(self) -> list[dict]: + try: + from datasets import load_dataset + except ImportError: + raise ImportError("pip install datasets") + + split = self.cfg.get("split", "validation") + ds = load_dataset("gaia-benchmark/GAIA", "2023_all", split=split, trust_remote_code=True) + return self._filter_levels(list(ds)) + + def _filter_levels(self, samples: list[dict]) -> list[dict]: + levels = self.cfg.get("levels") + if not levels: + return samples + return [s for s in samples if s.get("Level") in levels or s.get("level") in levels] + + # ── Prompt ──────────────────────────────────────────────────────────────── + + def build_prompt(self, sample: dict) -> str: + question = sample.get("Question", sample.get("question", "")) + tool_block = ( + AVAILABLE_TOOLS_DESCRIPTION + if self.cfg.get("tool_mode", "describe") == "describe" + else "" + ) + return ( + f"{SYSTEM_PROMPT}\n" + f"{tool_block}\n" + f"Question: {question}\n" + f"Assistant:" + ) + + # ── Evaluation ──────────────────────────────────────────────────────────── + + def evaluate_sample(self, sample: dict, prediction: str) -> dict: + ground_truth = ( + sample.get("Final answer") + or sample.get("final_answer") + or sample.get("answer") + or "" + ) + if not isinstance(ground_truth, str): + ground_truth = str(ground_truth) + + extracted = self._extract_answer(prediction) + exact_ok = self._normalize(extracted) == self._normalize(ground_truth) + + if exact_ok: + return {"passed": True, "score": 1.0, "note": "exact match"} + + # Partial: is ground truth contained in extracted (or vice versa)? + n_gt = self._normalize(ground_truth) + n_pred = self._normalize(extracted) + partial = (n_gt in n_pred) or (n_pred and n_pred in n_gt) + score = 0.5 if partial else 0.0 + + return { + "passed": False, + "score": score, + "note": f"pred='{extracted[:60]}' gt='{ground_truth[:60]}'", + } + + # ── Helpers ─────────────────────────────────────────────────────────────── + + @staticmethod + def _extract_answer(text: str) -> str: + """Pull the text after 'ANSWER:' on its own line.""" + match = re.search(r"(?i)^ANSWER:\s*(.+)", text, re.MULTILINE) + if match: + return match.group(1).strip() + # Fallback: last non-empty line + lines = [l.strip() for l in text.splitlines() if l.strip()] + return lines[-1] if lines else text.strip() + + @staticmethod + def _normalize(s: str) -> str: + """Lower-case, strip punctuation and extra whitespace.""" + s = unicodedata.normalize("NFKC", s).lower().strip() + s = re.sub(r"[^\w\s]", "", s) + return re.sub(r"\s+", " ", s) diff --git a/research/evals/src/slm_evals/benchmarks/swe_bench.py b/research/evals/src/slm_evals/benchmarks/swe_bench.py new file mode 100644 index 0000000000000000000000000000000000000000..8d994a8b07e472ac910d09491312456b03816815 --- /dev/null +++ b/research/evals/src/slm_evals/benchmarks/swe_bench.py @@ -0,0 +1,188 @@ +""" +benchmarks/swe_bench.py +──────────────────────── +SWE-bench Verified — agentic coding benchmark. + +What it tests: Given a GitHub issue description + repository context, +can the model produce a patch that fixes the bug (passes test suite)? + +Dataset: princeton-nlp/SWE-bench_Verified on HF Hub (500 human-verified tasks). + +Scoring: + Offline mode (default): Checks patch structural validity + keyword heuristics. + Full mode (cfg["full_eval"]=True): Runs the patch in a Docker sandbox and + executes the test suite. Requires Docker + swebench[eval] installed. + +Note: Full end-to-end SWE-bench eval requires the official harness + (https://github.com/princeton-nlp/SWE-bench). This adapter wraps + the offline/structural scoring path for lightweight local use, + and delegates to the harness when full_eval is requested. +""" + +from __future__ import annotations +import re +from typing import Any + +from slm_evals.benchmarks.base import BaseBenchmark + + +SYSTEM_PROMPT = """\ +You are an expert software engineer. +You will be given a GitHub issue and the relevant source code. +Produce a unified diff patch that fixes the issue. +Output ONLY the patch, starting with --- and ending with the last +++ hunk. +Do not include any explanation. +""" + + +class SWEBenchmark(BaseBenchmark): + """ + SWE-bench Verified adapter. + + Config keys (benchmark_overrides.swe_bench): + data_path – local JSONL + full_eval – bool (default False); run actual test harness + context_lines – int, how many lines of file context to include (default 80) + difficulty – list[str] filter by difficulty label (optional) + """ + + name = "swe_bench" + + def load_dataset(self) -> list[dict]: + data_path = self.cfg.get("data_path") + if data_path: + return self._load_local(data_path) + return self._load_from_hub() + + def _load_local(self, path: str) -> list[dict]: + import json + samples = [] + with open(path) as f: + for line in f: + line = line.strip() + if line: + samples.append(json.loads(line)) + return samples + + def _load_from_hub(self) -> list[dict]: + try: + from datasets import load_dataset + except ImportError: + raise ImportError("pip install datasets") + + ds = load_dataset( + "princeton-nlp/SWE-bench_Verified", + split="test", + trust_remote_code=True, + ) + return list(ds) + + # ── Prompt ──────────────────────────────────────────────────────────────── + + def build_prompt(self, sample: dict) -> str: + issue_text = sample.get("problem_statement", sample.get("issue", "")) + repo = sample.get("repo", "unknown/repo") + hints = sample.get("hints_text", "") + context_snip = self._build_context_snippet(sample) + + return ( + f"{SYSTEM_PROMPT}\n" + f"Repository: {repo}\n\n" + f"Issue:\n{issue_text}\n\n" + f"{'Hints: ' + hints + chr(10) if hints else ''}" + f"Relevant code:\n{context_snip}\n\n" + f"Patch:" + ) + + def _build_context_snippet(self, sample: dict) -> str: + """Pull relevant file snippets from the sample if available.""" + n = self.cfg.get("context_lines", 80) + + # SWE-bench Verified includes patch/test files fields + base_commit = sample.get("base_commit", "") + patch = sample.get("patch", "") # ground truth patch (don't expose to model) + test_patch = sample.get("test_patch", "") + + # We expose only the files mentioned in the issue, not the patch itself + file_names = re.findall(r"[\w/]+\.py", sample.get("problem_statement", "")) + if file_names: + return f"[Files likely relevant: {', '.join(set(file_names[:5]))}]\n(Fetch via repo checkout at {base_commit})" + return "(No inline context available — use repo checkout for full context)" + + # ── Evaluation ──────────────────────────────────────────────────────────── + + def evaluate_sample(self, sample: dict, prediction: str) -> dict: + if self.cfg.get("full_eval", False): + return self._full_harness_eval(sample, prediction) + return self._structural_eval(sample, prediction) + + def _structural_eval(self, sample: dict, prediction: str) -> dict: + """ + Lightweight offline scoring: + - Is the output a valid unified diff? + - Does it touch any of the expected files? + - Does it contain meaningful change lines (+/-)? + """ + is_diff = self._looks_like_diff(prediction) + expected_f = self._expected_files(sample) + touches_f = self._patch_touches_files(prediction, expected_f) + has_changes = bool(re.search(r"^[+-][^+-]", prediction, re.MULTILINE)) + + score = sum([is_diff * 0.4, touches_f * 0.4, has_changes * 0.2]) + passed = score >= 0.6 + + return { + "passed": passed, + "score": round(score, 3), + "note": ( + f"valid_diff={is_diff} " + f"touches_expected_files={touches_f} " + f"has_changes={has_changes}" + ), + } + + def _full_harness_eval(self, sample: dict, prediction: str) -> dict: + """ + Delegate to the official SWE-bench evaluation harness. + Requires: pip install swebench AND Docker running. + + Returns pass/fail based on whether tests pass after applying the patch. + """ + try: + from swebench.harness.run_evaluation import run_instances + except ImportError: + raise ImportError( + "pip install swebench (and ensure Docker is running)" + ) + + instance_id = sample.get("instance_id", sample.get("id", "unknown")) + result = run_instances( + predictions={instance_id: {"model_patch": prediction}}, + instances=[sample], + run_id="slm_bench_eval", + ) + resolved = result.get(instance_id, {}).get("resolved", False) + return { + "passed": resolved, + "score": 1.0 if resolved else 0.0, + "note": "full harness eval", + } + + # ── Helpers ─────────────────────────────────────────────────────────────── + + @staticmethod + def _looks_like_diff(text: str) -> bool: + return bool(re.search(r"^(---|\+\+\+|@@)", text, re.MULTILINE)) + + @staticmethod + def _expected_files(sample: dict) -> list[str]: + patch = sample.get("patch", "") + return re.findall(r"(?:---|\+\+\+) [ab]/(.+\.py)", patch) + + @staticmethod + def _patch_touches_files(prediction: str, expected_files: list[str]) -> float: + if not expected_files: + return 0.5 # can't verify, give benefit of doubt + pred_files = re.findall(r"(?:---|\+\+\+) [ab]/(.+\.py)", prediction) + hits = set(pred_files) & set(expected_files) + return len(hits) / len(expected_files) diff --git a/research/evals/src/slm_evals/benchmarks/tau_bench.py b/research/evals/src/slm_evals/benchmarks/tau_bench.py new file mode 100644 index 0000000000000000000000000000000000000000..bbc7e505ef67f737c710ee95cf69210f0caa4a75 --- /dev/null +++ b/research/evals/src/slm_evals/benchmarks/tau_bench.py @@ -0,0 +1,232 @@ +""" +benchmarks/tau_bench.py +──────────────────────── +τ-bench: Tool-Agent-User interaction benchmark. + +What it tests: Multi-turn agentic loops where the model must call +retail/airline domain tools to satisfy a simulated user, and the +database state must exactly match ground truth at episode end. + +Dataset: ShishirPatil/tau-bench on HF Hub, or local JSONL. + +Scoring: Pass@1 task success (database state matches ground truth). + +Note: Full τ-bench uses a live GPT-4o user simulator. This adapter + uses a lightweight rule-based user simulator by default, which + is deterministic, free, and reproducible for offline/local evals. + Set cfg["use_llm_user"] = True to enable the LLM user (costs tokens). +""" + +from __future__ import annotations +import json +import re +import copy +from typing import Any + +from slm_evals.benchmarks.base import BaseBenchmark + + +AGENT_SYSTEM = """\ +You are a helpful customer service agent. +You have access to the following tools. To call a tool, output ONLY: +TOOL_CALL: {"name": "", "args": {: }} + +After receiving the tool result, continue the conversation. +When the task is complete, output: DONE +""" + + +class TauBenchmark(BaseBenchmark): + """ + τ-bench multi-turn tool-agent-user benchmark. + + Config keys (benchmark_overrides.tau_bench): + data_path – local JSONL + domain – "retail" | "airline" | "both" (default: "retail") + max_turns – max dialogue turns per episode (default: 15) + use_llm_user – use an LLM to simulate user (default: False) + """ + + name = "tau_bench" + + def load_dataset(self) -> list[dict]: + data_path = self.cfg.get("data_path") + if data_path: + return self._load_local(data_path) + return self._load_from_hub() + + def _load_local(self, path: str) -> list[dict]: + samples = [] + with open(path) as f: + for line in f: + line = line.strip() + if line: + samples.append(json.loads(line)) + return samples + + def _load_from_hub(self) -> list[dict]: + try: + from datasets import load_dataset + except ImportError: + raise ImportError("pip install datasets") + + domain = self.cfg.get("domain", "retail") + config = "retail" if domain in ("retail", "both") else "airline" + ds = load_dataset("ShishirPatil/tau-bench", config, split="test", trust_remote_code=True) + return list(ds) + + # ── Prompt is used only for the first turn ──────────────────────────────── + + def build_prompt(self, sample: dict) -> str: + tools_block = json.dumps(sample.get("tools", []), indent=2) + context = sample.get("context", "") + user_goal = sample.get("instruction", sample.get("user_goal", "")) + return ( + f"{AGENT_SYSTEM}\n" + f"Available tools:\n{tools_block}\n\n" + f"Context:\n{context}\n\n" + f"[User]: {user_goal}\n" + f"[Agent]:" + ) + + # ── Multi-turn simulation ───────────────────────────────────────────────── + + def run(self) -> dict[str, Any]: + """Override run() to support multi-turn episodes.""" + dataset = self.load_dataset() + if self.max_samples: + dataset = dataset[: self.max_samples] + + import time + samples_out = [] + errors = 0 + + for sample in dataset: + t0 = time.perf_counter() + try: + result = self._run_episode(sample) + except Exception as exc: + errors += 1 + result = {"passed": False, "score": 0.0, "note": f"ERROR: {exc}"} + + latency = round(time.perf_counter() - t0, 3) + samples_out.append({"id": sample.get("id", ""), "latency_s": latency, **result}) + + passed = sum(1 for s in samples_out if s["passed"]) + total = len(samples_out) + + return { + "benchmark": self.name, + "passed": passed, + "total": total, + "score": (passed / total) if total else 0.0, + "error_count": errors, + "avg_latency_s": round( + sum(s["latency_s"] for s in samples_out) / total, 3 + ) if total else 0.0, + "samples": samples_out, + } + + def _run_episode(self, sample: dict) -> dict: + """Simulate one full τ-bench episode.""" + max_turns = self.cfg.get("max_turns", 15) + tools = {t["name"]: t for t in sample.get("tools", [])} + db_state = copy.deepcopy(sample.get("initial_state", {})) + ground_truth_state = sample.get("ground_truth_state", {}) + conversation: list[str] = [] + + # First user message + user_msg = sample.get("instruction", sample.get("user_goal", "")) + conversation.append(f"[User]: {user_msg}") + + for _turn in range(max_turns): + # Build prompt from conversation history + tools_block = json.dumps(list(tools.values()), indent=2) + history = "\n".join(conversation) + prompt = ( + f"{AGENT_SYSTEM}\n" + f"Available tools:\n{tools_block}\n\n" + f"{history}\n[Agent]:" + ) + + agent_reply = self.generate( + prompt, + max_new_tokens=self.max_new_tokens, + temperature=self.temperature, + ) + conversation.append(f"[Agent]: {agent_reply}") + + # ── Check for termination ───────────────────────────────────────── + if "DONE" in agent_reply.upper(): + break + + # ── Check for tool call ─────────────────────────────────────────── + tool_result = self._try_execute_tool(agent_reply, tools, db_state) + if tool_result is not None: + conversation.append(f"[Tool]: {json.dumps(tool_result)}") + # Simulated user follow-up + conversation.append(self._simulated_user_reply(sample, db_state)) + + # ── Score: does final db state match ground truth? ──────────────────── + state_match = self._compare_states(db_state, ground_truth_state) + return { + "passed": state_match >= 1.0, + "score": round(state_match, 3), + "note": f"state_match={state_match:.2f} turns={len(conversation)}", + "prediction": "\n".join(conversation[-4:]), # last 4 lines for logging + } + + @staticmethod + def _try_execute_tool( + agent_reply: str, + tools: dict, + db_state: dict, + ) -> dict | None: + """ + Extract a TOOL_CALL from the agent reply and execute it + against the in-memory db_state (simple rule-based simulation). + """ + match = re.search(r"TOOL_CALL:\s*(\{.*?\})", agent_reply, re.DOTALL) + if not match: + return None + + try: + call = json.loads(match.group(1)) + except json.JSONDecodeError: + return {"error": "Malformed tool call JSON"} + + tool_name = call.get("name", "") + args = call.get("args", {}) + + # ── Simple state-mutation rules ─────────────────────────────────────── + # Extend this section with real tool logic or mock implementations. + if tool_name not in tools: + return {"error": f"Unknown tool: {tool_name}"} + + # Generic: apply key-value updates from args to db_state + for key, value in args.items(): + if key in db_state: + db_state[key] = value + + return {"status": "ok", "tool": tool_name, "args": args} + + @staticmethod + def _simulated_user_reply(sample: dict, db_state: dict) -> str: + """Minimal rule-based user simulator (replace with LLM if cfg allows).""" + # Real τ-bench uses GPT-4o here; we return a generic confirmation. + return "[User]: Thank you, please continue." + + @staticmethod + def _compare_states(actual: dict, expected: dict) -> float: + """0–1 fraction of expected keys that match in actual state.""" + if not expected: + return 1.0 + hits = sum( + 1 for k, v in expected.items() + if str(actual.get(k, "")).strip() == str(v).strip() + ) + return hits / len(expected) + + # ── Not used (multi-turn overrides run()) ───────────────────────────────── + def evaluate_sample(self, sample, prediction): + pass diff --git a/research/evals/src/slm_evals/lm_eval/__init__.py b/research/evals/src/slm_evals/lm_eval/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..21044de1a2ff0c4e3ac4260fb3fc0a788f16b061 --- /dev/null +++ b/research/evals/src/slm_evals/lm_eval/__init__.py @@ -0,0 +1,5 @@ +"""lm-evaluation-harness integration for slm-evals.""" + +from slm_evals.lm_eval.preset_resolver import LMEvalModelSpec, resolve_model_spec + +__all__ = ["LMEvalModelSpec", "resolve_model_spec"] diff --git a/research/evals/src/slm_evals/lm_eval/ensemble_lm.py b/research/evals/src/slm_evals/lm_eval/ensemble_lm.py new file mode 100644 index 0000000000000000000000000000000000000000..963195301d332ea5ed495f82a928d7cd26fbede8 --- /dev/null +++ b/research/evals/src/slm_evals/lm_eval/ensemble_lm.py @@ -0,0 +1,111 @@ +"""Custom lm-eval backend for saved JEPA ensemble checkpoints.""" + +from __future__ import annotations + +import logging +from typing import TYPE_CHECKING, Any + +from lm_eval.api.model import LM +from lm_eval.api.registry import register_model + +if TYPE_CHECKING: + from lm_eval.api.instance import Instance + +eval_logger = logging.getLogger(__name__) + + +@register_model("ensemble-lm") +class EnsembleLM(LM): + """Evaluate ensemble checkpoints with full-stack generation and LLM-head scoring.""" + + def __init__( + self, + checkpoint_path: str, + device: str | None = "auto", + dtype: str = "bfloat16", + batch_size: int | str = 1, + max_batch_size: int | None = 64, + trust_remote_code: bool = True, + **kwargs: Any, + ) -> None: + super().__init__() + from ensemble.checkpoint import load_checkpoint + + load_in_4bit = dtype == "int4" + resolved_device = None if device in (None, "auto") else device + self._ens = load_checkpoint( + checkpoint_path, + device=resolved_device, + load_in_4bit=load_in_4bit, + ) + self._checkpoint_path = checkpoint_path + self._dtype = dtype + self._device = device or "auto" + self._batch_size = batch_size + self._max_batch_size = max_batch_size + self._trust_remote_code = trust_remote_code + + backend_name = type(self._ens.llm).__name__ + if backend_name == "TinyBackend": + self._hf_lm = None + eval_logger.warning( + "ensemble-lm: tiny backend checkpoint — loglikelihood tasks " + "are not supported; use generate_until tasks only." + ) + else: + from lm_eval.models.huggingface import HFLM + + self._hf_lm = HFLM( + pretrained=self._ens.llm.model, + tokenizer=self._ens.llm.tokenizer, + device=self._device, + dtype=dtype if dtype != "int4" else "auto", + batch_size=batch_size, + max_batch_size=max_batch_size, + trust_remote_code=trust_remote_code, + ) + self._device = self._hf_lm.device + + @property + def tokenizer_name(self) -> str: + if self._hf_lm is not None: + return self._hf_lm.tokenizer_name + return f"ensemble:{self._checkpoint_path}" + + def loglikelihood(self, requests: list[Instance]) -> list[tuple[float, bool]]: + if self._hf_lm is None: + raise NotImplementedError( + "loglikelihood is not supported for tiny ensemble checkpoints" + ) + return self._hf_lm.loglikelihood(requests) + + def loglikelihood_rolling(self, requests: list[Instance]) -> list[float]: + if self._hf_lm is None: + raise NotImplementedError( + "loglikelihood_rolling is not supported for tiny ensemble checkpoints" + ) + return self._hf_lm.loglikelihood_rolling(requests) + + def generate_until( + self, requests: list[Instance], disable_tqdm: bool = False + ) -> list[str]: + del disable_tqdm + outputs: list[str] = [] + for req in requests: + context, gen_kwargs = req.args + until = gen_kwargs.get("until", []) + max_gen_toks = gen_kwargs.get("max_gen_toks", 256) + do_sample = gen_kwargs.get("do_sample", False) + temperature = float(gen_kwargs.get("temperature", 0.0 if not do_sample else 1.0)) + + text = self._ens.generate_text( + context, + max_new_tokens=int(max_gen_toks), + temperature=temperature, + ) + if until: + for stop in until: + if stop and stop in text: + text = text.split(stop, 1)[0] + outputs.append(text.strip()) + return outputs diff --git a/research/evals/src/slm_evals/lm_eval/preset_resolver.py b/research/evals/src/slm_evals/lm_eval/preset_resolver.py new file mode 100644 index 0000000000000000000000000000000000000000..f07d5161a6130dd35f5b526fa09058cf8891e5c2 --- /dev/null +++ b/research/evals/src/slm_evals/lm_eval/preset_resolver.py @@ -0,0 +1,294 @@ +"""Resolve models.yaml presets and paths into lm-eval model specifications.""" + +from __future__ import annotations + +import os +import sys +from dataclasses import dataclass +from pathlib import Path +from typing import Any + + +_REPO_ROOT = Path(__file__).resolve().parents[5] +_MANIFEST_FILE = "manifest.json" +_HF_HUB_PREFIXES = ( + "openbmb/", + "google/", + "meta-llama/", + "Qwen/", + "HuggingFaceTB/", +) + + +def _ensure_inference_on_path() -> None: + libs = _REPO_ROOT / "libs" / "inference" / "src" + if str(libs) not in sys.path: + sys.path.insert(0, str(libs)) + + +def _is_ensemble_checkpoint(path: str | Path) -> bool: + try: + from ensemble.checkpoint import is_ensemble_checkpoint + + return is_ensemble_checkpoint(path) + except ImportError: + return (Path(path) / _MANIFEST_FILE).is_file() + + +def _looks_like_hf_hub_id(model_path: str) -> bool: + if model_path.startswith(("./", "../")) or os.path.isabs(model_path): + return False + if "\\" in model_path: + return False + if Path(model_path).exists(): + return False + if model_path.startswith(_HF_HUB_PREFIXES): + return True + parts = model_path.split("/") + return len(parts) <= 2 and all(parts) + + +def _resolve_model_path(model_path: str) -> Path: + path = Path(model_path) + if _looks_like_hf_hub_id(model_path): + return path + if not path.is_absolute(): + path = (_REPO_ROOT / path).resolve() + return path + + +def _list_ensemble_checkpoint_names() -> list[str]: + root = _REPO_ROOT / "models" / "ensemble" + if not root.is_dir(): + return [] + names: list[str] = [] + for child in sorted(root.iterdir()): + if child.is_dir() and (child / _MANIFEST_FILE).is_file(): + names.append(child.name) + return names + + +def _missing_local_model_message(path: Path) -> str: + msg = f"Local model path not found: {path}" + ensembles = _list_ensemble_checkpoint_names() + if ensembles: + msg += ( + "\nEnsemble checkpoints under models/ensemble/: " + + ", ".join(ensembles) + ) + return msg + + +def _invalid_local_model_message(path: Path) -> str: + return ( + f"Local path is not a recognized checkpoint: {path}\n" + f"Expected ensemble manifest at {path / _MANIFEST_FILE} " + "or HuggingFace config.json / adapter_config.json." + ) + + +def _resolve_explicit_model_path( + model_path: str, + *, + adapter_path: str | None, + trust_remote_code: bool | None, + dtype: str | None, + device: str | None, +) -> LMEvalModelSpec: + path = _resolve_model_path(model_path) + resolved = str(path) + + if not _looks_like_hf_hub_id(model_path): + if not path.exists(): + raise FileNotFoundError(_missing_local_model_message(path)) + if path.is_dir() and _is_ensemble_checkpoint(path): + args: dict[str, Any] = {"checkpoint_path": resolved} + if dtype: + args["dtype"] = dtype + if device: + args["device"] = device + return LMEvalModelSpec( + lm_eval_model="ensemble-lm", + model_args=args, + preset_key=None, + base_model=resolved, + adapter_path=None, + checkpoint_path=resolved, + trust_remote_code=trust_remote_code or False, + ) + if path.is_dir() and not ( + (path / "config.json").is_file() + or (path / "adapter_config.json").is_file() + ): + raise ValueError(_invalid_local_model_message(path)) + + base = resolved if not _looks_like_hf_hub_id(model_path) else model_path + peft = adapter_path + if peft and not Path(peft).is_absolute(): + peft = str((_REPO_ROOT / peft).resolve()) + if not Path(peft).exists(): + raise FileNotFoundError(f"LoRA adapter path not found: {peft}") + + args = { + "pretrained": base, + "trust_remote_code": trust_remote_code + if trust_remote_code is not None + else True, + } + if peft: + args["peft"] = peft + if dtype: + args["dtype"] = dtype + + return LMEvalModelSpec( + lm_eval_model="hf", + model_args=args, + preset_key=None, + base_model=base, + adapter_path=peft, + checkpoint_path=None, + trust_remote_code=bool(args["trust_remote_code"]), + ) + + +def _is_lm_evalable_preset(model) -> bool: + if model.backend != "transformers": + return False + if model.multimodal: + return False + if not model.model_id: + return False + return True + + +@dataclass(frozen=True) +class LMEvalModelSpec: + """Resolved model target for lm-evaluation-harness.""" + + lm_eval_model: str + model_args: dict[str, Any] + preset_key: str | None + base_model: str + adapter_path: str | None + checkpoint_path: str | None + trust_remote_code: bool + + def model_args_string(self) -> str: + parts = [] + for key, value in self.model_args.items(): + if value is None: + continue + if isinstance(value, bool): + parts.append(f"{key}={str(value).lower()}") + else: + parts.append(f"{key}={value}") + return ",".join(parts) + + +def resolve_model_spec( + *, + preset: str | None = None, + model_path: str | None = None, + adapter_path: str | None = None, + trust_remote_code: bool | None = None, + dtype: str | None = None, + device: str | None = None, +) -> LMEvalModelSpec: + """Resolve preset or explicit paths into an lm-eval model specification.""" + if preset and model_path: + raise ValueError("Pass only one of --preset or --model, not both.") + + if preset: + return _resolve_from_preset( + preset, + adapter_override=adapter_path, + trust_remote_code=trust_remote_code, + dtype=dtype, + device=device, + ) + + if not model_path: + raise ValueError("One of --preset or --model is required.") + + return _resolve_explicit_model_path( + model_path, + adapter_path=adapter_path, + trust_remote_code=trust_remote_code, + dtype=dtype, + device=device, + ) + + +def _resolve_from_preset( + preset_key: str, + *, + adapter_override: str | None, + trust_remote_code: bool | None, + dtype: str | None, + device: str | None, +) -> LMEvalModelSpec: + _ensure_inference_on_path() + from inference.config import get_app_config, get_model_config + + app_config = get_app_config(reload=True) + if preset_key not in app_config.models: + raise ValueError( + f"Unknown preset {preset_key!r}. " + f"Available: {', '.join(sorted(app_config.models))}" + ) + + model = get_model_config(preset_key).resolve_paths(_REPO_ROOT) + if not _is_lm_evalable_preset(model): + raise ValueError( + f"Preset {preset_key!r} uses backend={model.backend!r} " + f"(multimodal={model.multimodal}); only text transformers presets " + "are supported for lm-eval." + ) + + model_id = model.model_id + if not model_id: + raise ValueError(f"Preset {preset_key!r} has no model_id.") + + if not _looks_like_hf_hub_id(model_id): + path = _resolve_model_path(model_id) + if not path.exists(): + raise FileNotFoundError(_missing_local_model_message(path)) + model_id = str(path) + + if _is_ensemble_checkpoint(model_id): + args: dict[str, Any] = {"checkpoint_path": model_id} + if dtype: + args["dtype"] = dtype + if device: + args["device"] = device + return LMEvalModelSpec( + lm_eval_model="ensemble-lm", + model_args=args, + preset_key=preset_key, + base_model=model_id, + adapter_path=None, + checkpoint_path=model_id, + trust_remote_code=model.trust_remote_code, + ) + + adapter = adapter_override or model.adapter_path + trust = ( + trust_remote_code + if trust_remote_code is not None + else model.trust_remote_code + ) + args = {"pretrained": model_id, "trust_remote_code": trust} + if adapter: + args["peft"] = adapter + if dtype: + args["dtype"] = dtype + + return LMEvalModelSpec( + lm_eval_model="hf", + model_args=args, + preset_key=preset_key, + base_model=model_id, + adapter_path=adapter, + checkpoint_path=None, + trust_remote_code=trust, + ) diff --git a/research/evals/src/slm_evals/lm_eval/profiles.py b/research/evals/src/slm_evals/lm_eval/profiles.py new file mode 100644 index 0000000000000000000000000000000000000000..d1cd55747a10b2f3d3dbf73140feda5bfd52cf4e --- /dev/null +++ b/research/evals/src/slm_evals/lm_eval/profiles.py @@ -0,0 +1,180 @@ +"""Load and display eval profile catalog from configs/eval_profiles.yaml.""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +import yaml + +_REPO_ROOT = Path(__file__).resolve().parents[5] +_CONFIGS_DIR = Path(__file__).resolve().parents[3] / "configs" +_PROFILES_FILE = _CONFIGS_DIR / "eval_profiles.yaml" + + +def load_profiles_catalog() -> dict[str, Any]: + if not _PROFILES_FILE.is_file(): + raise FileNotFoundError(f"Profile catalog not found: {_PROFILES_FILE}") + with open(_PROFILES_FILE) as f: + return yaml.safe_load(f) or {} + + +def config_path_for_profile(name: str) -> Path: + catalog = load_profiles_catalog() + profiles = catalog.get("profiles", {}) + if name not in profiles: + known = ", ".join(sorted(profiles)) + raise ValueError(f"Unknown lm-eval profile {name!r}. Known: {known}") + rel = profiles[name].get("config") + if not rel: + raise ValueError(f"Profile {name!r} has no config file in eval_profiles.yaml") + path = _CONFIGS_DIR / rel + if not path.is_file(): + raise FileNotFoundError(f"Profile config missing: {path}") + return path + + +def _format_tasks(tasks: list[str] | None) -> str: + if not tasks: + return "—" + return ", ".join(tasks) + + +def format_profiles_table(*, include_suites: bool = False, include_external: bool = False) -> str: + catalog = load_profiles_catalog() + lines = [ + "Eval profiles (slm-lm-eval — use --profile NAME or --config PATH)", + "", + f"{'Profile':<16} {'Claim':<28} {'Tasks'}", + f"{'-'*16} {'-'*28} {'-'*40}", + ] + for name, meta in sorted(catalog.get("profiles", {}).items()): + claim = meta.get("claim", "") + tasks = _format_tasks(meta.get("tasks")) + config = meta.get("config", "") + lines.append(f"{name:<16} {claim:<28} {tasks}") + if meta.get("description"): + lines.append(f"{'':16} config: {config}") + lines.append(f"{'':16} {meta['description']}") + lines.append("") + + if include_suites: + lines.extend( + [ + "", + "Other suites (separate CLI — see docs/eval_profiles.md)", + "", + f"{'Suite':<16} {'Tool':<14} {'Claim':<28}", + f"{'-'*16} {'-'*14} {'-'*28}", + ] + ) + for name, meta in sorted(catalog.get("suites", {}).items()): + tool = meta.get("tool", "") + claim = meta.get("claim", "") + lines.append(f"{name:<16} {tool:<14} {claim:<28}") + benchmarks = meta.get("benchmarks") + if benchmarks: + lines.append(f"{'':16} benchmarks: {', '.join(benchmarks)}") + if meta.get("command"): + lines.append(f"{'':16} {meta['command']}") + lines.append("") + + if include_external: + lines.extend( + [ + "", + "External (not integrated in this repo)", + "", + ] + ) + for name, meta in sorted(catalog.get("external", {}).items()): + lines.append(f"- {name}: {meta.get('claim', '')} — {meta.get('description', '')}") + + lines.extend( + [ + "", + "Examples:", + " slm-lm-eval --list-profiles", + " slm-lm-eval --profile reasoning --preset minicpm5-1b --experiment-name baseline", + " slm-benchmark --list-benchmarks", + ] + ) + return "\n".join(lines) + + +def list_lm_eval_task_names(limit: int = 80) -> list[str]: + try: + from lm_eval.tasks import TaskManager + + tm = TaskManager() + names = sorted(tm.all_tasks) + return names[:limit] if limit else names + except ImportError: + return [] + + +def format_lm_eval_tasks(*, limit: int = 80) -> str: + tasks = list_lm_eval_task_names(limit=limit) + if not tasks: + catalog = load_profiles_catalog() + seen: set[str] = set() + for section in ("profiles",): + for meta in catalog.get(section, {}).values(): + for task in meta.get("tasks") or []: + seen.add(task) + tasks = sorted(seen) + header = ( + "lm-eval not installed — showing tasks referenced in eval_profiles.yaml only.\n" + "Install: uv sync --group lm-eval\n" + ) + else: + header = f"lm-eval tasks (first {len(tasks)}; run with --list-tasks --all for full list):\n" + + lines = [header.rstrip(), ""] + col = 4 + for i in range(0, len(tasks), col): + chunk = tasks[i : i + col] + lines.append(" ".join(f"{t:<22}" for t in chunk)) + lines.append("") + lines.append("Use in a run: slm-lm-eval --profile reasoning --preset minicpm5-1b") + lines.append("Or override: slm-lm-eval --config ... --tasks gsm8k arc_easy") + return "\n".join(lines) + + +def format_agentic_benchmarks() -> str: + from slm_evals.run_benchmark import BENCHMARK_REGISTRY + + catalog = load_profiles_catalog() + lines = [ + "Agentic benchmarks (slm-benchmark — use --benchmarks NAME)", + "", + f"{'Key':<12} {'Measures'}", + f"{'-'*12} {'-'*50}", + "bfcl Single-turn function calling (Berkeley FC Leaderboard)", + "tau_bench Multi-turn tool + user simulation", + "gaia End-to-end assistant tasks (reasoning + tools)", + "swe_bench Code patch generation (SWE-bench Verified)", + "all Run all four benchmarks", + "", + "Registered in code:", + ] + for key in sorted(BENCHMARK_REGISTRY): + lines.append(f" - {key}") + + lines.append("") + lines.append("Preset suites from eval_profiles.yaml:") + for name, meta in sorted(catalog.get("suites", {}).items()): + if meta.get("tool") != "slm-benchmark": + continue + bms = meta.get("benchmarks") or [] + lines.append(f" {name}: {', '.join(bms)}") + lines.append(f" {meta.get('description', '')}") + + lines.extend( + [ + "", + "Example:", + " slm-benchmark --model openbmb/MiniCPM5-1B --benchmarks bfcl --max-samples 20", + ] + ) + return "\n".join(lines) diff --git a/research/evals/src/slm_evals/run_benchmark.py b/research/evals/src/slm_evals/run_benchmark.py new file mode 100644 index 0000000000000000000000000000000000000000..9ffe7c3939ee7a8ade788646efb10b5bc054391b --- /dev/null +++ b/research/evals/src/slm_evals/run_benchmark.py @@ -0,0 +1,188 @@ +""" +SLM Agentic Benchmark Suite +============================ +Run BFCL, τ-bench, GAIA, SWE-bench against a local HuggingFace model checkpoint. + +Usage: + uv run --package slm-evals slm-benchmark --model ./path/to/model --benchmarks bfcl + uv run --package slm-evals python -m slm_evals.run_benchmark --model ./path/to/model + uv run --package slm-evals slm-benchmark --config configs/experiment_001.yaml +""" + +from __future__ import annotations + +import argparse +import sys + +from slm_evals.benchmarks.bfcl import BFCLBenchmark +from slm_evals.benchmarks.gaia import GAIABenchmark +from slm_evals.benchmarks.swe_bench import SWEBenchmark +from slm_evals.benchmarks.tau_bench import TauBenchmark +from slm_evals.utils.config_loader import build_config_from_args, load_config +from slm_evals.utils.model_loader import load_model +from slm_evals.utils.reporter import Reporter + +BENCHMARK_REGISTRY = { + "bfcl": BFCLBenchmark, + "tau_bench": TauBenchmark, + "gaia": GAIABenchmark, + "swe_bench": SWEBenchmark, +} + + +def parse_args(): + parser = argparse.ArgumentParser( + description="SLM Agentic Benchmark Suite — HuggingFace backend" + ) + parser.add_argument( + "--model", + type=str, + help="Path to local HuggingFace model directory (or HF Hub ID)", + ) + parser.add_argument( + "--model-type", + type=str, + default="auto", + choices=["auto", "hf", "ensemble"], + help="auto detects ensemble checkpoints via manifest.json", + ) + parser.add_argument( + "--benchmarks", + nargs="+", + choices=list(BENCHMARK_REGISTRY.keys()) + ["all"], + default=["all"], + help="Which benchmarks to run (default: all)", + ) + parser.add_argument( + "--config", + type=str, + default=None, + help="Optional YAML config file (overrides other flags)", + ) + parser.add_argument( + "--max-samples", + type=int, + default=None, + help="Cap number of samples per benchmark (useful for quick smoke tests)", + ) + parser.add_argument( + "--output-dir", + type=str, + default="results", + help="Directory to write results (default: ./results)", + ) + parser.add_argument( + "--experiment-name", + type=str, + default=None, + help="Name tag for this run (auto-generated from timestamp if omitted)", + ) + parser.add_argument( + "--device", + type=str, + default="auto", + help="Device map for HF: 'auto', 'cpu', 'cuda', 'cuda:0' etc.", + ) + parser.add_argument( + "--dtype", + type=str, + default="bfloat16", + choices=["float32", "float16", "bfloat16", "int8", "int4"], + help="Model dtype / quantization level", + ) + parser.add_argument( + "--max-new-tokens", + type=int, + default=512, + help="Max tokens to generate per inference call", + ) + parser.add_argument( + "--temperature", + type=float, + default=0.0, + help="Sampling temperature (0.0 = greedy)", + ) + parser.add_argument( + "--list-benchmarks", + action="store_true", + help="Show agentic benchmark keys and preset suites from eval_profiles.yaml", + ) + return parser.parse_args() + + +def main(): + args = parse_args() + + if args.list_benchmarks: + from slm_evals.lm_eval.profiles import format_agentic_benchmarks + + print(format_agentic_benchmarks()) + return + + if args.config: + cfg = load_config(args.config) + else: + if not args.model: + print("error: --model is required unless --config is provided", file=sys.stderr) + sys.exit(2) + cfg = build_config_from_args(args) + + print(f"\n{'='*60}") + print(" SLM Benchmark Suite") + print(f" Model : {cfg['model_path']} ({cfg.get('model_type', 'auto')})") + print(f" Runs : {', '.join(cfg['benchmarks'])}") + print(f" Out : {cfg['output_dir']}") + print(f"{'='*60}\n") + + print("⏳ Loading model …") + model_bundle = load_model( + model_path=cfg["model_path"], + device=cfg["device"], + dtype=cfg["dtype"], + model_type=cfg.get("model_type", "auto"), + ) + print(f"✅ Model loaded — {model_bundle['param_count']:.2f}B parameters\n") + + reporter = Reporter( + output_dir=cfg["output_dir"], + experiment_name=cfg["experiment_name"], + model_path=cfg["model_path"], + ) + + benchmark_names = ( + list(BENCHMARK_REGISTRY.keys()) + if "all" in cfg["benchmarks"] + else cfg["benchmarks"] + ) + + all_results = {} + for name in benchmark_names: + print(f"▶ Running benchmark: {name.upper()}") + print(f" {'─'*50}") + + bench_cls = BENCHMARK_REGISTRY[name] + bench = bench_cls( + model_bundle=model_bundle, + max_samples=cfg.get("max_samples"), + max_new_tokens=cfg.get("max_new_tokens", 512), + temperature=cfg.get("temperature", 0.0), + benchmark_cfg=cfg.get("benchmark_overrides", {}).get(name, {}), + ) + + result = bench.run() + all_results[name] = result + + print(f" Score : {result['score']:.2%}") + print(f" Passed: {result['passed']} / {result['total']}") + print() + + paths = reporter.save(all_results) + print(f"\n{'='*60}") + print(" Results saved:") + for fmt, path in paths.items(): + print(f" {fmt:<8} → {path}") + print(f"{'='*60}\n") + + +if __name__ == "__main__": + main() diff --git a/research/evals/src/slm_evals/run_lm_eval.py b/research/evals/src/slm_evals/run_lm_eval.py new file mode 100644 index 0000000000000000000000000000000000000000..3faccd0a054a15319b91f43d917cd076d202d717 --- /dev/null +++ b/research/evals/src/slm_evals/run_lm_eval.py @@ -0,0 +1,430 @@ +""" +slm-lm-eval — Academic benchmarks via lm-evaluation-harness +============================================================ +Run GSM8K, ARC, HellaSwag, and related tasks against presets, finetuned +checkpoints, or ensemble manifests. + +Usage: + uv run --package slm-evals slm-lm-eval \\ + --config research/evals/configs/lm_eval_minicpm5.yaml \\ + --preset minicpm5-1b \\ + --experiment-name minicpm5-1b__baseline +""" + +from __future__ import annotations + +import argparse +import datetime +import json +import subprocess +import sys +from pathlib import Path +from typing import Any + +import yaml + +# Register custom ensemble backend before simple_evaluate(). +import slm_evals.lm_eval.ensemble_lm # noqa: F401 +from slm_evals.lm_eval.preset_resolver import resolve_model_spec +from slm_evals.lm_eval.profiles import ( + config_path_for_profile, + format_lm_eval_tasks, + format_profiles_table, +) + + +def _ensure_lm_eval_models_registered() -> None: + """Import lm-eval model backends so registry includes hf and ensemble-lm.""" + import lm_eval.models # noqa: F401 — registers bundled backends when available + + try: + import lm_eval.models.huggingface # noqa: F401 + except ImportError: + pass + +_REPO_ROOT = Path(__file__).resolve().parents[4] +_DEFAULT_OUTPUT = _REPO_ROOT / "results" / "lm_eval" + +_METRIC_PRIORITY = ( + "acc,none", + "acc_norm,none", + "exact_match,strict-match", + "exact_match,flexible-extract", + "f1,none", + "bleu,none", +) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Run lm-evaluation-harness benchmarks via slm-evals", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=( + "Profiles: slm-lm-eval --list-profiles\n" + " slm-lm-eval --profile reasoning --preset minicpm5-1b\n" + "All tasks: slm-lm-eval --list-tasks (requires uv sync --group lm-eval)" + ), + ) + parser.add_argument( + "--list-profiles", + action="store_true", + help="Show claim-matched lm-eval profiles and other eval suites", + ) + parser.add_argument( + "--list-profiles-all", + action="store_true", + help="Like --list-profiles but include agentic suites and external notes", + ) + parser.add_argument( + "--list-tasks", + action="store_true", + help="List lm-eval task names (from harness, or catalog fallback)", + ) + parser.add_argument( + "--list-tasks-all", + action="store_true", + help="List all lm-eval task names (can be long)", + ) + parser.add_argument( + "--profile", + type=str, + default=None, + metavar="NAME", + help="Shorthand for --config (e.g. reasoning, understanding, code, smoke)", + ) + parser.add_argument("--config", type=str, default=None, help="YAML config path") + parser.add_argument("--preset", type=str, default=None, help="models.yaml preset key") + parser.add_argument( + "--model", + type=str, + default=None, + help="HF Hub id, merged checkpoint dir, or ensemble manifest dir", + ) + parser.add_argument("--adapter", type=str, default=None, help="LoRA adapter path") + parser.add_argument( + "--tasks", + nargs="+", + default=None, + help="Task names (overrides config)", + ) + parser.add_argument("--num-fewshot", type=int, default=None) + parser.add_argument("--limit", type=int, default=None, help="Max samples per task") + parser.add_argument("--seed", type=int, default=None) + parser.add_argument("--batch-size", default=None) + parser.add_argument("--device", type=str, default=None) + parser.add_argument("--dtype", type=str, default=None) + parser.add_argument( + "--output-dir", + type=str, + default=str(_DEFAULT_OUTPUT), + help="Root directory for lm-eval results", + ) + parser.add_argument("--experiment-name", type=str, default=None) + parser.add_argument( + "--compare-to", + type=str, + default=None, + help="Path to baseline results.json for delta table", + ) + return parser.parse_args() + + +def load_lm_eval_config(path: str) -> dict[str, Any]: + with open(path) as f: + cfg = yaml.safe_load(f) or {} + cfg.setdefault("tasks", ["arc_easy", "hellaswag"]) + cfg.setdefault("num_fewshot", 0) + cfg.setdefault("limit", None) + cfg.setdefault("seed", 42) + cfg.setdefault("batch_size", "auto") + cfg.setdefault("device", "auto") + cfg.setdefault("dtype", "bfloat16") + cfg.setdefault("trust_remote_code", True) + cfg.setdefault("output_dir", str(_DEFAULT_OUTPUT)) + return cfg + + +def merge_config(args: argparse.Namespace) -> dict[str, Any]: + cfg: dict[str, Any] = {} + config_path = args.config + if args.profile: + if config_path: + raise SystemExit("Pass only one of --profile or --config, not both.") + config_path = str(config_path_for_profile(args.profile)) + if config_path: + cfg = load_lm_eval_config(config_path) + + if args.tasks: + cfg["tasks"] = args.tasks + if args.num_fewshot is not None: + cfg["num_fewshot"] = args.num_fewshot + if args.limit is not None: + cfg["limit"] = args.limit + if args.seed is not None: + cfg["seed"] = args.seed + if args.batch_size is not None: + cfg["batch_size"] = args.batch_size + if args.device is not None: + cfg["device"] = args.device + if args.dtype is not None: + cfg["dtype"] = args.dtype + if args.output_dir: + cfg["output_dir"] = args.output_dir + + cfg["preset"] = args.preset + cfg["model_path"] = args.model + cfg["adapter_path"] = args.adapter + cfg["compare_to"] = args.compare_to or cfg.get("compare_to") + + if not cfg.get("experiment_name"): + if args.experiment_name: + cfg["experiment_name"] = args.experiment_name + else: + tag = args.preset or Path(args.model or "model").name + ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S") + cfg["experiment_name"] = f"{tag}__lm-eval__{ts}" + elif args.experiment_name: + cfg["experiment_name"] = args.experiment_name + + return cfg + + +def _git_hash() -> str | None: + try: + out = subprocess.check_output( + ["git", "rev-parse", "HEAD"], + cwd=_REPO_ROOT, + stderr=subprocess.DEVNULL, + text=True, + ) + return out.strip() + except (subprocess.CalledProcessError, FileNotFoundError): + return None + + +def _primary_metric(task_metrics: dict[str, Any]) -> tuple[str, float] | None: + for key in _METRIC_PRIORITY: + if key in task_metrics and isinstance(task_metrics[key], (int, float)): + return key, float(task_metrics[key]) + for key, value in task_metrics.items(): + if isinstance(value, (int, float)): + return key, float(value) + return None + + +def write_summary_md( + path: Path, + *, + spec, + cfg: dict[str, Any], + results_payload: dict[str, Any], +) -> None: + lines = [ + "# lm-eval summary", + "", + f"- experiment: `{cfg['experiment_name']}`", + f"- model backend: `{spec.lm_eval_model}`", + f"- base model: `{spec.base_model}`", + ] + if spec.adapter_path: + lines.append(f"- adapter: `{spec.adapter_path}`") + if spec.checkpoint_path: + lines.append(f"- ensemble checkpoint: `{spec.checkpoint_path}`") + lines.extend( + [ + f"- tasks: {', '.join(cfg['tasks'])}", + f"- num_fewshot: {cfg.get('num_fewshot')}", + f"- limit: {cfg.get('limit')}", + f"- seed: {cfg.get('seed')}", + "", + "| task | metric | score |", + "| --- | --- | ---: |", + ] + ) + + task_results = results_payload.get("results", {}) + for task, metrics in sorted(task_results.items()): + picked = _primary_metric(metrics) + if picked: + metric_name, score = picked + lines.append(f"| {task} | {metric_name} | {score:.4f} |") + else: + lines.append(f"| {task} | — | — |") + + path.write_text("\n".join(lines) + "\n") + + +def compare_results( + baseline_path: Path, + candidate_path: Path, + *, + cfg: dict[str, Any], +) -> str: + baseline = json.loads(baseline_path.read_text()) + candidate = json.loads(candidate_path.read_text()) + + warnings: list[str] = [] + for key in ("seed", "limit", "num_fewshot"): + b_cfg = baseline.get("run_meta", {}).get(key, baseline.get("config", {}).get(key)) + c_cfg = candidate.get("run_meta", {}).get(key, candidate.get("config", {}).get(key)) + if b_cfg != c_cfg and b_cfg is not None and c_cfg is not None: + warnings.append(f"Mismatch on {key}: baseline={b_cfg!r} candidate={c_cfg!r}") + + b_tasks = set(baseline.get("results", {})) + c_tasks = set(candidate.get("results", {})) + shared = sorted(b_tasks & c_tasks) + if not shared: + warnings.append("No shared tasks between baseline and candidate.") + + lines = [ + "# lm-eval comparison", + "", + f"- baseline: `{baseline_path}`", + f"- candidate: `{candidate_path}`", + f"- candidate experiment: `{cfg['experiment_name']}`", + "", + ] + if warnings: + lines.append("## Warnings") + lines.extend(f"- {w}" for w in warnings) + lines.append("") + + lines.extend(["| task | baseline | candidate | delta |", "| --- | ---: | ---: | ---: |"]) + for task in shared: + b_metric = _primary_metric(baseline["results"][task]) + c_metric = _primary_metric(candidate["results"][task]) + if not b_metric or not c_metric: + continue + _, b_score = b_metric + _, c_score = c_metric + delta = c_score - b_score + sign = "+" if delta >= 0 else "" + lines.append( + f"| {task} | {b_score:.4f} | {c_score:.4f} | {sign}{delta:.4f} |" + ) + + return "\n".join(lines) + "\n" + + +def main() -> int: + args = parse_args() + + if args.list_profiles or args.list_profiles_all: + print( + format_profiles_table( + include_suites=args.list_profiles_all, + include_external=args.list_profiles_all, + ) + ) + return 0 + + if args.list_tasks or args.list_tasks_all: + print(format_lm_eval_tasks(limit=0 if args.list_tasks_all else 80)) + return 0 + + cfg = merge_config(args) + + if not cfg.get("preset") and not cfg.get("model_path"): + print("Error: pass --preset or --model (or set in config).", file=sys.stderr) + return 1 + + spec = resolve_model_spec( + preset=cfg.get("preset"), + model_path=cfg.get("model_path"), + adapter_path=cfg.get("adapter_path"), + trust_remote_code=cfg.get("trust_remote_code"), + dtype=cfg.get("dtype"), + device=cfg.get("device"), + ) + + out_dir = Path(cfg["output_dir"]) / cfg["experiment_name"] + out_dir.mkdir(parents=True, exist_ok=True) + + try: + import lm_eval + except ImportError as exc: + print( + "lm-eval is not installed. Run: uv sync --group lm-eval", + file=sys.stderr, + ) + raise SystemExit(1) from exc + + _ensure_lm_eval_models_registered() + + seed = int(cfg.get("seed", 42)) + model_args = dict(spec.model_args) + eval_device = cfg.get("device") + if spec.lm_eval_model == "hf": + model_args.pop("device", None) + else: + eval_device = None + + eval_results = lm_eval.simple_evaluate( + model=spec.lm_eval_model, + model_args=model_args, + tasks=cfg["tasks"], + num_fewshot=cfg.get("num_fewshot"), + batch_size=cfg.get("batch_size"), + device=eval_device, + limit=cfg.get("limit"), + random_seed=seed, + numpy_random_seed=seed, + torch_random_seed=seed, + fewshot_random_seed=seed, + log_samples=False, + ) + + if eval_results is None: + print("lm-eval returned no results.", file=sys.stderr) + return 1 + + run_meta = { + "experiment_name": cfg["experiment_name"], + "preset": spec.preset_key, + "lm_eval_model": spec.lm_eval_model, + "base_model": spec.base_model, + "adapter_path": spec.adapter_path, + "checkpoint_path": spec.checkpoint_path, + "tasks": cfg["tasks"], + "num_fewshot": cfg.get("num_fewshot"), + "limit": cfg.get("limit"), + "seed": seed, + "batch_size": cfg.get("batch_size"), + "device": cfg.get("device"), + "dtype": cfg.get("dtype"), + "git_hash": _git_hash(), + } + + payload = dict(eval_results) + payload["run_meta"] = run_meta + + results_path = out_dir / "results.json" + results_path.write_text(json.dumps(payload, indent=2, default=str)) + + summary_path = out_dir / "summary.md" + write_summary_md(summary_path, spec=spec, cfg=cfg, results_payload=payload) + + meta_path = out_dir / "run_meta.json" + meta_path.write_text(json.dumps(run_meta, indent=2)) + + print(f"Wrote {results_path}") + print(f"Wrote {summary_path}") + + compare_to = cfg.get("compare_to") + if compare_to: + compare_path = out_dir / "comparison.md" + compare_text = compare_results( + Path(compare_to), + results_path, + cfg=cfg, + ) + compare_path.write_text(compare_text) + print(f"Wrote {compare_path}") + print() + print(compare_text) + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/research/evals/src/slm_evals/utils/__init__.py b/research/evals/src/slm_evals/utils/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..7c4063edfa0f9cdd076e1d50aa82e402b87be597 --- /dev/null +++ b/research/evals/src/slm_evals/utils/__init__.py @@ -0,0 +1 @@ +"""Benchmark utilities.""" diff --git a/research/evals/src/slm_evals/utils/config_loader.py b/research/evals/src/slm_evals/utils/config_loader.py new file mode 100644 index 0000000000000000000000000000000000000000..ae3fec8b55474a795bf64d5d8f849d0150417683 --- /dev/null +++ b/research/evals/src/slm_evals/utils/config_loader.py @@ -0,0 +1,61 @@ +""" +utils/config_loader.py +─────────────────────── +Load experiment config from a YAML file OR build one from CLI args. +""" + +from __future__ import annotations +import datetime +from pathlib import Path +from typing import Any + + +def load_config(path: str) -> dict[str, Any]: + """Parse a YAML config file into a flat config dict.""" + try: + import yaml + except ImportError: + raise ImportError("PyYAML required: pip install pyyaml") + + with open(path) as f: + cfg = yaml.safe_load(f) + + _fill_defaults(cfg) + return cfg + + +def build_config_from_args(args) -> dict[str, Any]: + """Convert argparse Namespace into a config dict.""" + benchmarks = args.benchmarks if args.benchmarks else ["all"] + cfg: dict[str, Any] = { + "model_path": args.model, + "model_type": getattr(args, "model_type", "auto"), + "benchmarks": benchmarks, + "max_samples": args.max_samples, + "output_dir": args.output_dir, + "experiment_name": args.experiment_name, + "device": args.device, + "dtype": args.dtype, + "max_new_tokens": args.max_new_tokens, + "temperature": args.temperature, + "benchmark_overrides": {}, + } + _fill_defaults(cfg) + return cfg + + +def _fill_defaults(cfg: dict[str, Any]) -> None: + """In-place: fill any missing keys with sensible defaults.""" + if not cfg.get("experiment_name"): + ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S") + model_tag = Path(cfg.get("model_path", "unknown")).name + cfg["experiment_name"] = f"{model_tag}__{ts}" + + cfg.setdefault("model_type", "auto") + cfg.setdefault("device", "auto") + cfg.setdefault("dtype", "bfloat16") + cfg.setdefault("max_new_tokens", 512) + cfg.setdefault("temperature", 0.0) + cfg.setdefault("max_samples", None) + cfg.setdefault("output_dir", "results") + cfg.setdefault("benchmark_overrides", {}) diff --git a/research/evals/src/slm_evals/utils/model_loader.py b/research/evals/src/slm_evals/utils/model_loader.py new file mode 100644 index 0000000000000000000000000000000000000000..67a1e7d2110c61f2dba2d3becaa0121af0cf8da6 --- /dev/null +++ b/research/evals/src/slm_evals/utils/model_loader.py @@ -0,0 +1,206 @@ +""" +utils/model_loader.py +───────────────────── +Load a local (or HF Hub) model via HuggingFace Transformers. +Returns a model_bundle dict that every benchmark consumes. +""" + +from __future__ import annotations +import torch +from pathlib import Path +from typing import Any + + +DTYPE_MAP = { + "float32": torch.float32, + "float16": torch.float16, + "bfloat16": torch.bfloat16, +} + + +def _resolve_model_type(model_path: str, model_type: str | None) -> str: + if model_type and model_type != "auto": + return model_type + try: + from ensemble.checkpoint import is_ensemble_checkpoint + + if is_ensemble_checkpoint(model_path): + return "ensemble" + except ImportError: + pass + return "hf" + + +def load_ensemble_model( + model_path: str, + device: str = "auto", + dtype: str = "bfloat16", +) -> dict[str, Any]: + """Load a saved JEPA ensemble checkpoint (LLM + emb + JEPA + bridge).""" + from ensemble.checkpoint import load_checkpoint + + resolved_device = ( + "cpu" + if device == "cpu" + else None + if device == "auto" + else device + ) + load_in_4bit = dtype == "int4" + ens = load_checkpoint( + model_path, + device=resolved_device, + load_in_4bit=load_in_4bit, + ) + + param_count = sum(p.numel() for p in ens.parameters()) / 1e9 + llm_device = str(ens.llm.device) + tokenizer = getattr(ens.llm, "tokenizer", None) + + def generate_fn( + prompt: str, + max_new_tokens: int = 512, + temperature: float = 0.0, + stop_strings: list[str] | None = None, + ) -> str: + text = ens.generate_text( + prompt, + max_new_tokens=max_new_tokens, + temperature=temperature, + ) + if stop_strings: + for stop in stop_strings: + if stop in text: + text = text.split(stop, 1)[0] + return text.strip() + + return { + "model": ens, + "tokenizer": tokenizer, + "device": llm_device, + "dtype": dtype, + "param_count": param_count, + "model_path": str(model_path), + "model_type": "ensemble", + "generate_fn": generate_fn, + } + + +def load_model( + model_path: str, + device: str = "auto", + dtype: str = "bfloat16", + model_type: str | None = "auto", +) -> dict[str, Any]: + """ + Load model + tokenizer from a local path or HF Hub ID. + + Set model_type to ``ensemble`` (or use a path with manifest.json) for + saved JEPA ensemble checkpoints under models/ensemble/. + + Returns + ------- + model_bundle : dict with keys + model – the loaded AutoModelForCausalLM + tokenizer – the matching AutoTokenizer + device – resolved torch device string + dtype – resolved torch dtype + param_count – float (billions) + model_path – original path string + generate_fn – convenience callable (prompt → str) + """ + resolved = _resolve_model_type(model_path, model_type) + if resolved == "ensemble": + return load_ensemble_model(model_path, device=device, dtype=dtype) + # ── lazy imports so the module is importable without torch installed ────── + try: + from transformers import ( + AutoModelForCausalLM, + AutoTokenizer, + BitsAndBytesConfig, + ) + except ImportError as e: + raise ImportError( + "transformers is required: pip install transformers accelerate" + ) from e + + model_path = str(model_path) + + # ── Quantization config ─────────────────────────────────────────────────── + quant_cfg = None + torch_dtype = DTYPE_MAP.get(dtype, torch.bfloat16) + + if dtype == "int4": + quant_cfg = BitsAndBytesConfig( + load_in_4bit=True, + bnb_4bit_quant_type="nf4", + bnb_4bit_compute_dtype=torch.bfloat16, + ) + torch_dtype = None + elif dtype == "int8": + quant_cfg = BitsAndBytesConfig(load_in_8bit=True) + torch_dtype = None + + # ── Load tokenizer ──────────────────────────────────────────────────────── + tokenizer = AutoTokenizer.from_pretrained( + model_path, + trust_remote_code=True, + padding_side="left", + ) + if tokenizer.pad_token is None: + tokenizer.pad_token = tokenizer.eos_token + + # ── Load model ──────────────────────────────────────────────────────────── + model = AutoModelForCausalLM.from_pretrained( + model_path, + device_map=device, + torch_dtype=torch_dtype, + quantization_config=quant_cfg, + trust_remote_code=True, + ) + model.eval() + + # ── Parameter count ─────────────────────────────────────────────────────── + param_count = sum(p.numel() for p in model.parameters()) / 1e9 + + # ── Resolve actual device ───────────────────────────────────────────────── + resolved_device = next(model.parameters()).device + + # ── Convenience generate function ───────────────────────────────────────── + def generate_fn( + prompt: str, + max_new_tokens: int = 512, + temperature: float = 0.0, + stop_strings: list[str] | None = None, + ) -> str: + """Run inference and return the decoded completion (without prompt).""" + inputs = tokenizer(prompt, return_tensors="pt").to(resolved_device) + + gen_kwargs: dict[str, Any] = dict( + **inputs, + max_new_tokens=max_new_tokens, + pad_token_id=tokenizer.pad_token_id, + eos_token_id=tokenizer.eos_token_id, + ) + if temperature > 0: + gen_kwargs.update(do_sample=True, temperature=temperature, top_p=0.95) + else: + gen_kwargs["do_sample"] = False + + with torch.no_grad(): + output_ids = model.generate(**gen_kwargs) + + # Strip the input tokens from output + new_ids = output_ids[0][inputs["input_ids"].shape[1]:] + return tokenizer.decode(new_ids, skip_special_tokens=True).strip() + + return { + "model": model, + "tokenizer": tokenizer, + "device": str(resolved_device), + "dtype": dtype, + "param_count": param_count, + "model_path": model_path, + "model_type": "hf", + "generate_fn": generate_fn, + } diff --git a/research/evals/src/slm_evals/utils/reporter.py b/research/evals/src/slm_evals/utils/reporter.py new file mode 100644 index 0000000000000000000000000000000000000000..9f7136ffd0adc8de8d1d84e95069cdf0db029967 --- /dev/null +++ b/research/evals/src/slm_evals/utils/reporter.py @@ -0,0 +1,136 @@ +""" +utils/reporter.py +────────────────── +Persist benchmark results to JSON, Markdown, and CSV. +""" + +from __future__ import annotations +import csv +import json +import datetime +from pathlib import Path +from typing import Any + + +class Reporter: + def __init__( + self, + output_dir: str, + experiment_name: str, + model_path: str, + ): + self.output_dir = Path(output_dir) + self.experiment_name = experiment_name + self.model_path = model_path + self.timestamp = datetime.datetime.now().isoformat(timespec="seconds") + + self.run_dir = self.output_dir / experiment_name + self.run_dir.mkdir(parents=True, exist_ok=True) + + # ── Public ──────────────────────────────────────────────────────────────── + + def save(self, all_results: dict[str, Any]) -> dict[str, str]: + """Write all three output formats. Returns dict of format → filepath.""" + payload = self._build_payload(all_results) + paths = { + "json": self._write_json(payload), + "csv": self._write_csv(payload), + "markdown": self._write_markdown(payload), + } + return paths + + # ── Private ─────────────────────────────────────────────────────────────── + + def _build_payload(self, all_results: dict[str, Any]) -> dict[str, Any]: + aggregate_score = ( + sum(r["score"] for r in all_results.values()) / len(all_results) + if all_results else 0.0 + ) + return { + "experiment_name": self.experiment_name, + "timestamp": self.timestamp, + "model_path": self.model_path, + "aggregate_score": round(aggregate_score, 4), + "benchmarks": all_results, + } + + def _write_json(self, payload: dict) -> str: + path = self.run_dir / "results.json" + with open(path, "w") as f: + json.dump(payload, f, indent=2, default=str) + return str(path) + + def _write_csv(self, payload: dict) -> str: + path = self.run_dir / "results.csv" + rows = [] + for bench_name, result in payload["benchmarks"].items(): + base_row = { + "experiment": payload["experiment_name"], + "timestamp": payload["timestamp"], + "model_path": payload["model_path"], + "benchmark": bench_name, + "score": round(result["score"], 4), + "passed": result["passed"], + "total": result["total"], + "error_count": result.get("error_count", 0), + "avg_latency_s": result.get("avg_latency_s", ""), + } + rows.append(base_row) + + if rows: + with open(path, "w", newline="") as f: + writer = csv.DictWriter(f, fieldnames=rows[0].keys()) + writer.writeheader() + writer.writerows(rows) + return str(path) + + def _write_markdown(self, payload: dict) -> str: + path = self.run_dir / "report.md" + lines = [ + f"# Benchmark Report — {payload['experiment_name']}", + "", + f"| Field | Value |", + f"|---|---|", + f"| **Timestamp** | {payload['timestamp']} |", + f"| **Model** | `{payload['model_path']}` |", + f"| **Aggregate Score** | **{payload['aggregate_score']:.2%}** |", + "", + "## Results by Benchmark", + "", + "| Benchmark | Score | Passed | Total | Errors | Avg Latency |", + "|---|---|---|---|---|---|", + ] + + for bench_name, result in payload["benchmarks"].items(): + lines.append( + f"| {bench_name} " + f"| {result['score']:.2%} " + f"| {result['passed']} " + f"| {result['total']} " + f"| {result.get('error_count', 0)} " + f"| {result.get('avg_latency_s', 'n/a')}s |" + ) + + lines += ["", "## Per-Sample Details", ""] + + for bench_name, result in payload["benchmarks"].items(): + lines += [ + f"### {bench_name.upper()}", + "", + "| # | Sample ID | Status | Score | Note |", + "|---|---|---|---|---|", + ] + for i, sample in enumerate(result.get("samples", []), 1): + status = "✅" if sample.get("passed") else "❌" + lines.append( + f"| {i} " + f"| {sample.get('id', '—')} " + f"| {status} " + f"| {sample.get('score', ''):.2f} " + f"| {sample.get('note', '')} |" + ) + lines.append("") + + with open(path, "w") as f: + f.write("\n".join(lines)) + return str(path) diff --git a/research/finetune.py b/research/finetune.py new file mode 100644 index 0000000000000000000000000000000000000000..2f798d87deefe00c00b52a1201777b361bb30c09 --- /dev/null +++ b/research/finetune.py @@ -0,0 +1,963 @@ +""" +finetune.py — Fine-tune a small LLM: FULL, LoRA, or QLoRA, one script. +====================================================================== + +Install: + pip install torch transformers datasets peft accelerate + pip install bitsandbytes # only needed for --mode qlora + +Model resolution (first match wins) +------------------------------------ +1. --model +2. --preset from models.yaml (or FINETUNE_PRESET env) +3. MODEL_ID / BASE env (raw Hugging Face id or local path) +4. ACTIVE_MODEL preset from models.yaml (text transformers presets only) + +Outputs are saved under ./models/finetuned/-/ by default. + +Examples +-------- +# LoRA on the lesson-agent chat dataset using models.yaml preset +python research/finetune.py --preset minicpm5-1b --mode lora --epochs 1 + +# Same, but read ACTIVE_MODEL / BASE from .env (auto-loaded from repo root) +python research/finetune.py --mode lora --max_steps 50 + +# LoRA on an instruction dataset from the Hub +python research/finetune.py \ + --model Qwen/Qwen2.5-0.5B-Instruct \ + --dataset tatsu-lab/alpaca --format alpaca \ + --mode lora --epochs 1 + +# QLoRA (4-bit) on a local JSONL chat file: {"messages": [{"role":..,"content":..}, ...]} +python research/finetune.py \ + --model meta-llama/Llama-3.2-1B-Instruct \ + --dataset ./data/chats.jsonl --format chat \ + --mode qlora + +# Hugging Face Hub datasets (--dataset is the repo id; optional --dataset-config / --split) +python research/finetune.py \ + --preset minicpm5-1b --mode qlora \ + --dataset tatsu-lab/alpaca --format alpaca --dataset-split train + +python research/finetune.py \ + --preset minicpm5-1b --mode lora \ + --dataset HuggingFaceTB/smoltalk --format chat \ + --dataset-config all --dataset-split train[:500] + +# Env vars also work: FINETUNE_DATASET, FINETUNE_DATASET_CONFIG, FINETUNE_DATASET_SPLIT + +# FULL fine-tune on raw text files (continued pretraining style) +python research/finetune.py \ + --model HuggingFaceTB/SmolLM2-360M \ + --dataset ./data/corpus.txt --format text \ + --mode full --lr 2e-5 + +# After LoRA training, merge adapter into standalone weights: +python research/finetune.py --merge ./models/finetuned/minicpm5-1b-lora \ + --out ./models/finetuned/minicpm5-1b-merged + +Dataset formats (--format) +-------------------------- + alpaca : columns instruction / input(optional) / output + chat : column messages = [{"role": "...", "content": "..."}] + prompt : columns prompt / completion (or prompt / response) + text : column text — or a plain .txt file (one doc per line / whole file) + +Local files: .json, .jsonl, .csv, .txt. Hub ids: any datasets repo. + +Hub datasets useful for the lesson / teacher agent (--format must match columns): + tatsu-lab/alpaca alpaca instruction tuning (general) + HuggingFaceTB/smoltalk chat multi-turn chat (use config: all) + Open-Orca/OpenOrca prompt instruction + response pairs + databricks/databricks-dolly-15k alpaca short Q&A, good for small models + +After training, metrics are written to /training_results.json +(train/eval loss, perplexity, result_score 0–100). +""" + +import argparse +import gc +import json +import math +import os +import subprocess +import sys +from pathlib import Path + +os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True") + +import torch +from datasets import load_dataset +from transformers import ( + AutoModelForCausalLM, + AutoTokenizer, + DataCollatorForLanguageModeling, + Trainer, + TrainingArguments, +) + +IGNORE_INDEX = -100 + +_REPO_ROOT = Path(__file__).resolve().parents[1] +_DEFAULT_DATASET = _REPO_ROOT / "research/data/education-lesson-chat.jsonl" +_FINETUNE_ROOT = _REPO_ROOT / "models/finetuned" +_FALLBACK_FINETUNE_PRESET = "minicpm5-1b" + + +def _load_dotenv(path: Path) -> None: + """Load KEY=VALUE pairs from .env without overriding existing env vars.""" + if not path.is_file(): + return + for line in path.read_text().splitlines(): + line = line.strip() + if not line or line.startswith("#") or "=" not in line: + continue + key, _, value = line.partition("=") + key = key.strip() + value = value.strip().strip('"').strip("'") + if key: + os.environ.setdefault(key, value) + + +def _ensure_repo_on_path() -> None: + libs = _REPO_ROOT / "libs" / "inference" / "src" + if str(libs) not in sys.path: + sys.path.insert(0, str(libs)) + + +def _is_finetuneable_preset(model) -> bool: + return model.backend == "transformers" and not model.multimodal and bool( + model.model_id + ) + + +def resolve_model_and_preset( + *, + model_arg: str | None, + preset_arg: str | None, +) -> tuple[str, str | None, bool]: + """Return (model_id_or_path, preset_key, trust_remote_code).""" + if model_arg: + trust = os.environ.get("TRUST_REMOTE_CODE", "").lower() in { + "1", + "true", + "yes", + } + return model_arg, preset_arg, trust + + for env_name in ("FINETUNE_MODEL", "MODEL_ID", "BASE"): + raw = os.environ.get(env_name) + if raw: + trust = os.environ.get("TRUST_REMOTE_CODE", "").lower() in { + "1", + "true", + "yes", + } + return raw, preset_arg, trust + + _ensure_repo_on_path() + from inference.config import get_app_config, get_model_config + + app_config = get_app_config(reload=True) + preset_key = ( + preset_arg + or os.environ.get("FINETUNE_PRESET") + or os.environ.get("ACTIVE_MODEL") + ) + + if preset_key and preset_key in app_config.models: + model = get_model_config(preset_key) + if not _is_finetuneable_preset(model): + print( + f"Preset {preset_key!r} is {model.backend}" + + (" multimodal" if model.multimodal else "") + + "; falling back to a text transformers preset for fine-tuning." + ) + preset_key = None + + if preset_key is None: + for candidate in (_FALLBACK_FINETUNE_PRESET, *app_config.models): + if candidate not in app_config.models: + continue + model = get_model_config(candidate) + if _is_finetuneable_preset(model): + preset_key = candidate + break + + if not preset_key: + raise SystemExit( + "No fine-tunable transformers preset found. Pass --model or set BASE/MODEL_ID." + ) + + model = get_model_config(preset_key) + if not _is_finetuneable_preset(model): + raise SystemExit( + f"Preset {preset_key!r} cannot be fine-tuned " + f"(backend={model.backend}, multimodal={model.multimodal})." + ) + return model.model_id, preset_key, model.trust_remote_code + + +def default_output_dir(preset_key: str | None, mode: str) -> str: + label = preset_key or "custom" + return str((_FINETUNE_ROOT / f"{label}-{mode}").resolve()) + + +# ---------------------------------------------------------------------------- +# Args +# ---------------------------------------------------------------------------- +def parse_args(): + p = argparse.ArgumentParser() + p.add_argument( + "--model", + type=str, + default=None, + help="HF id or local path (overrides models.yaml / env)", + ) + p.add_argument( + "--preset", + type=str, + default=None, + help="Preset key from models.yaml (default: FINETUNE_PRESET or ACTIVE_MODEL)", + ) + p.add_argument( + "--dataset", + type=str, + default=None, + help="HF Hub repo id (e.g. tatsu-lab/alpaca) or local file path", + ) + p.add_argument( + "--dataset-config", + type=str, + default=os.environ.get("FINETUNE_DATASET_CONFIG"), + help="HF dataset config/subset name (optional)", + ) + p.add_argument( + "--dataset-split", + type=str, + default=os.environ.get("FINETUNE_DATASET_SPLIT", "train"), + help="HF split name or slice, e.g. train or train[:1000]", + ) + p.add_argument( + "--dataset-max-samples", + type=int, + default=int(os.environ["FINETUNE_MAX_SAMPLES"]) + if os.environ.get("FINETUNE_MAX_SAMPLES") + else None, + help="Cap examples after loading (useful for Hub smoke tests)", + ) + p.add_argument( + "--format", + type=str, + default=os.environ.get("FINETUNE_FORMAT", "chat"), + choices=["alpaca", "chat", "prompt", "text"], + ) + p.add_argument("--mode", type=str, default="lora", + choices=["full", "lora", "qlora"]) + p.add_argument( + "--out", + type=str, + default=None, + help="Output directory (default: ./models/finetuned/-)", + ) + # training hparams + p.add_argument("--epochs", type=float, default=1.0) + p.add_argument("--max_steps", type=int, default=-1) + p.add_argument("--batch_size", type=int, default=4) + p.add_argument("--grad_accum", type=int, default=4) + p.add_argument("--lr", type=float, default=None, + help="default: 2e-4 for (q)lora, 2e-5 for full") + p.add_argument("--max_len", type=int, default=1024) + p.add_argument("--warmup_ratio", type=float, default=0.03) + p.add_argument("--mask_prompt", action="store_true", default=True, + help="compute loss only on the response tokens") + p.add_argument("--no_mask_prompt", dest="mask_prompt", action="store_false") + # lora hparams + p.add_argument("--lora_r", type=int, default=16) + p.add_argument("--lora_alpha", type=int, default=32) + p.add_argument("--lora_dropout", type=float, default=0.05) + p.add_argument("--lora_targets", type=str, + default="q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj", + help="comma list; 'all-linear' also works") + # misc + p.add_argument("--val_split", type=float, default=0.02) + p.add_argument("--seed", type=int, default=42) + p.add_argument("--bf16", action="store_true", default=None) + p.add_argument("--gradient_checkpointing", action="store_true", default=True) + p.add_argument( + "--device", + type=str, + default=os.environ.get("FINETUNE_DEVICE", "auto"), + choices=["auto", "cpu", "cuda"], + help="Training device (default: auto; set FINETUNE_DEVICE=cpu to avoid GPU OOM)", + ) + p.add_argument("--resume", type=str, default=None) + # merge mode + p.add_argument("--merge", type=str, default=None, + help="path to a LoRA adapter dir to merge into its base model") + p.add_argument( + "--lm-eval-after", + action="store_true", + help="run slm-lm-eval on the saved checkpoint after training", + ) + p.add_argument( + "--lm-eval-config", + type=str, + default=str(_REPO_ROOT / "research/evals/configs/lm_eval_smoke.yaml"), + help="YAML config for post-training lm-eval (default: lm_eval_smoke.yaml)", + ) + p.add_argument( + "--lm-eval-baseline", + type=str, + default=None, + help="optional baseline preset key; runs lm-eval on base model and compares", + ) + return p.parse_args() + + +# ---------------------------------------------------------------------------- +# Dataset loading + normalization to (prompt, response) or raw text +# ---------------------------------------------------------------------------- +def load_raw_dataset( + path: str, + *, + config: str | None = None, + split: str = "train", + max_samples: int | None = None, +): + """Load from a local file or Hugging Face Hub (datasets.load_dataset).""" + if os.path.exists(path): + ext = os.path.splitext(path)[1].lower() + if ext in (".json", ".jsonl"): + ds = load_dataset("json", data_files=path, split="train") + elif ext == ".csv": + ds = load_dataset("csv", data_files=path, split="train") + elif ext == ".txt": + ds = load_dataset("text", data_files=path, split="train") + else: + raise ValueError(f"Unsupported local file type: {ext}") + else: + kwargs: dict = {"path": path, "split": split} + if config: + kwargs["name"] = config + print(f"Loading Hub dataset: {path}" + (f" (config={config})" if config else "") + + f" split={split}") + ds = load_dataset(**kwargs) + + if max_samples is not None and max_samples > 0: + ds = ds.select(range(min(max_samples, len(ds)))) + return ds + + +def _last_metric(history: list[dict], key: str) -> float | None: + for entry in reversed(history): + if key in entry: + return float(entry[key]) + return None + + +def _result_score(eval_loss: float | None, train_loss: float | None) -> float | None: + """Higher is better (0–100). Derived from eval loss, else train loss.""" + loss = eval_loss if eval_loss is not None else train_loss + if loss is None: + return None + # exp(-loss) maps typical LM losses (~0.5–3) into a readable 0–100 band. + return round(min(100.0, max(0.0, 100.0 * math.exp(-loss))), 2) + + +def save_training_results( + out_dir: str, + *, + args, + preset_key: str | None, + train_count: int, + eval_count: int, + train_result, + log_history: list[dict], + eval_metrics: dict | None, +) -> Path: + history = train_result.metrics if hasattr(train_result, "metrics") else {} + + final_train_loss = _last_metric(log_history, "loss") + if final_train_loss is None and "train_loss" in history: + final_train_loss = float(history["train_loss"]) + + eval_loss = None + perplexity = None + if eval_metrics: + eval_loss = float(eval_metrics.get("eval_loss", 0)) + if eval_loss < 20: + perplexity = round(math.exp(eval_loss), 4) + + result_score = _result_score(eval_loss, final_train_loss) + + payload = { + "model": args.model, + "preset": preset_key, + "dataset": args.dataset, + "dataset_config": args.dataset_config, + "dataset_split": args.dataset_split, + "format": args.format, + "mode": args.mode, + "output_dir": out_dir, + "samples": {"train": train_count, "eval": eval_count}, + "metrics": { + "final_train_loss": round(final_train_loss, 6) + if final_train_loss is not None + else None, + "eval_loss": round(eval_loss, 6) if eval_loss is not None else None, + "perplexity": perplexity, + "loss_score": round(eval_loss, 6) + if eval_loss is not None + else ( + round(final_train_loss, 6) if final_train_loss is not None else None + ), + "result_score": result_score, + }, + "training": { + "epochs": args.epochs, + "max_steps": args.max_steps, + "global_step": getattr(train_result, "global_step", None), + "train_runtime_sec": round(history.get("train_runtime", 0), 2) + if history + else None, + "train_samples_per_second": history.get("train_samples_per_second"), + }, + } + + path = Path(out_dir) / "training_results.json" + path.write_text(json.dumps(payload, indent=2) + "\n") + return path + + +def to_prompt_response(example, fmt, tokenizer): + """Normalize any supported format into a single training string, + returning (full_text, prompt_text). prompt_text is None for raw text.""" + if fmt == "text": + return example["text"], None + + if fmt == "alpaca": + instr = example.get("instruction", "") + inp = example.get("input", "") or "" + out = example.get("output", "") + user = instr if not inp else f"{instr}\n\n{inp}" + messages = [{"role": "user", "content": user}, + {"role": "assistant", "content": out}] + + elif fmt == "prompt": + prompt = example.get("prompt", "") + resp = example.get("completion", example.get("response", "")) + messages = [{"role": "user", "content": prompt}, + {"role": "assistant", "content": resp}] + + elif fmt == "chat": + messages = example["messages"] + + else: + raise ValueError(fmt) + + # Use the model's chat template when it has one; else simple fallback. + if tokenizer.chat_template: + full = tokenizer.apply_chat_template( + messages, tokenize=False, add_generation_prompt=False) + prompt_only = tokenizer.apply_chat_template( + messages[:-1], tokenize=False, add_generation_prompt=True) + else: + prompt_only = "".join( + f"### {m['role'].capitalize()}:\n{m['content']}\n\n" + for m in messages[:-1]) + "### Assistant:\n" + full = prompt_only + messages[-1]["content"] + (tokenizer.eos_token or "") + return full, prompt_only + + +def build_tokenize_fn(tokenizer, fmt, max_len, mask_prompt): + def fn(example): + full, prompt = to_prompt_response(example, fmt, tokenizer) + ids = tokenizer(full, truncation=True, max_length=max_len, + add_special_tokens=(fmt == "text"))["input_ids"] + labels = list(ids) + if mask_prompt and prompt is not None: + p_len = len(tokenizer(prompt, add_special_tokens=False)["input_ids"]) + p_len = min(p_len, len(labels)) + labels[:p_len] = [IGNORE_INDEX] * p_len # no loss on prompt + return {"input_ids": ids, "labels": labels} + return fn + + +class CausalCollator: + """Pads input_ids with pad_token and labels with IGNORE_INDEX.""" + def __init__(self, tokenizer): + self.tok = tokenizer + + def __call__(self, batch): + max_len = max(len(b["input_ids"]) for b in batch) + input_ids, labels, attn = [], [], [] + pad = self.tok.pad_token_id + for b in batch: + n = max_len - len(b["input_ids"]) + input_ids.append(b["input_ids"] + [pad] * n) + labels.append(b["labels"] + [IGNORE_INDEX] * n) + attn.append([1] * len(b["input_ids"]) + [0] * n) + return { + "input_ids": torch.tensor(input_ids), + "labels": torch.tensor(labels), + "attention_mask": torch.tensor(attn), + } + + +# ---------------------------------------------------------------------------- +# Model loading for each mode +# ---------------------------------------------------------------------------- +def _training_uses_cuda(args) -> bool: + if args.device == "cpu": + return False + if args.device == "cuda": + return True + return torch.cuda.is_available() + + +def _gpu_memory_summary() -> str: + if not torch.cuda.is_available(): + return "CUDA not available" + free, total = torch.cuda.mem_get_info() + alloc = torch.cuda.memory_allocated() + reserved = torch.cuda.memory_reserved() + return ( + f"{free // 2**20} MiB free / {total // 2**20} MiB total " + f"(allocated {alloc // 2**20} MiB, reserved {reserved // 2**20} MiB)" + ) + + +def _gpu_total_gib() -> float | None: + if not torch.cuda.is_available(): + return None + _, total = torch.cuda.mem_get_info() + return total / (1024**3) + + +def _apply_low_vram_defaults(args) -> None: + """Cap batch/seq length and prefer QLoRA on GPUs that cannot fit full LoRA.""" + if not _training_uses_cuda(args): + return + total_gib = _gpu_total_gib() + if total_gib is None or total_gib >= 6.0: + return + + orig_batch, orig_max_len, orig_mode = args.batch_size, args.max_len, args.mode + args.batch_size = min(args.batch_size, 1) + args.max_len = min(args.max_len, 512) + args.gradient_checkpointing = True + + if total_gib < 4.5 and args.mode == "lora": + try: + import bitsandbytes # noqa: F401 + args.mode = "qlora" + except ImportError: + print( + f"Warning: {total_gib:.1f} GiB GPU — full LoRA may OOM. " + "Install finetune extras and use --mode qlora:\n" + " uv sync --group finetune" + ) + + if ( + args.batch_size != orig_batch + or args.max_len != orig_max_len + or args.mode != orig_mode + ): + print( + f"Low VRAM ({total_gib:.1f} GiB): adjusted training defaults — " + f"batch_size {orig_batch}->{args.batch_size}, " + f"max_len {orig_max_len}->{args.max_len}" + + (f", mode {orig_mode}->{args.mode}" if args.mode != orig_mode else "") + ) + + +def _validate_cuda_device(args) -> None: + if not _training_uses_cuda(args): + return + if torch.cuda.is_available(): + return + raise SystemExit( + "CUDA training was requested (--device cuda or auto with a visible GPU) " + "but PyTorch cannot use the GPU.\n" + f" torch.cuda.is_available() = False\n" + f" torch.cuda.device_count() = {torch.cuda.device_count()}\n" + "Run `nvidia-smi` and check for driver errors (ERR! fields). " + "If the GPU is busy or broken, free it or reboot, then retry.\n" + "Fallback: pass --device cpu (slower, higher RAM use)." + ) + + +def clear_gpu_memory(*, reset_peak: bool = True) -> None: + """Release cached CUDA allocations before loading a model.""" + gc.collect() + if not torch.cuda.is_available(): + return + torch.cuda.empty_cache() + try: + torch.cuda.ipc_collect() + except Exception: + pass + if reset_peak: + torch.cuda.reset_peak_memory_stats() + torch.cuda.synchronize() + + +def _cuda_device_map() -> str | dict[str, int]: + """Keep weights on one GPU; avoid CPU offload on small cards.""" + if torch.cuda.device_count() <= 1: + return {"": 0} + return "auto" + + +def load_model_and_tokenizer(args): + common = {"trust_remote_code": args.trust_remote_code} + tokenizer = AutoTokenizer.from_pretrained(args.model, **common) + if tokenizer.pad_token is None: + tokenizer.pad_token = tokenizer.eos_token + + use_cuda = _training_uses_cuda(args) + bf16_ok = ( + args.bf16 + if args.bf16 is not None + else use_cuda and torch.cuda.is_bf16_supported() + ) + dtype = torch.bfloat16 if bf16_ok else torch.float32 + + if args.mode == "qlora": + if not use_cuda: + raise SystemExit("QLoRA requires CUDA. Use --mode lora with --device cpu.") + try: + import bitsandbytes # noqa: F401 + except ImportError as exc: + raise SystemExit( + "QLoRA requires bitsandbytes. Install with:\n" + " uv sync --group finetune" + ) from exc + from transformers import BitsAndBytesConfig + bnb = BitsAndBytesConfig( + load_in_4bit=True, + bnb_4bit_quant_type="nf4", + bnb_4bit_use_double_quant=True, + bnb_4bit_compute_dtype=torch.bfloat16 if bf16_ok else torch.float16, + ) + model = AutoModelForCausalLM.from_pretrained( + args.model, + quantization_config=bnb, + device_map=_cuda_device_map(), + **common, + ) + from peft import prepare_model_for_kbit_training + model = prepare_model_for_kbit_training(model) + else: + model = AutoModelForCausalLM.from_pretrained( + args.model, + dtype=dtype, + device_map=_cuda_device_map() if use_cuda else None, + **common, + ) + if not use_cuda: + model.to("cpu") + + if args.mode in ("lora", "qlora"): + from peft import LoraConfig, get_peft_model + targets = ("all-linear" if args.lora_targets == "all-linear" + else [t.strip() for t in args.lora_targets.split(",")]) + cfg = LoraConfig( + r=args.lora_r, lora_alpha=args.lora_alpha, + lora_dropout=args.lora_dropout, + target_modules=targets, task_type="CAUSAL_LM") + model = get_peft_model(model, cfg) + model.print_trainable_parameters() + + if args.gradient_checkpointing: + model.gradient_checkpointing_enable() + model.config.use_cache = False + + return model, tokenizer, bf16_ok + + +# ---------------------------------------------------------------------------- +# Merge a trained LoRA adapter back into base weights +# ---------------------------------------------------------------------------- +def merge_adapter(adapter_dir, out_dir): + from peft import PeftModel, PeftConfig + cfg = PeftConfig.from_pretrained(adapter_dir) + base = AutoModelForCausalLM.from_pretrained( + cfg.base_model_name_or_path, torch_dtype=torch.bfloat16) + tok = AutoTokenizer.from_pretrained(cfg.base_model_name_or_path) + model = PeftModel.from_pretrained(base, adapter_dir) + model = model.merge_and_unload() + model.save_pretrained(out_dir) + tok.save_pretrained(out_dir) + print(f"Merged model saved to {out_dir}") + + +def run_post_lm_eval( + *, + checkpoint_path: str, + config_path: str, + experiment_name: str, + baseline_preset: str | None = None, + adapter_path: str | None = None, +) -> dict | None: + """Run slm-lm-eval via subprocess; return paths written under post_eval.""" + baseline_results: Path | None = None + if baseline_preset: + baseline_name = f"{baseline_preset}__lm-eval-baseline" + baseline_cmd = [ + "uv", + "run", + "--package", + "slm-evals", + "slm-lm-eval", + "--config", + config_path, + "--preset", + baseline_preset, + "--experiment-name", + baseline_name, + ] + print(f"\n--- lm-eval baseline ({baseline_preset}) ---") + subprocess.run(baseline_cmd, cwd=_REPO_ROOT, check=False) + baseline_results = ( + _REPO_ROOT / "results" / "lm_eval" / baseline_name / "results.json" + ) + + cmd = [ + "uv", + "run", + "--package", + "slm-evals", + "slm-lm-eval", + "--config", + config_path, + "--model", + checkpoint_path, + "--experiment-name", + experiment_name, + ] + if adapter_path: + cmd.extend(["--adapter", adapter_path]) + if baseline_results and baseline_results.is_file(): + cmd.extend(["--compare-to", str(baseline_results)]) + + print(f"\n--- lm-eval candidate ({experiment_name}) ---") + proc = subprocess.run(cmd, cwd=_REPO_ROOT, check=False) + out_root = _REPO_ROOT / "results" / "lm_eval" / experiment_name + post_eval = { + "experiment_name": experiment_name, + "config": config_path, + "checkpoint_path": checkpoint_path, + "adapter_path": adapter_path, + "baseline_preset": baseline_preset, + "results_json": str(out_root / "results.json"), + "summary_md": str(out_root / "summary.md"), + "comparison_md": str(out_root / "comparison.md") + if (out_root / "comparison.md").is_file() + else None, + "exit_code": proc.returncode, + } + return post_eval if proc.returncode == 0 else post_eval + + +# ---------------------------------------------------------------------------- +# Main +# ---------------------------------------------------------------------------- +def main(): + _load_dotenv(_REPO_ROOT / ".env") + args = parse_args() + + if args.merge: + out_dir = args.out or default_output_dir(None, "merged") + merge_adapter(args.merge, out_dir) + return + + model_id, preset_key, trust_remote_code = resolve_model_and_preset( + model_arg=args.model, + preset_arg=args.preset, + ) + args.model = model_id + args.trust_remote_code = trust_remote_code + if not args.dataset: + args.dataset = ( + os.environ.get("FINETUNE_DATASET") + or str(_DEFAULT_DATASET) + ) + if not args.out: + args.out = os.environ.get("FINETUNE_OUT") or default_output_dir( + preset_key, args.mode + ) + + Path(args.out).mkdir(parents=True, exist_ok=True) + + print(f"Base model: {args.model}") + if preset_key: + print(f"Preset: {preset_key}") + print(f"Dataset: {args.dataset}") + print(f"Output: {args.out}") + print(f"Device: {args.device}") + + _validate_cuda_device(args) + _apply_low_vram_defaults(args) + if _training_uses_cuda(args): + print(f"GPU before cleanup: {_gpu_memory_summary()}") + clear_gpu_memory() + print(f"GPU after cleanup: {_gpu_memory_summary()}") + + lr = args.lr or (2e-5 if args.mode == "full" else 2e-4) + + model, tokenizer, bf16_ok = load_model_and_tokenizer(args) + + if _training_uses_cuda(args): + print(f"GPU after model load: {_gpu_memory_summary()}") + + ds = load_raw_dataset( + args.dataset, + config=args.dataset_config, + split=args.dataset_split, + max_samples=args.dataset_max_samples, + ) + ds = ds.shuffle(seed=args.seed) + tokenize = build_tokenize_fn(tokenizer, args.format, args.max_len, + args.mask_prompt) + ds = ds.map(tokenize, remove_columns=ds.column_names, desc="tokenizing") + ds = ds.filter(lambda e: len(e["input_ids"]) > 1) + + if args.val_split > 0: + split = ds.train_test_split(test_size=args.val_split, seed=args.seed) + train_ds, eval_ds = split["train"], split["test"] + else: + train_ds, eval_ds = ds, None + + targs = TrainingArguments( + output_dir=args.out, + num_train_epochs=args.epochs, + max_steps=args.max_steps, + per_device_train_batch_size=args.batch_size, + per_device_eval_batch_size=args.batch_size, + gradient_accumulation_steps=args.grad_accum, + learning_rate=lr, + lr_scheduler_type="cosine", + warmup_ratio=args.warmup_ratio, + weight_decay=0.01, + logging_steps=10, + eval_strategy="steps" if eval_ds is not None else "no", + eval_steps=200, + save_strategy="steps", + save_steps=500, + save_total_limit=2, + bf16=bf16_ok, + fp16=(not bf16_ok and _training_uses_cuda(args)), + gradient_checkpointing=args.gradient_checkpointing, + report_to="none", + seed=args.seed, + ) + + trainer = Trainer( + model=model, + args=targs, + train_dataset=train_ds, + eval_dataset=eval_ds, + data_collator=CausalCollator(tokenizer), + ) + + train_result = trainer.train(resume_from_checkpoint=args.resume) + + # ---- save ----------------------------------------------------------- + model.config.use_cache = True + trainer.save_model(args.out) # full weights OR adapter only + tokenizer.save_pretrained(args.out) + + eval_metrics = None + if eval_ds is not None: + eval_metrics = trainer.evaluate() + ppl = ( + math.exp(eval_metrics["eval_loss"]) + if eval_metrics["eval_loss"] < 20 + else float("inf") + ) + print( + f"\neval_loss={eval_metrics['eval_loss']:.4f} " + f"perplexity={ppl:.2f}" + ) + + results_path = save_training_results( + args.out, + args=args, + preset_key=preset_key, + train_count=len(train_ds), + eval_count=len(eval_ds) if eval_ds is not None else 0, + train_result=train_result, + log_history=trainer.state.log_history, + eval_metrics=eval_metrics, + ) + m = json.loads(results_path.read_text())["metrics"] + print(f"\n--- scores ---") + print(f"loss_score = {m['loss_score']} (lower is better)") + print(f"result_score = {m['result_score']} (0–100, higher is better)") + print(f"Saved to {results_path}") + + if args.mode in ("lora", "qlora"): + merged = f"{args.out}-merged" + print(f"\nAdapter saved to {args.out}") + print( + "Use in Gradio: set ACTIVE_MODEL to the matching *-lora preset " + "in models.yaml, or merge with:\n" + f" python research/finetune.py --merge {args.out} --out {merged}" + ) + else: + print(f"\nFull model saved to {args.out}") + + # quick smoke generation + try: + model.eval() + prompt = "Hello! Briefly introduce yourself." + if tokenizer.chat_template: + text = tokenizer.apply_chat_template( + [{"role": "user", "content": prompt}], + tokenize=False, add_generation_prompt=True) + else: + text = prompt + device = next(model.parameters()).device + ids = tokenizer(text, return_tensors="pt").to(device) + out = model.generate(**ids, max_new_tokens=60, do_sample=True, + temperature=0.7, + pad_token_id=tokenizer.pad_token_id) + print("\n--- sample ---\n" + + tokenizer.decode(out[0][ids["input_ids"].shape[1]:], + skip_special_tokens=True)) + except Exception as e: # smoke test is best-effort + print(f"(sample generation skipped: {e})") + + if args.lm_eval_after: + exp_name = f"{Path(args.out).name}__lm-eval-posttrain" + if args.mode in ("lora", "qlora"): + post_eval = run_post_lm_eval( + checkpoint_path=args.model, + config_path=args.lm_eval_config, + experiment_name=exp_name, + baseline_preset=args.lm_eval_baseline or preset_key, + adapter_path=args.out, + ) + else: + post_eval = run_post_lm_eval( + checkpoint_path=args.out, + config_path=args.lm_eval_config, + experiment_name=exp_name, + baseline_preset=args.lm_eval_baseline or preset_key, + ) + if post_eval: + payload = json.loads(results_path.read_text()) + payload["post_eval"] = post_eval + results_path.write_text(json.dumps(payload, indent=2)) + print(f"Appended post_eval to {results_path}") + + if _training_uses_cuda(args): + clear_gpu_memory() + print(f"GPU after training: {_gpu_memory_summary()}") + + +if __name__ == "__main__": + main() diff --git a/research/llm_emb_jepa_ensemble_pluggable.py b/research/llm_emb_jepa_ensemble_pluggable.py new file mode 100644 index 0000000000000000000000000000000000000000..5f39ae839603791d9696162bd042a45faef150e9 --- /dev/null +++ b/research/llm_emb_jepa_ensemble_pluggable.py @@ -0,0 +1,23 @@ +"""Deprecated shim — use `ensemble.jepa_ensemble` instead.""" + +from ensemble.backends import HFBackend, LLMBackend, TinyBackend, make_backend +from ensemble.bridge import Bridge +from ensemble.jepa import JEPA +from ensemble.jepa_ensemble import Ensemble, demo_hf, demo_tiny, segment_pairs_from_texts +from ensemble.memory import Embedder, Router, VectorStore + +__all__ = [ + "LLMBackend", + "HFBackend", + "TinyBackend", + "make_backend", + "Embedder", + "JEPA", + "Bridge", + "VectorStore", + "Router", + "Ensemble", + "segment_pairs_from_texts", + "demo_tiny", + "demo_hf", +] diff --git a/research/world_model_ensemble.py b/research/world_model_ensemble.py new file mode 100644 index 0000000000000000000000000000000000000000..64d3b190d80e522128c8c4798d4034182d9daba3 --- /dev/null +++ b/research/world_model_ensemble.py @@ -0,0 +1,20 @@ +"""Deprecated shim — use `ensemble.world_ensemble` instead.""" + +from ensemble.world_ensemble import ( + WorldEnsemble, + demo, + hf_segment_sequences, + toy_segment_sequences, +) + +__all__ = [ + "WorldEnsemble", + "toy_segment_sequences", + "hf_segment_sequences", + "demo", +] + +if __name__ == "__main__": + import sys + + demo(sys.argv[1] if len(sys.argv) > 1 else "tiny") diff --git a/results/lm_eval/ensemble-jepa__manual/summary.md b/results/lm_eval/ensemble-jepa__manual/summary.md new file mode 100644 index 0000000000000000000000000000000000000000..8bd4e0a354896b9f89d039cb62ab21efd9f099e1 --- /dev/null +++ b/results/lm_eval/ensemble-jepa__manual/summary.md @@ -0,0 +1,15 @@ +# lm-eval summary + +- experiment: `ensemble-jepa__manual` +- model backend: `ensemble-lm` +- base model: `/home/msg-encrypted/Documents/dev/hackathon/small-model-hackathon/models/ensemble/custom-jepa-pretrain` +- ensemble checkpoint: `/home/msg-encrypted/Documents/dev/hackathon/small-model-hackathon/models/ensemble/custom-jepa-pretrain` +- tasks: arc_easy, hellaswag +- num_fewshot: 0 +- limit: 25 +- seed: 42 + +| task | metric | score | +| --- | --- | ---: | +| arc_easy | acc,none | 0.5200 | +| hellaswag | acc,none | 0.2800 | diff --git a/uv.lock b/uv.lock index 77b563160c68a1eb8e9a75124f078fd3e2b4ec4c..6f04a05d5a0550bae94064b3b5d22bdcd64a1b67 100644 --- a/uv.lock +++ b/uv.lock @@ -16,11 +16,22 @@ resolution-markers = [ [manifest] members = [ "agent", + "ensemble", "gradio-space", "inference", + "slm-evals", "small-model-hackathon", ] +[[package]] +name = "absl-py" +version = "2.4.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/64/c7/8de93764ad66968d19329a7e0c147a2bb3c7054c554d4a119111b8f9440f/absl_py-2.4.0.tar.gz", hash = "sha256:8c6af82722b35cf71e0f4d1d47dcaebfff286e27110a99fc359349b247dfb5d4", size = 116543, upload-time = "2026-01-28T10:17:05.322Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/18/a6/907a406bb7d359e6a63f99c313846d9eec4f7e6f7437809e03aa00fa3074/absl_py-2.4.0-py3-none-any.whl", hash = "sha256:88476fd881ca8aab94ffa78b7b6c632a782ab3ba1cd19c9bd423abc4fb4cd28d", size = 135750, upload-time = "2026-01-28T10:17:04.19Z" }, +] + [[package]] name = "accelerate" version = "1.13.0" @@ -62,6 +73,128 @@ requires-dist = [ { name = "pyyaml", specifier = ">=6.0.2" }, ] +[[package]] +name = "aiohappyeyeballs" +version = "2.6.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/33/c6/61a2d7b7572279226bb2e7f61d7a19ca7c90da0329c93fa0d560cbf288d8/aiohappyeyeballs-2.6.2.tar.gz", hash = "sha256:e202810ee718bd01fc6ef49e8ea53d023d5cb6b581076d7925aa499fa55dbe64", size = 22591, upload-time = "2026-05-20T15:12:24.631Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5f/fc/a7bf5b6e4e617b45f90f2d9d2a68519c249c81dd4fc2658c7a2a61c4f4b7/aiohappyeyeballs-2.6.2-py3-none-any.whl", hash = "sha256:4708045e2d7a6c6bdf8aafa8ed39649eaf926a4543b54560659129e3365953c4", size = 15062, upload-time = "2026-05-20T15:12:23.328Z" }, +] + +[[package]] +name = "aiohttp" +version = "3.14.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "aiohappyeyeballs" }, + { name = "aiosignal" }, + { name = "attrs" }, + { name = "frozenlist" }, + { name = "multidict" }, + { name = "propcache" }, + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, + { name = "yarl" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/82/78/8ea7308cac6934de8c74a14f3d5f65d1c89287426688be79538d0e5c013d/aiohttp-3.14.1.tar.gz", hash = "sha256:307f2cff90a764d329e77040603fa032db89c5c24fdad50c4c15334cba744035", size = 7955794, upload-time = "2026-06-07T21:09:35.529Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1d/21/151624b51cd92553d95424daf4bf19f19ce9be9002d19253e7e7ce67197b/aiohttp-3.14.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:d35143e27778b4bb0fb189562d7f275bff79c62ab8e98459717c0ea617ff2480", size = 757402, upload-time = "2026-06-07T21:06:40.311Z" }, + { url = "https://files.pythonhosted.org/packages/c2/82/280619e0bd7bf2454987e19282616e84762255dd9c8468f62382e8c191f1/aiohttp-3.14.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:bcfb80a2cc36fba2534e5e5b5264dc7ae6fcd9bf15256da3e53d2f499e6fa29d", size = 512310, upload-time = "2026-06-07T21:06:42.207Z" }, + { url = "https://files.pythonhosted.org/packages/55/b2/2aac325583aaa1353045f96dffa586d8a34e8322e14a7ba49cffeb103ab4/aiohttp-3.14.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:27fd7c91e51729b4f7e1577865fa6d34c9adccbc39aabe9000285b48af9f0ec2", size = 512448, upload-time = "2026-06-07T21:06:43.813Z" }, + { url = "https://files.pythonhosted.org/packages/8a/72/a60607cb849faa8af8a356c9329ea2eb6f395d49e82cc82ccba1fd8deb8f/aiohttp-3.14.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:64c567bf9eaf664280116a8688f63016e6b32db2505908e2bdaca1b6438142f2", size = 1766854, upload-time = "2026-06-07T21:06:45.391Z" }, + { url = "https://files.pythonhosted.org/packages/b5/d3/d9fe1c9ec7557ab4d0d82bebaa728c6418f0b93295ec2f4ab015f7710cc7/aiohttp-3.14.1-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:f5e6ff2bdbb8f4cd3fbe41f99e25bbcd58e3bf9f13d3dd31a11e7917251cc77a", size = 1740884, upload-time = "2026-06-07T21:06:47.413Z" }, + { url = "https://files.pythonhosted.org/packages/c1/dc/f2cecfaf9337ba3e63f181500814ff502aa3d00d9c7ec93a9d23d10a27b2/aiohttp-3.14.1-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:2f73e01dc37122325caf079982621262f96d74823c179038a82fddfc50359264", size = 1810034, upload-time = "2026-06-07T21:06:50.165Z" }, + { url = "https://files.pythonhosted.org/packages/66/d7/2ff65c5e65c0d7476daf7e15c032e0805e36811185b9623e3238ad6c763e/aiohttp-3.14.1-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:bb2c0c80d431c0d03f2c7dbf125150fedd4f0de17366a7ca33f7ccb822391842", size = 1904054, upload-time = "2026-06-07T21:06:52.035Z" }, + { url = "https://files.pythonhosted.org/packages/20/9c/d445818389df371f56d141d881153ba23183c4735a03f7356ffb43f7757d/aiohttp-3.14.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3e6fc1a85fa7194a1a7d19f44e8609180f4a8eb5fa4c7ed8b4355f080fad235c", size = 1790278, upload-time = "2026-06-07T21:06:54.049Z" }, + { url = "https://files.pythonhosted.org/packages/4d/aa/bf04cb4d865fc6101c2229a294ad744973b72e513fdc5a6b791e6983d72a/aiohttp-3.14.1-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:686b6c0d3911ec387b444ddf5dc62fb7f7c0a7d5186a7861626496a5ab4aff95", size = 1591795, upload-time = "2026-06-07T21:06:55.911Z" }, + { url = "https://files.pythonhosted.org/packages/dc/b4/4dac0038960427ba832f6609dfb4ea5437d7fd80c72001b9e48f834f428b/aiohttp-3.14.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:c6fa4dc7ad6f8109c70bb1499e589f76b0b792baf39f9b017eb92c8a81d0a199", size = 1728397, upload-time = "2026-06-07T21:06:57.777Z" }, + { url = "https://files.pythonhosted.org/packages/2b/f9/7cd4e8ad7aa3b75f17d56bb5498dd604a93d4e6eece822ba0568c413fff0/aiohttp-3.14.1-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:87a5eea1b2a5e21e1ebdbb33ad4165359189327e63fc4e4894693e7f821ac817", size = 1766504, upload-time = "2026-06-07T21:07:00.009Z" }, + { url = "https://files.pythonhosted.org/packages/f9/df/fc01d9fcad0f73fed3f3d361f1f94f975947b50dff82919f6dc2bf4316cc/aiohttp-3.14.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:1c1421eb01d4fd608d88cc8290211d177a58532b55ad94076fb349c5bf467f0a", size = 1777806, upload-time = "2026-06-07T21:07:02.064Z" }, + { url = "https://files.pythonhosted.org/packages/41/09/47e2d090bddcc8fb4ccb4c314aadc32d7c5d9bb55f50f6ad1c92fc15d501/aiohttp-3.14.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:34b257ec41345c1e8f2df68fa908a7952f5de932723871eb633ecbbff396c9a4", size = 1580707, upload-time = "2026-06-07T21:07:03.942Z" }, + { url = "https://files.pythonhosted.org/packages/3d/36/f1a4ce904ae0b6930cfe9afc96d0896f7ec1a620c400405d63783bb95a9c/aiohttp-3.14.1-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:de538791a80e5d862addbc183f70f0158ac9b9bb872bb147f1fd2a683691e087", size = 1798121, upload-time = "2026-06-07T21:07:05.987Z" }, + { url = "https://files.pythonhosted.org/packages/70/0a/e0075ce9ca0279ee1d4f0c0b85f54fea02ebc83c3007651a72bece658fec/aiohttp-3.14.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:6f71173be42d3241d428f760122febb748de0623f44308a6f120d0dd9ec572e3", size = 1767580, upload-time = "2026-06-07T21:07:07.873Z" }, + { url = "https://files.pythonhosted.org/packages/3e/61/a0c0a8f327a9c52095cdd8e312391b00d3ed64ab6c72bb5c33d8ec251cf7/aiohttp-3.14.1-cp312-cp312-win32.whl", hash = "sha256:ec8dc383ee57ea3e883477dcca3f11b65d58199f1080acaf4cd6ad9a99698be4", size = 452771, upload-time = "2026-06-07T21:07:09.669Z" }, + { url = "https://files.pythonhosted.org/packages/df/d9/ea367c75f16ac9c6cdc8febb25e8318fa21a2b1bc8d6514d4b2d890bface/aiohttp-3.14.1-cp312-cp312-win_amd64.whl", hash = "sha256:2aa92c87868cd13674989f9ee83e5f9f7ea4237589b728048e1f0c8f6caa3271", size = 479873, upload-time = "2026-06-07T21:07:11.538Z" }, + { url = "https://files.pythonhosted.org/packages/03/64/8d96784a7851156db8a4c6c3f6f91042fdf39fb15a4cc38c8b3c14833c45/aiohttp-3.14.1-cp312-cp312-win_arm64.whl", hash = "sha256:2c840c90759922cb5e6dda94596e079a30fb5a5ba548e7e0dc00574703940847", size = 448073, upload-time = "2026-06-07T21:07:13.637Z" }, + { url = "https://files.pythonhosted.org/packages/bc/97/bd137012dd97e1649162b099135a80e1fd59aaa807b2430fc448d1029aff/aiohttp-3.14.1-cp313-cp313-android_21_arm64_v8a.whl", hash = "sha256:b3a03285a7f9c7b016324574a6d92a1c895da6b978cb8f1deee3ac72bc6da178", size = 506882, upload-time = "2026-06-07T21:07:15.501Z" }, + { url = "https://files.pythonhosted.org/packages/ef/79/e5cc690e9d922a66887ceeaca53a8ffd5a7b0be3816142b7abc433742d89/aiohttp-3.14.1-cp313-cp313-android_21_x86_64.whl", hash = "sha256:2a73f487ab8ef5abbb24b7aa9b73e98eaba9e9e031804ff2416f02eca315ccaf", size = 515270, upload-time = "2026-06-07T21:07:17.53Z" }, + { url = "https://files.pythonhosted.org/packages/fe/22/a73ccbf9dbd6e26dda0b24d5fd5db7da92ee3383a79f47677ffb834c5c5b/aiohttp-3.14.1-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:915fbb7b41b115192259f8c9ae58f3ddc444d2b5579917270211858e606a4afd", size = 485841, upload-time = "2026-06-07T21:07:19.555Z" }, + { url = "https://files.pythonhosted.org/packages/3b/b9/57ed8eaf596321c2ad747bd480fb1700dbd7177c60dfc9e4c187f629662e/aiohttp-3.14.1-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:7fb4bdf95b0561a79f259f9d28fbc109728c5ee7f27aff6391f0ca703a329abe", size = 492088, upload-time = "2026-06-07T21:07:21.581Z" }, + { url = "https://files.pythonhosted.org/packages/78/c0/5ebe5270a7c140d7c6f79dcb018640225f14d406c149e4eec04a7d82fe71/aiohttp-3.14.1-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:1b9748363260121d2927704f5d4fc498150669ca3ae93625986ee89c8f80dcd4", size = 501564, upload-time = "2026-06-07T21:07:23.388Z" }, + { url = "https://files.pythonhosted.org/packages/75/7f/8cdaa24fc7983865e0915153b96a9ac5bcdd3548d64c5a27d17cecccad2d/aiohttp-3.14.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:86a6dab78b0e43e2897a3bbe15745aa60dc5423ca437b7b0b164c069bf91b876", size = 751998, upload-time = "2026-06-07T21:07:25.046Z" }, + { url = "https://files.pythonhosted.org/packages/b2/f4/c4227aacfacc5cb0cc2d119b65301d177912a6842cd64e120c47af76064f/aiohttp-3.14.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:4dfd6e47d3c44c2279907607f73a4240b88c69eb8b90da7e2441a8045dfd21da", size = 510918, upload-time = "2026-06-07T21:07:27.28Z" }, + { url = "https://files.pythonhosted.org/packages/ab/01/a2d5f96cd4e74424864d30bc0a7e44d0a12dacdcfa91b5b2d1bd3dca6bf3/aiohttp-3.14.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:317acd9f8602858dc7d59679812c376c7f0b97bcbbf16e0d6237f54141d8a8a6", size = 508657, upload-time = "2026-06-07T21:07:29.252Z" }, + { url = "https://files.pythonhosted.org/packages/e8/ed/3c0fb5c500fdd8e7ebc10d1889c04384fffa1a9163eac1356088ca9da1b1/aiohttp-3.14.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bd869c427324e5cb15195793de951295710db28be7d818247f3097b4ab5d4b96", size = 1757907, upload-time = "2026-06-07T21:07:31.03Z" }, + { url = "https://files.pythonhosted.org/packages/0b/ab/d4c924d9bd5be3050c226612413ce68cb54c70d2c31b661bfc8d9a5b6a70/aiohttp-3.14.1-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:93b032b5ec3255473c143627d21a69ac74ae12f7f33974cb587c564d11b1066f", size = 1737565, upload-time = "2026-06-07T21:07:33.031Z" }, + { url = "https://files.pythonhosted.org/packages/19/2a/37326821ff779084020cdc33224d20b19f42f4183a500ff92022a739eda7/aiohttp-3.14.1-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f234b4deb12f3ad59127e037bc57c40c21e45b45282df7d3a55a0f409f595296", size = 1799018, upload-time = "2026-06-07T21:07:35.003Z" }, + { url = "https://files.pythonhosted.org/packages/b3/4f/6e947ba73e4ce09070761c05ed3a8ceb7c21f5e46798671d8b2aac0e4626/aiohttp-3.14.1-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:9af6779bfb46abf124068327abcdf9ce95c9ef8287a3e8da76ccf2d0f16c28fa", size = 1894416, upload-time = "2026-06-07T21:07:36.956Z" }, + { url = "https://files.pythonhosted.org/packages/9d/6e/dbf1d0625dc711fb2851f4f3c3055c39ed58bae92082d8c627dbe6013736/aiohttp-3.14.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:faccab372e66bc76d5731525e7f1143c922271725b9d38c9f97edcc66266b451", size = 1783881, upload-time = "2026-06-07T21:07:39.063Z" }, + { url = "https://files.pythonhosted.org/packages/44/c2/5e25098a67268ed369483ae7d1a58bd0a13d03aab860d2a0e4a6eb25b046/aiohttp-3.14.1-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f380468b09d2a81633ee863b0ec5648d364bd17bb8ecfb8c2f387f7ac1faf42c", size = 1587572, upload-time = "2026-06-07T21:07:41.058Z" }, + { url = "https://files.pythonhosted.org/packages/2a/bd/cf9cee17e140f942a3de73e658a543aa8fbf35a5fc67a9d2538d52d77f0b/aiohttp-3.14.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:97e704dcd26271f5bda3fa07c3ce0fb76d6d3f8659f4baa1a24442cc9ba177ca", size = 1722137, upload-time = "2026-06-07T21:07:43.014Z" }, + { url = "https://files.pythonhosted.org/packages/89/6d/5684f8c59045c96f81a18cefbc1fbbd79d25b88f1c622f2a5c5c08fcb632/aiohttp-3.14.1-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:269b76ac5394092b95bc4a098f4fc6c191c083c3bd12775d1e30e663132f6a09", size = 1755953, upload-time = "2026-06-07T21:07:45.933Z" }, + { url = "https://files.pythonhosted.org/packages/a8/40/35caf3170f8359760740a7d9aa0fff2e344bef98e1d1186f5a0f6dec17e6/aiohttp-3.14.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:5c0b3e614340c889d575451696374c9d17affd54cd607ca0babed8f8c37b9397", size = 1766479, upload-time = "2026-06-07T21:07:48.047Z" }, + { url = "https://files.pythonhosted.org/packages/6d/a1/b0c61e7a137f0d81de49a82023a6df73c3c16d6fefb0f8e4a93d21639002/aiohttp-3.14.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:5663ee9257cfa1add7253a7da3035a02f31b6600ec48261585e1800a81533080", size = 1580077, upload-time = "2026-06-07T21:07:50.069Z" }, + { url = "https://files.pythonhosted.org/packages/0b/41/194ea4623693009fcefebef7aef63c141754f153e9cd0d39d3b9e36c175c/aiohttp-3.14.1-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:603a2c834142172ffddc054067f5ec0ca65d57a0aa98a71bc81952573208e345", size = 1791688, upload-time = "2026-06-07T21:07:52.106Z" }, + { url = "https://files.pythonhosted.org/packages/ba/45/4de841f005cfe1fd63e2a2fe011262c515e2a62aa6994b15947e7d717ac9/aiohttp-3.14.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:cb21957bb8aca671c1765e32f58164cf0c50e6bf41c0bbbd16da20732ecaf588", size = 1761094, upload-time = "2026-06-07T21:07:54.113Z" }, + { url = "https://files.pythonhosted.org/packages/e4/ae/dbce10533d3896d544d5053939ed75b7dc31a1b0973d959b1b5ae21028d6/aiohttp-3.14.1-cp313-cp313-win32.whl", hash = "sha256:e509a55f681e6158c20f70f102f9cf61fb20fbc382272bc6d94b7343f2582780", size = 452662, upload-time = "2026-06-07T21:07:56.06Z" }, + { url = "https://files.pythonhosted.org/packages/7b/d9/0bf1a19362c32f06229da5e7ddfcec91f93474d6307f7a2d3135e9c674dc/aiohttp-3.14.1-cp313-cp313-win_amd64.whl", hash = "sha256:1ac8531b638959718e18c2207fbfe297819875da46a740b29dfa29beba64355a", size = 479748, upload-time = "2026-06-07T21:07:58.319Z" }, + { url = "https://files.pythonhosted.org/packages/22/0a/62e7232dc9484fbec112ceb32efb6a624cc7994ec6e2b019286f17c4e8f2/aiohttp-3.14.1-cp313-cp313-win_arm64.whl", hash = "sha256:250d14af67f6b6a1a4a811049b1afa69d61d617fca6bf33149b3ab1a6dbcf7b8", size = 447723, upload-time = "2026-06-07T21:08:00.154Z" }, + { url = "https://files.pythonhosted.org/packages/c4/a1/5fafa04e1ca91ddb47608699d60649c1c6db3cf41c99e78fc4056f9513db/aiohttp-3.14.1-cp314-cp314-android_24_arm64_v8a.whl", hash = "sha256:7c106c26852ca1c2047c6b80384f17100b4e439af276f21ef3d4e2f450ae7e15", size = 508531, upload-time = "2026-06-07T21:08:02.093Z" }, + { url = "https://files.pythonhosted.org/packages/fa/2e/bfa02f699d87ffc86d5959270b28f1cb410add3ccaced8ed2e0b8a5238fc/aiohttp-3.14.1-cp314-cp314-android_24_x86_64.whl", hash = "sha256:20205f7f5ade7aaec9f4b500549bbc071b046453aed72f9c06dcab87896a83e8", size = 514718, upload-time = "2026-06-07T21:08:04.476Z" }, + { url = "https://files.pythonhosted.org/packages/85/a5/9594ad6289eebbc97d167c44213d557807f90e59115caad24de21ad2c3b1/aiohttp-3.14.1-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:62a759436b29e677181a9e76bab8b8f689a29cb9c535f45f7c48c9c830d3f8c3", size = 487918, upload-time = "2026-06-07T21:08:06.377Z" }, + { url = "https://files.pythonhosted.org/packages/b4/61/16a32c36c3c49edec122a3dc811f2057df2f94d3b14aa107c8017d981618/aiohttp-3.14.1-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:2964cbf553df4d7a57348da44d961d871895fc1ee4e8c322b2a95612c7b17fba", size = 494014, upload-time = "2026-06-07T21:08:08.263Z" }, + { url = "https://files.pythonhosted.org/packages/9b/89/3ebcf96ed99c05bec9c434aaac6963fd3cbab4a786ae739908a144d9ce44/aiohttp-3.14.1-cp314-cp314-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:237651caadc3a59badd39319c54642b5299e9cc98a3a194310e55d5bb9f5e397", size = 502398, upload-time = "2026-06-07T21:08:10.244Z" }, + { url = "https://files.pythonhosted.org/packages/fd/3d/b74870a0c2d40c355928cd5b96c7a11fa821b8a40fc41365e64479b151fb/aiohttp-3.14.1-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:896e12dfdbbab9d8f7e16d2b28c6769a60126fa92095d1ebf9473d02593a2448", size = 758018, upload-time = "2026-06-07T21:08:12.447Z" }, + { url = "https://files.pythonhosted.org/packages/d3/66/f42f5c984d99e49c6cff5f26f590750f2e2f7ef1fcfb99966ab5be1b632e/aiohttp-3.14.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:d03f281ed22579314ba00821ce20115a7c0ac430660b4cc05704a3f818b3e004", size = 512462, upload-time = "2026-06-07T21:08:14.624Z" }, + { url = "https://files.pythonhosted.org/packages/e9/a7/248e1aebe0c7810b0271e021a0f2a5eb6e78a051885b3c9df49f42a5802d/aiohttp-3.14.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:07eabb979d236335fed927e137a928c9adfb7df3b9ec7aa31726f133a62be983", size = 512824, upload-time = "2026-06-07T21:08:16.572Z" }, + { url = "https://files.pythonhosted.org/packages/26/97/2aa0e5ba0727dc3bd5aaebb7ccbc510f7dfb7fb961ec87497cd496635ab1/aiohttp-3.14.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4fe1f1087cbadb280b5e1bb054a4f00d1423c74d6626c5e48400d871d34ecefe", size = 1749898, upload-time = "2026-06-07T21:08:18.635Z" }, + { url = "https://files.pythonhosted.org/packages/00/8d/e97f6c96c891d457c8479d92a514ba194d0412f981d72c70341ee18488ed/aiohttp-3.14.1-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:367a9314fdc79dab0fac96e216cb41dd73c85bdca85306ce8999118ba7e0f333", size = 1710114, upload-time = "2026-06-07T21:08:20.892Z" }, + { url = "https://files.pythonhosted.org/packages/6f/e6/aa8d7e863048c8fceb5cd6ce74017311cec3ead07847387e12265fb4444e/aiohttp-3.14.1-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a24f677ebe83749039e7bdf862ff0bbb16818ae4193d4ef96505e269375bcce0", size = 1802541, upload-time = "2026-06-07T21:08:23.044Z" }, + { url = "https://files.pythonhosted.org/packages/83/a8/72193137de57fda4ebfae4563182d082c8856e3b6e9871d0b46f028fb369/aiohttp-3.14.1-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:c83afe0ba876be7e943d2e0ba645809ad441575d2840c895c21ee5de93b9377a", size = 1875776, upload-time = "2026-06-07T21:08:25.288Z" }, + { url = "https://files.pythonhosted.org/packages/a0/18/938441025db6769a3464596b2410af3afde0b21eb2f204c6f766f68af4bd/aiohttp-3.14.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:634e385930fb6d2d479cf3aa66515955863b77a5e3c2b5894ca259a25b308602", size = 1760329, upload-time = "2026-06-07T21:08:27.363Z" }, + { url = "https://files.pythonhosted.org/packages/60/29/bf2496b4065e76e09fe48015aaffe5ce161d8f089b06ac6982070f653076/aiohttp-3.14.1-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:eeea07c4397bbc57719c4eed8f9c284874d4f175f9b6d57f7a1546b976d455ca", size = 1587293, upload-time = "2026-06-07T21:08:29.805Z" }, + { url = "https://files.pythonhosted.org/packages/49/a2/2136674d52123b1354bd05dd5753c318db47dc0c927cc70b27bab3755456/aiohttp-3.14.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:335c0cc3e3545ce98dcb9cfcb836f40c3411f43fa03dab757597d80c89af8a35", size = 1714756, upload-time = "2026-06-07T21:08:32.094Z" }, + { url = "https://files.pythonhosted.org/packages/a7/b9/e5fd2e6f915503081c0f9b1e8540947037929c70c191da2e4d54b31a21a1/aiohttp-3.14.1-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:ae6be797afdef264e8a84864a85b196ca06045586481b3df8a967322fd2fa844", size = 1721052, upload-time = "2026-06-07T21:08:34.167Z" }, + { url = "https://files.pythonhosted.org/packages/63/5a/2833e324a2263e104e31e2e91bc5bbee81bc499afd32203faee048a883f0/aiohttp-3.14.1-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:8560b4d712474335d08907db7973f71912d3a9a8f1dee992ec06b5d2fe359496", size = 1766888, upload-time = "2026-06-07T21:08:36.95Z" }, + { url = "https://files.pythonhosted.org/packages/57/fa/dea6511870913162f3b2e8c42a7614eb203a4540b8c2da43e0bfb0548f3c/aiohttp-3.14.1-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:2b7edd08e0a5deb1e8564a2fcd8f4561014a3f05252334671bbf55ddd47db0e5", size = 1581679, upload-time = "2026-06-07T21:08:39.292Z" }, + { url = "https://files.pythonhosted.org/packages/14/bd/3cf0d55e71784b33534e9710a67d382d900598b4787fbce6cc7317f8c42a/aiohttp-3.14.1-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:b6ff7fcee63287ae57b5df3e4f5957ce032122802509246dec1a5bcc55904c95", size = 1782021, upload-time = "2026-06-07T21:08:41.407Z" }, + { url = "https://files.pythonhosted.org/packages/c1/af/14bb5843eccbe234f4dfb78ab73e549d99727247e62ae5d62cbd22eaf5b0/aiohttp-3.14.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:6ffbb2f4ec1ceaff7e07d43922954da26b223d188bf30658e561b98e23089444", size = 1742574, upload-time = "2026-06-07T21:08:43.795Z" }, + { url = "https://files.pythonhosted.org/packages/f2/1e/fbeb7af9210a67ac0f9c9bec0f8f4568497924e33137a3d5b48e1cf85f3f/aiohttp-3.14.1-cp314-cp314-win32.whl", hash = "sha256:a9875b46d910cff3ea2f5962f9d266b465459fe634e22556ab9bd6fc1192eea0", size = 457773, upload-time = "2026-06-07T21:08:46.168Z" }, + { url = "https://files.pythonhosted.org/packages/f0/2b/13e8d741a9ec5db7d900c060554cf8352ab85e44e2a4469ebb9d377bda17/aiohttp-3.14.1-cp314-cp314-win_amd64.whl", hash = "sha256:af8b4b81a960eeaf1234971ac3cd0ba5901f3cd42eae42a46b4d089a8b492719", size = 485001, upload-time = "2026-06-07T21:08:48.401Z" }, + { url = "https://files.pythonhosted.org/packages/df/30/491acfa2c4d6c3ff59c49a14fc1b50be3241e25bbb0c84c09e2da4d11395/aiohttp-3.14.1-cp314-cp314-win_arm64.whl", hash = "sha256:cf4491381b1b57425c315a56a439251b1bdac07b2275f19a8c44bc57744532ec", size = 453809, upload-time = "2026-06-07T21:08:50.7Z" }, + { url = "https://files.pythonhosted.org/packages/34/e3/19dbe1a1f4cc6230eb9e314de7fe68053b0992f9302b27d12141a0b5db53/aiohttp-3.14.1-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:819c054312f1af92947e6a55883d1b66feefab11531a7fc45e0fb9b63880b5c2", size = 793320, upload-time = "2026-06-07T21:08:52.775Z" }, + { url = "https://files.pythonhosted.org/packages/7f/20/1b7182219ba1b108430d6e4dc53d25ae02dcfcf5a045b33af4e8c5167527/aiohttp-3.14.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:10ee9c1753a8f706345b22496c79fbddb5be0599e0823f3738b1534058e25340", size = 529077, upload-time = "2026-06-07T21:08:55Z" }, + { url = "https://files.pythonhosted.org/packages/b9/c8/14ce60ec31a2e5f5274bb17d383a6f7a3aabca31ac04eee05585bbadab16/aiohttp-3.14.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:1601cc37baf5750ccacae618ec2daf020769581695550e3b654a911f859c563d", size = 532476, upload-time = "2026-06-07T21:08:57.176Z" }, + { url = "https://files.pythonhosted.org/packages/7e/02/9ac85e081e53da2e061b02fa7758fe0a12d17b8ce2d1f5e6c7cb76730328/aiohttp-3.14.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4d6e0ac9da31c9c04c84e1c0182ad8d6df35965a85cae29cd71d089621b3ae94", size = 1922347, upload-time = "2026-06-07T21:08:59.563Z" }, + { url = "https://files.pythonhosted.org/packages/c0/3e/d3ba07a0ab38b5389e10bec4362d21e10a4f667cba2d79ba30837b3a5059/aiohttp-3.14.1-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:9e8f2d660c350b3d0e259c7a7e3d9b7fc8b41210cbcc3d4a7076ff0a5e5c2fdc", size = 1786465, upload-time = "2026-06-07T21:09:01.909Z" }, + { url = "https://files.pythonhosted.org/packages/0b/cb/e2ee978a00cfb2df829704a69528b18154eba5939f45bc1efa8f33aee4c5/aiohttp-3.14.1-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:4691802dda97be727f79d86818acaad7eb8e9252626a1d6b519fedbb92d5e251", size = 1909423, upload-time = "2026-06-07T21:09:04.357Z" }, + { url = "https://files.pythonhosted.org/packages/73/5d/1430334858b1022b58ae50399a918f0bd6fe8fa7fa183598d657ff61e040/aiohttp-3.14.1-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:c389c482a7e9b9dc3ee2701ac46c4125297a3818875b9c305ddb603c04828fd1", size = 2001906, upload-time = "2026-06-07T21:09:06.722Z" }, + { url = "https://files.pythonhosted.org/packages/66/4e/560c7472d3d198a23aa5c8b19a5115bf6a9b77b7d3e4bb363da320430ad2/aiohttp-3.14.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fc0cacab7ba4e56f0f81c82a98c09bed2f39c940107b03a34b168bdf7597edd3", size = 1877095, upload-time = "2026-06-07T21:09:09.011Z" }, + { url = "https://files.pythonhosted.org/packages/0d/f1/4745806578d447db4a784a8591e2dae3afdfc2bcb96f8f81271b13df6543/aiohttp-3.14.1-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:979ed4717f59b8bb12e3963378fa285d93d367e15bcd66c721311826d3c44a6c", size = 1676222, upload-time = "2026-06-07T21:09:11.461Z" }, + { url = "https://files.pythonhosted.org/packages/6a/c9/48255813cca749a229ef0ab476004ec623728ad79a9c0840616f6c076325/aiohttp-3.14.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:38e1e7daaea81df51c952e18483f323d878499a1e2bfe564790e0f9701d6f203", size = 1842922, upload-time = "2026-06-07T21:09:14.118Z" }, + { url = "https://files.pythonhosted.org/packages/3d/c0/bbd054e2bee909f529523a5af3891052606af5143c09f5f183ec3b234676/aiohttp-3.14.1-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:4132e72c608fe9fecb8f409113567605915b83e9bdd3ea56538d2f9cd35002f1", size = 1825035, upload-time = "2026-06-07T21:09:16.447Z" }, + { url = "https://files.pythonhosted.org/packages/a8/ae/90395d4376deceb74e09ec26b6adf7d2015a6f8802d6d84446af860fef04/aiohttp-3.14.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:eefd9cc9b6d4a2db5f00a26bc3e4f9acf71926a6ec557cd56c9c6f27c290b665", size = 1849512, upload-time = "2026-06-07T21:09:18.742Z" }, + { url = "https://files.pythonhosted.org/packages/93/bd/fb25f3049957553d4ce0ba6ae480aa2f592a6985497fca590837d16c1be0/aiohttp-3.14.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:b165790117eea512d7f3fb22f1f6dad3d55a7189571993eb015591c1401276d1", size = 1668571, upload-time = "2026-06-07T21:09:21.458Z" }, + { url = "https://files.pythonhosted.org/packages/3f/22/7f73303d64dd567ff3addca90b556690ed1233a47b8f55d242fb90af3681/aiohttp-3.14.1-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:ed09c7eb1c391271c2ed0314a51903e72a3acb653d5ccfc264cdf3ef11f8269d", size = 1881159, upload-time = "2026-06-07T21:09:23.813Z" }, + { url = "https://files.pythonhosted.org/packages/44/be/0474c5a8b5640e1e4aa1923430a91f4151be82e511373fe764189b89aef5/aiohttp-3.14.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:99abd37084b82f5830c635fddd0b4993b9742a66eb746dacf433c8590e8f9e3c", size = 1841409, upload-time = "2026-06-07T21:09:26.207Z" }, + { url = "https://files.pythonhosted.org/packages/7b/3c/bb4a7cba26956cb3da4553cc2056cf67be5b5ff6e6d8fa4fbdff73bfb7ae/aiohttp-3.14.1-cp314-cp314t-win32.whl", hash = "sha256:47ddf841cdecc810749921d25606dee45857d12d2ad5ddb7b5bd7eab12e4b365", size = 494166, upload-time = "2026-06-07T21:09:28.505Z" }, + { url = "https://files.pythonhosted.org/packages/8a/84/ec80c2c1f66a952555a9f86df6b33af65108a6febfa0471b69013a12f807/aiohttp-3.14.1-cp314-cp314t-win_amd64.whl", hash = "sha256:5e78b522b7a6e27e0b25d19b247b75039ac4c94f99823e3c9e53ae1603a9f7e9", size = 530255, upload-time = "2026-06-07T21:09:30.843Z" }, + { url = "https://files.pythonhosted.org/packages/2a/71/6e22be134a4061ada85a92951b842f2657f17d926b727f3f94c56ae963d6/aiohttp-3.14.1-cp314-cp314t-win_arm64.whl", hash = "sha256:90d53f1609c29ccc2193945ef732428382a28f78d0456ae4d3daf0d48b74f0f6", size = 469640, upload-time = "2026-06-07T21:09:33.028Z" }, +] + +[[package]] +name = "aiosignal" +version = "1.4.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "frozenlist" }, + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/61/62/06741b579156360248d1ec624842ad0edf697050bbaf7c3e46394e106ad1/aiosignal-1.4.0.tar.gz", hash = "sha256:f47eecd9468083c2029cc99945502cb7708b082c232f9aca65da147157b251c7", size = 25007, upload-time = "2025-07-03T22:54:43.528Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fb/76/641ae371508676492379f16e2fa48f4e2c11741bd63c48be4b12a6b09cba/aiosignal-1.4.0-py3-none-any.whl", hash = "sha256:053243f8b92b990551949e63930a839ff0cf0b0ebbe0597b0f3fb19e1a0fe82e", size = 7490, upload-time = "2025-07-03T22:54:42.156Z" }, +] + [[package]] name = "annotated-doc" version = "0.0.4" @@ -93,6 +226,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/da/42/e921fccf5015463e32a3cf6ee7f980a6ed0f395ceeaa45060b61d86486c2/anyio-4.13.0-py3-none-any.whl", hash = "sha256:08b310f9e24a9594186fd75b4f73f4a4152069e3853f1ed8bfbf58369f4ad708", size = 114353, upload-time = "2026-03-24T12:59:08.246Z" }, ] +[[package]] +name = "attrs" +version = "26.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/9a/8e/82a0fe20a541c03148528be8cac2408564a6c9a0cc7e9171802bc1d26985/attrs-26.1.0.tar.gz", hash = "sha256:d03ceb89cb322a8fd706d4fb91940737b6642aa36998fe130a9bc96c985eff32", size = 952055, upload-time = "2026-03-19T14:22:25.026Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/64/b4/17d4b0b2a2dc85a6df63d1157e028ed19f90d4cd97c36717afef2bc2f395/attrs-26.1.0-py3-none-any.whl", hash = "sha256:c647aa4a12dfbad9333ca4e71fe62ddc36f4e63b2d260a37a8b83d2f043ac309", size = 67548, upload-time = "2026-03-19T14:22:23.645Z" }, +] + [[package]] name = "audioop-lts" version = "0.2.2" @@ -149,6 +291,22 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/f6/22/91616fe707a5c5510de2cac9b046a30defe7007ba8a0c04f9c08f27df312/audioop_lts-0.2.2-cp314-cp314t-win_arm64.whl", hash = "sha256:b492c3b040153e68b9fdaff5913305aaaba5bb433d8a7f73d5cf6a64ed3cc1dd", size = 25206, upload-time = "2025-08-05T16:43:16.444Z" }, ] +[[package]] +name = "bitsandbytes" +version = "0.49.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy" }, + { name = "packaging" }, + { name = "torch" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/d8/7d/f1fe0992334b18cd8494f89aeec1dcc674635584fcd9f115784fea3a1d05/bitsandbytes-0.49.2-py3-none-macosx_14_0_arm64.whl", hash = "sha256:87be5975edeac5396d699ecbc39dfc47cf2c026daaf2d5852a94368611a6823f", size = 131940, upload-time = "2026-02-16T21:26:04.572Z" }, + { url = "https://files.pythonhosted.org/packages/29/71/acff7af06c818664aa87ff73e17a52c7788ad746b72aea09d3cb8e424348/bitsandbytes-0.49.2-py3-none-manylinux_2_24_aarch64.whl", hash = "sha256:2fc0830c5f7169be36e60e11f2be067c8f812dfcb829801a8703735842450750", size = 31442815, upload-time = "2026-02-16T21:26:06.783Z" }, + { url = "https://files.pythonhosted.org/packages/19/57/3443d6f183436fbdaf5000aac332c4d5ddb056665d459244a5608e98ae92/bitsandbytes-0.49.2-py3-none-manylinux_2_24_x86_64.whl", hash = "sha256:54b771f06e1a3c73af5c7f16ccf0fc23a846052813d4b008d10cb6e017dd1c8c", size = 60651714, upload-time = "2026-02-16T21:26:11.579Z" }, + { url = "https://files.pythonhosted.org/packages/b6/d4/501655842ad6771fb077f576d78cbedb5445d15b1c3c91343ed58ca46f0e/bitsandbytes-0.49.2-py3-none-win_amd64.whl", hash = "sha256:2e0ddd09cd778155388023cbe81f00afbb7c000c214caef3ce83386e7144df7d", size = 55372289, upload-time = "2026-02-16T21:26:16.267Z" }, +] + [[package]] name = "brotli" version = "1.2.0" @@ -196,6 +354,88 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/59/8c/57e832b7af6d7c5abe66eb3fbe3a3a32f4d11ea23a1aa7131371035be991/certifi-2026.5.20-py3-none-any.whl", hash = "sha256:3c52e209ba0a4ad7aebe60436a4ab349c39e1e602e8c134221e546902ad25897", size = 134134, upload-time = "2026-05-20T11:46:48.578Z" }, ] +[[package]] +name = "chardet" +version = "6.0.0.post1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7f/42/fb9436c103a881a377e34b9f58d77b5f503461c702ff654ebe86151bcfe9/chardet-6.0.0.post1.tar.gz", hash = "sha256:6b78048c3c97c7b2ed1fbad7a18f76f5a6547f7d34dbab536cc13887c9a92fa4", size = 12521798, upload-time = "2026-02-22T15:09:17.925Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/66/42/5de54f632c2de53cd3415b3703383d5fff43a94cbc0567ef362515261a21/chardet-6.0.0.post1-py3-none-any.whl", hash = "sha256:c894a36800549adf7bb5f2af47033281b75fdfcd2aa0f0243be0ad22a52e2dcb", size = 627245, upload-time = "2026-02-22T15:09:15.876Z" }, +] + +[[package]] +name = "charset-normalizer" +version = "3.4.7" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e7/a1/67fe25fac3c7642725500a3f6cfe5821ad557c3abb11c9d20d12c7008d3e/charset_normalizer-3.4.7.tar.gz", hash = "sha256:ae89db9e5f98a11a4bf50407d4363e7b09b31e55bc117b4f7d80aab97ba009e5", size = 144271, upload-time = "2026-04-02T09:28:39.342Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0c/eb/4fc8d0a7110eb5fc9cc161723a34a8a6c200ce3b4fbf681bc86feee22308/charset_normalizer-3.4.7-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:eca9705049ad3c7345d574e3510665cb2cf844c2f2dcfe675332677f081cbd46", size = 311328, upload-time = "2026-04-02T09:26:24.331Z" }, + { url = "https://files.pythonhosted.org/packages/f8/e3/0fadc706008ac9d7b9b5be6dc767c05f9d3e5df51744ce4cc9605de7b9f4/charset_normalizer-3.4.7-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6178f72c5508bfc5fd446a5905e698c6212932f25bcdd4b47a757a50605a90e2", size = 208061, upload-time = "2026-04-02T09:26:25.568Z" }, + { url = "https://files.pythonhosted.org/packages/42/f0/3dd1045c47f4a4604df85ec18ad093912ae1344ac706993aff91d38773a2/charset_normalizer-3.4.7-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e1421b502d83040e6d7fb2fb18dff63957f720da3d77b2fbd3187ceb63755d7b", size = 229031, upload-time = "2026-04-02T09:26:26.865Z" }, + { url = "https://files.pythonhosted.org/packages/dc/67/675a46eb016118a2fbde5a277a5d15f4f69d5f3f5f338e5ee2f8948fcf43/charset_normalizer-3.4.7-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:edac0f1ab77644605be2cbba52e6b7f630731fc42b34cb0f634be1a6eface56a", size = 225239, upload-time = "2026-04-02T09:26:28.044Z" }, + { url = "https://files.pythonhosted.org/packages/4b/f8/d0118a2f5f23b02cd166fa385c60f9b0d4f9194f574e2b31cef350ad7223/charset_normalizer-3.4.7-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5649fd1c7bade02f320a462fdefd0b4bd3ce036065836d4f42e0de958038e116", size = 216589, upload-time = "2026-04-02T09:26:29.239Z" }, + { url = "https://files.pythonhosted.org/packages/b1/f1/6d2b0b261b6c4ceef0fcb0d17a01cc5bc53586c2d4796fa04b5c540bc13d/charset_normalizer-3.4.7-cp312-cp312-manylinux_2_31_armv7l.whl", hash = "sha256:203104ed3e428044fd943bc4bf45fa73c0730391f9621e37fe39ecf477b128cb", size = 202733, upload-time = "2026-04-02T09:26:30.5Z" }, + { url = "https://files.pythonhosted.org/packages/6f/c0/7b1f943f7e87cc3db9626ba17807d042c38645f0a1d4415c7a14afb5591f/charset_normalizer-3.4.7-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:298930cec56029e05497a76988377cbd7457ba864beeea92ad7e844fe74cd1f1", size = 212652, upload-time = "2026-04-02T09:26:31.709Z" }, + { url = "https://files.pythonhosted.org/packages/38/dd/5a9ab159fe45c6e72079398f277b7d2b523e7f716acc489726115a910097/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:708838739abf24b2ceb208d0e22403dd018faeef86ddac04319a62ae884c4f15", size = 211229, upload-time = "2026-04-02T09:26:33.282Z" }, + { url = "https://files.pythonhosted.org/packages/d5/ff/531a1cad5ca855d1c1a8b69cb71abfd6d85c0291580146fda7c82857caa1/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:0f7eb884681e3938906ed0434f20c63046eacd0111c4ba96f27b76084cd679f5", size = 203552, upload-time = "2026-04-02T09:26:34.845Z" }, + { url = "https://files.pythonhosted.org/packages/c1/4c/a5fb52d528a8ca41f7598cb619409ece30a169fbdf9cdce592e53b46c3a6/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:4dc1e73c36828f982bfe79fadf5919923f8a6f4df2860804db9a98c48824ce8d", size = 230806, upload-time = "2026-04-02T09:26:36.152Z" }, + { url = "https://files.pythonhosted.org/packages/59/7a/071feed8124111a32b316b33ae4de83d36923039ef8cf48120266844285b/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:aed52fea0513bac0ccde438c188c8a471c4e0f457c2dd20cdbf6ea7a450046c7", size = 212316, upload-time = "2026-04-02T09:26:37.672Z" }, + { url = "https://files.pythonhosted.org/packages/fd/35/f7dba3994312d7ba508e041eaac39a36b120f32d4c8662b8814dab876431/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:fea24543955a6a729c45a73fe90e08c743f0b3334bbf3201e6c4bc1b0c7fa464", size = 227274, upload-time = "2026-04-02T09:26:38.93Z" }, + { url = "https://files.pythonhosted.org/packages/8a/2d/a572df5c9204ab7688ec1edc895a73ebded3b023bb07364710b05dd1c9be/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:bb6d88045545b26da47aa879dd4a89a71d1dce0f0e549b1abcb31dfe4a8eac49", size = 218468, upload-time = "2026-04-02T09:26:40.17Z" }, + { url = "https://files.pythonhosted.org/packages/86/eb/890922a8b03a568ca2f336c36585a4713c55d4d67bf0f0c78924be6315ca/charset_normalizer-3.4.7-cp312-cp312-win32.whl", hash = "sha256:2257141f39fe65a3fdf38aeccae4b953e5f3b3324f4ff0daf9f15b8518666a2c", size = 148460, upload-time = "2026-04-02T09:26:41.416Z" }, + { url = "https://files.pythonhosted.org/packages/35/d9/0e7dffa06c5ab081f75b1b786f0aefc88365825dfcd0ac544bdb7b2b6853/charset_normalizer-3.4.7-cp312-cp312-win_amd64.whl", hash = "sha256:5ed6ab538499c8644b8a3e18debabcd7ce684f3fa91cf867521a7a0279cab2d6", size = 159330, upload-time = "2026-04-02T09:26:42.554Z" }, + { url = "https://files.pythonhosted.org/packages/9e/5d/481bcc2a7c88ea6b0878c299547843b2521ccbc40980cb406267088bc701/charset_normalizer-3.4.7-cp312-cp312-win_arm64.whl", hash = "sha256:56be790f86bfb2c98fb742ce566dfb4816e5a83384616ab59c49e0604d49c51d", size = 147828, upload-time = "2026-04-02T09:26:44.075Z" }, + { url = "https://files.pythonhosted.org/packages/c1/3b/66777e39d3ae1ddc77ee606be4ec6d8cbd4c801f65e5a1b6f2b11b8346dd/charset_normalizer-3.4.7-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:f496c9c3cc02230093d8330875c4c3cdfc3b73612a5fd921c65d39cbcef08063", size = 309627, upload-time = "2026-04-02T09:26:45.198Z" }, + { url = "https://files.pythonhosted.org/packages/2e/4e/b7f84e617b4854ade48a1b7915c8ccfadeba444d2a18c291f696e37f0d3b/charset_normalizer-3.4.7-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0ea948db76d31190bf08bd371623927ee1339d5f2a0b4b1b4a4439a65298703c", size = 207008, upload-time = "2026-04-02T09:26:46.824Z" }, + { url = "https://files.pythonhosted.org/packages/c4/bb/ec73c0257c9e11b268f018f068f5d00aa0ef8c8b09f7753ebd5f2880e248/charset_normalizer-3.4.7-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a277ab8928b9f299723bc1a2dabb1265911b1a76341f90a510368ca44ad9ab66", size = 228303, upload-time = "2026-04-02T09:26:48.397Z" }, + { url = "https://files.pythonhosted.org/packages/85/fb/32d1f5033484494619f701e719429c69b766bfc4dbc61aa9e9c8c166528b/charset_normalizer-3.4.7-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:3bec022aec2c514d9cf199522a802bd007cd588ab17ab2525f20f9c34d067c18", size = 224282, upload-time = "2026-04-02T09:26:49.684Z" }, + { url = "https://files.pythonhosted.org/packages/fa/07/330e3a0dda4c404d6da83b327270906e9654a24f6c546dc886a0eb0ffb23/charset_normalizer-3.4.7-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e044c39e41b92c845bc815e5ae4230804e8e7bc29e399b0437d64222d92809dd", size = 215595, upload-time = "2026-04-02T09:26:50.915Z" }, + { url = "https://files.pythonhosted.org/packages/e3/7c/fc890655786e423f02556e0216d4b8c6bcb6bdfa890160dc66bf52dee468/charset_normalizer-3.4.7-cp313-cp313-manylinux_2_31_armv7l.whl", hash = "sha256:f495a1652cf3fbab2eb0639776dad966c2fb874d79d87ca07f9d5f059b8bd215", size = 201986, upload-time = "2026-04-02T09:26:52.197Z" }, + { url = "https://files.pythonhosted.org/packages/d8/97/bfb18b3db2aed3b90cf54dc292ad79fdd5ad65c4eae454099475cbeadd0d/charset_normalizer-3.4.7-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e712b419df8ba5e42b226c510472b37bd57b38e897d3eca5e8cfd410a29fa859", size = 211711, upload-time = "2026-04-02T09:26:53.49Z" }, + { url = "https://files.pythonhosted.org/packages/6f/a5/a581c13798546a7fd557c82614a5c65a13df2157e9ad6373166d2a3e645d/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:7804338df6fcc08105c7745f1502ba68d900f45fd770d5bdd5288ddccb8a42d8", size = 210036, upload-time = "2026-04-02T09:26:54.975Z" }, + { url = "https://files.pythonhosted.org/packages/8c/bf/b3ab5bcb478e4193d517644b0fb2bf5497fbceeaa7a1bc0f4d5b50953861/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:481551899c856c704d58119b5025793fa6730adda3571971af568f66d2424bb5", size = 202998, upload-time = "2026-04-02T09:26:56.303Z" }, + { url = "https://files.pythonhosted.org/packages/e7/4e/23efd79b65d314fa320ec6017b4b5834d5c12a58ba4610aa353af2e2f577/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:f59099f9b66f0d7145115e6f80dd8b1d847176df89b234a5a6b3f00437aa0832", size = 230056, upload-time = "2026-04-02T09:26:57.554Z" }, + { url = "https://files.pythonhosted.org/packages/b9/9f/1e1941bc3f0e01df116e68dc37a55c4d249df5e6fa77f008841aef68264f/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:f59ad4c0e8f6bba240a9bb85504faa1ab438237199d4cce5f622761507b8f6a6", size = 211537, upload-time = "2026-04-02T09:26:58.843Z" }, + { url = "https://files.pythonhosted.org/packages/80/0f/088cbb3020d44428964a6c97fe1edfb1b9550396bf6d278330281e8b709c/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:3dedcc22d73ec993f42055eff4fcfed9318d1eeb9a6606c55892a26964964e48", size = 226176, upload-time = "2026-04-02T09:27:00.437Z" }, + { url = "https://files.pythonhosted.org/packages/6a/9f/130394f9bbe06f4f63e22641d32fc9b202b7e251c9aef4db044324dac493/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:64f02c6841d7d83f832cd97ccf8eb8a906d06eb95d5276069175c696b024b60a", size = 217723, upload-time = "2026-04-02T09:27:02.021Z" }, + { url = "https://files.pythonhosted.org/packages/73/55/c469897448a06e49f8fa03f6caae97074fde823f432a98f979cc42b90e69/charset_normalizer-3.4.7-cp313-cp313-win32.whl", hash = "sha256:4042d5c8f957e15221d423ba781e85d553722fc4113f523f2feb7b188cc34c5e", size = 148085, upload-time = "2026-04-02T09:27:03.192Z" }, + { url = "https://files.pythonhosted.org/packages/5d/78/1b74c5bbb3f99b77a1715c91b3e0b5bdb6fe302d95ace4f5b1bec37b0167/charset_normalizer-3.4.7-cp313-cp313-win_amd64.whl", hash = "sha256:3946fa46a0cf3e4c8cb1cc52f56bb536310d34f25f01ca9b6c16afa767dab110", size = 158819, upload-time = "2026-04-02T09:27:04.454Z" }, + { url = "https://files.pythonhosted.org/packages/68/86/46bd42279d323deb8687c4a5a811fd548cb7d1de10cf6535d099877a9a9f/charset_normalizer-3.4.7-cp313-cp313-win_arm64.whl", hash = "sha256:80d04837f55fc81da168b98de4f4b797ef007fc8a79ab71c6ec9bc4dd662b15b", size = 147915, upload-time = "2026-04-02T09:27:05.971Z" }, + { url = "https://files.pythonhosted.org/packages/97/c8/c67cb8c70e19ef1960b97b22ed2a1567711de46c4ddf19799923adc836c2/charset_normalizer-3.4.7-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:c36c333c39be2dbca264d7803333c896ab8fa7d4d6f0ab7edb7dfd7aea6e98c0", size = 309234, upload-time = "2026-04-02T09:27:07.194Z" }, + { url = "https://files.pythonhosted.org/packages/99/85/c091fdee33f20de70d6c8b522743b6f831a2f1cd3ff86de4c6a827c48a76/charset_normalizer-3.4.7-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1c2aed2e5e41f24ea8ef1590b8e848a79b56f3a5564a65ceec43c9d692dc7d8a", size = 208042, upload-time = "2026-04-02T09:27:08.749Z" }, + { url = "https://files.pythonhosted.org/packages/87/1c/ab2ce611b984d2fd5d86a5a8a19c1ae26acac6bad967da4967562c75114d/charset_normalizer-3.4.7-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:54523e136b8948060c0fa0bc7b1b50c32c186f2fceee897a495406bb6e311d2b", size = 228706, upload-time = "2026-04-02T09:27:09.951Z" }, + { url = "https://files.pythonhosted.org/packages/a8/29/2b1d2cb00bf085f59d29eb773ce58ec2d325430f8c216804a0a5cd83cbca/charset_normalizer-3.4.7-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:715479b9a2802ecac752a3b0efa2b0b60285cf962ee38414211abdfccc233b41", size = 224727, upload-time = "2026-04-02T09:27:11.175Z" }, + { url = "https://files.pythonhosted.org/packages/47/5c/032c2d5a07fe4d4855fea851209cca2b6f03ebeb6d4e3afdb3358386a684/charset_normalizer-3.4.7-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bd6c2a1c7573c64738d716488d2cdd3c00e340e4835707d8fdb8dc1a66ef164e", size = 215882, upload-time = "2026-04-02T09:27:12.446Z" }, + { url = "https://files.pythonhosted.org/packages/2c/c2/356065d5a8b78ed04499cae5f339f091946a6a74f91e03476c33f0ab7100/charset_normalizer-3.4.7-cp314-cp314-manylinux_2_31_armv7l.whl", hash = "sha256:c45e9440fb78f8ddabcf714b68f936737a121355bf59f3907f4e17721b9d1aae", size = 200860, upload-time = "2026-04-02T09:27:13.721Z" }, + { url = "https://files.pythonhosted.org/packages/0c/cd/a32a84217ced5039f53b29f460962abb2d4420def55afabe45b1c3c7483d/charset_normalizer-3.4.7-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:3534e7dcbdcf757da6b85a0bbf5b6868786d5982dd959b065e65481644817a18", size = 211564, upload-time = "2026-04-02T09:27:15.272Z" }, + { url = "https://files.pythonhosted.org/packages/44/86/58e6f13ce26cc3b8f4a36b94a0f22ae2f00a72534520f4ae6857c4b81f89/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:e8ac484bf18ce6975760921bb6148041faa8fef0547200386ea0b52b5d27bf7b", size = 211276, upload-time = "2026-04-02T09:27:16.834Z" }, + { url = "https://files.pythonhosted.org/packages/8f/fe/d17c32dc72e17e155e06883efa84514ca375f8a528ba2546bee73fc4df81/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:a5fe03b42827c13cdccd08e6c0247b6a6d4b5e3cdc53fd1749f5896adcdc2356", size = 201238, upload-time = "2026-04-02T09:27:18.229Z" }, + { url = "https://files.pythonhosted.org/packages/6a/29/f33daa50b06525a237451cdb6c69da366c381a3dadcd833fa5676bc468b3/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:2d6eb928e13016cea4f1f21d1e10c1cebd5a421bc57ddf5b1142ae3f86824fab", size = 230189, upload-time = "2026-04-02T09:27:19.445Z" }, + { url = "https://files.pythonhosted.org/packages/b6/6e/52c84015394a6a0bdcd435210a7e944c5f94ea1055f5cc5d56c5fe368e7b/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:e74327fb75de8986940def6e8dee4f127cc9752bee7355bb323cc5b2659b6d46", size = 211352, upload-time = "2026-04-02T09:27:20.79Z" }, + { url = "https://files.pythonhosted.org/packages/8c/d7/4353be581b373033fb9198bf1da3cf8f09c1082561e8e922aa7b39bf9fe8/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:d6038d37043bced98a66e68d3aa2b6a35505dc01328cd65217cefe82f25def44", size = 227024, upload-time = "2026-04-02T09:27:22.063Z" }, + { url = "https://files.pythonhosted.org/packages/30/45/99d18aa925bd1740098ccd3060e238e21115fffbfdcb8f3ece837d0ace6c/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:7579e913a5339fb8fa133f6bbcfd8e6749696206cf05acdbdca71a1b436d8e72", size = 217869, upload-time = "2026-04-02T09:27:23.486Z" }, + { url = "https://files.pythonhosted.org/packages/5c/05/5ee478aa53f4bb7996482153d4bfe1b89e0f087f0ab6b294fcf92d595873/charset_normalizer-3.4.7-cp314-cp314-win32.whl", hash = "sha256:5b77459df20e08151cd6f8b9ef8ef1f961ef73d85c21a555c7eed5b79410ec10", size = 148541, upload-time = "2026-04-02T09:27:25.146Z" }, + { url = "https://files.pythonhosted.org/packages/48/77/72dcb0921b2ce86420b2d79d454c7022bf5be40202a2a07906b9f2a35c97/charset_normalizer-3.4.7-cp314-cp314-win_amd64.whl", hash = "sha256:92a0a01ead5e668468e952e4238cccd7c537364eb7d851ab144ab6627dbbe12f", size = 159634, upload-time = "2026-04-02T09:27:26.642Z" }, + { url = "https://files.pythonhosted.org/packages/c6/a3/c2369911cd72f02386e4e340770f6e158c7980267da16af8f668217abaa0/charset_normalizer-3.4.7-cp314-cp314-win_arm64.whl", hash = "sha256:67f6279d125ca0046a7fd386d01b311c6363844deac3e5b069b514ba3e63c246", size = 148384, upload-time = "2026-04-02T09:27:28.271Z" }, + { url = "https://files.pythonhosted.org/packages/94/09/7e8a7f73d24dba1f0035fbbf014d2c36828fc1bf9c88f84093e57d315935/charset_normalizer-3.4.7-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:effc3f449787117233702311a1b7d8f59cba9ced946ba727bdc329ec69028e24", size = 330133, upload-time = "2026-04-02T09:27:29.474Z" }, + { url = "https://files.pythonhosted.org/packages/8d/da/96975ddb11f8e977f706f45cddd8540fd8242f71ecdb5d18a80723dcf62c/charset_normalizer-3.4.7-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:fbccdc05410c9ee21bbf16a35f4c1d16123dcdeb8a1d38f33654fa21d0234f79", size = 216257, upload-time = "2026-04-02T09:27:30.793Z" }, + { url = "https://files.pythonhosted.org/packages/e5/e8/1d63bf8ef2d388e95c64b2098f45f84758f6d102a087552da1485912637b/charset_normalizer-3.4.7-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:733784b6d6def852c814bce5f318d25da2ee65dd4839a0718641c696e09a2960", size = 234851, upload-time = "2026-04-02T09:27:32.44Z" }, + { url = "https://files.pythonhosted.org/packages/9b/40/e5ff04233e70da2681fa43969ad6f66ca5611d7e669be0246c4c7aaf6dc8/charset_normalizer-3.4.7-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a89c23ef8d2c6b27fd200a42aa4ac72786e7c60d40efdc76e6011260b6e949c4", size = 233393, upload-time = "2026-04-02T09:27:34.03Z" }, + { url = "https://files.pythonhosted.org/packages/be/c1/06c6c49d5a5450f76899992f1ee40b41d076aee9279b49cf9974d2f313d5/charset_normalizer-3.4.7-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6c114670c45346afedc0d947faf3c7f701051d2518b943679c8ff88befe14f8e", size = 223251, upload-time = "2026-04-02T09:27:35.369Z" }, + { url = "https://files.pythonhosted.org/packages/2b/9f/f2ff16fb050946169e3e1f82134d107e5d4ae72647ec8a1b1446c148480f/charset_normalizer-3.4.7-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:a180c5e59792af262bf263b21a3c49353f25945d8d9f70628e73de370d55e1e1", size = 206609, upload-time = "2026-04-02T09:27:36.661Z" }, + { url = "https://files.pythonhosted.org/packages/69/d5/a527c0cd8d64d2eab7459784fb4169a0ac76e5a6fc5237337982fd61347e/charset_normalizer-3.4.7-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:3c9a494bc5ec77d43cea229c4f6db1e4d8fe7e1bbffa8b6f0f0032430ff8ab44", size = 220014, upload-time = "2026-04-02T09:27:38.019Z" }, + { url = "https://files.pythonhosted.org/packages/7e/80/8a7b8104a3e203074dc9aa2c613d4b726c0e136bad1cc734594b02867972/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:8d828b6667a32a728a1ad1d93957cdf37489c57b97ae6c4de2860fa749b8fc1e", size = 218979, upload-time = "2026-04-02T09:27:39.37Z" }, + { url = "https://files.pythonhosted.org/packages/02/9a/b759b503d507f375b2b5c153e4d2ee0a75aa215b7f2489cf314f4541f2c0/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:cf1493cd8607bec4d8a7b9b004e699fcf8f9103a9284cc94962cb73d20f9d4a3", size = 209238, upload-time = "2026-04-02T09:27:40.722Z" }, + { url = "https://files.pythonhosted.org/packages/c2/4e/0f3f5d47b86bdb79256e7290b26ac847a2832d9a4033f7eb2cd4bcf4bb5b/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:0c96c3b819b5c3e9e165495db84d41914d6894d55181d2d108cc1a69bfc9cce0", size = 236110, upload-time = "2026-04-02T09:27:42.33Z" }, + { url = "https://files.pythonhosted.org/packages/96/23/bce28734eb3ed2c91dcf93abeb8a5cf393a7b2749725030bb630e554fdd8/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:752a45dc4a6934060b3b0dab47e04edc3326575f82be64bc4fc293914566503e", size = 219824, upload-time = "2026-04-02T09:27:43.924Z" }, + { url = "https://files.pythonhosted.org/packages/2c/6f/6e897c6984cc4d41af319b077f2f600fc8214eb2fe2d6bcb79141b882400/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:8778f0c7a52e56f75d12dae53ae320fae900a8b9b4164b981b9c5ce059cd1fcb", size = 233103, upload-time = "2026-04-02T09:27:45.348Z" }, + { url = "https://files.pythonhosted.org/packages/76/22/ef7bd0fe480a0ae9b656189ec00744b60933f68b4f42a7bb06589f6f576a/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:ce3412fbe1e31eb81ea42f4169ed94861c56e643189e1e75f0041f3fe7020abe", size = 225194, upload-time = "2026-04-02T09:27:46.706Z" }, + { url = "https://files.pythonhosted.org/packages/c5/a7/0e0ab3e0b5bc1219bd80a6a0d4d72ca74d9250cb2382b7c699c147e06017/charset_normalizer-3.4.7-cp314-cp314t-win32.whl", hash = "sha256:c03a41a8784091e67a39648f70c5f97b5b6a37f216896d44d2cdcb82615339a0", size = 159827, upload-time = "2026-04-02T09:27:48.053Z" }, + { url = "https://files.pythonhosted.org/packages/7a/1d/29d32e0fb40864b1f878c7f5a0b343ae676c6e2b271a2d55cc3a152391da/charset_normalizer-3.4.7-cp314-cp314t-win_amd64.whl", hash = "sha256:03853ed82eeebbce3c2abfdbc98c96dc205f32a79627688ac9a27370ea61a49c", size = 174168, upload-time = "2026-04-02T09:27:49.795Z" }, + { url = "https://files.pythonhosted.org/packages/de/32/d92444ad05c7a6e41fb2036749777c163baf7a0301a040cb672d6b2b1ae9/charset_normalizer-3.4.7-cp314-cp314t-win_arm64.whl", hash = "sha256:c35abb8bfff0185efac5878da64c45dafd2b37fb0383add1be155a763c1f083d", size = 153018, upload-time = "2026-04-02T09:27:51.116Z" }, + { url = "https://files.pythonhosted.org/packages/db/8f/61959034484a4a7c527811f4721e75d02d653a35afb0b6054474d8185d4c/charset_normalizer-3.4.7-py3-none-any.whl", hash = "sha256:3dce51d0f5e7951f8bb4900c257dad282f49190fdbebecd4ba99bcc41fef404d", size = 61958, upload-time = "2026-04-02T09:28:37.794Z" }, +] + [[package]] name = "click" version = "8.4.1" @@ -283,6 +523,53 @@ nvtx = [ { name = "nvidia-nvtx", marker = "sys_platform == 'linux'" }, ] +[[package]] +name = "dataproperty" +version = "1.1.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "mbstrdecoder" }, + { name = "typepy", extra = ["datetime"] }, +] +sdist = { url = "https://files.pythonhosted.org/packages/f1/6f/a801320bb388d965be9c370ec753cc33120e6cbe0069fa05644f05821975/dataproperty-1.1.1.tar.gz", hash = "sha256:a83af82a234edda5378a36fb092bc90dd554646c5e58202a310acf468ae81bc8", size = 42954, upload-time = "2026-05-09T10:33:42.212Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/03/41/eab7fe313820578b341a2a1d6aeeedd2c38ec1e3f3d51e57e2735b5beac0/dataproperty-1.1.1-py3-none-any.whl", hash = "sha256:cf026aa002dbd6c57c619ec6741ffd61ae7bf2f20481951d8af2dff44480340e", size = 27691, upload-time = "2026-05-09T10:33:40.468Z" }, +] + +[[package]] +name = "datasets" +version = "5.0.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "dill" }, + { name = "filelock" }, + { name = "fsspec", extra = ["http"] }, + { name = "httpx" }, + { name = "huggingface-hub" }, + { name = "multiprocess" }, + { name = "numpy" }, + { name = "packaging" }, + { name = "pandas" }, + { name = "pyarrow" }, + { name = "pyyaml" }, + { name = "requests" }, + { name = "tqdm" }, + { name = "xxhash" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/d9/85/ce4f780c32f7e36d71257f1c27e8ba898ebe379cb54f211f5f2013f2c219/datasets-5.0.0.tar.gz", hash = "sha256:83dbbbdb07a33b82192b8c419deb18739b138ee2ce1a322d55ce6b100954ec1a", size = 631708, upload-time = "2026-06-05T13:18:26.124Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/05/66/73034ad30b59f13439b75e620989dacba4c047256e358ba7c2e9ec98ea22/datasets-5.0.0-py3-none-any.whl", hash = "sha256:7dd34927a0fd7046e98aad5cb9430e699c373238a15befa7b9bf22b991a7fee6", size = 555084, upload-time = "2026-06-05T13:18:24.435Z" }, +] + +[[package]] +name = "dill" +version = "0.4.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/81/e1/56027a71e31b02ddc53c7d65b01e68edf64dea2932122fe7746a516f75d5/dill-0.4.1.tar.gz", hash = "sha256:423092df4182177d4d8ba8290c8a5b640c66ab35ec7da59ccfa00f6fa3eea5fa", size = 187315, upload-time = "2026-01-19T02:36:56.85Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1e/77/dc8c558f7593132cf8fefec57c4f60c83b16941c574ac5f619abb3ae7933/dill-0.4.1-py3-none-any.whl", hash = "sha256:1e1ce33e978ae97fcfcff5638477032b801c46c7c65cf717f95fbc2248f79a9d", size = 120019, upload-time = "2026-01-19T02:36:55.663Z" }, +] + [[package]] name = "diskcache" version = "5.6.3" @@ -292,6 +579,39 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/3f/27/4570e78fc0bf5ea0ca45eb1de3818a23787af9b390c0b0a0033a1b8236f9/diskcache-5.6.3-py3-none-any.whl", hash = "sha256:5e31b2d5fbad117cc363ebaf6b689474db18a1f6438bc82358b024abd4c2ca19", size = 45550, upload-time = "2023-08-31T06:11:58.822Z" }, ] +[[package]] +name = "ensemble" +version = "0.1.0" +source = { editable = "research/ensemble" } +dependencies = [ + { name = "torch" }, +] + +[package.metadata] +requires-dist = [{ name = "torch", specifier = ">=2.5.0" }] + +[[package]] +name = "evaluate" +version = "0.4.6" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "datasets" }, + { name = "dill" }, + { name = "fsspec", extra = ["http"] }, + { name = "huggingface-hub" }, + { name = "multiprocess" }, + { name = "numpy" }, + { name = "packaging" }, + { name = "pandas" }, + { name = "requests" }, + { name = "tqdm" }, + { name = "xxhash" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ad/d0/0c17a8e6e8dc7245f22dea860557c32bae50fc4d287ae030cb0e8ab8720f/evaluate-0.4.6.tar.gz", hash = "sha256:e07036ca12b3c24331f83ab787f21cc2dbf3631813a1631e63e40897c69a3f21", size = 65716, upload-time = "2025-09-18T13:06:30.581Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3e/af/3e990d8d4002bbc9342adb4facd59506e653da93b2417de0fa6027cb86b1/evaluate-0.4.6-py3-none-any.whl", hash = "sha256:bca85bc294f338377b7ac2f861e21c308b11b2a285f510d7d5394d5df437db29", size = 84069, upload-time = "2025-09-18T13:06:29.265Z" }, +] + [[package]] name = "fastapi" version = "0.136.3" @@ -317,6 +637,95 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/4c/a0/614c5fe402fd88951df45f4dda2fa3b4e17a99ecd92340771929169b3b95/filelock-3.29.1-py3-none-any.whl", hash = "sha256:85199dfd706869641b72b2e8955d5416a4b2b7dc4b0e8e6d97b4cc1299a6983b", size = 40750, upload-time = "2026-06-03T15:19:02.959Z" }, ] +[[package]] +name = "frozenlist" +version = "1.8.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/2d/f5/c831fac6cc817d26fd54c7eaccd04ef7e0288806943f7cc5bbf69f3ac1f0/frozenlist-1.8.0.tar.gz", hash = "sha256:3ede829ed8d842f6cd48fc7081d7a41001a56f1f38603f9d49bf3020d59a31ad", size = 45875, upload-time = "2025-10-06T05:38:17.865Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/69/29/948b9aa87e75820a38650af445d2ef2b6b8a6fab1a23b6bb9e4ef0be2d59/frozenlist-1.8.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:78f7b9e5d6f2fdb88cdde9440dc147259b62b9d3b019924def9f6478be254ac1", size = 87782, upload-time = "2025-10-06T05:36:06.649Z" }, + { url = "https://files.pythonhosted.org/packages/64/80/4f6e318ee2a7c0750ed724fa33a4bdf1eacdc5a39a7a24e818a773cd91af/frozenlist-1.8.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:229bf37d2e4acdaf808fd3f06e854a4a7a3661e871b10dc1f8f1896a3b05f18b", size = 50594, upload-time = "2025-10-06T05:36:07.69Z" }, + { url = "https://files.pythonhosted.org/packages/2b/94/5c8a2b50a496b11dd519f4a24cb5496cf125681dd99e94c604ccdea9419a/frozenlist-1.8.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f833670942247a14eafbb675458b4e61c82e002a148f49e68257b79296e865c4", size = 50448, upload-time = "2025-10-06T05:36:08.78Z" }, + { url = "https://files.pythonhosted.org/packages/6a/bd/d91c5e39f490a49df14320f4e8c80161cfcce09f1e2cde1edd16a551abb3/frozenlist-1.8.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:494a5952b1c597ba44e0e78113a7266e656b9794eec897b19ead706bd7074383", size = 242411, upload-time = "2025-10-06T05:36:09.801Z" }, + { url = "https://files.pythonhosted.org/packages/8f/83/f61505a05109ef3293dfb1ff594d13d64a2324ac3482be2cedc2be818256/frozenlist-1.8.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:96f423a119f4777a4a056b66ce11527366a8bb92f54e541ade21f2374433f6d4", size = 243014, upload-time = "2025-10-06T05:36:11.394Z" }, + { url = "https://files.pythonhosted.org/packages/d8/cb/cb6c7b0f7d4023ddda30cf56b8b17494eb3a79e3fda666bf735f63118b35/frozenlist-1.8.0-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:3462dd9475af2025c31cc61be6652dfa25cbfb56cbbf52f4ccfe029f38decaf8", size = 234909, upload-time = "2025-10-06T05:36:12.598Z" }, + { url = "https://files.pythonhosted.org/packages/31/c5/cd7a1f3b8b34af009fb17d4123c5a778b44ae2804e3ad6b86204255f9ec5/frozenlist-1.8.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c4c800524c9cd9bac5166cd6f55285957fcfc907db323e193f2afcd4d9abd69b", size = 250049, upload-time = "2025-10-06T05:36:14.065Z" }, + { url = "https://files.pythonhosted.org/packages/c0/01/2f95d3b416c584a1e7f0e1d6d31998c4a795f7544069ee2e0962a4b60740/frozenlist-1.8.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:d6a5df73acd3399d893dafc71663ad22534b5aa4f94e8a2fabfe856c3c1b6a52", size = 256485, upload-time = "2025-10-06T05:36:15.39Z" }, + { url = "https://files.pythonhosted.org/packages/ce/03/024bf7720b3abaebcff6d0793d73c154237b85bdf67b7ed55e5e9596dc9a/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:405e8fe955c2280ce66428b3ca55e12b3c4e9c336fb2103a4937e891c69a4a29", size = 237619, upload-time = "2025-10-06T05:36:16.558Z" }, + { url = "https://files.pythonhosted.org/packages/69/fa/f8abdfe7d76b731f5d8bd217827cf6764d4f1d9763407e42717b4bed50a0/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:908bd3f6439f2fef9e85031b59fd4f1297af54415fb60e4254a95f75b3cab3f3", size = 250320, upload-time = "2025-10-06T05:36:17.821Z" }, + { url = "https://files.pythonhosted.org/packages/f5/3c/b051329f718b463b22613e269ad72138cc256c540f78a6de89452803a47d/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:294e487f9ec720bd8ffcebc99d575f7eff3568a08a253d1ee1a0378754b74143", size = 246820, upload-time = "2025-10-06T05:36:19.046Z" }, + { url = "https://files.pythonhosted.org/packages/0f/ae/58282e8f98e444b3f4dd42448ff36fa38bef29e40d40f330b22e7108f565/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:74c51543498289c0c43656701be6b077f4b265868fa7f8a8859c197006efb608", size = 250518, upload-time = "2025-10-06T05:36:20.763Z" }, + { url = "https://files.pythonhosted.org/packages/8f/96/007e5944694d66123183845a106547a15944fbbb7154788cbf7272789536/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:776f352e8329135506a1d6bf16ac3f87bc25b28e765949282dcc627af36123aa", size = 239096, upload-time = "2025-10-06T05:36:22.129Z" }, + { url = "https://files.pythonhosted.org/packages/66/bb/852b9d6db2fa40be96f29c0d1205c306288f0684df8fd26ca1951d461a56/frozenlist-1.8.0-cp312-cp312-win32.whl", hash = "sha256:433403ae80709741ce34038da08511d4a77062aa924baf411ef73d1146e74faf", size = 39985, upload-time = "2025-10-06T05:36:23.661Z" }, + { url = "https://files.pythonhosted.org/packages/b8/af/38e51a553dd66eb064cdf193841f16f077585d4d28394c2fa6235cb41765/frozenlist-1.8.0-cp312-cp312-win_amd64.whl", hash = "sha256:34187385b08f866104f0c0617404c8eb08165ab1272e884abc89c112e9c00746", size = 44591, upload-time = "2025-10-06T05:36:24.958Z" }, + { url = "https://files.pythonhosted.org/packages/a7/06/1dc65480ab147339fecc70797e9c2f69d9cea9cf38934ce08df070fdb9cb/frozenlist-1.8.0-cp312-cp312-win_arm64.whl", hash = "sha256:fe3c58d2f5db5fbd18c2987cba06d51b0529f52bc3a6cdc33d3f4eab725104bd", size = 40102, upload-time = "2025-10-06T05:36:26.333Z" }, + { url = "https://files.pythonhosted.org/packages/2d/40/0832c31a37d60f60ed79e9dfb5a92e1e2af4f40a16a29abcc7992af9edff/frozenlist-1.8.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:8d92f1a84bb12d9e56f818b3a746f3efba93c1b63c8387a73dde655e1e42282a", size = 85717, upload-time = "2025-10-06T05:36:27.341Z" }, + { url = "https://files.pythonhosted.org/packages/30/ba/b0b3de23f40bc55a7057bd38434e25c34fa48e17f20ee273bbde5e0650f3/frozenlist-1.8.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:96153e77a591c8adc2ee805756c61f59fef4cf4073a9275ee86fe8cba41241f7", size = 49651, upload-time = "2025-10-06T05:36:28.855Z" }, + { url = "https://files.pythonhosted.org/packages/0c/ab/6e5080ee374f875296c4243c381bbdef97a9ac39c6e3ce1d5f7d42cb78d6/frozenlist-1.8.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:f21f00a91358803399890ab167098c131ec2ddd5f8f5fd5fe9c9f2c6fcd91e40", size = 49417, upload-time = "2025-10-06T05:36:29.877Z" }, + { url = "https://files.pythonhosted.org/packages/d5/4e/e4691508f9477ce67da2015d8c00acd751e6287739123113a9fca6f1604e/frozenlist-1.8.0-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:fb30f9626572a76dfe4293c7194a09fb1fe93ba94c7d4f720dfae3b646b45027", size = 234391, upload-time = "2025-10-06T05:36:31.301Z" }, + { url = "https://files.pythonhosted.org/packages/40/76/c202df58e3acdf12969a7895fd6f3bc016c642e6726aa63bd3025e0fc71c/frozenlist-1.8.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:eaa352d7047a31d87dafcacbabe89df0aa506abb5b1b85a2fb91bc3faa02d822", size = 233048, upload-time = "2025-10-06T05:36:32.531Z" }, + { url = "https://files.pythonhosted.org/packages/f9/c0/8746afb90f17b73ca5979c7a3958116e105ff796e718575175319b5bb4ce/frozenlist-1.8.0-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:03ae967b4e297f58f8c774c7eabcce57fe3c2434817d4385c50661845a058121", size = 226549, upload-time = "2025-10-06T05:36:33.706Z" }, + { url = "https://files.pythonhosted.org/packages/7e/eb/4c7eefc718ff72f9b6c4893291abaae5fbc0c82226a32dcd8ef4f7a5dbef/frozenlist-1.8.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f6292f1de555ffcc675941d65fffffb0a5bcd992905015f85d0592201793e0e5", size = 239833, upload-time = "2025-10-06T05:36:34.947Z" }, + { url = "https://files.pythonhosted.org/packages/c2/4e/e5c02187cf704224f8b21bee886f3d713ca379535f16893233b9d672ea71/frozenlist-1.8.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:29548f9b5b5e3460ce7378144c3010363d8035cea44bc0bf02d57f5a685e084e", size = 245363, upload-time = "2025-10-06T05:36:36.534Z" }, + { url = "https://files.pythonhosted.org/packages/1f/96/cb85ec608464472e82ad37a17f844889c36100eed57bea094518bf270692/frozenlist-1.8.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:ec3cc8c5d4084591b4237c0a272cc4f50a5b03396a47d9caaf76f5d7b38a4f11", size = 229314, upload-time = "2025-10-06T05:36:38.582Z" }, + { url = "https://files.pythonhosted.org/packages/5d/6f/4ae69c550e4cee66b57887daeebe006fe985917c01d0fff9caab9883f6d0/frozenlist-1.8.0-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:517279f58009d0b1f2e7c1b130b377a349405da3f7621ed6bfae50b10adf20c1", size = 243365, upload-time = "2025-10-06T05:36:40.152Z" }, + { url = "https://files.pythonhosted.org/packages/7a/58/afd56de246cf11780a40a2c28dc7cbabbf06337cc8ddb1c780a2d97e88d8/frozenlist-1.8.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:db1e72ede2d0d7ccb213f218df6a078a9c09a7de257c2fe8fcef16d5925230b1", size = 237763, upload-time = "2025-10-06T05:36:41.355Z" }, + { url = "https://files.pythonhosted.org/packages/cb/36/cdfaf6ed42e2644740d4a10452d8e97fa1c062e2a8006e4b09f1b5fd7d63/frozenlist-1.8.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:b4dec9482a65c54a5044486847b8a66bf10c9cb4926d42927ec4e8fd5db7fed8", size = 240110, upload-time = "2025-10-06T05:36:42.716Z" }, + { url = "https://files.pythonhosted.org/packages/03/a8/9ea226fbefad669f11b52e864c55f0bd57d3c8d7eb07e9f2e9a0b39502e1/frozenlist-1.8.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:21900c48ae04d13d416f0e1e0c4d81f7931f73a9dfa0b7a8746fb2fe7dd970ed", size = 233717, upload-time = "2025-10-06T05:36:44.251Z" }, + { url = "https://files.pythonhosted.org/packages/1e/0b/1b5531611e83ba7d13ccc9988967ea1b51186af64c42b7a7af465dcc9568/frozenlist-1.8.0-cp313-cp313-win32.whl", hash = "sha256:8b7b94a067d1c504ee0b16def57ad5738701e4ba10cec90529f13fa03c833496", size = 39628, upload-time = "2025-10-06T05:36:45.423Z" }, + { url = "https://files.pythonhosted.org/packages/d8/cf/174c91dbc9cc49bc7b7aab74d8b734e974d1faa8f191c74af9b7e80848e6/frozenlist-1.8.0-cp313-cp313-win_amd64.whl", hash = "sha256:878be833caa6a3821caf85eb39c5ba92d28e85df26d57afb06b35b2efd937231", size = 43882, upload-time = "2025-10-06T05:36:46.796Z" }, + { url = "https://files.pythonhosted.org/packages/c1/17/502cd212cbfa96eb1388614fe39a3fc9ab87dbbe042b66f97acb57474834/frozenlist-1.8.0-cp313-cp313-win_arm64.whl", hash = "sha256:44389d135b3ff43ba8cc89ff7f51f5a0bb6b63d829c8300f79a2fe4fe61bcc62", size = 39676, upload-time = "2025-10-06T05:36:47.8Z" }, + { url = "https://files.pythonhosted.org/packages/d2/5c/3bbfaa920dfab09e76946a5d2833a7cbdf7b9b4a91c714666ac4855b88b4/frozenlist-1.8.0-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:e25ac20a2ef37e91c1b39938b591457666a0fa835c7783c3a8f33ea42870db94", size = 89235, upload-time = "2025-10-06T05:36:48.78Z" }, + { url = "https://files.pythonhosted.org/packages/d2/d6/f03961ef72166cec1687e84e8925838442b615bd0b8854b54923ce5b7b8a/frozenlist-1.8.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:07cdca25a91a4386d2e76ad992916a85038a9b97561bf7a3fd12d5d9ce31870c", size = 50742, upload-time = "2025-10-06T05:36:49.837Z" }, + { url = "https://files.pythonhosted.org/packages/1e/bb/a6d12b7ba4c3337667d0e421f7181c82dda448ce4e7ad7ecd249a16fa806/frozenlist-1.8.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:4e0c11f2cc6717e0a741f84a527c52616140741cd812a50422f83dc31749fb52", size = 51725, upload-time = "2025-10-06T05:36:50.851Z" }, + { url = "https://files.pythonhosted.org/packages/bc/71/d1fed0ffe2c2ccd70b43714c6cab0f4188f09f8a67a7914a6b46ee30f274/frozenlist-1.8.0-cp313-cp313t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:b3210649ee28062ea6099cfda39e147fa1bc039583c8ee4481cb7811e2448c51", size = 284533, upload-time = "2025-10-06T05:36:51.898Z" }, + { url = "https://files.pythonhosted.org/packages/c9/1f/fb1685a7b009d89f9bf78a42d94461bc06581f6e718c39344754a5d9bada/frozenlist-1.8.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:581ef5194c48035a7de2aefc72ac6539823bb71508189e5de01d60c9dcd5fa65", size = 292506, upload-time = "2025-10-06T05:36:53.101Z" }, + { url = "https://files.pythonhosted.org/packages/e6/3b/b991fe1612703f7e0d05c0cf734c1b77aaf7c7d321df4572e8d36e7048c8/frozenlist-1.8.0-cp313-cp313t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:3ef2d026f16a2b1866e1d86fc4e1291e1ed8a387b2c333809419a2f8b3a77b82", size = 274161, upload-time = "2025-10-06T05:36:54.309Z" }, + { url = "https://files.pythonhosted.org/packages/ca/ec/c5c618767bcdf66e88945ec0157d7f6c4a1322f1473392319b7a2501ded7/frozenlist-1.8.0-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:5500ef82073f599ac84d888e3a8c1f77ac831183244bfd7f11eaa0289fb30714", size = 294676, upload-time = "2025-10-06T05:36:55.566Z" }, + { url = "https://files.pythonhosted.org/packages/7c/ce/3934758637d8f8a88d11f0585d6495ef54b2044ed6ec84492a91fa3b27aa/frozenlist-1.8.0-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:50066c3997d0091c411a66e710f4e11752251e6d2d73d70d8d5d4c76442a199d", size = 300638, upload-time = "2025-10-06T05:36:56.758Z" }, + { url = "https://files.pythonhosted.org/packages/fc/4f/a7e4d0d467298f42de4b41cbc7ddaf19d3cfeabaf9ff97c20c6c7ee409f9/frozenlist-1.8.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:5c1c8e78426e59b3f8005e9b19f6ff46e5845895adbde20ece9218319eca6506", size = 283067, upload-time = "2025-10-06T05:36:57.965Z" }, + { url = "https://files.pythonhosted.org/packages/dc/48/c7b163063d55a83772b268e6d1affb960771b0e203b632cfe09522d67ea5/frozenlist-1.8.0-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:eefdba20de0d938cec6a89bd4d70f346a03108a19b9df4248d3cf0d88f1b0f51", size = 292101, upload-time = "2025-10-06T05:36:59.237Z" }, + { url = "https://files.pythonhosted.org/packages/9f/d0/2366d3c4ecdc2fd391e0afa6e11500bfba0ea772764d631bbf82f0136c9d/frozenlist-1.8.0-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:cf253e0e1c3ceb4aaff6df637ce033ff6535fb8c70a764a8f46aafd3d6ab798e", size = 289901, upload-time = "2025-10-06T05:37:00.811Z" }, + { url = "https://files.pythonhosted.org/packages/b8/94/daff920e82c1b70e3618a2ac39fbc01ae3e2ff6124e80739ce5d71c9b920/frozenlist-1.8.0-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:032efa2674356903cd0261c4317a561a6850f3ac864a63fc1583147fb05a79b0", size = 289395, upload-time = "2025-10-06T05:37:02.115Z" }, + { url = "https://files.pythonhosted.org/packages/e3/20/bba307ab4235a09fdcd3cc5508dbabd17c4634a1af4b96e0f69bfe551ebd/frozenlist-1.8.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:6da155091429aeba16851ecb10a9104a108bcd32f6c1642867eadaee401c1c41", size = 283659, upload-time = "2025-10-06T05:37:03.711Z" }, + { url = "https://files.pythonhosted.org/packages/fd/00/04ca1c3a7a124b6de4f8a9a17cc2fcad138b4608e7a3fc5877804b8715d7/frozenlist-1.8.0-cp313-cp313t-win32.whl", hash = "sha256:0f96534f8bfebc1a394209427d0f8a63d343c9779cda6fc25e8e121b5fd8555b", size = 43492, upload-time = "2025-10-06T05:37:04.915Z" }, + { url = "https://files.pythonhosted.org/packages/59/5e/c69f733a86a94ab10f68e496dc6b7e8bc078ebb415281d5698313e3af3a1/frozenlist-1.8.0-cp313-cp313t-win_amd64.whl", hash = "sha256:5d63a068f978fc69421fb0e6eb91a9603187527c86b7cd3f534a5b77a592b888", size = 48034, upload-time = "2025-10-06T05:37:06.343Z" }, + { url = "https://files.pythonhosted.org/packages/16/6c/be9d79775d8abe79b05fa6d23da99ad6e7763a1d080fbae7290b286093fd/frozenlist-1.8.0-cp313-cp313t-win_arm64.whl", hash = "sha256:bf0a7e10b077bf5fb9380ad3ae8ce20ef919a6ad93b4552896419ac7e1d8e042", size = 41749, upload-time = "2025-10-06T05:37:07.431Z" }, + { url = "https://files.pythonhosted.org/packages/f1/c8/85da824b7e7b9b6e7f7705b2ecaf9591ba6f79c1177f324c2735e41d36a2/frozenlist-1.8.0-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:cee686f1f4cadeb2136007ddedd0aaf928ab95216e7691c63e50a8ec066336d0", size = 86127, upload-time = "2025-10-06T05:37:08.438Z" }, + { url = "https://files.pythonhosted.org/packages/8e/e8/a1185e236ec66c20afd72399522f142c3724c785789255202d27ae992818/frozenlist-1.8.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:119fb2a1bd47307e899c2fac7f28e85b9a543864df47aa7ec9d3c1b4545f096f", size = 49698, upload-time = "2025-10-06T05:37:09.48Z" }, + { url = "https://files.pythonhosted.org/packages/a1/93/72b1736d68f03fda5fdf0f2180fb6caaae3894f1b854d006ac61ecc727ee/frozenlist-1.8.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:4970ece02dbc8c3a92fcc5228e36a3e933a01a999f7094ff7c23fbd2beeaa67c", size = 49749, upload-time = "2025-10-06T05:37:10.569Z" }, + { url = "https://files.pythonhosted.org/packages/a7/b2/fabede9fafd976b991e9f1b9c8c873ed86f202889b864756f240ce6dd855/frozenlist-1.8.0-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:cba69cb73723c3f329622e34bdbf5ce1f80c21c290ff04256cff1cd3c2036ed2", size = 231298, upload-time = "2025-10-06T05:37:11.993Z" }, + { url = "https://files.pythonhosted.org/packages/3a/3b/d9b1e0b0eed36e70477ffb8360c49c85c8ca8ef9700a4e6711f39a6e8b45/frozenlist-1.8.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:778a11b15673f6f1df23d9586f83c4846c471a8af693a22e066508b77d201ec8", size = 232015, upload-time = "2025-10-06T05:37:13.194Z" }, + { url = "https://files.pythonhosted.org/packages/dc/94/be719d2766c1138148564a3960fc2c06eb688da592bdc25adcf856101be7/frozenlist-1.8.0-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:0325024fe97f94c41c08872db482cf8ac4800d80e79222c6b0b7b162d5b13686", size = 225038, upload-time = "2025-10-06T05:37:14.577Z" }, + { url = "https://files.pythonhosted.org/packages/e4/09/6712b6c5465f083f52f50cf74167b92d4ea2f50e46a9eea0523d658454ae/frozenlist-1.8.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:97260ff46b207a82a7567b581ab4190bd4dfa09f4db8a8b49d1a958f6aa4940e", size = 240130, upload-time = "2025-10-06T05:37:15.781Z" }, + { url = "https://files.pythonhosted.org/packages/f8/d4/cd065cdcf21550b54f3ce6a22e143ac9e4836ca42a0de1022da8498eac89/frozenlist-1.8.0-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:54b2077180eb7f83dd52c40b2750d0a9f175e06a42e3213ce047219de902717a", size = 242845, upload-time = "2025-10-06T05:37:17.037Z" }, + { url = "https://files.pythonhosted.org/packages/62/c3/f57a5c8c70cd1ead3d5d5f776f89d33110b1addae0ab010ad774d9a44fb9/frozenlist-1.8.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:2f05983daecab868a31e1da44462873306d3cbfd76d1f0b5b69c473d21dbb128", size = 229131, upload-time = "2025-10-06T05:37:18.221Z" }, + { url = "https://files.pythonhosted.org/packages/6c/52/232476fe9cb64f0742f3fde2b7d26c1dac18b6d62071c74d4ded55e0ef94/frozenlist-1.8.0-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:33f48f51a446114bc5d251fb2954ab0164d5be02ad3382abcbfe07e2531d650f", size = 240542, upload-time = "2025-10-06T05:37:19.771Z" }, + { url = "https://files.pythonhosted.org/packages/5f/85/07bf3f5d0fb5414aee5f47d33c6f5c77bfe49aac680bfece33d4fdf6a246/frozenlist-1.8.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:154e55ec0655291b5dd1b8731c637ecdb50975a2ae70c606d100750a540082f7", size = 237308, upload-time = "2025-10-06T05:37:20.969Z" }, + { url = "https://files.pythonhosted.org/packages/11/99/ae3a33d5befd41ac0ca2cc7fd3aa707c9c324de2e89db0e0f45db9a64c26/frozenlist-1.8.0-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:4314debad13beb564b708b4a496020e5306c7333fa9a3ab90374169a20ffab30", size = 238210, upload-time = "2025-10-06T05:37:22.252Z" }, + { url = "https://files.pythonhosted.org/packages/b2/60/b1d2da22f4970e7a155f0adde9b1435712ece01b3cd45ba63702aea33938/frozenlist-1.8.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:073f8bf8becba60aa931eb3bc420b217bb7d5b8f4750e6f8b3be7f3da85d38b7", size = 231972, upload-time = "2025-10-06T05:37:23.5Z" }, + { url = "https://files.pythonhosted.org/packages/3f/ab/945b2f32de889993b9c9133216c068b7fcf257d8595a0ac420ac8677cab0/frozenlist-1.8.0-cp314-cp314-win32.whl", hash = "sha256:bac9c42ba2ac65ddc115d930c78d24ab8d4f465fd3fc473cdedfccadb9429806", size = 40536, upload-time = "2025-10-06T05:37:25.581Z" }, + { url = "https://files.pythonhosted.org/packages/59/ad/9caa9b9c836d9ad6f067157a531ac48b7d36499f5036d4141ce78c230b1b/frozenlist-1.8.0-cp314-cp314-win_amd64.whl", hash = "sha256:3e0761f4d1a44f1d1a47996511752cf3dcec5bbdd9cc2b4fe595caf97754b7a0", size = 44330, upload-time = "2025-10-06T05:37:26.928Z" }, + { url = "https://files.pythonhosted.org/packages/82/13/e6950121764f2676f43534c555249f57030150260aee9dcf7d64efda11dd/frozenlist-1.8.0-cp314-cp314-win_arm64.whl", hash = "sha256:d1eaff1d00c7751b7c6662e9c5ba6eb2c17a2306ba5e2a37f24ddf3cc953402b", size = 40627, upload-time = "2025-10-06T05:37:28.075Z" }, + { url = "https://files.pythonhosted.org/packages/c0/c7/43200656ecc4e02d3f8bc248df68256cd9572b3f0017f0a0c4e93440ae23/frozenlist-1.8.0-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:d3bb933317c52d7ea5004a1c442eef86f426886fba134ef8cf4226ea6ee1821d", size = 89238, upload-time = "2025-10-06T05:37:29.373Z" }, + { url = "https://files.pythonhosted.org/packages/d1/29/55c5f0689b9c0fb765055629f472c0de484dcaf0acee2f7707266ae3583c/frozenlist-1.8.0-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:8009897cdef112072f93a0efdce29cd819e717fd2f649ee3016efd3cd885a7ed", size = 50738, upload-time = "2025-10-06T05:37:30.792Z" }, + { url = "https://files.pythonhosted.org/packages/ba/7d/b7282a445956506fa11da8c2db7d276adcbf2b17d8bb8407a47685263f90/frozenlist-1.8.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:2c5dcbbc55383e5883246d11fd179782a9d07a986c40f49abe89ddf865913930", size = 51739, upload-time = "2025-10-06T05:37:32.127Z" }, + { url = "https://files.pythonhosted.org/packages/62/1c/3d8622e60d0b767a5510d1d3cf21065b9db874696a51ea6d7a43180a259c/frozenlist-1.8.0-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:39ecbc32f1390387d2aa4f5a995e465e9e2f79ba3adcac92d68e3e0afae6657c", size = 284186, upload-time = "2025-10-06T05:37:33.21Z" }, + { url = "https://files.pythonhosted.org/packages/2d/14/aa36d5f85a89679a85a1d44cd7a6657e0b1c75f61e7cad987b203d2daca8/frozenlist-1.8.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:92db2bf818d5cc8d9c1f1fc56b897662e24ea5adb36ad1f1d82875bd64e03c24", size = 292196, upload-time = "2025-10-06T05:37:36.107Z" }, + { url = "https://files.pythonhosted.org/packages/05/23/6bde59eb55abd407d34f77d39a5126fb7b4f109a3f611d3929f14b700c66/frozenlist-1.8.0-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:2dc43a022e555de94c3b68a4ef0b11c4f747d12c024a520c7101709a2144fb37", size = 273830, upload-time = "2025-10-06T05:37:37.663Z" }, + { url = "https://files.pythonhosted.org/packages/d2/3f/22cff331bfad7a8afa616289000ba793347fcd7bc275f3b28ecea2a27909/frozenlist-1.8.0-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:cb89a7f2de3602cfed448095bab3f178399646ab7c61454315089787df07733a", size = 294289, upload-time = "2025-10-06T05:37:39.261Z" }, + { url = "https://files.pythonhosted.org/packages/a4/89/5b057c799de4838b6c69aa82b79705f2027615e01be996d2486a69ca99c4/frozenlist-1.8.0-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:33139dc858c580ea50e7e60a1b0ea003efa1fd42e6ec7fdbad78fff65fad2fd2", size = 300318, upload-time = "2025-10-06T05:37:43.213Z" }, + { url = "https://files.pythonhosted.org/packages/30/de/2c22ab3eb2a8af6d69dc799e48455813bab3690c760de58e1bf43b36da3e/frozenlist-1.8.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:168c0969a329b416119507ba30b9ea13688fafffac1b7822802537569a1cb0ef", size = 282814, upload-time = "2025-10-06T05:37:45.337Z" }, + { url = "https://files.pythonhosted.org/packages/59/f7/970141a6a8dbd7f556d94977858cfb36fa9b66e0892c6dd780d2219d8cd8/frozenlist-1.8.0-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:28bd570e8e189d7f7b001966435f9dac6718324b5be2990ac496cf1ea9ddb7fe", size = 291762, upload-time = "2025-10-06T05:37:46.657Z" }, + { url = "https://files.pythonhosted.org/packages/c1/15/ca1adae83a719f82df9116d66f5bb28bb95557b3951903d39135620ef157/frozenlist-1.8.0-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:b2a095d45c5d46e5e79ba1e5b9cb787f541a8dee0433836cea4b96a2c439dcd8", size = 289470, upload-time = "2025-10-06T05:37:47.946Z" }, + { url = "https://files.pythonhosted.org/packages/ac/83/dca6dc53bf657d371fbc88ddeb21b79891e747189c5de990b9dfff2ccba1/frozenlist-1.8.0-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:eab8145831a0d56ec9c4139b6c3e594c7a83c2c8be25d5bcf2d86136a532287a", size = 289042, upload-time = "2025-10-06T05:37:49.499Z" }, + { url = "https://files.pythonhosted.org/packages/96/52/abddd34ca99be142f354398700536c5bd315880ed0a213812bc491cff5e4/frozenlist-1.8.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:974b28cf63cc99dfb2188d8d222bc6843656188164848c4f679e63dae4b0708e", size = 283148, upload-time = "2025-10-06T05:37:50.745Z" }, + { url = "https://files.pythonhosted.org/packages/af/d3/76bd4ed4317e7119c2b7f57c3f6934aba26d277acc6309f873341640e21f/frozenlist-1.8.0-cp314-cp314t-win32.whl", hash = "sha256:342c97bf697ac5480c0a7ec73cd700ecfa5a8a40ac923bd035484616efecc2df", size = 44676, upload-time = "2025-10-06T05:37:52.222Z" }, + { url = "https://files.pythonhosted.org/packages/89/76/c615883b7b521ead2944bb3480398cbb07e12b7b4e4d073d3752eb721558/frozenlist-1.8.0-cp314-cp314t-win_amd64.whl", hash = "sha256:06be8f67f39c8b1dc671f5d83aaefd3358ae5cdcf8314552c57e7ed3e6475bdd", size = 49451, upload-time = "2025-10-06T05:37:53.425Z" }, + { url = "https://files.pythonhosted.org/packages/e0/a3/5982da14e113d07b325230f95060e2169f5311b1017ea8af2a29b374c289/frozenlist-1.8.0-cp314-cp314t-win_arm64.whl", hash = "sha256:102e6314ca4da683dca92e3b1355490fed5f313b768500084fbe6371fddfdb79", size = 42507, upload-time = "2025-10-06T05:37:54.513Z" }, + { url = "https://files.pythonhosted.org/packages/9a/9a/e35b4a917281c0b8419d4207f4334c8e8c5dbf4f3f5f9ada73958d937dcc/frozenlist-1.8.0-py3-none-any.whl", hash = "sha256:0c18a16eab41e82c295618a77502e17b195883241c563b00f0aa5106fc4eaa0d", size = 13409, upload-time = "2025-10-06T05:38:16.721Z" }, +] + [[package]] name = "fsspec" version = "2026.4.0" @@ -326,6 +735,11 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d5/0c/043d5e551459da400957a1395e0febbf771446ff34291afcbe3d8be2a279/fsspec-2026.4.0-py3-none-any.whl", hash = "sha256:11ef7bb35dab8a394fde6e608221d5cf3e8499401c249bebaeaad760a1a8dec2", size = 203402, upload-time = "2026-04-29T20:42:36.842Z" }, ] +[package.optional-dependencies] +http = [ + { name = "aiohttp" }, +] + [[package]] name = "gradio" version = "6.16.0" @@ -527,6 +941,7 @@ dependencies = [ { name = "accelerate" }, { name = "huggingface-hub" }, { name = "llama-cpp-python" }, + { name = "peft" }, { name = "pyyaml" }, { name = "torch" }, { name = "torchvision" }, @@ -538,6 +953,7 @@ requires-dist = [ { name = "accelerate", specifier = ">=1.2.0" }, { name = "huggingface-hub", specifier = ">=0.27.0" }, { name = "llama-cpp-python", specifier = ">=0.3.0" }, + { name = "peft", specifier = ">=0.14.0" }, { name = "pyyaml", specifier = ">=6.0.2" }, { name = "torch", specifier = ">=2.5.0" }, { name = "torchvision", specifier = ">=0.20.0" }, @@ -565,6 +981,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/62/a1/3d680cbfd5f4b8f15abc1d571870c5fc3e594bb582bc3b64ea099db13e56/jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", size = 134899, upload-time = "2025-03-05T20:05:00.369Z" }, ] +[[package]] +name = "joblib" +version = "1.5.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/41/f2/d34e8b3a08a9cc79a50b2208a93dce981fe615b64d5a4d4abee421d898df/joblib-1.5.3.tar.gz", hash = "sha256:8561a3269e6801106863fd0d6d84bb737be9e7631e33aaed3fb9ce5953688da3", size = 331603, upload-time = "2025-12-15T08:41:46.427Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7b/91/984aca2ec129e2757d1e4e3c81c3fcda9d0f85b74670a094cc443d9ee949/joblib-1.5.3-py3-none-any.whl", hash = "sha256:5fc3c5039fc5ca8c0276333a188bbd59d6b7ab37fe6632daa76bc7f9ec18e713", size = 309071, upload-time = "2025-12-15T08:41:44.973Z" }, +] + [[package]] name = "llama-cpp-python" version = "0.3.26" @@ -577,6 +1002,39 @@ dependencies = [ ] sdist = { url = "https://files.pythonhosted.org/packages/6e/1a/4cf485fccebb768cb07b54a497335127235852e2d9a50b219e754c7a4c8d/llama_cpp_python-0.3.26.tar.gz", hash = "sha256:71b004d52a274273ae03ec954d62771d26c86f32193fd30be57601508bec0064", size = 68011908, upload-time = "2026-06-05T02:14:43.27Z" } +[[package]] +name = "lm-eval" +version = "0.4.12" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "datasets" }, + { name = "dill" }, + { name = "evaluate" }, + { name = "jinja2" }, + { name = "more-itertools" }, + { name = "numpy" }, + { name = "pytablewriter" }, + { name = "rouge-score" }, + { name = "sacrebleu" }, + { name = "scikit-learn" }, + { name = "sqlitedict" }, + { name = "tqdm" }, + { name = "typing-extensions" }, + { name = "word2number" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/12/32/783623b1d1fe7c89ebd774db46bef4c787fe93e7c5e1b4877c7dfbf1abbe/lm_eval-0.4.12.tar.gz", hash = "sha256:45938d7c8024d5eee5c8a0da36abe7cc09c252413c577cf81666f0fcd28b4345", size = 3360517, upload-time = "2026-05-11T13:05:52.297Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/bd/43/944724465ed73ec2889f58007c114ce8dd2023324bc86d3e6f41ce00e2e8/lm_eval-0.4.12-py3-none-any.whl", hash = "sha256:02971ff68284dd14cfa7fce9310a58452c4162e8d413ba96aa7988a0ff9352ef", size = 8903672, upload-time = "2026-05-11T13:05:48.902Z" }, +] + +[package.optional-dependencies] +hf = [ + { name = "accelerate" }, + { name = "peft" }, + { name = "torch" }, + { name = "transformers" }, +] + [[package]] name = "lxml" version = "6.1.1" @@ -732,6 +1190,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/70/bc/6f1c2f612465f5fa89b95bead1f44dcb607670fd42891d8fdcd5d039f4f4/markupsafe-3.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:32001d6a8fc98c8cb5c947787c5d08b0a50663d139f1305bac5885d98d9b40fa", size = 14146, upload-time = "2025-09-27T18:37:28.327Z" }, ] +[[package]] +name = "mbstrdecoder" +version = "1.1.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "chardet" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/4b/9c/dd6e38d747a62ead27f9abef32f4ca4311d4e40ac28e76bcc9ffb5dd0329/mbstrdecoder-1.1.5.tar.gz", hash = "sha256:8cbfba26938befd8a35e3cc06ca0632f61320b7b2be7df32550b895e1725b1ce", size = 14529, upload-time = "2026-05-05T04:17:58.23Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4c/eb/711270faab7b7df702339a2c68b31fd3ed4fffc68b0e99e5bdf49b1e87e4/mbstrdecoder-1.1.5-py3-none-any.whl", hash = "sha256:4a50fe113d4abecfd86e8f716b2e413cce03d63af83ec3c7cdbe81dec0e519ed", size = 7966, upload-time = "2026-05-05T04:17:56.78Z" }, +] + [[package]] name = "mdurl" version = "0.1.2" @@ -741,6 +1211,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl", hash = "sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8", size = 9979, upload-time = "2022-08-14T12:40:09.779Z" }, ] +[[package]] +name = "more-itertools" +version = "11.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/de/1d/f4da6f02cdffe04d6362210b807146a26044c88d839208aec273bb0d9184/more_itertools-11.1.0.tar.gz", hash = "sha256:48e8f4d9e7e5878571ecf6f2b4e57634f93cd474cc8cfbd2376f2d11b396e30d", size = 145772, upload-time = "2026-05-22T14:14:29.909Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e8/3d/1087453384dbde46a8c7f9356eead2c58be8a7bf156bca40243377c85715/more_itertools-11.1.0-py3-none-any.whl", hash = "sha256:4b65538ae22f6fed0ce4874efd317463a7489796a0939fa66824dd542125a192", size = 72226, upload-time = "2026-05-22T14:14:28.824Z" }, +] + [[package]] name = "mpmath" version = "1.3.0" @@ -750,6 +1229,131 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl", hash = "sha256:a0b2b9fe80bbcd81a6647ff13108738cfb482d481d826cc0e02f5b35e5c88d2c", size = 536198, upload-time = "2023-03-07T16:47:09.197Z" }, ] +[[package]] +name = "multidict" +version = "6.7.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/1a/c2/c2d94cbe6ac1753f3fc980da97b3d930efe1da3af3c9f5125354436c073d/multidict-6.7.1.tar.gz", hash = "sha256:ec6652a1bee61c53a3e5776b6049172c53b6aaba34f18c9ad04f82712bac623d", size = 102010, upload-time = "2026-01-26T02:46:45.979Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/8d/9c/f20e0e2cf80e4b2e4b1c365bf5fe104ee633c751a724246262db8f1a0b13/multidict-6.7.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:a90f75c956e32891a4eda3639ce6dd86e87105271f43d43442a3aedf3cddf172", size = 76893, upload-time = "2026-01-26T02:43:52.754Z" }, + { url = "https://files.pythonhosted.org/packages/fe/cf/18ef143a81610136d3da8193da9d80bfe1cb548a1e2d1c775f26b23d024a/multidict-6.7.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:3fccb473e87eaa1382689053e4a4618e7ba7b9b9b8d6adf2027ee474597128cd", size = 45456, upload-time = "2026-01-26T02:43:53.893Z" }, + { url = "https://files.pythonhosted.org/packages/a9/65/1caac9d4cd32e8433908683446eebc953e82d22b03d10d41a5f0fefe991b/multidict-6.7.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:b0fa96985700739c4c7853a43c0b3e169360d6855780021bfc6d0f1ce7c123e7", size = 43872, upload-time = "2026-01-26T02:43:55.041Z" }, + { url = "https://files.pythonhosted.org/packages/cf/3b/d6bd75dc4f3ff7c73766e04e705b00ed6dbbaccf670d9e05a12b006f5a21/multidict-6.7.1-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:cb2a55f408c3043e42b40cc8eecd575afa27b7e0b956dfb190de0f8499a57a53", size = 251018, upload-time = "2026-01-26T02:43:56.198Z" }, + { url = "https://files.pythonhosted.org/packages/fd/80/c959c5933adedb9ac15152e4067c702a808ea183a8b64cf8f31af8ad3155/multidict-6.7.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:eb0ce7b2a32d09892b3dd6cc44877a0d02a33241fafca5f25c8b6b62374f8b75", size = 258883, upload-time = "2026-01-26T02:43:57.499Z" }, + { url = "https://files.pythonhosted.org/packages/86/85/7ed40adafea3d4f1c8b916e3b5cc3a8e07dfcdcb9cd72800f4ed3ca1b387/multidict-6.7.1-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:c3a32d23520ee37bf327d1e1a656fec76a2edd5c038bf43eddfa0572ec49c60b", size = 242413, upload-time = "2026-01-26T02:43:58.755Z" }, + { url = "https://files.pythonhosted.org/packages/d2/57/b8565ff533e48595503c785f8361ff9a4fde4d67de25c207cd0ba3befd03/multidict-6.7.1-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9c90fed18bffc0189ba814749fdcc102b536e83a9f738a9003e569acd540a733", size = 268404, upload-time = "2026-01-26T02:44:00.216Z" }, + { url = "https://files.pythonhosted.org/packages/e0/50/9810c5c29350f7258180dfdcb2e52783a0632862eb334c4896ac717cebcb/multidict-6.7.1-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:da62917e6076f512daccfbbde27f46fed1c98fee202f0559adec8ee0de67f71a", size = 269456, upload-time = "2026-01-26T02:44:02.202Z" }, + { url = "https://files.pythonhosted.org/packages/f3/8d/5e5be3ced1d12966fefb5c4ea3b2a5b480afcea36406559442c6e31d4a48/multidict-6.7.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bfde23ef6ed9db7eaee6c37dcec08524cb43903c60b285b172b6c094711b3961", size = 256322, upload-time = "2026-01-26T02:44:03.56Z" }, + { url = "https://files.pythonhosted.org/packages/31/6e/d8a26d81ac166a5592782d208dd90dfdc0a7a218adaa52b45a672b46c122/multidict-6.7.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:3758692429e4e32f1ba0df23219cd0b4fc0a52f476726fff9337d1a57676a582", size = 253955, upload-time = "2026-01-26T02:44:04.845Z" }, + { url = "https://files.pythonhosted.org/packages/59/4c/7c672c8aad41534ba619bcd4ade7a0dc87ed6b8b5c06149b85d3dd03f0cd/multidict-6.7.1-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:398c1478926eca669f2fd6a5856b6de9c0acf23a2cb59a14c0ba5844fa38077e", size = 251254, upload-time = "2026-01-26T02:44:06.133Z" }, + { url = "https://files.pythonhosted.org/packages/7b/bd/84c24de512cbafbdbc39439f74e967f19570ce7924e3007174a29c348916/multidict-6.7.1-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:c102791b1c4f3ab36ce4101154549105a53dc828f016356b3e3bcae2e3a039d3", size = 252059, upload-time = "2026-01-26T02:44:07.518Z" }, + { url = "https://files.pythonhosted.org/packages/fa/ba/f5449385510825b73d01c2d4087bf6d2fccc20a2d42ac34df93191d3dd03/multidict-6.7.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:a088b62bd733e2ad12c50dad01b7d0166c30287c166e137433d3b410add807a6", size = 263588, upload-time = "2026-01-26T02:44:09.382Z" }, + { url = "https://files.pythonhosted.org/packages/d7/11/afc7c677f68f75c84a69fe37184f0f82fce13ce4b92f49f3db280b7e92b3/multidict-6.7.1-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:3d51ff4785d58d3f6c91bdbffcb5e1f7ddfda557727043aa20d20ec4f65e324a", size = 259642, upload-time = "2026-01-26T02:44:10.73Z" }, + { url = "https://files.pythonhosted.org/packages/2b/17/ebb9644da78c4ab36403739e0e6e0e30ebb135b9caf3440825001a0bddcb/multidict-6.7.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:fc5907494fccf3e7d3f94f95c91d6336b092b5fc83811720fae5e2765890dfba", size = 251377, upload-time = "2026-01-26T02:44:12.042Z" }, + { url = "https://files.pythonhosted.org/packages/ca/a4/840f5b97339e27846c46307f2530a2805d9d537d8b8bd416af031cad7fa0/multidict-6.7.1-cp312-cp312-win32.whl", hash = "sha256:28ca5ce2fd9716631133d0e9a9b9a745ad7f60bac2bccafb56aa380fc0b6c511", size = 41887, upload-time = "2026-01-26T02:44:14.245Z" }, + { url = "https://files.pythonhosted.org/packages/80/31/0b2517913687895f5904325c2069d6a3b78f66cc641a86a2baf75a05dcbb/multidict-6.7.1-cp312-cp312-win_amd64.whl", hash = "sha256:fcee94dfbd638784645b066074b338bc9cc155d4b4bffa4adce1615c5a426c19", size = 46053, upload-time = "2026-01-26T02:44:15.371Z" }, + { url = "https://files.pythonhosted.org/packages/0c/5b/aba28e4ee4006ae4c7df8d327d31025d760ffa992ea23812a601d226e682/multidict-6.7.1-cp312-cp312-win_arm64.whl", hash = "sha256:ba0a9fb644d0c1a2194cf7ffb043bd852cea63a57f66fbd33959f7dae18517bf", size = 43307, upload-time = "2026-01-26T02:44:16.852Z" }, + { url = "https://files.pythonhosted.org/packages/f2/22/929c141d6c0dba87d3e1d38fbdf1ba8baba86b7776469f2bc2d3227a1e67/multidict-6.7.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:2b41f5fed0ed563624f1c17630cb9941cf2309d4df00e494b551b5f3e3d67a23", size = 76174, upload-time = "2026-01-26T02:44:18.509Z" }, + { url = "https://files.pythonhosted.org/packages/c7/75/bc704ae15fee974f8fccd871305e254754167dce5f9e42d88a2def741a1d/multidict-6.7.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:84e61e3af5463c19b67ced91f6c634effb89ef8bfc5ca0267f954451ed4bb6a2", size = 45116, upload-time = "2026-01-26T02:44:19.745Z" }, + { url = "https://files.pythonhosted.org/packages/79/76/55cd7186f498ed080a18440c9013011eb548f77ae1b297206d030eb1180a/multidict-6.7.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:935434b9853c7c112eee7ac891bc4cb86455aa631269ae35442cb316790c1445", size = 43524, upload-time = "2026-01-26T02:44:21.571Z" }, + { url = "https://files.pythonhosted.org/packages/e9/3c/414842ef8d5a1628d68edee29ba0e5bcf235dbfb3ccd3ea303a7fe8c72ff/multidict-6.7.1-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:432feb25a1cb67fe82a9680b4d65fb542e4635cb3166cd9c01560651ad60f177", size = 249368, upload-time = "2026-01-26T02:44:22.803Z" }, + { url = "https://files.pythonhosted.org/packages/f6/32/befed7f74c458b4a525e60519fe8d87eef72bb1e99924fa2b0f9d97a221e/multidict-6.7.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e82d14e3c948952a1a85503817e038cba5905a3352de76b9a465075d072fba23", size = 256952, upload-time = "2026-01-26T02:44:24.306Z" }, + { url = "https://files.pythonhosted.org/packages/03/d6/c878a44ba877f366630c860fdf74bfb203c33778f12b6ac274936853c451/multidict-6.7.1-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:4cfb48c6ea66c83bcaaf7e4dfa7ec1b6bbcf751b7db85a328902796dfde4c060", size = 240317, upload-time = "2026-01-26T02:44:25.772Z" }, + { url = "https://files.pythonhosted.org/packages/68/49/57421b4d7ad2e9e60e25922b08ceb37e077b90444bde6ead629095327a6f/multidict-6.7.1-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:1d540e51b7e8e170174555edecddbd5538105443754539193e3e1061864d444d", size = 267132, upload-time = "2026-01-26T02:44:27.648Z" }, + { url = "https://files.pythonhosted.org/packages/b7/fe/ec0edd52ddbcea2a2e89e174f0206444a61440b40f39704e64dc807a70bd/multidict-6.7.1-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:273d23f4b40f3dce4d6c8a821c741a86dec62cded82e1175ba3d99be128147ed", size = 268140, upload-time = "2026-01-26T02:44:29.588Z" }, + { url = "https://files.pythonhosted.org/packages/b0/73/6e1b01cbeb458807aa0831742232dbdd1fa92bfa33f52a3f176b4ff3dc11/multidict-6.7.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9d624335fd4fa1c08a53f8b4be7676ebde19cd092b3895c421045ca87895b429", size = 254277, upload-time = "2026-01-26T02:44:30.902Z" }, + { url = "https://files.pythonhosted.org/packages/6a/b2/5fb8c124d7561a4974c342bc8c778b471ebbeb3cc17df696f034a7e9afe7/multidict-6.7.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:12fad252f8b267cc75b66e8fc51b3079604e8d43a75428ffe193cd9e2195dfd6", size = 252291, upload-time = "2026-01-26T02:44:32.31Z" }, + { url = "https://files.pythonhosted.org/packages/5a/96/51d4e4e06bcce92577fcd488e22600bd38e4fd59c20cb49434d054903bd2/multidict-6.7.1-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:03ede2a6ffbe8ef936b92cb4529f27f42be7f56afcdab5ab739cd5f27fb1cbf9", size = 250156, upload-time = "2026-01-26T02:44:33.734Z" }, + { url = "https://files.pythonhosted.org/packages/db/6b/420e173eec5fba721a50e2a9f89eda89d9c98fded1124f8d5c675f7a0c0f/multidict-6.7.1-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:90efbcf47dbe33dcf643a1e400d67d59abeac5db07dc3f27d6bdeae497a2198c", size = 249742, upload-time = "2026-01-26T02:44:35.222Z" }, + { url = "https://files.pythonhosted.org/packages/44/a3/ec5b5bd98f306bc2aa297b8c6f11a46714a56b1e6ef5ebda50a4f5d7c5fb/multidict-6.7.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:5c4b9bfc148f5a91be9244d6264c53035c8a0dcd2f51f1c3c6e30e30ebaa1c84", size = 262221, upload-time = "2026-01-26T02:44:36.604Z" }, + { url = "https://files.pythonhosted.org/packages/cd/f7/e8c0d0da0cd1e28d10e624604e1a36bcc3353aaebdfdc3a43c72bc683a12/multidict-6.7.1-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:401c5a650f3add2472d1d288c26deebc540f99e2fb83e9525007a74cd2116f1d", size = 258664, upload-time = "2026-01-26T02:44:38.008Z" }, + { url = "https://files.pythonhosted.org/packages/52/da/151a44e8016dd33feed44f730bd856a66257c1ee7aed4f44b649fb7edeb3/multidict-6.7.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:97891f3b1b3ffbded884e2916cacf3c6fc87b66bb0dde46f7357404750559f33", size = 249490, upload-time = "2026-01-26T02:44:39.386Z" }, + { url = "https://files.pythonhosted.org/packages/87/af/a3b86bf9630b732897f6fc3f4c4714b90aa4361983ccbdcd6c0339b21b0c/multidict-6.7.1-cp313-cp313-win32.whl", hash = "sha256:e1c5988359516095535c4301af38d8a8838534158f649c05dd1050222321bcb3", size = 41695, upload-time = "2026-01-26T02:44:41.318Z" }, + { url = "https://files.pythonhosted.org/packages/b2/35/e994121b0e90e46134673422dd564623f93304614f5d11886b1b3e06f503/multidict-6.7.1-cp313-cp313-win_amd64.whl", hash = "sha256:960c83bf01a95b12b08fd54324a4eb1d5b52c88932b5cba5d6e712bb3ed12eb5", size = 45884, upload-time = "2026-01-26T02:44:42.488Z" }, + { url = "https://files.pythonhosted.org/packages/ca/61/42d3e5dbf661242a69c97ea363f2d7b46c567da8eadef8890022be6e2ab0/multidict-6.7.1-cp313-cp313-win_arm64.whl", hash = "sha256:563fe25c678aaba333d5399408f5ec3c383ca5b663e7f774dd179a520b8144df", size = 43122, upload-time = "2026-01-26T02:44:43.664Z" }, + { url = "https://files.pythonhosted.org/packages/6d/b3/e6b21c6c4f314bb956016b0b3ef2162590a529b84cb831c257519e7fde44/multidict-6.7.1-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:c76c4bec1538375dad9d452d246ca5368ad6e1c9039dadcf007ae59c70619ea1", size = 83175, upload-time = "2026-01-26T02:44:44.894Z" }, + { url = "https://files.pythonhosted.org/packages/fb/76/23ecd2abfe0957b234f6c960f4ade497f55f2c16aeb684d4ecdbf1c95791/multidict-6.7.1-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:57b46b24b5d5ebcc978da4ec23a819a9402b4228b8a90d9c656422b4bdd8a963", size = 48460, upload-time = "2026-01-26T02:44:46.106Z" }, + { url = "https://files.pythonhosted.org/packages/c4/57/a0ed92b23f3a042c36bc4227b72b97eca803f5f1801c1ab77c8a212d455e/multidict-6.7.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:e954b24433c768ce78ab7929e84ccf3422e46deb45a4dc9f93438f8217fa2d34", size = 46930, upload-time = "2026-01-26T02:44:47.278Z" }, + { url = "https://files.pythonhosted.org/packages/b5/66/02ec7ace29162e447f6382c495dc95826bf931d3818799bbef11e8f7df1a/multidict-6.7.1-cp313-cp313t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:3bd231490fa7217cc832528e1cd8752a96f0125ddd2b5749390f7c3ec8721b65", size = 242582, upload-time = "2026-01-26T02:44:48.604Z" }, + { url = "https://files.pythonhosted.org/packages/58/18/64f5a795e7677670e872673aca234162514696274597b3708b2c0d276cce/multidict-6.7.1-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:253282d70d67885a15c8a7716f3a73edf2d635793ceda8173b9ecc21f2fb8292", size = 250031, upload-time = "2026-01-26T02:44:50.544Z" }, + { url = "https://files.pythonhosted.org/packages/c8/ed/e192291dbbe51a8290c5686f482084d31bcd9d09af24f63358c3d42fd284/multidict-6.7.1-cp313-cp313t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:0b4c48648d7649c9335cf1927a8b87fa692de3dcb15faa676c6a6f1f1aabda43", size = 228596, upload-time = "2026-01-26T02:44:51.951Z" }, + { url = "https://files.pythonhosted.org/packages/1e/7e/3562a15a60cf747397e7f2180b0a11dc0c38d9175a650e75fa1b4d325e15/multidict-6.7.1-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:98bc624954ec4d2c7cb074b8eefc2b5d0ce7d482e410df446414355d158fe4ca", size = 257492, upload-time = "2026-01-26T02:44:53.902Z" }, + { url = "https://files.pythonhosted.org/packages/24/02/7d0f9eae92b5249bb50ac1595b295f10e263dd0078ebb55115c31e0eaccd/multidict-6.7.1-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:1b99af4d9eec0b49927b4402bcbb58dea89d3e0db8806a4086117019939ad3dd", size = 255899, upload-time = "2026-01-26T02:44:55.316Z" }, + { url = "https://files.pythonhosted.org/packages/00/e3/9b60ed9e23e64c73a5cde95269ef1330678e9c6e34dd4eb6b431b85b5a10/multidict-6.7.1-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6aac4f16b472d5b7dc6f66a0d49dd57b0e0902090be16594dc9ebfd3d17c47e7", size = 247970, upload-time = "2026-01-26T02:44:56.783Z" }, + { url = "https://files.pythonhosted.org/packages/3e/06/538e58a63ed5cfb0bd4517e346b91da32fde409d839720f664e9a4ae4f9d/multidict-6.7.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:21f830fe223215dffd51f538e78c172ed7c7f60c9b96a2bf05c4848ad49921c3", size = 245060, upload-time = "2026-01-26T02:44:58.195Z" }, + { url = "https://files.pythonhosted.org/packages/b2/2f/d743a3045a97c895d401e9bd29aaa09b94f5cbdf1bd561609e5a6c431c70/multidict-6.7.1-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:f5dd81c45b05518b9aa4da4aa74e1c93d715efa234fd3e8a179df611cc85e5f4", size = 235888, upload-time = "2026-01-26T02:44:59.57Z" }, + { url = "https://files.pythonhosted.org/packages/38/83/5a325cac191ab28b63c52f14f1131f3b0a55ba3b9aa65a6d0bf2a9b921a0/multidict-6.7.1-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:eb304767bca2bb92fb9c5bd33cedc95baee5bb5f6c88e63706533a1c06ad08c8", size = 243554, upload-time = "2026-01-26T02:45:01.054Z" }, + { url = "https://files.pythonhosted.org/packages/20/1f/9d2327086bd15da2725ef6aae624208e2ef828ed99892b17f60c344e57ed/multidict-6.7.1-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:c9035dde0f916702850ef66460bc4239d89d08df4d02023a5926e7446724212c", size = 252341, upload-time = "2026-01-26T02:45:02.484Z" }, + { url = "https://files.pythonhosted.org/packages/e8/2c/2a1aa0280cf579d0f6eed8ee5211c4f1730bd7e06c636ba2ee6aafda302e/multidict-6.7.1-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:af959b9beeb66c822380f222f0e0a1889331597e81f1ded7f374f3ecb0fd6c52", size = 246391, upload-time = "2026-01-26T02:45:03.862Z" }, + { url = "https://files.pythonhosted.org/packages/e5/03/7ca022ffc36c5a3f6e03b179a5ceb829be9da5783e6fe395f347c0794680/multidict-6.7.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:41f2952231456154ee479651491e94118229844dd7226541788be783be2b5108", size = 243422, upload-time = "2026-01-26T02:45:05.296Z" }, + { url = "https://files.pythonhosted.org/packages/dc/1d/b31650eab6c5778aceed46ba735bd97f7c7d2f54b319fa916c0f96e7805b/multidict-6.7.1-cp313-cp313t-win32.whl", hash = "sha256:df9f19c28adcb40b6aae30bbaa1478c389efd50c28d541d76760199fc1037c32", size = 47770, upload-time = "2026-01-26T02:45:06.754Z" }, + { url = "https://files.pythonhosted.org/packages/ac/5b/2d2d1d522e51285bd61b1e20df8f47ae1a9d80839db0b24ea783b3832832/multidict-6.7.1-cp313-cp313t-win_amd64.whl", hash = "sha256:d54ecf9f301853f2c5e802da559604b3e95bb7a3b01a9c295c6ee591b9882de8", size = 53109, upload-time = "2026-01-26T02:45:08.044Z" }, + { url = "https://files.pythonhosted.org/packages/3d/a3/cc409ba012c83ca024a308516703cf339bdc4b696195644a7215a5164a24/multidict-6.7.1-cp313-cp313t-win_arm64.whl", hash = "sha256:5a37ca18e360377cfda1d62f5f382ff41f2b8c4ccb329ed974cc2e1643440118", size = 45573, upload-time = "2026-01-26T02:45:09.349Z" }, + { url = "https://files.pythonhosted.org/packages/91/cc/db74228a8be41884a567e88a62fd589a913708fcf180d029898c17a9a371/multidict-6.7.1-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:8f333ec9c5eb1b7105e3b84b53141e66ca05a19a605368c55450b6ba208cb9ee", size = 75190, upload-time = "2026-01-26T02:45:10.651Z" }, + { url = "https://files.pythonhosted.org/packages/d5/22/492f2246bb5b534abd44804292e81eeaf835388901f0c574bac4eeec73c5/multidict-6.7.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:a407f13c188f804c759fc6a9f88286a565c242a76b27626594c133b82883b5c2", size = 44486, upload-time = "2026-01-26T02:45:11.938Z" }, + { url = "https://files.pythonhosted.org/packages/f1/4f/733c48f270565d78b4544f2baddc2fb2a245e5a8640254b12c36ac7ac68e/multidict-6.7.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:0e161ddf326db5577c3a4cc2d8648f81456e8a20d40415541587a71620d7a7d1", size = 43219, upload-time = "2026-01-26T02:45:14.346Z" }, + { url = "https://files.pythonhosted.org/packages/24/bb/2c0c2287963f4259c85e8bcbba9182ced8d7fca65c780c38e99e61629d11/multidict-6.7.1-cp314-cp314-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:1e3a8bb24342a8201d178c3b4984c26ba81a577c80d4d525727427460a50c22d", size = 245132, upload-time = "2026-01-26T02:45:15.712Z" }, + { url = "https://files.pythonhosted.org/packages/a7/f9/44d4b3064c65079d2467888794dea218d1601898ac50222ab8a9a8094460/multidict-6.7.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:97231140a50f5d447d3164f994b86a0bed7cd016e2682f8650d6a9158e14fd31", size = 252420, upload-time = "2026-01-26T02:45:17.293Z" }, + { url = "https://files.pythonhosted.org/packages/8b/13/78f7275e73fa17b24c9a51b0bd9d73ba64bb32d0ed51b02a746eb876abe7/multidict-6.7.1-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:6b10359683bd8806a200fd2909e7c8ca3a7b24ec1d8132e483d58e791d881048", size = 233510, upload-time = "2026-01-26T02:45:19.356Z" }, + { url = "https://files.pythonhosted.org/packages/4b/25/8167187f62ae3cbd52da7893f58cb036b47ea3fb67138787c76800158982/multidict-6.7.1-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:283ddac99f7ac25a4acadbf004cb5ae34480bbeb063520f70ce397b281859362", size = 264094, upload-time = "2026-01-26T02:45:20.834Z" }, + { url = "https://files.pythonhosted.org/packages/a1/e7/69a3a83b7b030cf283fb06ce074a05a02322359783424d7edf0f15fe5022/multidict-6.7.1-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:538cec1e18c067d0e6103aa9a74f9e832904c957adc260e61cd9d8cf0c3b3d37", size = 260786, upload-time = "2026-01-26T02:45:22.818Z" }, + { url = "https://files.pythonhosted.org/packages/fe/3b/8ec5074bcfc450fe84273713b4b0a0dd47c0249358f5d82eb8104ffe2520/multidict-6.7.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7eee46ccb30ff48a1e35bb818cc90846c6be2b68240e42a78599166722cea709", size = 248483, upload-time = "2026-01-26T02:45:24.368Z" }, + { url = "https://files.pythonhosted.org/packages/48/5a/d5a99e3acbca0e29c5d9cba8f92ceb15dce78bab963b308ae692981e3a5d/multidict-6.7.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:fa263a02f4f2dd2d11a7b1bb4362aa7cb1049f84a9235d31adf63f30143469a0", size = 248403, upload-time = "2026-01-26T02:45:25.982Z" }, + { url = "https://files.pythonhosted.org/packages/35/48/e58cd31f6c7d5102f2a4bf89f96b9cf7e00b6c6f3d04ecc44417c00a5a3c/multidict-6.7.1-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:2e1425e2f99ec5bd36c15a01b690a1a2456209c5deed58f95469ffb46039ccbb", size = 240315, upload-time = "2026-01-26T02:45:27.487Z" }, + { url = "https://files.pythonhosted.org/packages/94/33/1cd210229559cb90b6786c30676bb0c58249ff42f942765f88793b41fdce/multidict-6.7.1-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:497394b3239fc6f0e13a78a3e1b61296e72bf1c5f94b4c4eb80b265c37a131cd", size = 245528, upload-time = "2026-01-26T02:45:28.991Z" }, + { url = "https://files.pythonhosted.org/packages/64/f2/6e1107d226278c876c783056b7db43d800bb64c6131cec9c8dfb6903698e/multidict-6.7.1-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:233b398c29d3f1b9676b4b6f75c518a06fcb2ea0b925119fb2c1bc35c05e1601", size = 258784, upload-time = "2026-01-26T02:45:30.503Z" }, + { url = "https://files.pythonhosted.org/packages/4d/c1/11f664f14d525e4a1b5327a82d4de61a1db604ab34c6603bb3c2cc63ad34/multidict-6.7.1-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:93b1818e4a6e0930454f0f2af7dfce69307ca03cdcfb3739bf4d91241967b6c1", size = 251980, upload-time = "2026-01-26T02:45:32.603Z" }, + { url = "https://files.pythonhosted.org/packages/e1/9f/75a9ac888121d0c5bbd4ecf4eead45668b1766f6baabfb3b7f66a410e231/multidict-6.7.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:f33dc2a3abe9249ea5d8360f969ec7f4142e7ac45ee7014d8f8d5acddf178b7b", size = 243602, upload-time = "2026-01-26T02:45:34.043Z" }, + { url = "https://files.pythonhosted.org/packages/9a/e7/50bf7b004cc8525d80dbbbedfdc7aed3e4c323810890be4413e589074032/multidict-6.7.1-cp314-cp314-win32.whl", hash = "sha256:3ab8b9d8b75aef9df299595d5388b14530839f6422333357af1339443cff777d", size = 40930, upload-time = "2026-01-26T02:45:36.278Z" }, + { url = "https://files.pythonhosted.org/packages/e0/bf/52f25716bbe93745595800f36fb17b73711f14da59ed0bb2eba141bc9f0f/multidict-6.7.1-cp314-cp314-win_amd64.whl", hash = "sha256:5e01429a929600e7dab7b166062d9bb54a5eed752384c7384c968c2afab8f50f", size = 45074, upload-time = "2026-01-26T02:45:37.546Z" }, + { url = "https://files.pythonhosted.org/packages/97/ab/22803b03285fa3a525f48217963da3a65ae40f6a1b6f6cf2768879e208f9/multidict-6.7.1-cp314-cp314-win_arm64.whl", hash = "sha256:4885cb0e817aef5d00a2e8451d4665c1808378dc27c2705f1bf4ef8505c0d2e5", size = 42471, upload-time = "2026-01-26T02:45:38.889Z" }, + { url = "https://files.pythonhosted.org/packages/e0/6d/f9293baa6146ba9507e360ea0292b6422b016907c393e2f63fc40ab7b7b5/multidict-6.7.1-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:0458c978acd8e6ea53c81eefaddbbee9c6c5e591f41b3f5e8e194780fe026581", size = 82401, upload-time = "2026-01-26T02:45:40.254Z" }, + { url = "https://files.pythonhosted.org/packages/7a/68/53b5494738d83558d87c3c71a486504d8373421c3e0dbb6d0db48ad42ee0/multidict-6.7.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:c0abd12629b0af3cf590982c0b413b1e7395cd4ec026f30986818ab95bfaa94a", size = 48143, upload-time = "2026-01-26T02:45:41.635Z" }, + { url = "https://files.pythonhosted.org/packages/37/e8/5284c53310dcdc99ce5d66563f6e5773531a9b9fe9ec7a615e9bc306b05f/multidict-6.7.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:14525a5f61d7d0c94b368a42cff4c9a4e7ba2d52e2672a7b23d84dc86fb02b0c", size = 46507, upload-time = "2026-01-26T02:45:42.99Z" }, + { url = "https://files.pythonhosted.org/packages/e4/fc/6800d0e5b3875568b4083ecf5f310dcf91d86d52573160834fb4bfcf5e4f/multidict-6.7.1-cp314-cp314t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:17307b22c217b4cf05033dabefe68255a534d637c6c9b0cc8382718f87be4262", size = 239358, upload-time = "2026-01-26T02:45:44.376Z" }, + { url = "https://files.pythonhosted.org/packages/41/75/4ad0973179361cdf3a113905e6e088173198349131be2b390f9fa4da5fc6/multidict-6.7.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7a7e590ff876a3eaf1c02a4dfe0724b6e69a9e9de6d8f556816f29c496046e59", size = 246884, upload-time = "2026-01-26T02:45:47.167Z" }, + { url = "https://files.pythonhosted.org/packages/c3/9c/095bb28b5da139bd41fb9a5d5caff412584f377914bd8787c2aa98717130/multidict-6.7.1-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:5fa6a95dfee63893d80a34758cd0e0c118a30b8dcb46372bf75106c591b77889", size = 225878, upload-time = "2026-01-26T02:45:48.698Z" }, + { url = "https://files.pythonhosted.org/packages/07/d0/c0a72000243756e8f5a277b6b514fa005f2c73d481b7d9e47cd4568aa2e4/multidict-6.7.1-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a0543217a6a017692aa6ae5cc39adb75e587af0f3a82288b1492eb73dd6cc2a4", size = 253542, upload-time = "2026-01-26T02:45:50.164Z" }, + { url = "https://files.pythonhosted.org/packages/c0/6b/f69da15289e384ecf2a68837ec8b5ad8c33e973aa18b266f50fe55f24b8c/multidict-6.7.1-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:f99fe611c312b3c1c0ace793f92464d8cd263cc3b26b5721950d977b006b6c4d", size = 252403, upload-time = "2026-01-26T02:45:51.779Z" }, + { url = "https://files.pythonhosted.org/packages/a2/76/b9669547afa5a1a25cd93eaca91c0da1c095b06b6d2d8ec25b713588d3a1/multidict-6.7.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9004d8386d133b7e6135679424c91b0b854d2d164af6ea3f289f8f2761064609", size = 244889, upload-time = "2026-01-26T02:45:53.27Z" }, + { url = "https://files.pythonhosted.org/packages/7e/a9/a50d2669e506dad33cfc45b5d574a205587b7b8a5f426f2fbb2e90882588/multidict-6.7.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:e628ef0e6859ffd8273c69412a2465c4be4a9517d07261b33334b5ec6f3c7489", size = 241982, upload-time = "2026-01-26T02:45:54.919Z" }, + { url = "https://files.pythonhosted.org/packages/c5/bb/1609558ad8b456b4827d3c5a5b775c93b87878fd3117ed3db3423dfbce1b/multidict-6.7.1-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:841189848ba629c3552035a6a7f5bf3b02eb304e9fea7492ca220a8eda6b0e5c", size = 232415, upload-time = "2026-01-26T02:45:56.981Z" }, + { url = "https://files.pythonhosted.org/packages/d8/59/6f61039d2aa9261871e03ab9dc058a550d240f25859b05b67fd70f80d4b3/multidict-6.7.1-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:ce1bbd7d780bb5a0da032e095c951f7014d6b0a205f8318308140f1a6aba159e", size = 240337, upload-time = "2026-01-26T02:45:58.698Z" }, + { url = "https://files.pythonhosted.org/packages/a1/29/fdc6a43c203890dc2ae9249971ecd0c41deaedfe00d25cb6564b2edd99eb/multidict-6.7.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:b26684587228afed0d50cf804cc71062cc9c1cdf55051c4c6345d372947b268c", size = 248788, upload-time = "2026-01-26T02:46:00.862Z" }, + { url = "https://files.pythonhosted.org/packages/a9/14/a153a06101323e4cf086ecee3faadba52ff71633d471f9685c42e3736163/multidict-6.7.1-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:9f9af11306994335398293f9958071019e3ab95e9a707dc1383a35613f6abcb9", size = 242842, upload-time = "2026-01-26T02:46:02.824Z" }, + { url = "https://files.pythonhosted.org/packages/41/5f/604ae839e64a4a6efc80db94465348d3b328ee955e37acb24badbcd24d83/multidict-6.7.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:b4938326284c4f1224178a560987b6cf8b4d38458b113d9b8c1db1a836e640a2", size = 240237, upload-time = "2026-01-26T02:46:05.898Z" }, + { url = "https://files.pythonhosted.org/packages/5f/60/c3a5187bf66f6fb546ff4ab8fb5a077cbdd832d7b1908d4365c7f74a1917/multidict-6.7.1-cp314-cp314t-win32.whl", hash = "sha256:98655c737850c064a65e006a3df7c997cd3b220be4ec8fe26215760b9697d4d7", size = 48008, upload-time = "2026-01-26T02:46:07.468Z" }, + { url = "https://files.pythonhosted.org/packages/0c/f7/addf1087b860ac60e6f382240f64fb99f8bfb532bb06f7c542b83c29ca61/multidict-6.7.1-cp314-cp314t-win_amd64.whl", hash = "sha256:497bde6223c212ba11d462853cfa4f0ae6ef97465033e7dc9940cdb3ab5b48e5", size = 53542, upload-time = "2026-01-26T02:46:08.809Z" }, + { url = "https://files.pythonhosted.org/packages/4c/81/4629d0aa32302ef7b2ec65c75a728cc5ff4fa410c50096174c1632e70b3e/multidict-6.7.1-cp314-cp314t-win_arm64.whl", hash = "sha256:2bbd113e0d4af5db41d5ebfe9ccaff89de2120578164f86a5d17d5a576d1e5b2", size = 44719, upload-time = "2026-01-26T02:46:11.146Z" }, + { url = "https://files.pythonhosted.org/packages/81/08/7036c080d7117f28a4af526d794aab6a84463126db031b007717c1a6676e/multidict-6.7.1-py3-none-any.whl", hash = "sha256:55d97cc6dae627efa6a6e548885712d4864b81110ac76fa4e534c03819fa4a56", size = 12319, upload-time = "2026-01-26T02:46:44.004Z" }, +] + +[[package]] +name = "multiprocess" +version = "0.70.19" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "dill" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a2/f2/e783ac7f2aeeed14e9e12801f22529cc7e6b7ab80928d6dcce4e9f00922d/multiprocess-0.70.19.tar.gz", hash = "sha256:952021e0e6c55a4a9fe4cd787895b86e239a40e76802a789d6305398d3975897", size = 2079989, upload-time = "2026-01-19T06:47:39.744Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e3/45/8004d1e6b9185c1a444d6b55ac5682acf9d98035e54386d967366035a03a/multiprocess-0.70.19-py310-none-any.whl", hash = "sha256:97404393419dcb2a8385910864eedf47a3cadf82c66345b44f036420eb0b5d87", size = 134948, upload-time = "2026-01-19T06:47:32.325Z" }, + { url = "https://files.pythonhosted.org/packages/86/c2/dec9722dc3474c164a0b6bcd9a7ed7da542c98af8cabce05374abab35edd/multiprocess-0.70.19-py311-none-any.whl", hash = "sha256:928851ae7973aea4ce0eaf330bbdafb2e01398a91518d5c8818802845564f45c", size = 144457, upload-time = "2026-01-19T06:47:33.711Z" }, + { url = "https://files.pythonhosted.org/packages/71/70/38998b950a97ea279e6bd657575d22d1a2047256caf707d9a10fbce4f065/multiprocess-0.70.19-py312-none-any.whl", hash = "sha256:3a56c0e85dd5025161bac5ce138dcac1e49174c7d8e74596537e729fd5c53c28", size = 150281, upload-time = "2026-01-19T06:47:35.037Z" }, + { url = "https://files.pythonhosted.org/packages/7f/74/d2c27e03cb84251dfe7249b8e82923643c6d48fa4883b9476b025e7dc7eb/multiprocess-0.70.19-py313-none-any.whl", hash = "sha256:8d5eb4ec5017ba2fab4e34a747c6d2c2b6fecfe9e7236e77988db91580ada952", size = 156414, upload-time = "2026-01-19T06:47:35.915Z" }, + { url = "https://files.pythonhosted.org/packages/a0/61/af9115673a5870fd885247e2f1b68c4f1197737da315b520a91c757a861a/multiprocess-0.70.19-py314-none-any.whl", hash = "sha256:e8cc7fbdff15c0613f0a1f1f8744bef961b0a164c0ca29bdff53e9d2d93c5e5f", size = 160318, upload-time = "2026-01-19T06:47:37.497Z" }, + { url = "https://files.pythonhosted.org/packages/7e/82/69e539c4c2027f1e1697e09aaa2449243085a0edf81ae2c6341e84d769b6/multiprocess-0.70.19-py39-none-any.whl", hash = "sha256:0d4b4397ed669d371c81dcd1ef33fd384a44d6c3de1bd0ca7ac06d837720d3c5", size = 133477, upload-time = "2026-01-19T06:47:38.619Z" }, +] + +[[package]] +name = "narwhals" +version = "2.22.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/62/3c/c4ef2164a71c1a63d7f1ae411c4082c5fa872405106db60a4b7114989ad7/narwhals-2.22.1.tar.gz", hash = "sha256:d62920805a0a43b7ff8b54b0c0d3142d796f8a9301836ada37e573d6a33cbcd9", size = 647493, upload-time = "2026-06-05T12:34:34.051Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/48/ca/36339329c4604adbcc99c899b7eb1ce1a555c499b6a6860757dc9bfed36d/narwhals-2.22.1-py3-none-any.whl", hash = "sha256:60567d774edf77db53906f89d9fbd164e66e56d66d388e1e6990f17ac33cfb53", size = 454815, upload-time = "2026-06-05T12:34:32.289Z" }, +] + [[package]] name = "networkx" version = "3.6.1" @@ -759,6 +1363,21 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/9e/c9/b2622292ea83fbb4ec318f5b9ab867d0a28ab43c5717bb85b0a5f6b3b0a4/networkx-3.6.1-py3-none-any.whl", hash = "sha256:d47fbf302e7d9cbbb9e2555a0d267983d2aa476bac30e90dfbe5669bd57f3762", size = 2068504, upload-time = "2025-12-08T17:02:38.159Z" }, ] +[[package]] +name = "nltk" +version = "3.9.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "click" }, + { name = "joblib" }, + { name = "regex" }, + { name = "tqdm" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/74/a1/b3b4adf15585a5bc4c357adde150c01ebeeb642173ded4d871e89468767c/nltk-3.9.4.tar.gz", hash = "sha256:ed03bc098a40481310320808b2db712d95d13ca65b27372f8a403949c8b523d0", size = 2946864, upload-time = "2026-03-24T06:13:40.641Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9d/91/04e965f8e717ba0ab4bdca5c112deeab11c9e750d94c4d4602f050295d39/nltk-3.9.4-py3-none-any.whl", hash = "sha256:f2fa301c3a12718ce4a0e9305c5675299da5ad9e26068218b69d692fda84828f", size = 1552087, upload-time = "2026-03-24T06:13:38.47Z" }, +] + [[package]] name = "numpy" version = "2.4.6" @@ -1086,6 +1705,36 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/0f/54/68a0978d1ef8502b8492099beaa6e7a0c1b32e3b5d4f677f5810cb08711c/pandas-3.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:b2c95f8bfc1ee412bf482605d7bfd30c12d1d26bd59fdd91efeef1d4718decb1", size = 9466464, upload-time = "2026-05-11T18:54:22.754Z" }, ] +[[package]] +name = "pathvalidate" +version = "3.3.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/fa/2a/52a8da6fe965dea6192eb716b357558e103aea0a1e9a8352ad575a8406ca/pathvalidate-3.3.1.tar.gz", hash = "sha256:b18c07212bfead624345bb8e1d6141cdcf15a39736994ea0b94035ad2b1ba177", size = 63262, upload-time = "2025-06-15T09:07:20.736Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9a/70/875f4a23bfc4731703a5835487d0d2fb999031bd415e7d17c0ae615c18b7/pathvalidate-3.3.1-py3-none-any.whl", hash = "sha256:5263baab691f8e1af96092fa5137ee17df5bdfbd6cff1fcac4d6ef4bc2e1735f", size = 24305, upload-time = "2025-06-15T09:07:19.117Z" }, +] + +[[package]] +name = "peft" +version = "0.19.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "accelerate" }, + { name = "huggingface-hub" }, + { name = "numpy" }, + { name = "packaging" }, + { name = "psutil" }, + { name = "pyyaml" }, + { name = "safetensors" }, + { name = "torch" }, + { name = "tqdm" }, + { name = "transformers" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/86/cf/037f1e3d5186496c05513a6754639e2dab3038a05f384284d49a9bd06a2d/peft-0.19.1.tar.gz", hash = "sha256:0d97542fe96dcdaa20d3b81c06f26f988618f416a73544ab23c3618ccb674a40", size = 763738, upload-time = "2026-04-16T15:46:45.105Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e8/b6/f54d676ed93cc2dd2234c3b172ea9c8c3d7d29361e66b1b23dec57a67465/peft-0.19.1-py3-none-any.whl", hash = "sha256:2113f72a81621b5913ef28f9022204c742df111890c5f49d812716a4a301e356", size = 680692, upload-time = "2026-04-16T15:46:42.886Z" }, +] + [[package]] name = "pillow" version = "12.2.0" @@ -1164,6 +1813,112 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" }, ] +[[package]] +name = "portalocker" +version = "3.2.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "pywin32", marker = "sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/5e/77/65b857a69ed876e1951e88aaba60f5ce6120c33703f7cb61a3c894b8c1b6/portalocker-3.2.0.tar.gz", hash = "sha256:1f3002956a54a8c3730586c5c77bf18fae4149e07eaf1c29fc3faf4d5a3f89ac", size = 95644, upload-time = "2025-06-14T13:20:40.03Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4b/a6/38c8e2f318bf67d338f4d629e93b0b4b9af331f455f0390ea8ce4a099b26/portalocker-3.2.0-py3-none-any.whl", hash = "sha256:3cdc5f565312224bc570c49337bd21428bba0ef363bbcf58b9ef4a9f11779968", size = 22424, upload-time = "2025-06-14T13:20:38.083Z" }, +] + +[[package]] +name = "propcache" +version = "0.5.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ec/44/c87281c333769159c50594f22610f77398a47ccbfbbf23074e744e86f87c/propcache-0.5.2.tar.gz", hash = "sha256:01c4fc7480cd0598bb4b57022df55b9ca296da7fc5a8760bd8451a7e63a7d427", size = 50208, upload-time = "2026-05-08T21:02:12.199Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4a/cb/e27bc2b2737a0bb49962b275efa051e8f1c35a936df7d5139b6b658b7dc9/propcache-0.5.2-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:806719138ecd720339a12410fb9614ac9b2b2d3a5fdf8235d56981c36f4039ba", size = 95887, upload-time = "2026-05-08T21:00:11.277Z" }, + { url = "https://files.pythonhosted.org/packages/e6/13/b8ae04c59392f8d11c6cd9fb4011d1dc7c86b81225c770280300e259ffe1/propcache-0.5.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:db2b80ea58eab4f86b2beec3cc8b39e8ff9276ac20e96b7cce43c8ae84cd6b5a", size = 54654, upload-time = "2026-05-08T21:00:12.604Z" }, + { url = "https://files.pythonhosted.org/packages/2c/7d/49777a3e20b55863d4794384a38acd460c04157b0a00f8602b0d508b8431/propcache-0.5.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:e5cbfac9f61484f7e9f3597775500cd3ebe8274e9b050c38f9525c77c97520bf", size = 55190, upload-time = "2026-05-08T21:00:13.935Z" }, + { url = "https://files.pythonhosted.org/packages/44/c7/085d0cd63062e84044e3f05797749c3f8e3938ff3aeb0eb2f69d43fafc91/propcache-0.5.2-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5dbc581d2814337da56222fab8dc5f161cd798a434e49bac27930aaef798e144", size = 59995, upload-time = "2026-05-08T21:00:15.526Z" }, + { url = "https://files.pythonhosted.org/packages/9c/42/32cf8e3009e92b2645cf1e944f701e8ea4e924dffde1ee26db860bcbf7e4/propcache-0.5.2-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:857187f381f88c8e2fa2fe56ab94879d011b883d5a2ee5a1b60a8cd2a06846d9", size = 63422, upload-time = "2026-05-08T21:00:16.824Z" }, + { url = "https://files.pythonhosted.org/packages/9e/1b/f112433f99fc979431b87a39ef169e3f8df070d99a72792c56d6937ac48b/propcache-0.5.2-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:178b4a2cdaac1818e2bf1c5a99b94383fa73ea5382e032a48dec07dc5668dc42", size = 64342, upload-time = "2026-05-08T21:00:18.362Z" }, + { url = "https://files.pythonhosted.org/packages/14/15/5574111ae50dd6e879456888c0eadd4c5a869959775854e18e18a6b345f3/propcache-0.5.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6f328175a2cde1f0ff2c4ed8ce968b9dcfb55f3a7153f39e2957ed994da13476", size = 61639, upload-time = "2026-05-08T21:00:19.692Z" }, + { url = "https://files.pythonhosted.org/packages/cc/da/4d775080b1490c0ae604acda868bd71aabe3a89ed16f2aa4339eb8a283e7/propcache-0.5.2-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:5671d09a36b06d0fd4a3da0fccbcae360e9b1570924171a15e9e0997f0249fba", size = 61588, upload-time = "2026-05-08T21:00:21.155Z" }, + { url = "https://files.pythonhosted.org/packages/04/ac/f076982cbe2195ee9cf32de5a1e46951d9fb399fc207f390562dd0fd8fb2/propcache-0.5.2-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:80168e2ebe4d3ec6599d10ad8f520304ae1cad9b6c5a95372aef1b66b7bfb53a", size = 60029, upload-time = "2026-05-08T21:00:22.713Z" }, + { url = "https://files.pythonhosted.org/packages/70/60/189be62e0dd898dce3b331e1b8c7a543cd3a405ac0c81fe8ee8a9d5d77e1/propcache-0.5.2-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:45f11346f884bc47444f6e6647131055844134c3175b629f84952e2b5cd62b64", size = 56774, upload-time = "2026-05-08T21:00:24.001Z" }, + { url = "https://files.pythonhosted.org/packages/ea/9e/93377b9c7939c1ffae98f878dee955efadfd638078bc86dbc21f9d52f651/propcache-0.5.2-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:8e778ebd44ef4f66ed60a0416b06b489687db264a9c0b3620362f26489492913", size = 63532, upload-time = "2026-05-08T21:00:25.545Z" }, + { url = "https://files.pythonhosted.org/packages/14/f9/590ef6cfb9b8028d516d287812ece32bb0bc5f11fbb9c8bf6b2e6313fec8/propcache-0.5.2-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:c0cb9ed24c8964e172768d455a38254c2dd8a552905729ce006cad3d3dda59b1", size = 61592, upload-time = "2026-05-08T21:00:27.186Z" }, + { url = "https://files.pythonhosted.org/packages/b4/5e/70958b3034c297a630bba2f17ca7abc2d5f39a803ad7e370ab79d1ecd022/propcache-0.5.2-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:1d1ad32d9d4355e2be65574fd0bfd3677e7066b009cd5b9b2dee8aa6a6393b33", size = 64788, upload-time = "2026-05-08T21:00:28.8Z" }, + { url = "https://files.pythonhosted.org/packages/12/fd/77fe5936d8c3086ca9048f7f415f122ed82e53884a9ec193646b42deef06/propcache-0.5.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:c80f4ba3e8f00189165999a742ee526ebeccedf6c3f7beb0c7df821e9772435a", size = 62514, upload-time = "2026-05-08T21:00:30.098Z" }, + { url = "https://files.pythonhosted.org/packages/cf/74/66bd798b5b3be70aa1b391f5cc9d6a0a5532d7fd3b19ec0b213e72e6ad9d/propcache-0.5.2-cp312-cp312-win32.whl", hash = "sha256:8c7972d8f193740d9175f0998ab38717e6cd322d5935c5b0fef8c0d323fd9031", size = 39018, upload-time = "2026-05-08T21:00:31.622Z" }, + { url = "https://files.pythonhosted.org/packages/61/7c/5c0d34aa3024694d6dcb9271cdbdd08c4e47c1c0ad95ec7e7bc74cdea145/propcache-0.5.2-cp312-cp312-win_amd64.whl", hash = "sha256:d9ee8826a7d47863a08ac44e1a5f611a462eefc3a194b492da242128bec75b42", size = 42322, upload-time = "2026-05-08T21:00:32.918Z" }, + { url = "https://files.pythonhosted.org/packages/4d/91/875812f1a3feb20ceba818ef39fbe4d92f1081e04ac815c822496d0d038b/propcache-0.5.2-cp312-cp312-win_arm64.whl", hash = "sha256:2800a4a8ead6b28cccd1ec54b59346f0def7922ee1c7598e8499c733cfbb7c84", size = 38172, upload-time = "2026-05-08T21:00:35.124Z" }, + { url = "https://files.pythonhosted.org/packages/c5/09/f049e45385503fe67db75a6b6186a7b9f0c3930366dc960522c312a825b1/propcache-0.5.2-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:099aaf4b4d1a02265b92a977edf00b5c4f63b3b17ac6de39b0d637c9cac0188a", size = 94457, upload-time = "2026-05-08T21:00:36.355Z" }, + { url = "https://files.pythonhosted.org/packages/6b/65/83d1d05655baf63113731bd5a1008435e14f8d1e5a06cbe4ec5b23ad7a31/propcache-0.5.2-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:68ce1c44c7a813a7f71ea04315a8c7b330b63db99d059a797a4651bb6f69f117", size = 53835, upload-time = "2026-05-08T21:00:38.072Z" }, + { url = "https://files.pythonhosted.org/packages/a9/12/a6ba6482bb5ea3260c000c9b20881c95fa11c6b30173715668259f844ed7/propcache-0.5.2-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:fc299c129490f55f254cd90be0deca4764e36e9a7c08b4aa588479a3bbed3098", size = 54545, upload-time = "2026-05-08T21:00:39.319Z" }, + { url = "https://files.pythonhosted.org/packages/a9/19/7fa086f5764c59ec8a8e157cd93aa8497acc00aba9dcdec56bfffb32602d/propcache-0.5.2-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a6ae2198be502c10f09b2516e7b5d019816924bc3183a43ce792a7bd6625e6f4", size = 59886, upload-time = "2026-05-08T21:00:40.621Z" }, + { url = "https://files.pythonhosted.org/packages/a1/e4/5d7663dc8235956c8f5281698a3af1d351d8820341ddd890f59d9a9127f2/propcache-0.5.2-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:6041d31504dc1779d700e1edcfb08eea334b357620b06681a4eabb57a74e574e", size = 63261, upload-time = "2026-05-08T21:00:41.775Z" }, + { url = "https://files.pythonhosted.org/packages/4a/4a/15a03adee24d6350da4292caeac44c34c033d2afe5e87eb370f38854560f/propcache-0.5.2-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:f7eabc04151c78a9f4d5bbb5f1faf571e4defeb4b585e0fe95b60ff2dbe4d3d7", size = 64184, upload-time = "2026-05-08T21:00:43.018Z" }, + { url = "https://files.pythonhosted.org/packages/8b/c6/979176efdaa3d239e36d503d5af63a0a773b36662ed8f52e5b6a6d9fd40e/propcache-0.5.2-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4db0ba63d693afd40d249bd93f842b5f144f8fcbb83de05660373bcf30517b1d", size = 61534, upload-time = "2026-05-08T21:00:44.507Z" }, + { url = "https://files.pythonhosted.org/packages/c8/22/63e8cd1bae4c2d2be6493b6b7d10566ddafad88137cfbc99964a1119853c/propcache-0.5.2-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:1dbcf7675229b35d31abb6547d8ebc8c27a830ac3f9a794edff6254873ec7c0a", size = 61500, upload-time = "2026-05-08T21:00:45.796Z" }, + { url = "https://files.pythonhosted.org/packages/60/5a/28e5d9acbac1cc9ccb67045e8c1b943aa8d79fdf39c93bd73cacd68008ea/propcache-0.5.2-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:d310c013aad2c72f1c3f2f8dd3279d460a858c551f97aeb8c63e4693cca7b4d2", size = 59994, upload-time = "2026-05-08T21:00:47.093Z" }, + { url = "https://files.pythonhosted.org/packages/f3/40/db650677f554a95b9c01a7c9d93d629e93a15562f5deb4573c9ee136fed2/propcache-0.5.2-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:06187263ddad280d05b4d8a8b3bb7d164cbebd469236544a42e6d9b28ac6a4fa", size = 56884, upload-time = "2026-05-08T21:00:48.376Z" }, + { url = "https://files.pythonhosted.org/packages/80/45/70b39b89516ff8b96bf732fa6fded8cef20f293cb1508690101c3c07ec51/propcache-0.5.2-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:3115559b8effafd63b142ea5ed53d63a16ea6469cbc63dce4ee194b42db5d853", size = 63464, upload-time = "2026-05-08T21:00:49.954Z" }, + { url = "https://files.pythonhosted.org/packages/f9/e2/fa59d3a89eac5534293124af4f1d0d0ada091ce4a0ab4610ce03fd2bdd8d/propcache-0.5.2-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:c60462af8e6dc30c35407c7237ea908d777b22862bbee27bc4699c0d8bcdc45a", size = 61588, upload-time = "2026-05-08T21:00:51.281Z" }, + { url = "https://files.pythonhosted.org/packages/0b/97/efb547a55c4bc7381cfb202d6a2239ac621045277bc1ea5dfd3a7f0516c0/propcache-0.5.2-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:40314bca9ac559716fe374094fc81c11dcc34b64fd6c585360f5775690505704", size = 64667, upload-time = "2026-05-08T21:00:52.602Z" }, + { url = "https://files.pythonhosted.org/packages/92/56/f5c7d9b4b7595d5127da38974d791b2153f3d1eae6c674af3583ace92ad3/propcache-0.5.2-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:cfa21e036ce1e1db2be04ba3b85d2df1bb1702fa01932d984c5464c665228ff4", size = 62463, upload-time = "2026-05-08T21:00:54.303Z" }, + { url = "https://files.pythonhosted.org/packages/bd/3b/484a3a65fc9f9f60c41dcd17b428bace5389544e2c680994534a20755066/propcache-0.5.2-cp313-cp313-win32.whl", hash = "sha256:f156a3529f38063b6dbaf356e15602a7f95f8055b1295a438433a6386f10463d", size = 38621, upload-time = "2026-05-08T21:00:55.808Z" }, + { url = "https://files.pythonhosted.org/packages/1c/fd/3f0f10dba4dabad3bf53102be007abf55481067952bde0fdddff439e7c61/propcache-0.5.2-cp313-cp313-win_amd64.whl", hash = "sha256:dfed59d0a5aeb01e242e66ff0300bc4a265a7c05f612d30016f0b60b1017d757", size = 41649, upload-time = "2026-05-08T21:00:57.061Z" }, + { url = "https://files.pythonhosted.org/packages/90/ec/6ce619cc32bb500a482f811f9cd509368b4e58e638d13f2c68f370d6b475/propcache-0.5.2-cp313-cp313-win_arm64.whl", hash = "sha256:ba338430e87ceb9c8f0cf754de38a9860560261e56c00376debd628698a7364f", size = 37636, upload-time = "2026-05-08T21:00:58.646Z" }, + { url = "https://files.pythonhosted.org/packages/1b/82/c1d268bbbf2ef981c5bf0fbbe746db617c66e3bcefe431a1aa8943fbe23a/propcache-0.5.2-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:a592f5f3da71c8691c788c13cb6734b6d17663d2e1cb8caddf0673d01ef8847d", size = 98872, upload-time = "2026-05-08T21:00:59.889Z" }, + { url = "https://files.pythonhosted.org/packages/f4/d4/52c871e73e864e6b34c0e2d58ac1ec5ccd149497ddc7ad2137ae98323a35/propcache-0.5.2-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:6a997d0489e9668a384fcfd5061b857aa5361de73191cac204d04b889cfbbafa", size = 56257, upload-time = "2026-05-08T21:01:01.195Z" }, + { url = "https://files.pythonhosted.org/packages/67/f0/9b90ca2a210b3d09bcfcd96ecd0f55545c091535abce2a45de2775cfd357/propcache-0.5.2-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:10734b5484ea113152ee25a91dccedf81631791805d2c9ccb054958e51842c94", size = 56696, upload-time = "2026-05-08T21:01:02.941Z" }, + { url = "https://files.pythonhosted.org/packages/9d/0e/6e9d4ba07c8e56e21ddec1e75f12148142b21ca83a51871babce095334f4/propcache-0.5.2-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:cafca7e56c12bb02ae16d283742bef25a61122e9dab2b5b3f2ccbe589ce32164", size = 62378, upload-time = "2026-05-08T21:01:04.475Z" }, + { url = "https://files.pythonhosted.org/packages/65/19/c10badaa463dde8a27ce884f8ee2ec37e6035b7c9f5ff0c8f74f06f08dac/propcache-0.5.2-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f064f8d2b59177878b7615df1735cd8fe3462ed6be8c7b217d17a276489c2b7f", size = 65283, upload-time = "2026-05-08T21:01:05.959Z" }, + { url = "https://files.pythonhosted.org/packages/b0/b6/93bea99ca80e19cef6512a8580e5b7857bbe09422d9daa7fd4ef5723306c/propcache-0.5.2-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:f78abfa8dfc32376fd1aacf597b2f2fbbe0ea751419aee718af5d4f82537ef8c", size = 66616, upload-time = "2026-05-08T21:01:07.228Z" }, + { url = "https://files.pythonhosted.org/packages/83/e4/5c7462e50625f051f37fb38b8224f7639f667184bbd34424ec83819bb1b7/propcache-0.5.2-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f7467da8a9822bf1a55336f877340c5bcbd3c482afc43a99771169f74a26dedc", size = 63773, upload-time = "2026-05-08T21:01:08.514Z" }, + { url = "https://files.pythonhosted.org/packages/ca/b6/99238894047b13c823be25027e736626cd414a52a5e30d2c3347c2733529/propcache-0.5.2-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:a6ddc6ac9e25de626c1f129c1b467d7ecd33ce2237d3fd0c4e429feef0a7ee1f", size = 63664, upload-time = "2026-05-08T21:01:09.874Z" }, + { url = "https://files.pythonhosted.org/packages/85/1e/a3a1a63116a2b8edb415a8bb9a6f0c34bd03830b1e18e8ce2904e1dc1cf4/propcache-0.5.2-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:2f22cbbac9e26a8e864c0985ff1268d5d939d53d9d9411a9824279097e03a2cb", size = 62643, upload-time = "2026-05-08T21:01:11.132Z" }, + { url = "https://files.pythonhosted.org/packages/e4/03/893cf147de2fc6543c5eaa07ad833170e7e2a2385725bbebe8c0503723bb/propcache-0.5.2-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:fc76378c62a0f04d0cd82fbb1a2cd2d7e28fcb40d5873f28a6c44e388aaa2751", size = 59595, upload-time = "2026-05-08T21:01:12.387Z" }, + { url = "https://files.pythonhosted.org/packages/86/3b/04c1a2e12c57766568ba75ba72b3bf2042818d4c1425fab6fc07155c7cff/propcache-0.5.2-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:acd2c8edba48e31e58a363b8cf4e5c7db3b04b3f9e371f601df30d9b0d244836", size = 65711, upload-time = "2026-05-08T21:01:13.676Z" }, + { url = "https://files.pythonhosted.org/packages/1c/34/80f8d0099f8d6bacc4de1624c85672681c8cd1149ca2da0e38fd120b817f/propcache-0.5.2-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:452b5065457eb9991ec5eb38ff41d6cd4c991c9ac7c531c4d5849ae473a9a13f", size = 64247, upload-time = "2026-05-08T21:01:14.936Z" }, + { url = "https://files.pythonhosted.org/packages/f3/1a/8b08f3a5f1037e9e370c55883ceeeee0f6dd0416fb2d2d67b8bfc91f2a79/propcache-0.5.2-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:3430bb2bfe1331885c427745a751e774ee679fd4344f80b97bf879815fe8fa55", size = 67102, upload-time = "2026-05-08T21:01:16.281Z" }, + { url = "https://files.pythonhosted.org/packages/34/68/8bdb7bb7756d76e005490649d10e4a8369e610c74d619f71e1aedf889e9c/propcache-0.5.2-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:cef6cea3922890dd6c9654971001fa797b526c16ab5e1e46c05fd6f877be7568", size = 64964, upload-time = "2026-05-08T21:01:17.57Z" }, + { url = "https://files.pythonhosted.org/packages/0a/aa/50fb0b5d3968b61a510926ff8b8465f1d6e976b3ab74496d7a4b9fc42515/propcache-0.5.2-cp313-cp313t-win32.whl", hash = "sha256:72d61e16dd78228b58c5d47be830ff3da7e5f139abdf0aef9d86cde1c5cf2191", size = 42546, upload-time = "2026-05-08T21:01:18.946Z" }, + { url = "https://files.pythonhosted.org/packages/ae/4c/0ddbae64321bd4a95bcbfc19307238016b5b1fee645c84626c8d539e5b74/propcache-0.5.2-cp313-cp313t-win_amd64.whl", hash = "sha256:0958834041a0166d343b8d2cedcd8bcbaeb4fdbe0cf08320c5379f143c3be6e7", size = 46330, upload-time = "2026-05-08T21:01:20.162Z" }, + { url = "https://files.pythonhosted.org/packages/00/d9/9cddc8efb78d8af264c5ec9f6d10b62f57c515feda8d321595f56010fb23/propcache-0.5.2-cp313-cp313t-win_arm64.whl", hash = "sha256:6de8bd93ddde9b992cf2b2e0d796d501a19026b5b9fd87356d7d0779531a8d96", size = 40521, upload-time = "2026-05-08T21:01:21.399Z" }, + { url = "https://files.pythonhosted.org/packages/e2/ea/23ee535d90ce8bcc465a3028eb3cc0ce3bd1005f4bb27710b30587de798d/propcache-0.5.2-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:46088abff4cba581dea21ae0467a480526cb25aa5f3c269e909f800328bc3999", size = 94662, upload-time = "2026-05-08T21:01:22.683Z" }, + { url = "https://files.pythonhosted.org/packages/b5/06/c5a52f419b5d8972f8d46a7577476090d8e3263ff589ce40b5ca4968d5be/propcache-0.5.2-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:fc88b26f08d634f7bc819a7852e5214f5802641ab8d9fd5326892292eee1993e", size = 53928, upload-time = "2026-05-08T21:01:23.986Z" }, + { url = "https://files.pythonhosted.org/packages/63/b1/4260d67d6bd85e58a66b72d54ce15d5de789b6f3870cc6bedf8ff9667401/propcache-0.5.2-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:97797ebb098e670a2f92dd66f32897e30d7615b14e7f59711de23e30a9072539", size = 54650, upload-time = "2026-05-08T21:01:25.305Z" }, + { url = "https://files.pythonhosted.org/packages/70/06/2f46c318e3307cd7a6a7481def374ce838c0fe20084b39dd54b0879d0e99/propcache-0.5.2-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ba57fffe4ac99c5d30076161b5866336d97600769bad35cc68f7774b15298a4e", size = 59912, upload-time = "2026-05-08T21:01:26.545Z" }, + { url = "https://files.pythonhosted.org/packages/4c/29/fe1aebec2ce57ab985a9c382bded1124431f85078113aa222c5d278430d4/propcache-0.5.2-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:583c19759d9eec1e5b69e2fbef36a7d9c326041be9746cb822d335c8cedc2979", size = 63300, upload-time = "2026-05-08T21:01:27.937Z" }, + { url = "https://files.pythonhosted.org/packages/b4/18/2334b26768b6c82be8c69e83671b767d5ef426aa09b0cba6c2ea47816774/propcache-0.5.2-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:d0326e2e5e1f3163fa306c834e48e8d490e5fae607a097a40c0648109b47ba80", size = 64208, upload-time = "2026-05-08T21:01:29.484Z" }, + { url = "https://files.pythonhosted.org/packages/2b/76/7f1bfd6afff4c5e38e36a3c6d68eb5f4b7311ea80baf693db78d95b603c4/propcache-0.5.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e00820e192c8dbebcafb383ebbf99030895f09905e7a0eb2e0340a0bcc2bc825", size = 61633, upload-time = "2026-05-08T21:01:31.068Z" }, + { url = "https://files.pythonhosted.org/packages/c4/46/b3ff8aba2b4953a3e50de2cf72f1b5748b8eca93b15f3dc2c84339084c09/propcache-0.5.2-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c66afea89b1e43725731d2004732a046fe6fe955d51f952c3e95a7314a284a39", size = 61724, upload-time = "2026-05-08T21:01:32.374Z" }, + { url = "https://files.pythonhosted.org/packages/c5/01/814cfcafbcff954f94c01cf30e097ddc88a076b5440fbcf4570753437d40/propcache-0.5.2-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:d4dc37dec6c6cdad0b57881a5658fd14fbf53e333b1a86cf86559f190e1d9ec4", size = 60069, upload-time = "2026-05-08T21:01:33.67Z" }, + { url = "https://files.pythonhosted.org/packages/da/68/5c6f7622d510cc666a300687e06fd060c1a43361c0c9b20d284f06d8096a/propcache-0.5.2-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:5570dbcc97571c15f68068e529c92715a12f8d54030e272d264b377e22bd17a5", size = 57099, upload-time = "2026-05-08T21:01:34.915Z" }, + { url = "https://files.pythonhosted.org/packages/55/27/9cb0b4c679124085327957d42521c99dba04c88c90c3e55a6f0b633ebccc/propcache-0.5.2-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:f814362777a9f841adddb200ecdf8f5cb1e5a3c4b7a86378edbd6ccb26edd702", size = 63391, upload-time = "2026-05-08T21:01:36.231Z" }, + { url = "https://files.pythonhosted.org/packages/f0/9d/7258aaa5bdf60fc6f27591eef6fe52768cb0beda7140be477c8b12c9794a/propcache-0.5.2-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:196913dea116aeb5a2ba95af4ddcb7ea85559ae07d8eee8751688310d09168c3", size = 61626, upload-time = "2026-05-08T21:01:37.545Z" }, + { url = "https://files.pythonhosted.org/packages/8e/0d/41c602003e8a9b16fe1e7eadf62c7bfba9d5474370b24200bf48b315f45f/propcache-0.5.2-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:6e7b8719005dd1175be4ab1cd25e9b98659a5e0347331506ec6760d2773a7fb5", size = 64781, upload-time = "2026-05-08T21:01:38.83Z" }, + { url = "https://files.pythonhosted.org/packages/8b/f3/38e66b1856e9bd079deea015bc4a55f7767c0e4db2f7dcf69e7e680ba4ce/propcache-0.5.2-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:51f96d685ab16e88cab128cd37a52c5da540809c8b879fa047731bfcb4ad35a4", size = 62570, upload-time = "2026-05-08T21:01:40.415Z" }, + { url = "https://files.pythonhosted.org/packages/95/ca/bbfe9b910ce57dde8bb4876b4520fc02a4e89497c10de26be936758a3aaa/propcache-0.5.2-cp314-cp314-win32.whl", hash = "sha256:cc6fc3cc62e8501d3ed62894425040d2728ecddb1ed072737a5c70bd537aa9f0", size = 39436, upload-time = "2026-05-08T21:01:41.654Z" }, + { url = "https://files.pythonhosted.org/packages/61/d2/45c9defbaa1ea297035d9d4cce9e8f80daafbf19319c6007f157c6256ea9/propcache-0.5.2-cp314-cp314-win_amd64.whl", hash = "sha256:81e3a30b0bb60caa22033dd0f8a3618d1d67356212514f62c57db75cb0ef410c", size = 42373, upload-time = "2026-05-08T21:01:43.041Z" }, + { url = "https://files.pythonhosted.org/packages/44/68/9ea5103f41d5217d7d6ec24db90018e23aebec070c3f9a6e54d12b841fd8/propcache-0.5.2-cp314-cp314-win_arm64.whl", hash = "sha256:0d2c9bf8528f135dbb805ce027567e09164f7efa51a2be07458a2c0420f292d0", size = 38554, upload-time = "2026-05-08T21:01:44.336Z" }, + { url = "https://files.pythonhosted.org/packages/8a/81/fadf555f42d3b762eea8a53950b0489fdc0aa9da5f8ed9e10ce0a4e01b48/propcache-0.5.2-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:4bc8ff1feffc6a61c7002ffe84634c41b822e104990ae009f44a0834430070bb", size = 99395, upload-time = "2026-05-08T21:01:45.883Z" }, + { url = "https://files.pythonhosted.org/packages/f5/c9/c61e134a686949cf7971af3a390148b1156f7be81c73bc0cd12c873e2d48/propcache-0.5.2-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:79aa3ff0a9b566633b642fa9caf7e21ed1c13d6feca718187873f199e1514078", size = 56653, upload-time = "2026-05-08T21:01:47.307Z" }, + { url = "https://files.pythonhosted.org/packages/cb/73/daf935ea7048ddd7ec8eec5345b4a40b619d2d178b3c0a0900796bc3c794/propcache-0.5.2-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:1b31822f4474c4036bae62de9402710051d431a606d6a0f907fec79935a071aa", size = 56914, upload-time = "2026-05-08T21:01:48.573Z" }, + { url = "https://files.pythonhosted.org/packages/79/9f/aba959b435ea18617edd7cf0a7ad0b9c574b8fc7e3d2cd55fb59cb255d33/propcache-0.5.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:13fef48778b5a2a756523fdb781326b028ca75e32858b04f2cdd19f394564917", size = 62567, upload-time = "2026-05-08T21:01:49.903Z" }, + { url = "https://files.pythonhosted.org/packages/6c/a1/859942de9a791ff42f6141736f5b37749b8f53e65edfa49638c67dd67e6a/propcache-0.5.2-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:8b73ab70f1a3351fbc71f663b3e645af6dd0329100c353081cf69c37433fc6fe", size = 65542, upload-time = "2026-05-08T21:01:51.204Z" }, + { url = "https://files.pythonhosted.org/packages/b5/61/315bc0fd6c0fc7f80a528b8afd209e5fc4a875ea79571b91b8f50f442907/propcache-0.5.2-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5538d2c13d93e4698af7e092b57bc7298fd35d1d58e656ae18f23ee0d0378e03", size = 66845, upload-time = "2026-05-08T21:01:52.539Z" }, + { url = "https://files.pythonhosted.org/packages/47/f7/9f8122e3132e8e354ac41975ef8f1099be7d5a16bc7ae562734e993665c0/propcache-0.5.2-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:cd645f03898405cabe694fb8bc35241e3a9c332ec85627584fe3de201452b335", size = 63985, upload-time = "2026-05-08T21:01:53.847Z" }, + { url = "https://files.pythonhosted.org/packages/c8/54/c317819ec157cbf6f35df9df9657a6f82daf34d5faf15948b2f639c2192e/propcache-0.5.2-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:a473b3440261e0c60706e732b2ed2f517857344fc21bf48fdfe211e2d98eb285", size = 63999, upload-time = "2026-05-08T21:01:55.179Z" }, + { url = "https://files.pythonhosted.org/packages/5a/56/387e3f7dfce0a9233df41fb888aa1c30222cb4bbbf09537c02dd9bd85fe2/propcache-0.5.2-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:7afa37062e6650640e932e4cc9297d81f9f42d9944029cc386b8247dea4da837", size = 62779, upload-time = "2026-05-08T21:01:57.489Z" }, + { url = "https://files.pythonhosted.org/packages/a1/9c/596784cb5824ed61ee960d3f8655a3f0993e107c6e98ab6c818b7fb92ccb/propcache-0.5.2-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:8a90efd5777e996e42d568db9ac740b944d691e565cbfd31b2f7832f9184b2b8", size = 59796, upload-time = "2026-05-08T21:01:58.736Z" }, + { url = "https://files.pythonhosted.org/packages/c2/3d/1a6cfa1726a48542c1e8784a0761421476a5b68e09b7f36bf95eb954aaba/propcache-0.5.2-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:f19bb891234d72535764d703bfed1153cc34f4214d5bd7150aee1eec9e8f4366", size = 66023, upload-time = "2026-05-08T21:02:00.228Z" }, + { url = "https://files.pythonhosted.org/packages/e4/0e/05fd6990369477076e4e280bcb970de760fddf0161a46e988bc95f7940ec/propcache-0.5.2-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:32775082acd2d807ee3db715c7770d38767b817870acfa08c29e057f3c4d5b56", size = 64448, upload-time = "2026-05-08T21:02:01.888Z" }, + { url = "https://files.pythonhosted.org/packages/cd/86/5f8da315a4309c62c10c0b2516b17492d5d3bbe1bb862b96604db67e2a37/propcache-0.5.2-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:9282fb1a3bccd038da9f768b927b24a0c753e466c086b7c4f3c6982851eefb2d", size = 67329, upload-time = "2026-05-08T21:02:03.484Z" }, + { url = "https://files.pythonhosted.org/packages/da/d3/3368efe79ab21f0cdf86ef49895811c9cc933131d4cde1f28a624e22e712/propcache-0.5.2-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:cc49723e2f60d6b32a0f0b08a3fd6d13203c07f1cd9566cfce0f12a917c967a2", size = 65172, upload-time = "2026-05-08T21:02:04.745Z" }, + { url = "https://files.pythonhosted.org/packages/d5/07/127e8b0bacfb325396196f9d976a22453049b89b9b2b08477cc3145faa44/propcache-0.5.2-cp314-cp314t-win32.whl", hash = "sha256:2d7aa89ebca5acc98cba9d1472d976e394782f587bad6661003602a619fd1821", size = 43813, upload-time = "2026-05-08T21:02:06.025Z" }, + { url = "https://files.pythonhosted.org/packages/88/fb/46dad6c0ae49ed230ab1b16c890c2b6314e2403e6c412976f4a72d64a527/propcache-0.5.2-cp314-cp314t-win_amd64.whl", hash = "sha256:d447bb0b3054be5818458fbb171208b1d9ff11eba14e18ca18b90cbb45767370", size = 47764, upload-time = "2026-05-08T21:02:07.353Z" }, + { url = "https://files.pythonhosted.org/packages/e7/c4/a47d0a63aa309d10d59ede6e9d4cff03a344a79d1f0f4cd0cd74997b53e0/propcache-0.5.2-cp314-cp314t-win_arm64.whl", hash = "sha256:fe67a3d11cd9b4efabfa45c3d00ffba2b26811442a73a581a94b67c2b5faccf6", size = 41140, upload-time = "2026-05-08T21:02:09.065Z" }, + { url = "https://files.pythonhosted.org/packages/3a/ed/1cdcab6ba3d6ab7feca11fc14f0eeea80755bb53ef4e892079f31b10a25f/propcache-0.5.2-py3-none-any.whl", hash = "sha256:be1ddfcbb376e3de5d2e2db1d58d6d67463e6b4f9f040c000de8e300295465fe", size = 14036, upload-time = "2026-05-08T21:02:10.673Z" }, +] + [[package]] name = "psutil" version = "7.2.2" @@ -1192,6 +1947,49 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/8c/c7/7bb2e321574b10df20cbde462a94e2b71d05f9bbda251ef27d104668306a/psutil-7.2.2-cp37-abi3-win_arm64.whl", hash = "sha256:8c233660f575a5a89e6d4cb65d9f938126312bca76d8fe087b947b3a1aaac9ee", size = 134617, upload-time = "2026-01-28T18:15:36.514Z" }, ] +[[package]] +name = "pyarrow" +version = "24.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/91/13/13e1069b351bdc3881266e11147ffccf687505dbb0ea74036237f5d454a5/pyarrow-24.0.0.tar.gz", hash = "sha256:85fe721a14dd823aca09127acbb06c3ca723efbd436c004f16bca601b04dcc83", size = 1180261, upload-time = "2026-04-21T10:51:25.837Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b4/a9/9686d9f07837f91f775e8932659192e02c74f9d8920524b480b85212cc68/pyarrow-24.0.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:6233c9ed9ab9d1db47de57d9753256d9dcffbf42db341576099f0fd9f6bf4810", size = 34981559, upload-time = "2026-04-21T10:47:22.17Z" }, + { url = "https://files.pythonhosted.org/packages/80/b6/0ddf0e9b6ead3474ab087ae598c76b031fc45532bf6a63f3a553440fb258/pyarrow-24.0.0-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:f7616236ec1bc2b15bfdec22a71ab38851c86f8f05ff64f379e1278cf20c634a", size = 36663654, upload-time = "2026-04-21T10:47:28.315Z" }, + { url = "https://files.pythonhosted.org/packages/7c/3b/926382efe8ce27ba729071d3566ade6dfb86bdf112f366000196b2f5780a/pyarrow-24.0.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:1617043b99bd33e5318ae18eb2919af09c71322ef1ca46566cdafc6e6712fb66", size = 45679394, upload-time = "2026-04-21T10:47:34.821Z" }, + { url = "https://files.pythonhosted.org/packages/b3/7a/829f7d9dfd37c207206081d6dad474d81dde29952401f07f2ba507814818/pyarrow-24.0.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:6165461f55ef6314f026de6638d661188e3455d3ec49834556a0ebbdbace18bb", size = 48863122, upload-time = "2026-04-21T10:47:42.056Z" }, + { url = "https://files.pythonhosted.org/packages/5f/e8/f88ce625fe8babaae64e8db2d417c7653adb3019b08aae85c5ed787dc816/pyarrow-24.0.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:3b13dedfe76a0ad2d1d859b0811b53827a4e9d93a0bcb05cf59333ab4980cc7e", size = 49376032, upload-time = "2026-04-21T10:47:48.967Z" }, + { url = "https://files.pythonhosted.org/packages/36/7a/82c363caa145fff88fb475da50d3bf52bb024f61917be5424c3392eaf878/pyarrow-24.0.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:25ea65d868eb04015cd18e6df2fbe98f07e5bda2abefabcb88fce39a947716f6", size = 51929490, upload-time = "2026-04-21T10:47:55.981Z" }, + { url = "https://files.pythonhosted.org/packages/66/1c/e3e72c8014ad2743ca64a701652c733cc5cbcee15c0463a32a8c55518d9e/pyarrow-24.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:295f0a7f2e242dabd513737cf076007dc5b2d59237e3eca37b05c0c6446f3826", size = 27355660, upload-time = "2026-04-21T10:48:01.718Z" }, + { url = "https://files.pythonhosted.org/packages/6f/d3/a1abf004482026ddc17f4503db227787fa3cfe41ec5091ff20e4fea55e57/pyarrow-24.0.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:02b001b3ed4723caa44f6cd1af2d5c86aa2cf9971dacc2ffa55b21237713dfba", size = 34976759, upload-time = "2026-04-21T10:48:07.258Z" }, + { url = "https://files.pythonhosted.org/packages/4f/4a/34f0a36d28a2dd32225301b79daad44e243dc1a2bb77d43b60749be255c4/pyarrow-24.0.0-cp313-cp313-macosx_12_0_x86_64.whl", hash = "sha256:04920d6a71aabd08a0417709efce97d45ea8e6fb733d9ca9ecffb13c67839f68", size = 36658471, upload-time = "2026-04-21T10:48:13.347Z" }, + { url = "https://files.pythonhosted.org/packages/1f/78/543b94712ae8bb1a6023bcc1acf1a740fbff8286747c289cd9468fced2a5/pyarrow-24.0.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:a964266397740257f16f7bb2e4f08a0c81454004beab8ff59dd531b73610e9f2", size = 45675981, upload-time = "2026-04-21T10:48:20.201Z" }, + { url = "https://files.pythonhosted.org/packages/84/9f/8fb7c222b100d314137fa40ec050de56cd8c6d957d1cfff685ce72f15b17/pyarrow-24.0.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:6f066b179d68c413374294bc1735f68475457c933258df594443bb9d88ddc2a0", size = 48859172, upload-time = "2026-04-21T10:48:27.541Z" }, + { url = "https://files.pythonhosted.org/packages/a7/d3/1ea72538e6c8b3b475ed78d1049a2c518e655761ea50fe1171fc855fcab7/pyarrow-24.0.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:1183baeb14c5f587b1ec52831e665718ce632caab84b7cd6b85fd44f96114495", size = 49385733, upload-time = "2026-04-21T10:48:34.7Z" }, + { url = "https://files.pythonhosted.org/packages/c3/be/c3d8b06a1ba35f2260f8e1f771abbee7d5e345c0937aab90675706b1690a/pyarrow-24.0.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:806f24b4085453c197a5078218d1ee08783ebbba271badd153d1ae22a3ee804f", size = 51934335, upload-time = "2026-04-21T10:48:42.099Z" }, + { url = "https://files.pythonhosted.org/packages/9c/62/89e07a1e7329d2cde3e3c6994ba0839a24977a2beda8be6005ea3d860b99/pyarrow-24.0.0-cp313-cp313-win_amd64.whl", hash = "sha256:e4505fc6583f7b05ab854934896bcac8253b04ac1171a77dfb73efef92076d91", size = 27271748, upload-time = "2026-04-21T10:49:42.532Z" }, + { url = "https://files.pythonhosted.org/packages/17/1a/cff3a59f80b5b1658549d46611b67163f65e0664431c076ad728bf9d5af4/pyarrow-24.0.0-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:1a4e45017efbf115032e4475ee876d525e0e36c742214fbe405332480ecd6275", size = 35238554, upload-time = "2026-04-21T10:48:48.526Z" }, + { url = "https://files.pythonhosted.org/packages/a8/99/cce0f42a327bfef2c420fb6078a3eb834826e5d6697bf3009fe11d2ad051/pyarrow-24.0.0-cp313-cp313t-macosx_12_0_x86_64.whl", hash = "sha256:7986f1fa71cee060ad00758bcc79d3a93bab8559bf978fab9e53472a2e25a17b", size = 36782301, upload-time = "2026-04-21T10:48:55.181Z" }, + { url = "https://files.pythonhosted.org/packages/2a/66/8e560d5ff6793ca29aca213c53eec0dd482dd46cb93b2819e5aab52e4252/pyarrow-24.0.0-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:d3e0b61e8efb24ed38898e5cdc5fffa9124be480008d401a1f8071500494ae42", size = 45721929, upload-time = "2026-04-21T10:49:03.676Z" }, + { url = "https://files.pythonhosted.org/packages/27/0c/a26e25505d030716e078d9f16eb74973cbf0b33b672884e9f9da1c83b871/pyarrow-24.0.0-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:55a3bc1e3df3b5567b7d27ef551b2283f0c68a5e86f1cd56abc569da4f31335b", size = 48825365, upload-time = "2026-04-21T10:49:11.714Z" }, + { url = "https://files.pythonhosted.org/packages/5f/eb/771f9ecb0c65e73fe9dccdd1717901b9594f08c4515d000c7c62df573811/pyarrow-24.0.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:641f795b361874ac9da5294f8f443dfdbee355cf2bd9e3b8d97aaac2306b9b37", size = 49451819, upload-time = "2026-04-21T10:49:21.474Z" }, + { url = "https://files.pythonhosted.org/packages/48/da/61ae89a88732f5a785646f3ec6125dbb640fa98a540eb2b9889caa561403/pyarrow-24.0.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:8adc8e6ce5fccf5dc707046ae4914fd537def529709cc0d285d37a7f9cd442ca", size = 51909252, upload-time = "2026-04-21T10:49:31.164Z" }, + { url = "https://files.pythonhosted.org/packages/cb/1a/8dd5cafab7b66573fa91c03d06d213356ad4edd71813aa75e08ce2b3a844/pyarrow-24.0.0-cp313-cp313t-win_amd64.whl", hash = "sha256:9b18371ad2f44044b81a8d23bc2d8a9b6a6226dca775e8e16cfee640473d6c5d", size = 27388127, upload-time = "2026-04-21T10:49:37.334Z" }, + { url = "https://files.pythonhosted.org/packages/ad/80/d022a34ff05d2cbedd8ccf841fc1f532ecfa9eb5ed1711b56d0e0ea71fc9/pyarrow-24.0.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:1cc9057f0319e26333b357e17f3c2c022f1a83739b48a88b25bfd5fa2dc18838", size = 35007997, upload-time = "2026-04-21T10:49:48.796Z" }, + { url = "https://files.pythonhosted.org/packages/1a/ff/f01485fda6f4e5d441afb8dd5e7681e4db18826c1e271852f5d3957d6a80/pyarrow-24.0.0-cp314-cp314-macosx_12_0_x86_64.whl", hash = "sha256:e6f1278ee4785b6db21229374a1c9e54ec7c549de5d1efc9630b6207de7e170b", size = 36678720, upload-time = "2026-04-21T10:49:55.858Z" }, + { url = "https://files.pythonhosted.org/packages/9e/c2/2d2d5fea814237923f71b36495211f20b43a1576f9a4d6da7e751a64ec6f/pyarrow-24.0.0-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:adbbedc55506cbdabb830890444fb856bfb0060c46c6f8026c6c2f2cf86ae795", size = 45741852, upload-time = "2026-04-21T10:50:04.624Z" }, + { url = "https://files.pythonhosted.org/packages/8e/3a/28ba9c1c1ebdbb5f1b94dfebb46f207e52e6a554b7fe4132540fde29a3a0/pyarrow-24.0.0-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:ae8a1145af31d903fa9bb166824d7abe9b4681a000b0159c9fb99c11bc11ad26", size = 48889852, upload-time = "2026-04-21T10:50:12.293Z" }, + { url = "https://files.pythonhosted.org/packages/df/51/4a389acfd31dca009f8fb82d7f510bb4130f2b3a8e18cf00194d0687d8ac/pyarrow-24.0.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:d7027eba1df3b2069e2e8d80f644fa0918b68c46432af3d088ddd390d063ecde", size = 49445207, upload-time = "2026-04-21T10:50:20.677Z" }, + { url = "https://files.pythonhosted.org/packages/19/4b/0bab2b23d2ae901b1b9a03c0efd4b2d070256f8ce3fc43f6e58c167b2081/pyarrow-24.0.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:e56a1ffe9bf7b727432b89104cc0849c21582949dd7bdcb34f17b2001a351a76", size = 51954117, upload-time = "2026-04-21T10:50:29.14Z" }, + { url = "https://files.pythonhosted.org/packages/29/88/f4e9145da0417b3d2c12035a8492b35ff4a3dbc653e614fcfb51d9dedb38/pyarrow-24.0.0-cp314-cp314-win_amd64.whl", hash = "sha256:38be1808cdd068605b787e6ca9119b27eb275a0234e50212c3492331680c3b1e", size = 28001155, upload-time = "2026-04-21T10:51:22.337Z" }, + { url = "https://files.pythonhosted.org/packages/79/4f/46a49a63f43526da895b1a45bbb51d5baf8e4d77159f8528fc3e5490007f/pyarrow-24.0.0-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:418e48ce50a45a6a6c73c454677203a9c75c966cb1e92ca3370959185f197a05", size = 35250387, upload-time = "2026-04-21T10:50:35.552Z" }, + { url = "https://files.pythonhosted.org/packages/a0/da/d5e0cd5ef00796922404806d5f00325cdadc3441ce2c13fe7115f2df9a64/pyarrow-24.0.0-cp314-cp314t-macosx_12_0_x86_64.whl", hash = "sha256:2f16197705a230a78270cdd4ea8a1d57e86b2fdcbc34a1f6aebc72e65c986f9a", size = 36797102, upload-time = "2026-04-21T10:50:42.417Z" }, + { url = "https://files.pythonhosted.org/packages/34/c7/5904145b0a593a05236c882933d439b5720f0a145381179063722fbfc123/pyarrow-24.0.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:fb24ac194bfc5e86839d7dcd52092ee31e5fe6733fe11f5e3b06ef0812b20072", size = 45745118, upload-time = "2026-04-21T10:50:49.324Z" }, + { url = "https://files.pythonhosted.org/packages/13/d3/cca42fe166d1c6e4d5b80e530b7949104d10e17508a90ae202dac205ce2a/pyarrow-24.0.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:9700ebd9a51f5895ce75ff4ac4b3c47a7d4b42bc618be8e713e5d56bacf5f931", size = 48844765, upload-time = "2026-04-21T10:50:55.579Z" }, + { url = "https://files.pythonhosted.org/packages/b0/49/942c3b79878ba928324d1e17c274ed84581db8c0a749b24bcf4cbdf15bd3/pyarrow-24.0.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:d8ddd2768da81d3ee08cfea9b597f4abb4e8e1dc8ae7e204b608d23a0d3ab699", size = 49471890, upload-time = "2026-04-21T10:51:02.439Z" }, + { url = "https://files.pythonhosted.org/packages/76/97/ff71431000a75d84135a1ace5ca4ba11726a231a8007bbb320a4c54075d5/pyarrow-24.0.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:61a3d7eaa97a14768b542f3d284dc6400dd2470d9f080708b13cd46b6ae18136", size = 51932250, upload-time = "2026-04-21T10:51:10.576Z" }, + { url = "https://files.pythonhosted.org/packages/51/be/6f79d55816d5c22557cf27533543d5d70dfe692adfbee4b99f2760674f38/pyarrow-24.0.0-cp314-cp314t-win_amd64.whl", hash = "sha256:c91d00057f23b8d353039520dc3a6c09d8608164c692e9f59a175a42b2ae0c19", size = 28131282, upload-time = "2026-04-21T10:51:16.815Z" }, +] + [[package]] name = "pydantic" version = "2.13.4" @@ -1300,6 +2098,24 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/f4/7e/a72dd26f3b0f4f2bf1dd8923c85f7ceb43172af56d63c7383eb62b332364/pygments-2.20.0-py3-none-any.whl", hash = "sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176", size = 1231151, upload-time = "2026-03-29T13:29:30.038Z" }, ] +[[package]] +name = "pytablewriter" +version = "1.2.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "dataproperty" }, + { name = "mbstrdecoder" }, + { name = "pathvalidate" }, + { name = "setuptools" }, + { name = "tabledata" }, + { name = "tcolorpy" }, + { name = "typepy", extra = ["datetime"] }, +] +sdist = { url = "https://files.pythonhosted.org/packages/f6/a1/617730f290f04d347103ab40bf67d317df6691b14746f6e1ea039fb57062/pytablewriter-1.2.1.tar.gz", hash = "sha256:7bd0f4f397e070e3b8a34edcf1b9257ccbb18305493d8350a5dbc9957fced959", size = 619241, upload-time = "2025-01-01T15:37:00.04Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/21/4c/c199512f01c845dfe5a7840ab3aae6c60463b5dc2a775be72502dfd9170a/pytablewriter-1.2.1-py3-none-any.whl", hash = "sha256:e906ff7ff5151d70a5f66e0f7b75642a7f2dce8d893c265b79cc9cf6bc04ddb4", size = 91083, upload-time = "2025-01-01T15:36:55.63Z" }, +] + [[package]] name = "pytest" version = "9.0.3" @@ -1374,6 +2190,25 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/ec/dd/96da98f892250475bdf2328112d7468abdd4acc7b902b6af23f4ed958ea0/pytz-2026.2-py2.py3-none-any.whl", hash = "sha256:04156e608bee23d3792fd45c94ae47fae1036688e75032eea2e3bf0323d1f126", size = 510141, upload-time = "2026-05-04T01:35:27.408Z" }, ] +[[package]] +name = "pywin32" +version = "312" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/83/ff/32aa7d2ed0ab12b323aaa64f9b75e6ad4f8fd09f9ccfc28c79414d46838d/pywin32-312-cp312-cp312-win32.whl", hash = "sha256:dab4f65ac9c4e48400a2a0530c46c3c579cd5905ecd11b80692373915269208b", size = 6371877, upload-time = "2026-06-04T07:49:28.836Z" }, + { url = "https://files.pythonhosted.org/packages/03/d9/77040d3b43df3f3be32ea289433d660d2727f5ba327bc73be835127d9d60/pywin32-312-cp312-cp312-win_amd64.whl", hash = "sha256:b457f6d628a47e8a7346ce22acb7e1a46a4a78b52e1d17e1af56871bd19a93bc", size = 6914841, upload-time = "2026-06-04T07:49:31.85Z" }, + { url = "https://files.pythonhosted.org/packages/e3/cc/7b1ec671775756020a0ee7f4feeaf3c568f0ab86bd3900088cf986937a92/pywin32-312-cp312-cp312-win_arm64.whl", hash = "sha256:6017c58e12f6809fbb0555b75df144c2922a9ffd18e4b9b5afa863b6c1a9d950", size = 6727901, upload-time = "2026-06-04T07:49:34.244Z" }, + { url = "https://files.pythonhosted.org/packages/2d/41/12fbfd7f36ed2146d8bc9de96c2741296bf0d490b98508496cff322e274c/pywin32-312-cp313-cp313-win32.whl", hash = "sha256:7a27df850933d16a8eabfbaeb73d52b273e2da667f80d70b01a89d1f6828d02c", size = 6370184, upload-time = "2026-06-04T07:49:36.253Z" }, + { url = "https://files.pythonhosted.org/packages/ba/db/36a78e3403099d31d9746d13fdcde5accc43c1155f375a34d15983a479a7/pywin32-312-cp313-cp313-win_amd64.whl", hash = "sha256:c53e878d15a1c44788082bfe712a905433473aa38f86375b7cf8b45e3acbaaf9", size = 6914298, upload-time = "2026-06-04T07:49:38.876Z" }, + { url = "https://files.pythonhosted.org/packages/84/37/c1697194092b76de9ed47ca124323f02c57ffc8a45c06f88a3d5acaf01eb/pywin32-312-cp313-cp313-win_arm64.whl", hash = "sha256:59aba5d5940842075343a5ddc6b11f1cdf0d1567fe745290359dfbcc7c2eb831", size = 6727640, upload-time = "2026-06-04T07:49:41.083Z" }, + { url = "https://files.pythonhosted.org/packages/fc/2b/1f3cded5822fd49c02f40544cbb5f58c7cfd6b1694869fd476cb6170ee97/pywin32-312-cp314-cp314-win32.whl", hash = "sha256:a77a90fbb6881238d2ca9c6fd797b25817f3768fe78d214a90137ff055a75f5b", size = 6468928, upload-time = "2026-06-04T07:49:43.188Z" }, + { url = "https://files.pythonhosted.org/packages/21/82/3bf86d2e2808902013132e1ce905a7da0da53790f3836c64bf44d55e24f3/pywin32-312-cp314-cp314-win_amd64.whl", hash = "sha256:a4dd3a848290ef724347b19f301045831d8e802fa4464f491b98b1e0a081432e", size = 7024157, upload-time = "2026-06-04T07:49:45.34Z" }, + { url = "https://files.pythonhosted.org/packages/a4/0e/73f6d6800b4f27655abd9e9f6aaeaefcddb2b946e4674efa2bab184a7f7b/pywin32-312-cp314-cp314-win_arm64.whl", hash = "sha256:9fce94568364e0155e6dfb781ac5d95903be8baf28670632beab1b523f300daa", size = 6839598, upload-time = "2026-06-04T07:49:47.613Z" }, + { url = "https://files.pythonhosted.org/packages/eb/61/caa39686032d2ebdd04ff0ab5cbe163126c0066d98e00c9018646e42393b/pywin32-312-cp315-cp315-win32.whl", hash = "sha256:5c1fbe4a937a73ae9297384a3da38518cbc694c68ad8a809b2e19acd350f03ed", size = 6471159, upload-time = "2026-06-04T07:49:50.035Z" }, + { url = "https://files.pythonhosted.org/packages/0f/cd/7e1de64a4a6f69c04214169657ccab0d93a670ea50e35eb8f489d7378249/pywin32-312-cp315-cp315-win_amd64.whl", hash = "sha256:c2f03a0f73f804a13c2735b99392b0cd426bb4f2c4d0178e5ac966a0f21618d5", size = 7025293, upload-time = "2026-06-04T07:49:54.857Z" }, + { url = "https://files.pythonhosted.org/packages/23/ed/4532e9388e65fa16b46776ef47ad631a64eda1631884488af707666350ed/pywin32-312-cp315-cp315-win_arm64.whl", hash = "sha256:a8597d28f267b39074aef51fa593530082b39cbe5a074226096857b1fed2dfb9", size = 6840337, upload-time = "2026-06-04T07:49:57.531Z" }, +] + [[package]] name = "pyyaml" version = "6.0.3" @@ -1508,6 +2343,21 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/da/80/35b4c33c804a165a7f55289afda3ea9e3eb6d15800341a2d66455c0f1f30/regex-2026.5.9-cp314-cp314t-win_arm64.whl", hash = "sha256:5e41809d2683fcde7d5a8c87a6567ba1fb1ce0de9f31bff578de00a4b2d76daa", size = 275713, upload-time = "2026-05-09T23:15:16.98Z" }, ] +[[package]] +name = "requests" +version = "2.34.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "certifi" }, + { name = "charset-normalizer" }, + { name = "idna" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ac/c3/e2a2b89f2d3e2179abd6d00ebd70bff6273f37fb3e0cc209f48b39d00cbf/requests-2.34.2.tar.gz", hash = "sha256:f288924cae4e29463698d6d60bc6a4da69c89185ad1e0bcc4104f584e960b9ed", size = 142856, upload-time = "2026-05-14T19:25:27.735Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a0/f4/c67b0b3f1b9245e8d266f0f112c500d50e5b4e83cb6f3b71b6528104182a/requests-2.34.2-py3-none-any.whl", hash = "sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0", size = 73075, upload-time = "2026-05-14T19:25:26.443Z" }, +] + [[package]] name = "rich" version = "15.0.0" @@ -1521,6 +2371,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/82/3b/64d4899d73f91ba49a8c18a8ff3f0ea8f1c1d75481760df8c68ef5235bf5/rich-15.0.0-py3-none-any.whl", hash = "sha256:33bd4ef74232fb73fe9279a257718407f169c09b78a87ad3d296f548e27de0bb", size = 310654, upload-time = "2026-04-12T08:24:02.83Z" }, ] +[[package]] +name = "rouge-score" +version = "0.1.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "absl-py" }, + { name = "nltk" }, + { name = "numpy" }, + { name = "six" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/e2/c5/9136736c37022a6ad27fea38f3111eb8f02fe75d067f9a985cc358653102/rouge_score-0.1.2.tar.gz", hash = "sha256:c7d4da2683e68c9abf0135ef915d63a46643666f848e558a1b9f7ead17ff0f04", size = 17400, upload-time = "2022-07-22T22:46:22.909Z" } + [[package]] name = "ruff" version = "0.15.16" @@ -1546,6 +2408,23 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/15/19/016553f86f207450aebebc2b2b5088d086b901cc8186c02ac4284db3bd88/ruff-0.15.16-py3-none-win_arm64.whl", hash = "sha256:8cd61783afb39638a7133ef0d2dfb1e91277593962f81b5a8423eb0b888a6121", size = 11134555, upload-time = "2026-06-04T16:33:00.136Z" }, ] +[[package]] +name = "sacrebleu" +version = "2.6.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama" }, + { name = "lxml" }, + { name = "numpy" }, + { name = "portalocker" }, + { name = "regex" }, + { name = "tabulate" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/d3/ed/d7acddcff74d690c56fe26a1f7828bdde548262828d0743414ea916c40c1/sacrebleu-2.6.0.tar.gz", hash = "sha256:91499b6cd46138d95154fff1e863c2f9be57e82f0c719d8dd718d0006cf6c566", size = 1893419, upload-time = "2026-01-12T17:17:20.799Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/06/f2/6c90ccf3ad1d09a7d662a405b274f3c93b92df59c8d6a025d26aaf34d302/sacrebleu-2.6.0-py3-none-any.whl", hash = "sha256:3edc1531575cfe4ad04ce53491a9307e234af1c3f805a1f491cbec844229a8a8", size = 100785, upload-time = "2026-01-12T17:17:18.868Z" }, +] + [[package]] name = "safehttpx" version = "0.1.7" @@ -1580,6 +2459,106 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/5d/e6/ec8471c8072382cb91233ba7267fd931219753bb43814cbc71757bfd4dab/safetensors-0.7.0-cp38-abi3-win_amd64.whl", hash = "sha256:d1239932053f56f3456f32eb9625590cc7582e905021f94636202a864d470755", size = 341380, upload-time = "2025-11-19T15:18:44.427Z" }, ] +[[package]] +name = "scikit-learn" +version = "1.9.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "joblib" }, + { name = "narwhals" }, + { name = "numpy" }, + { name = "scipy" }, + { name = "threadpoolctl" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/fa/6f/37092bdb25f712817231799fc5674d8e704066a8a70c1d2d40517e18b4ab/scikit_learn-1.9.0.tar.gz", hash = "sha256:8833266989d3a5110178a9fae30783675460724d0e1efb13b14901d2c660c557", size = 7750767, upload-time = "2026-06-02T11:54:32.706Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ac/20/75f915ff375d6249e6550ac740fdbbd66159a068fd3af1400ff62036b07a/scikit_learn-1.9.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:2bd41b0d201bc81575531b96b713d3eb5e5f50fb0b82101ff0f92294fdc236ac", size = 8741122, upload-time = "2026-06-02T11:53:24.08Z" }, + { url = "https://files.pythonhosted.org/packages/cc/d5/2b5148f2279196775e1db2aeb85d14b70ac80e7e32b3b28e7ebeafb0901d/scikit_learn-1.9.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:5be45aa4a42a68a533913a6ed736cf309de2226411c79ef8d609a5456f1939b1", size = 8261512, upload-time = "2026-06-02T11:53:27.183Z" }, + { url = "https://files.pythonhosted.org/packages/a0/ee/5adbc77656b71f9456a2f5a7a9fdb4bcf9207a6b962889f1c2f9323afa4e/scikit_learn-1.9.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5e50ed4da51974e86e940690e9a3d82e729b62b5a49f7c9bac534d515d39d86f", size = 8837603, upload-time = "2026-06-02T11:53:30.328Z" }, + { url = "https://files.pythonhosted.org/packages/6c/c2/63fdda36c56437eeb44aaf9493c8bcd62ce230ab1598924fc626ffbfa943/scikit_learn-1.9.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:056c92bb67ad4c28463c2f2653d9701449201e7e7a9e94e321be0f71c4fef2b8", size = 9132097, upload-time = "2026-06-02T11:53:33.456Z" }, + { url = "https://files.pythonhosted.org/packages/83/a4/c8e67227c680e2259c8864ae72ff48b06e16a6f51253a22167aa02a8aa4e/scikit_learn-1.9.0-cp312-cp312-win_amd64.whl", hash = "sha256:4306775fad04cc4b472a1b15af1ae9cede1540fbfcc17fbce3767cd8dc7ae283", size = 8211173, upload-time = "2026-06-02T11:53:36.602Z" }, + { url = "https://files.pythonhosted.org/packages/cf/fd/3c0863792e98e67e9184aa4029288a175935eb65443afcd30d4f143450cf/scikit_learn-1.9.0-cp312-cp312-win_arm64.whl", hash = "sha256:26e22435f63bcdcf396b574273f29f13dd531f5ea035801f5be10ba1540a4e60", size = 7867451, upload-time = "2026-06-02T11:53:39.075Z" }, + { url = "https://files.pythonhosted.org/packages/3c/01/cf3310626b6d48d3e9be69a1223f9180360b5e6edb045f50fade723ce494/scikit_learn-1.9.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:80746d63bd4b6eaca54d36fe5feaf4d28bb38dc6f9470f81c7cad7c40155f119", size = 8705188, upload-time = "2026-06-02T11:53:41.964Z" }, + { url = "https://files.pythonhosted.org/packages/3e/04/5acd7ae280c5f93b6ac5ef6cdec14eef4c8d1cd91d85b3292989c94d96b1/scikit_learn-1.9.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:5b934c45c252844a91d69fda3a34cff5e7307e1db10d77cb10a3980312c74713", size = 8228299, upload-time = "2026-06-02T11:53:44.817Z" }, + { url = "https://files.pythonhosted.org/packages/0c/39/ffe829a5b8ecb40a518724a997794657fdc354ada5e8fe8e64d998c0bac9/scikit_learn-1.9.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:38c3dcb9a1ffb85505ec53d54c7b4aea0cff70050425a7760c2af661ac85df05", size = 8789690, upload-time = "2026-06-02T11:53:47.461Z" }, + { url = "https://files.pythonhosted.org/packages/1f/88/8dab5de10c638c083772a6be83a3d8106ced492f74a928c8693638e5bb50/scikit_learn-1.9.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:da76d09304a4706db7cc1e3ebaa3b6b98a67365cc11d2996c4f1e58ba47df714", size = 9087723, upload-time = "2026-06-02T11:53:50.702Z" }, + { url = "https://files.pythonhosted.org/packages/20/3f/7917ca72464038f6240ec70c29f94862d08a34a74291ae4d4ec5eb8186a0/scikit_learn-1.9.0-cp313-cp313-win_amd64.whl", hash = "sha256:5808d98f15c6bf6d9d96d2348c1997392a5888ce7097e664105f930c4bca1277", size = 8184330, upload-time = "2026-06-02T11:53:53.396Z" }, + { url = "https://files.pythonhosted.org/packages/78/c7/15739eb2f61fda3c54639e9942414e5a19ad8a8d1f5a3266afad7cb7df80/scikit_learn-1.9.0-cp313-cp313-win_arm64.whl", hash = "sha256:d77f54c017633791bc0225a43e2f8d03745fdcfe4880268fcc4df15f505dec2e", size = 7840653, upload-time = "2026-06-02T11:53:56.035Z" }, + { url = "https://files.pythonhosted.org/packages/f4/7d/c9a35cf59b20a86fec24d306f1547b78dec194b08d367ce2a3e4854169d9/scikit_learn-1.9.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:9656acd4e93f74e0b66c8a36c88830a99252dfa900044d36bc2212ae89a47162", size = 8713289, upload-time = "2026-06-02T11:53:58.788Z" }, + { url = "https://files.pythonhosted.org/packages/3c/a7/552a7821597c632b907f7bfe8f36f9f572777af8ef8a48353041cf8e091a/scikit_learn-1.9.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:24360002ae845e7866522b0a5bbf690802e7bc388cac8663502e78aa98598aa2", size = 8245141, upload-time = "2026-06-02T11:54:01.694Z" }, + { url = "https://files.pythonhosted.org/packages/7d/79/f4a0c4fe9711154cddabf913471153af79056382ddc612cfe5ee0ff4b72e/scikit_learn-1.9.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5162ad10a418c8a282dde04c9aa06965de3e9a65f33c1440c0ae69bb1a09d913", size = 8847671, upload-time = "2026-06-02T11:54:04.448Z" }, + { url = "https://files.pythonhosted.org/packages/f0/af/4d72d9e475ac83719160c662619e4bf7b95c19507cd582e7d0167a3c3dae/scikit_learn-1.9.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1fea2cc5677ab49d6f5bade978c866da44957b712d92e9635e8b4f723013c3cb", size = 9118104, upload-time = "2026-06-02T11:54:07.205Z" }, + { url = "https://files.pythonhosted.org/packages/a2/d5/6a58eea2cb9abbb9b3f2bb8b2cfb3243d1152d69f442d256c7af71304769/scikit_learn-1.9.0-cp314-cp314-win_amd64.whl", hash = "sha256:64fa347efc1c839c487433e40c5144d38c336e8a2b59c81aa8660373945c2673", size = 8290674, upload-time = "2026-06-02T11:54:10.087Z" }, + { url = "https://files.pythonhosted.org/packages/65/5b/d4c879cf358f1187141cf90ced473f087183489090244f50c124a2ee478b/scikit_learn-1.9.0-cp314-cp314-win_arm64.whl", hash = "sha256:1b944b6db288f6b926e3650026ddafb988929de95d11fc2cc5fa117773c9ba42", size = 7978807, upload-time = "2026-06-02T11:54:12.769Z" }, + { url = "https://files.pythonhosted.org/packages/8a/43/bfae3121ec67ae09150d453c442c7c1cc166e9aefe056e6ab3b7728a5cfc/scikit_learn-1.9.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:4ccacf04ca5f4b492158a5f28afe0ace43f81b2571e4b9a66d34848b46128949", size = 9031941, upload-time = "2026-06-02T11:54:15.436Z" }, + { url = "https://files.pythonhosted.org/packages/75/b0/20a4546eb17f3b25d3c66df15810411c14ed5065bcfab50b53c96fb627b2/scikit_learn-1.9.0-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:ee1a8db2c18c08e34c7412d4b10be1cac214cd4ea7dc9715a6a327eb49a37c96", size = 8613528, upload-time = "2026-06-02T11:54:18.842Z" }, + { url = "https://files.pythonhosted.org/packages/18/3c/e440e039bb82cd19004edaaad00acbde0fb9b461083c3ecf37941c557312/scikit_learn-1.9.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:147e9329ef0e39f75d4cffa02b2aa48d827832684926cd5210d9a2cb5c57246b", size = 8855050, upload-time = "2026-06-02T11:54:21.699Z" }, + { url = "https://files.pythonhosted.org/packages/43/26/b341b8dab5998da6270a3a42c2152c578501354d36f944b5856757035ef8/scikit_learn-1.9.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5bad8f8b9950321b54c965fdcbac6c6c55e79e16646b49977bcf3668d3870a1a", size = 9097190, upload-time = "2026-06-02T11:54:24.454Z" }, + { url = "https://files.pythonhosted.org/packages/fb/de/b650b4d69b84468cfa2e28a3ff7b8103743029e6446ce1a97fe060ef688c/scikit_learn-1.9.0-cp314-cp314t-win_amd64.whl", hash = "sha256:78fc56eafd4edb9575d2d8950d1dd152061abb573341a1cb7e099fc40f6c6666", size = 8963204, upload-time = "2026-06-02T11:54:27.428Z" }, + { url = "https://files.pythonhosted.org/packages/ee/f3/ff83d76d7418112e5a61326443cdda87be3545dd8d6599c95b2481a4419e/scikit_learn-1.9.0-cp314-cp314t-win_arm64.whl", hash = "sha256:051075bda8b7aab87b1906ab3d4740a1e1224a19d7b3781a576736edc94e76aa", size = 8222661, upload-time = "2026-06-02T11:54:30.192Z" }, +] + +[[package]] +name = "scipy" +version = "1.17.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/7a/97/5a3609c4f8d58b039179648e62dd220f89864f56f7357f5d4f45c29eb2cc/scipy-1.17.1.tar.gz", hash = "sha256:95d8e012d8cb8816c226aef832200b1d45109ed4464303e997c5b13122b297c0", size = 30573822, upload-time = "2026-02-23T00:26:24.851Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/35/48/b992b488d6f299dbe3f11a20b24d3dda3d46f1a635ede1c46b5b17a7b163/scipy-1.17.1-cp312-cp312-macosx_10_14_x86_64.whl", hash = "sha256:35c3a56d2ef83efc372eaec584314bd0ef2e2f0d2adb21c55e6ad5b344c0dcb8", size = 31610954, upload-time = "2026-02-23T00:17:49.855Z" }, + { url = "https://files.pythonhosted.org/packages/b2/02/cf107b01494c19dc100f1d0b7ac3cc08666e96ba2d64db7626066cee895e/scipy-1.17.1-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:fcb310ddb270a06114bb64bbe53c94926b943f5b7f0842194d585c65eb4edd76", size = 28172662, upload-time = "2026-02-23T00:18:01.64Z" }, + { url = "https://files.pythonhosted.org/packages/cf/a9/599c28631bad314d219cf9ffd40e985b24d603fc8a2f4ccc5ae8419a535b/scipy-1.17.1-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:cc90d2e9c7e5c7f1a482c9875007c095c3194b1cfedca3c2f3291cdc2bc7c086", size = 20344366, upload-time = "2026-02-23T00:18:12.015Z" }, + { url = "https://files.pythonhosted.org/packages/35/f5/906eda513271c8deb5af284e5ef0206d17a96239af79f9fa0aebfe0e36b4/scipy-1.17.1-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:c80be5ede8f3f8eded4eff73cc99a25c388ce98e555b17d31da05287015ffa5b", size = 22704017, upload-time = "2026-02-23T00:18:21.502Z" }, + { url = "https://files.pythonhosted.org/packages/da/34/16f10e3042d2f1d6b66e0428308ab52224b6a23049cb2f5c1756f713815f/scipy-1.17.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e19ebea31758fac5893a2ac360fedd00116cbb7628e650842a6691ba7ca28a21", size = 32927842, upload-time = "2026-02-23T00:18:35.367Z" }, + { url = "https://files.pythonhosted.org/packages/01/8e/1e35281b8ab6d5d72ebe9911edcdffa3f36b04ed9d51dec6dd140396e220/scipy-1.17.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:02ae3b274fde71c5e92ac4d54bc06c42d80e399fec704383dcd99b301df37458", size = 35235890, upload-time = "2026-02-23T00:18:49.188Z" }, + { url = "https://files.pythonhosted.org/packages/c5/5c/9d7f4c88bea6e0d5a4f1bc0506a53a00e9fcb198de372bfe4d3652cef482/scipy-1.17.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8a604bae87c6195d8b1045eddece0514d041604b14f2727bbc2b3020172045eb", size = 35003557, upload-time = "2026-02-23T00:18:54.74Z" }, + { url = "https://files.pythonhosted.org/packages/65/94/7698add8f276dbab7a9de9fb6b0e02fc13ee61d51c7c3f85ac28b65e1239/scipy-1.17.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:f590cd684941912d10becc07325a3eeb77886fe981415660d9265c4c418d0bea", size = 37625856, upload-time = "2026-02-23T00:19:00.307Z" }, + { url = "https://files.pythonhosted.org/packages/a2/84/dc08d77fbf3d87d3ee27f6a0c6dcce1de5829a64f2eae85a0ecc1f0daa73/scipy-1.17.1-cp312-cp312-win_amd64.whl", hash = "sha256:41b71f4a3a4cab9d366cd9065b288efc4d4f3c0b37a91a8e0947fb5bd7f31d87", size = 36549682, upload-time = "2026-02-23T00:19:07.67Z" }, + { url = "https://files.pythonhosted.org/packages/bc/98/fe9ae9ffb3b54b62559f52dedaebe204b408db8109a8c66fdd04869e6424/scipy-1.17.1-cp312-cp312-win_arm64.whl", hash = "sha256:f4115102802df98b2b0db3cce5cb9b92572633a1197c77b7553e5203f284a5b3", size = 24547340, upload-time = "2026-02-23T00:19:12.024Z" }, + { url = "https://files.pythonhosted.org/packages/76/27/07ee1b57b65e92645f219b37148a7e7928b82e2b5dbeccecb4dff7c64f0b/scipy-1.17.1-cp313-cp313-macosx_10_14_x86_64.whl", hash = "sha256:5e3c5c011904115f88a39308379c17f91546f77c1667cea98739fe0fccea804c", size = 31590199, upload-time = "2026-02-23T00:19:17.192Z" }, + { url = "https://files.pythonhosted.org/packages/ec/ae/db19f8ab842e9b724bf5dbb7db29302a91f1e55bc4d04b1025d6d605a2c5/scipy-1.17.1-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:6fac755ca3d2c3edcb22f479fceaa241704111414831ddd3bc6056e18516892f", size = 28154001, upload-time = "2026-02-23T00:19:22.241Z" }, + { url = "https://files.pythonhosted.org/packages/5b/58/3ce96251560107b381cbd6e8413c483bbb1228a6b919fa8652b0d4090e7f/scipy-1.17.1-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:7ff200bf9d24f2e4d5dc6ee8c3ac64d739d3a89e2326ba68aaf6c4a2b838fd7d", size = 20325719, upload-time = "2026-02-23T00:19:26.329Z" }, + { url = "https://files.pythonhosted.org/packages/b2/83/15087d945e0e4d48ce2377498abf5ad171ae013232ae31d06f336e64c999/scipy-1.17.1-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:4b400bdc6f79fa02a4d86640310dde87a21fba0c979efff5248908c6f15fad1b", size = 22683595, upload-time = "2026-02-23T00:19:30.304Z" }, + { url = "https://files.pythonhosted.org/packages/b4/e0/e58fbde4a1a594c8be8114eb4aac1a55bcd6587047efc18a61eb1f5c0d30/scipy-1.17.1-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2b64ca7d4aee0102a97f3ba22124052b4bd2152522355073580bf4845e2550b6", size = 32896429, upload-time = "2026-02-23T00:19:35.536Z" }, + { url = "https://files.pythonhosted.org/packages/f5/5f/f17563f28ff03c7b6799c50d01d5d856a1d55f2676f537ca8d28c7f627cd/scipy-1.17.1-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:581b2264fc0aa555f3f435a5944da7504ea3a065d7029ad60e7c3d1ae09c5464", size = 35203952, upload-time = "2026-02-23T00:19:42.259Z" }, + { url = "https://files.pythonhosted.org/packages/8d/a5/9afd17de24f657fdfe4df9a3f1ea049b39aef7c06000c13db1530d81ccca/scipy-1.17.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:beeda3d4ae615106d7094f7e7cef6218392e4465cc95d25f900bebabfded0950", size = 34979063, upload-time = "2026-02-23T00:19:47.547Z" }, + { url = "https://files.pythonhosted.org/packages/8b/13/88b1d2384b424bf7c924f2038c1c409f8d88bb2a8d49d097861dd64a57b2/scipy-1.17.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6609bc224e9568f65064cfa72edc0f24ee6655b47575954ec6339534b2798369", size = 37598449, upload-time = "2026-02-23T00:19:53.238Z" }, + { url = "https://files.pythonhosted.org/packages/35/e5/d6d0e51fc888f692a35134336866341c08655d92614f492c6860dc45bb2c/scipy-1.17.1-cp313-cp313-win_amd64.whl", hash = "sha256:37425bc9175607b0268f493d79a292c39f9d001a357bebb6b88fdfaff13f6448", size = 36510943, upload-time = "2026-02-23T00:20:50.89Z" }, + { url = "https://files.pythonhosted.org/packages/2a/fd/3be73c564e2a01e690e19cc618811540ba5354c67c8680dce3281123fb79/scipy-1.17.1-cp313-cp313-win_arm64.whl", hash = "sha256:5cf36e801231b6a2059bf354720274b7558746f3b1a4efb43fcf557ccd484a87", size = 24545621, upload-time = "2026-02-23T00:20:55.871Z" }, + { url = "https://files.pythonhosted.org/packages/6f/6b/17787db8b8114933a66f9dcc479a8272e4b4da75fe03b0c282f7b0ade8cd/scipy-1.17.1-cp313-cp313t-macosx_10_14_x86_64.whl", hash = "sha256:d59c30000a16d8edc7e64152e30220bfbd724c9bbb08368c054e24c651314f0a", size = 31936708, upload-time = "2026-02-23T00:19:58.694Z" }, + { url = "https://files.pythonhosted.org/packages/38/2e/524405c2b6392765ab1e2b722a41d5da33dc5c7b7278184a8ad29b6cb206/scipy-1.17.1-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:010f4333c96c9bb1a4516269e33cb5917b08ef2166d5556ca2fd9f082a9e6ea0", size = 28570135, upload-time = "2026-02-23T00:20:03.934Z" }, + { url = "https://files.pythonhosted.org/packages/fd/c3/5bd7199f4ea8556c0c8e39f04ccb014ac37d1468e6cfa6a95c6b3562b76e/scipy-1.17.1-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:2ceb2d3e01c5f1d83c4189737a42d9cb2fc38a6eeed225e7515eef71ad301dce", size = 20741977, upload-time = "2026-02-23T00:20:07.935Z" }, + { url = "https://files.pythonhosted.org/packages/d9/b8/8ccd9b766ad14c78386599708eb745f6b44f08400a5fd0ade7cf89b6fc93/scipy-1.17.1-cp313-cp313t-macosx_14_0_x86_64.whl", hash = "sha256:844e165636711ef41f80b4103ed234181646b98a53c8f05da12ca5ca289134f6", size = 23029601, upload-time = "2026-02-23T00:20:12.161Z" }, + { url = "https://files.pythonhosted.org/packages/6d/a0/3cb6f4d2fb3e17428ad2880333cac878909ad1a89f678527b5328b93c1d4/scipy-1.17.1-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:158dd96d2207e21c966063e1635b1063cd7787b627b6f07305315dd73d9c679e", size = 33019667, upload-time = "2026-02-23T00:20:17.208Z" }, + { url = "https://files.pythonhosted.org/packages/f3/c3/2d834a5ac7bf3a0c806ad1508efc02dda3c8c61472a56132d7894c312dea/scipy-1.17.1-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:74cbb80d93260fe2ffa334efa24cb8f2f0f622a9b9febf8b483c0b865bfb3475", size = 35264159, upload-time = "2026-02-23T00:20:23.087Z" }, + { url = "https://files.pythonhosted.org/packages/4d/77/d3ed4becfdbd217c52062fafe35a72388d1bd82c2d0ba5ca19d6fcc93e11/scipy-1.17.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:dbc12c9f3d185f5c737d801da555fb74b3dcfa1a50b66a1a93e09190f41fab50", size = 35102771, upload-time = "2026-02-23T00:20:28.636Z" }, + { url = "https://files.pythonhosted.org/packages/bd/12/d19da97efde68ca1ee5538bb261d5d2c062f0c055575128f11a2730e3ac1/scipy-1.17.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:94055a11dfebe37c656e70317e1996dc197e1a15bbcc351bcdd4610e128fe1ca", size = 37665910, upload-time = "2026-02-23T00:20:34.743Z" }, + { url = "https://files.pythonhosted.org/packages/06/1c/1172a88d507a4baaf72c5a09bb6c018fe2ae0ab622e5830b703a46cc9e44/scipy-1.17.1-cp313-cp313t-win_amd64.whl", hash = "sha256:e30bdeaa5deed6bc27b4cc490823cd0347d7dae09119b8803ae576ea0ce52e4c", size = 36562980, upload-time = "2026-02-23T00:20:40.575Z" }, + { url = "https://files.pythonhosted.org/packages/70/b0/eb757336e5a76dfa7911f63252e3b7d1de00935d7705cf772db5b45ec238/scipy-1.17.1-cp313-cp313t-win_arm64.whl", hash = "sha256:a720477885a9d2411f94a93d16f9d89bad0f28ca23c3f8daa521e2dcc3f44d49", size = 24856543, upload-time = "2026-02-23T00:20:45.313Z" }, + { url = "https://files.pythonhosted.org/packages/cf/83/333afb452af6f0fd70414dc04f898647ee1423979ce02efa75c3b0f2c28e/scipy-1.17.1-cp314-cp314-macosx_10_14_x86_64.whl", hash = "sha256:a48a72c77a310327f6a3a920092fa2b8fd03d7deaa60f093038f22d98e096717", size = 31584510, upload-time = "2026-02-23T00:21:01.015Z" }, + { url = "https://files.pythonhosted.org/packages/ed/a6/d05a85fd51daeb2e4ea71d102f15b34fedca8e931af02594193ae4fd25f7/scipy-1.17.1-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:45abad819184f07240d8a696117a7aacd39787af9e0b719d00285549ed19a1e9", size = 28170131, upload-time = "2026-02-23T00:21:05.888Z" }, + { url = "https://files.pythonhosted.org/packages/db/7b/8624a203326675d7746a254083a187398090a179335b2e4a20e2ddc46e83/scipy-1.17.1-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:3fd1fcdab3ea951b610dc4cef356d416d5802991e7e32b5254828d342f7b7e0b", size = 20342032, upload-time = "2026-02-23T00:21:09.904Z" }, + { url = "https://files.pythonhosted.org/packages/c9/35/2c342897c00775d688d8ff3987aced3426858fd89d5a0e26e020b660b301/scipy-1.17.1-cp314-cp314-macosx_14_0_x86_64.whl", hash = "sha256:7bdf2da170b67fdf10bca777614b1c7d96ae3ca5794fd9587dce41eb2966e866", size = 22678766, upload-time = "2026-02-23T00:21:14.313Z" }, + { url = "https://files.pythonhosted.org/packages/ef/f2/7cdb8eb308a1a6ae1e19f945913c82c23c0c442a462a46480ce487fdc0ac/scipy-1.17.1-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:adb2642e060a6549c343603a3851ba76ef0b74cc8c079a9a58121c7ec9fe2350", size = 32957007, upload-time = "2026-02-23T00:21:19.663Z" }, + { url = "https://files.pythonhosted.org/packages/0b/2e/7eea398450457ecb54e18e9d10110993fa65561c4f3add5e8eccd2b9cd41/scipy-1.17.1-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:eee2cfda04c00a857206a4330f0c5e3e56535494e30ca445eb19ec624ae75118", size = 35221333, upload-time = "2026-02-23T00:21:25.278Z" }, + { url = "https://files.pythonhosted.org/packages/d9/77/5b8509d03b77f093a0d52e606d3c4f79e8b06d1d38c441dacb1e26cacf46/scipy-1.17.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:d2650c1fb97e184d12d8ba010493ee7b322864f7d3d00d3f9bb97d9c21de4068", size = 35042066, upload-time = "2026-02-23T00:21:31.358Z" }, + { url = "https://files.pythonhosted.org/packages/f9/df/18f80fb99df40b4070328d5ae5c596f2f00fffb50167e31439e932f29e7d/scipy-1.17.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:08b900519463543aa604a06bec02461558a6e1cef8fdbb8098f77a48a83c8118", size = 37612763, upload-time = "2026-02-23T00:21:37.247Z" }, + { url = "https://files.pythonhosted.org/packages/4b/39/f0e8ea762a764a9dc52aa7dabcfad51a354819de1f0d4652b6a1122424d6/scipy-1.17.1-cp314-cp314-win_amd64.whl", hash = "sha256:3877ac408e14da24a6196de0ddcace62092bfc12a83823e92e49e40747e52c19", size = 37290984, upload-time = "2026-02-23T00:22:35.023Z" }, + { url = "https://files.pythonhosted.org/packages/7c/56/fe201e3b0f93d1a8bcf75d3379affd228a63d7e2d80ab45467a74b494947/scipy-1.17.1-cp314-cp314-win_arm64.whl", hash = "sha256:f8885db0bc2bffa59d5c1b72fad7a6a92d3e80e7257f967dd81abb553a90d293", size = 25192877, upload-time = "2026-02-23T00:22:39.798Z" }, + { url = "https://files.pythonhosted.org/packages/96/ad/f8c414e121f82e02d76f310f16db9899c4fcde36710329502a6b2a3c0392/scipy-1.17.1-cp314-cp314t-macosx_10_14_x86_64.whl", hash = "sha256:1cc682cea2ae55524432f3cdff9e9a3be743d52a7443d0cba9017c23c87ae2f6", size = 31949750, upload-time = "2026-02-23T00:21:42.289Z" }, + { url = "https://files.pythonhosted.org/packages/7c/b0/c741e8865d61b67c81e255f4f0a832846c064e426636cd7de84e74d209be/scipy-1.17.1-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:2040ad4d1795a0ae89bfc7e8429677f365d45aa9fd5e4587cf1ea737f927b4a1", size = 28585858, upload-time = "2026-02-23T00:21:47.706Z" }, + { url = "https://files.pythonhosted.org/packages/ed/1b/3985219c6177866628fa7c2595bfd23f193ceebbe472c98a08824b9466ff/scipy-1.17.1-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:131f5aaea57602008f9822e2115029b55d4b5f7c070287699fe45c661d051e39", size = 20757723, upload-time = "2026-02-23T00:21:52.039Z" }, + { url = "https://files.pythonhosted.org/packages/c0/19/2a04aa25050d656d6f7b9e7b685cc83d6957fb101665bfd9369ca6534563/scipy-1.17.1-cp314-cp314t-macosx_14_0_x86_64.whl", hash = "sha256:9cdc1a2fcfd5c52cfb3045feb399f7b3ce822abdde3a193a6b9a60b3cb5854ca", size = 23043098, upload-time = "2026-02-23T00:21:56.185Z" }, + { url = "https://files.pythonhosted.org/packages/86/f1/3383beb9b5d0dbddd030335bf8a8b32d4317185efe495374f134d8be6cce/scipy-1.17.1-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6e3dcd57ab780c741fde8dc68619de988b966db759a3c3152e8e9142c26295ad", size = 33030397, upload-time = "2026-02-23T00:22:01.404Z" }, + { url = "https://files.pythonhosted.org/packages/41/68/8f21e8a65a5a03f25a79165ec9d2b28c00e66dc80546cf5eb803aeeff35b/scipy-1.17.1-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a9956e4d4f4a301ebf6cde39850333a6b6110799d470dbbb1e25326ac447f52a", size = 35281163, upload-time = "2026-02-23T00:22:07.024Z" }, + { url = "https://files.pythonhosted.org/packages/84/8d/c8a5e19479554007a5632ed7529e665c315ae7492b4f946b0deb39870e39/scipy-1.17.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:a4328d245944d09fd639771de275701ccadf5f781ba0ff092ad141e017eccda4", size = 35116291, upload-time = "2026-02-23T00:22:12.585Z" }, + { url = "https://files.pythonhosted.org/packages/52/52/e57eceff0e342a1f50e274264ed47497b59e6a4e3118808ee58ddda7b74a/scipy-1.17.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:a77cbd07b940d326d39a1d1b37817e2ee4d79cb30e7338f3d0cddffae70fcaa2", size = 37682317, upload-time = "2026-02-23T00:22:18.513Z" }, + { url = "https://files.pythonhosted.org/packages/11/2f/b29eafe4a3fbc3d6de9662b36e028d5f039e72d345e05c250e121a230dd4/scipy-1.17.1-cp314-cp314t-win_amd64.whl", hash = "sha256:eb092099205ef62cd1782b006658db09e2fed75bffcae7cc0d44052d8aa0f484", size = 37345327, upload-time = "2026-02-23T00:22:24.442Z" }, + { url = "https://files.pythonhosted.org/packages/07/39/338d9219c4e87f3e708f18857ecd24d22a0c3094752393319553096b98af/scipy-1.17.1-cp314-cp314t-win_arm64.whl", hash = "sha256:200e1050faffacc162be6a486a984a0497866ec54149a01270adc8a59b7c7d21", size = 25489165, upload-time = "2026-02-23T00:22:29.563Z" }, +] + [[package]] name = "semantic-version" version = "2.10.0" @@ -1616,12 +2595,53 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" }, ] +[[package]] +name = "slm-evals" +version = "0.1.0" +source = { editable = "research/evals" } +dependencies = [ + { name = "accelerate" }, + { name = "bitsandbytes" }, + { name = "datasets" }, + { name = "ensemble" }, + { name = "huggingface-hub" }, + { name = "pandas" }, + { name = "peft" }, + { name = "pyyaml" }, + { name = "torch" }, + { name = "tqdm" }, + { name = "transformers" }, +] + +[package.optional-dependencies] +lm-eval = [ + { name = "lm-eval", extra = ["hf"] }, +] + +[package.metadata] +requires-dist = [ + { name = "accelerate", specifier = ">=0.30.0" }, + { name = "bitsandbytes", specifier = ">=0.43.0" }, + { name = "datasets", specifier = ">=2.19.0" }, + { name = "ensemble", editable = "research/ensemble" }, + { name = "huggingface-hub", specifier = ">=0.22.0" }, + { name = "lm-eval", extras = ["hf"], marker = "extra == 'lm-eval'", specifier = ">=0.4.9" }, + { name = "pandas", specifier = ">=2.0.0" }, + { name = "peft", specifier = ">=0.14.0" }, + { name = "pyyaml", specifier = ">=6.0" }, + { name = "torch", specifier = ">=2.2.0" }, + { name = "tqdm", specifier = ">=4.66.0" }, + { name = "transformers", specifier = ">=4.41.0" }, +] +provides-extras = ["lm-eval"] + [[package]] name = "small-model-hackathon" version = "0.1.0" source = { virtual = "." } dependencies = [ { name = "agent" }, + { name = "ensemble" }, { name = "gradio-space" }, { name = "inference" }, ] @@ -1631,10 +2651,27 @@ dev = [ { name = "pytest" }, { name = "ruff" }, ] +ensemble = [ + { name = "accelerate" }, + { name = "peft" }, + { name = "transformers" }, +] +evals = [ + { name = "slm-evals" }, +] +finetune = [ + { name = "bitsandbytes" }, + { name = "datasets" }, + { name = "peft" }, +] +lm-eval = [ + { name = "slm-evals", extra = ["lm-eval"] }, +] [package.metadata] requires-dist = [ { name = "agent", editable = "libs/agent" }, + { name = "ensemble", editable = "research/ensemble" }, { name = "gradio-space", editable = "apps/gradio-space" }, { name = "inference", editable = "libs/inference" }, ] @@ -1644,6 +2681,24 @@ dev = [ { name = "pytest", specifier = ">=8.0.0" }, { name = "ruff", specifier = ">=0.9.0" }, ] +ensemble = [ + { name = "accelerate", specifier = ">=1.2.0" }, + { name = "peft", specifier = ">=0.14.0" }, + { name = "transformers", specifier = ">=5.7.0" }, +] +evals = [{ name = "slm-evals", editable = "research/evals" }] +finetune = [ + { name = "bitsandbytes", specifier = ">=0.46.1" }, + { name = "datasets", specifier = ">=3.0.0" }, + { name = "peft", specifier = ">=0.14.0" }, +] +lm-eval = [{ name = "slm-evals", extras = ["lm-eval"], editable = "research/evals" }] + +[[package]] +name = "sqlitedict" +version = "2.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/12/9a/7620d1e9dcb02839ed6d4b14064e609cdd7a8ae1e47289aa0456796dd9ca/sqlitedict-2.1.0.tar.gz", hash = "sha256:03d9cfb96d602996f1d4c2db2856f1224b96a9c431bdd16e78032a72940f9e8c", size = 21846, upload-time = "2022-12-03T13:39:13.102Z" } [[package]] name = "starlette" @@ -1670,6 +2725,46 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/a2/09/77d55d46fd61b4a135c444fc97158ef34a095e5681d0a6c10b75bf356191/sympy-1.14.0-py3-none-any.whl", hash = "sha256:e091cc3e99d2141a0ba2847328f5479b05d94a6635cb96148ccb3f34671bd8f5", size = 6299353, upload-time = "2025-04-27T18:04:59.103Z" }, ] +[[package]] +name = "tabledata" +version = "1.3.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "dataproperty" }, + { name = "typepy" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/75/65/2f54f0dedd775dde48e300023d20e13ad329a51e33dcadb6d47b4dc95768/tabledata-1.3.5.tar.gz", hash = "sha256:98c64d0ad6b520846b41000fb3f5b2f42fa7ca2675c2c669e5ccab6b93082a36", size = 25396, upload-time = "2026-05-11T12:03:26.367Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c7/86/37fa0e1437089f08b8b1b8c8ad93f6b57e9427753f002914299323300a9e/tabledata-1.3.5-py3-none-any.whl", hash = "sha256:a1e57afc4767b51bef551114c0df31f205d712dbb75e3caf9be7834a79f23136", size = 11919, upload-time = "2026-05-11T12:03:24.907Z" }, +] + +[[package]] +name = "tabulate" +version = "0.10.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/46/58/8c37dea7bbf769b20d58e7ace7e5edfe65b849442b00ffcdd56be88697c6/tabulate-0.10.0.tar.gz", hash = "sha256:e2cfde8f79420f6deeffdeda9aaec3b6bc5abce947655d17ac662b126e48a60d", size = 91754, upload-time = "2026-03-04T18:55:34.402Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/99/55/db07de81b5c630da5cbf5c7df646580ca26dfaefa593667fc6f2fe016d2e/tabulate-0.10.0-py3-none-any.whl", hash = "sha256:f0b0622e567335c8fabaaa659f1b33bcb6ddfe2e496071b743aa113f8774f2d3", size = 39814, upload-time = "2026-03-04T18:55:31.284Z" }, +] + +[[package]] +name = "tcolorpy" +version = "0.1.7" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/80/cc/44f2d81d8f9093aad81c3467a5bf5718d2b5f786e887b6e4adcfc17ec6b9/tcolorpy-0.1.7.tar.gz", hash = "sha256:0fbf6bf238890bbc2e32662aa25736769a29bf6d880328f310c910a327632614", size = 299437, upload-time = "2024-12-29T15:24:23.847Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/05/a2/ed023f2edd1e011b4d99b6727bce8253842d66c3fbf9ed0a26fc09a92571/tcolorpy-0.1.7-py3-none-any.whl", hash = "sha256:26a59d52027e175a37e0aba72efc99dda43f074db71f55b316d3de37d3251378", size = 8096, upload-time = "2024-12-29T15:24:21.33Z" }, +] + +[[package]] +name = "threadpoolctl" +version = "3.6.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/b7/4d/08c89e34946fce2aec4fbb45c9016efd5f4d7f24af8e5d93296e935631d8/threadpoolctl-3.6.0.tar.gz", hash = "sha256:8ab8b4aa3491d812b623328249fab5302a68d2d71745c8a4c719a2fcaba9f44e", size = 21274, upload-time = "2025-03-13T13:49:23.031Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/32/d5/f9a850d79b0851d1d4ef6456097579a9005b31fea68726a4ae5f2d82ddd9/threadpoolctl-3.6.0-py3-none-any.whl", hash = "sha256:43a0b8fd5a2928500110039e43a5eed8480b918967083ea48dc3ab9f13c4a7fb", size = 18638, upload-time = "2025-03-13T13:49:21.846Z" }, +] + [[package]] name = "tokenizers" version = "0.22.2" @@ -1830,6 +2925,25 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/c1/68/fa86e5a39608000f645535b2c124920126327ab731f8c4fafd5b07ff8d4b/triton-3.7.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ce061073102714b725f3660ec6939d94a1da7984b3aa99c921417cae273672f5", size = 201546766, upload-time = "2026-05-07T18:46:42.088Z" }, ] +[[package]] +name = "typepy" +version = "1.3.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "mbstrdecoder" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/8b/9f/ae119b0e0fd0fe8dcb0e1eeebfeb62f37fdc0b467267cff15cdb746ba38b/typepy-1.3.5.tar.gz", hash = "sha256:a1c5f54c41860f89bab175f512b11e8c9a57cfe7b8b3d5ae5d52d828b756b6dd", size = 39883, upload-time = "2026-05-04T14:04:32.835Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/64/71/75cf08c49b64a9419f1f2cef9be072ac34f6b784da2851489470b7c7ba15/typepy-1.3.5-py3-none-any.whl", hash = "sha256:de361b59609c7503efc2edbe9d7a4e053ae71307bf90ae1678ec4d6bcd807922", size = 31530, upload-time = "2026-05-04T14:04:31.46Z" }, +] + +[package.optional-dependencies] +datetime = [ + { name = "packaging" }, + { name = "python-dateutil" }, + { name = "pytz" }, +] + [[package]] name = "typer" version = "0.25.1" @@ -1875,6 +2989,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/ce/e4/dccd7f47c4b64213ac01ef921a1337ee6e30e8c6466046018326977efd95/tzdata-2026.2-py2.py3-none-any.whl", hash = "sha256:bbe9af844f658da81a5f95019480da3a89415801f6cc966806612cc7169bffe7", size = 349321, upload-time = "2026-04-24T15:22:05.876Z" }, ] +[[package]] +name = "urllib3" +version = "2.7.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/53/0c/06f8b233b8fd13b9e5ee11424ef85419ba0d8ba0b3138bf360be2ff56953/urllib3-2.7.0.tar.gz", hash = "sha256:231e0ec3b63ceb14667c67be60f2f2c40a518cb38b03af60abc813da26505f4c", size = 433602, upload-time = "2026-05-07T16:13:18.596Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl", hash = "sha256:9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897", size = 131087, upload-time = "2026-05-07T16:13:17.151Z" }, +] + [[package]] name = "uvicorn" version = "0.49.0" @@ -1888,6 +3011,12 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/88/fa/e1388bbcf24ef3274f45c0c1c7b501fd14971037c1b6ee23610553307497/uvicorn-0.49.0-py3-none-any.whl", hash = "sha256:ba3d14c3ee7e41c6c654c46c9eb489d33213cdd30aa1696eab1374337c13f68f", size = 71376, upload-time = "2026-06-03T22:01:29.037Z" }, ] +[[package]] +name = "word2number" +version = "1.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/4a/29/a31940c848521f0725f0df6b25dca8917f13a2025b0e8fcbe5d0457e45e6/word2number-1.1.zip", hash = "sha256:70e27a5d387f67b04c71fbb7621c05930b19bfd26efd6851e6e0f9969dcde7d0", size = 9723, upload-time = "2017-06-02T15:45:14.488Z" } + [[package]] name = "xlsxwriter" version = "3.2.9" @@ -1896,3 +3025,198 @@ sdist = { url = "https://files.pythonhosted.org/packages/46/2c/c06ef49dc36e7954e wheels = [ { url = "https://files.pythonhosted.org/packages/3a/0c/3662f4a66880196a590b202f0db82d919dd2f89e99a27fadef91c4a33d41/xlsxwriter-3.2.9-py3-none-any.whl", hash = "sha256:9a5db42bc5dff014806c58a20b9eae7322a134abb6fce3c92c181bfb275ec5b3", size = 175315, upload-time = "2025-09-16T00:16:20.108Z" }, ] + +[[package]] +name = "xxhash" +version = "3.7.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/24/2f/e183a1b407002f5af81822bee18b61cdb94b8670208ef34734d8d2b8ebe9/xxhash-3.7.0.tar.gz", hash = "sha256:6cc4eefbb542a5d6ffd6d70ea9c502957c925e800f998c5630ecc809d6702bae", size = 82022, upload-time = "2026-04-25T11:10:32.553Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f2/8a/51a14cdef4728c6c2337db8a7d8704422cc65676d9199d77215464c880af/xxhash-3.7.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:082c87bfdd2b9f457606c7a4a53457f4c4b48b0cdc48de0277f4349d79bb3d7a", size = 33357, upload-time = "2026-04-25T11:06:20.44Z" }, + { url = "https://files.pythonhosted.org/packages/b9/1b/0c2c933809421ffd9bf42b59315552c143c755db5d9a816b2f1ae273e884/xxhash-3.7.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:5e7ce913b61f35b0c1c839a49ac9c8e75dd8d860150688aed353b0ce1bf409d8", size = 30869, upload-time = "2026-04-25T11:06:21.989Z" }, + { url = "https://files.pythonhosted.org/packages/03/a8/89d5fdd6ee12d70ba99451de46dd0e8010167468dcd913ec855653f4dd50/xxhash-3.7.0-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:3beb1de3b1e9694fcdd853e570ee64c631c7062435d2f8c69c1adf809bc086f0", size = 194100, upload-time = "2026-04-25T11:06:23.586Z" }, + { url = "https://files.pythonhosted.org/packages/87/ee/2f9f2ed993e77206d1e66991290a1ebe22e843351ca3ebec8e49e01ba186/xxhash-3.7.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f3e7b689c3bce16699efcf736066f5c6cc4472c3840fe4b22bd8279daf4abdac", size = 212977, upload-time = "2026-04-25T11:06:25.019Z" }, + { url = "https://files.pythonhosted.org/packages/de/60/5a91644615a9e9d4e42c2e9925f1908e3a24e4e691d9de7340d565bea024/xxhash-3.7.0-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:a6545e6b409e3d5cbafc850fb84c55a1ca26ed15a6b11e3bf07a0e0cd84517c8", size = 236373, upload-time = "2026-04-25T11:06:26.482Z" }, + { url = "https://files.pythonhosted.org/packages/22/c0/f3a9384eaaed9d14d4d062a5d953aa0da489bfe9747877aa994caa87cd0b/xxhash-3.7.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:31ab1461c77a11461d703c88eb949e132a1c6515933cf675d97ec680f4bd18de", size = 212229, upload-time = "2026-04-25T11:06:28.065Z" }, + { url = "https://files.pythonhosted.org/packages/2e/67/02f07a9fd79726804190f2172c4894c3ed9a4ebccaca05653c84beb58025/xxhash-3.7.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:7c4d596b7676f811172687ec567cbafb9e4dea2f9be1bbb4f622410cb7f40f40", size = 445462, upload-time = "2026-04-25T11:06:30.048Z" }, + { url = "https://files.pythonhosted.org/packages/40/37/558f5a90c0672fc9b4402dc25d87ac5b7406616e8969430c9ca4e52ee74d/xxhash-3.7.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:13805f0461cba0a857924e70ff91ae6d52d2598f79a884e788db80532614a4a1", size = 193932, upload-time = "2026-04-25T11:06:31.857Z" }, + { url = "https://files.pythonhosted.org/packages/d5/90/aaa09cd58661d32044dbbad7df55bbe22a623032b810e7ed3b8c569a2a6f/xxhash-3.7.0-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:1d398f372496152f1c6933a33566373f8d1b37b98b8c9d608fa6edc0976f23b2", size = 284807, upload-time = "2026-04-25T11:06:33.697Z" }, + { url = "https://files.pythonhosted.org/packages/d6/f3/53df3719ab127a02c174f0c1c74924fcd110866e89c966bc7909cfa8fa84/xxhash-3.7.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:d610aa62cdb7d4d497740741772a24a794903bf3e79eaa51d2e800082abe11e5", size = 210445, upload-time = "2026-04-25T11:06:35.488Z" }, + { url = "https://files.pythonhosted.org/packages/72/33/d219975c0e8b6fa2eb9ccd486fe47e21bf1847985b878dd2fbc3126e0d5c/xxhash-3.7.0-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:073c23900a9fbf3d26616c17c830db28af9803677cd5b33aea3224d824111514", size = 241273, upload-time = "2026-04-25T11:06:37.24Z" }, + { url = "https://files.pythonhosted.org/packages/3e/50/49b1afe610eb3964cedcb90a4d4c3d46a261ee8669cbd4f060652619ae3c/xxhash-3.7.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:418a463c3e6a590c0cdc890f8be19adb44a8c8acd175ca5b2a6de77e61d0b386", size = 197950, upload-time = "2026-04-25T11:06:39.148Z" }, + { url = "https://files.pythonhosted.org/packages/c6/75/5f42a1a4c78717d906a4b6a140c6dbf837ab1f547a54d23c4e2903310936/xxhash-3.7.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:03f8ff4474ee61c845758ce00711d7087a770d77efb36f7e74a6e867301000b8", size = 210709, upload-time = "2026-04-25T11:06:40.958Z" }, + { url = "https://files.pythonhosted.org/packages/8a/85/237e446c25abced71e9c53d269f2cef5bab8a82b3f88a12e00c5368e7368/xxhash-3.7.0-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:44fba4a5f1d179b7ddc7b3dc40f56f9209046421679b57025d4d8821b376fd8d", size = 275345, upload-time = "2026-04-25T11:06:42.525Z" }, + { url = "https://files.pythonhosted.org/packages/62/34/c2c26c0a6a9cc739bc2a5f0ae03ba8b87deb12b8bce35f7ac495e790dc6d/xxhash-3.7.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:31e3516a0f829d06ded4a2c0f3c7c5561993256bfa1c493975fb9dc7bfa828a1", size = 414056, upload-time = "2026-04-25T11:06:44.343Z" }, + { url = "https://files.pythonhosted.org/packages/a0/aa/5c58e9bc8071b8afd8dcf297ff362f723c4892168faba149f19904132bf4/xxhash-3.7.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:b59ee2ac81de57771a09ecad09191e840a1d2fae1ef684208320591055768f83", size = 191485, upload-time = "2026-04-25T11:06:46.262Z" }, + { url = "https://files.pythonhosted.org/packages/d4/69/a929cf9d1e2e65a48b818cdce72cb6b69eab2e6877f21436d0a1942aff43/xxhash-3.7.0-cp312-cp312-win32.whl", hash = "sha256:74bbd92f8c7fcc397ba0a11bfdc106bc72ad7f11e3a60277753f87e7532b4d81", size = 30671, upload-time = "2026-04-25T11:06:48.039Z" }, + { url = "https://files.pythonhosted.org/packages/b9/1b/104b41a8947f4e1d4a66ce1e628eea752f37d1890bfd7453559ca7a3d950/xxhash-3.7.0-cp312-cp312-win_amd64.whl", hash = "sha256:7bd7bc82dd4f185f28f35193c2e968ef46131628e3cac62f639dadf321cba4d1", size = 31514, upload-time = "2026-04-25T11:06:49.279Z" }, + { url = "https://files.pythonhosted.org/packages/98/a0/1fd0ea1f1b886d9e7c73f0397571e22333a7d79e31da6d7127c2a4a71d75/xxhash-3.7.0-cp312-cp312-win_arm64.whl", hash = "sha256:7d7148180ec99ba36585b42c8c5de25e9b40191613bc4be68909b4d25a77a852", size = 27761, upload-time = "2026-04-25T11:06:50.448Z" }, + { url = "https://files.pythonhosted.org/packages/c1/ca/d5174b4c36d10f64d4ca7050563138c5a599efb01a765858ddefc9c1202a/xxhash-3.7.0-cp313-cp313-android_21_arm64_v8a.whl", hash = "sha256:4b6d6b33f141158692bd4eafbb96edbc5aa0dabdb593a962db01a91983d4f8fa", size = 36813, upload-time = "2026-04-25T11:06:51.73Z" }, + { url = "https://files.pythonhosted.org/packages/41/d0/abc6c9d347ba1f1e1e1d98125d0881a0452c7f9a76a9dd03a7b5d2197f23/xxhash-3.7.0-cp313-cp313-android_21_x86_64.whl", hash = "sha256:845d347df254d6c619f616afa921331bada8614b8d373d58725c663ba97c3605", size = 35121, upload-time = "2026-04-25T11:06:53.048Z" }, + { url = "https://files.pythonhosted.org/packages/bf/11/4cc834eb3d79f2f2b3a6ef7324195208bcdfbdcf7534d2b17267aa5f3a8f/xxhash-3.7.0-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:fddbbb69a6fff4f421e7a0d1fa28f894b20112e9e3fab306af451e2dfd0e459b", size = 29624, upload-time = "2026-04-25T11:06:54.311Z" }, + { url = "https://files.pythonhosted.org/packages/23/83/e97d3e7b635fe73a1dfb1e91f805324dd6d930bb42041cbf18f183bc0b6d/xxhash-3.7.0-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:54876a4e45101cec2bf8f31a973cda073a23e2e108538dad224ba07f85f22487", size = 30638, upload-time = "2026-04-25T11:06:55.864Z" }, + { url = "https://files.pythonhosted.org/packages/f4/40/d84951d80c35db1f4c40a29a64a8520eea5d56e764c603906b4fe763580f/xxhash-3.7.0-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:0c72fe9c7e3d6dfd7f1e21e224a877917fa09c465694ba4e06464b9511b65544", size = 33323, upload-time = "2026-04-25T11:06:57.336Z" }, + { url = "https://files.pythonhosted.org/packages/89/cc/c7dc6558d97e9ab023f663d69ab28b340ed9bf4d2d94f2c259cf896bb354/xxhash-3.7.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:a6d73a830b17ef49bc04e00182bd839164c1b3c59c127cd7c54fcb10c7ed8ee8", size = 33362, upload-time = "2026-04-25T11:06:58.656Z" }, + { url = "https://files.pythonhosted.org/packages/2a/6e/46b84017b1301d54091430353d4ad5901654a3e0871649877a416f7f1644/xxhash-3.7.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:91c3b07cf3362086d8f126c6aecd8e5e9396ad8b2f2219ea7e49a8250c318acd", size = 30874, upload-time = "2026-04-25T11:06:59.834Z" }, + { url = "https://files.pythonhosted.org/packages/df/5e/8f9158e3ab906ad3fec51e09b5ea0093e769f12207bfa42a368ca204e7ab/xxhash-3.7.0-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:50e879ebbac351c81565ca108db766d7832f5b8b6a5b14b8c0151f7190028e3d", size = 194185, upload-time = "2026-04-25T11:07:01.658Z" }, + { url = "https://files.pythonhosted.org/packages/f3/29/a804ded9f5d3d3758292678d23e7528b08fda7b7e750688d08b052322475/xxhash-3.7.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:921c14e93817842dd0dd9f372890a0f0c72e534650b6ab13c5be5cd0db11d47e", size = 213033, upload-time = "2026-04-25T11:07:03.606Z" }, + { url = "https://files.pythonhosted.org/packages/8b/91/1ce5a7d2fdc975267320e2c78fc1cecfe7ab735ccbcf6993ec5dd541cb2c/xxhash-3.7.0-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:e64a7c9d7dfca3e0fafcbc5e455519090706a3e36e95d655cec3e04e79f95aaa", size = 236140, upload-time = "2026-04-25T11:07:05.396Z" }, + { url = "https://files.pythonhosted.org/packages/34/04/fd595a4fd8617b05fa27bd9b684ecb4985bfed27917848eea85d54036d06/xxhash-3.7.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:2220af08163baf5fa36c2b8af079dc2cbe6e66ae061385267f9472362dfd53c6", size = 212291, upload-time = "2026-04-25T11:07:06.966Z" }, + { url = "https://files.pythonhosted.org/packages/03/fb/f1a379cbc372ae5b9f4ab36154c48a849ca6ebe3ac477067a57865bf3bc6/xxhash-3.7.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:f14bb8b22a4a91325813e3d553b8963c10cf8c756cff65ee50c194431296c655", size = 445532, upload-time = "2026-04-25T11:07:08.525Z" }, + { url = "https://files.pythonhosted.org/packages/65/59/172424b79f8cfd4b6d8a122b2193e6b8ad4b11f7159bb3b6f9b3191329bb/xxhash-3.7.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:496736f86a9bedaf64b0dc70e3539d0766df01c71ea22032698e88f3f04a1ce9", size = 193990, upload-time = "2026-04-25T11:07:10.315Z" }, + { url = "https://files.pythonhosted.org/packages/b9/19/aeac22161d953f139f07ba5586cb4a17c5b7b6dff985122803bb12933500/xxhash-3.7.0-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:0ff71596bd79816975b3de7130ab1ff4541410285a3c084584eeb1c8239996fd", size = 284876, upload-time = "2026-04-25T11:07:12.15Z" }, + { url = "https://files.pythonhosted.org/packages/77/d5/4fd0b59e7a02242953da05ff679fbb961b0a4368eac97a217e11dae110c1/xxhash-3.7.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:1ad86695c19b1d46fe106925db3c7a37f16be37669dcf58dcc70a9dd6e324676", size = 210495, upload-time = "2026-04-25T11:07:13.952Z" }, + { url = "https://files.pythonhosted.org/packages/aa/fb/976a3165c728c7faf74aa1b5ab3cf6a85e6d731612894741840524c7d28c/xxhash-3.7.0-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:970f9f8c50961d639cbd0d988c96f80ddf66006de93641719282c4fe7a87c5e6", size = 241331, upload-time = "2026-04-25T11:07:15.557Z" }, + { url = "https://files.pythonhosted.org/packages/4a/2c/6763d5901d53ac9e6ba296e5717ae599025c9d268396e8faa8b4b0a8e0ac/xxhash-3.7.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:5886ad85e9e347911783760a1d16cb6b393e8f9e3b52c982568226cb56927bdc", size = 198037, upload-time = "2026-04-25T11:07:17.563Z" }, + { url = "https://files.pythonhosted.org/packages/61/2b/876e722d533833f5f9a83473e6ba993e48745701096944e77bbecf29b2c3/xxhash-3.7.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:6e934bbae1e0ec74e27d5f0d7f37ef547ce5ff9f0a7e63fb39e559fc99526734", size = 210744, upload-time = "2026-04-25T11:07:19.055Z" }, + { url = "https://files.pythonhosted.org/packages/21/e6/d7e7baef7ce24166b4668d3c48557bb35a23b92ecadcac7e7718d099ab69/xxhash-3.7.0-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:3b6b3d28228af044ebcded71c4a3dd86e1dbd7e2f4645bf40f7b5da65bb5fb5a", size = 275406, upload-time = "2026-04-25T11:07:20.908Z" }, + { url = "https://files.pythonhosted.org/packages/92/fe/198b3763b2e01ca908f2154969a2352ec99bda892b574a11a9a151c5ede4/xxhash-3.7.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:6be4d70d9ab76c9f324ead9c01af6ff52c324745ea0c3731682a0cf99720f1fe", size = 414125, upload-time = "2026-04-25T11:07:23.037Z" }, + { url = "https://files.pythonhosted.org/packages/3a/6d/019a11affd5a5499137cacca53808659964785439855b5aa40dfd3412916/xxhash-3.7.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:151d7520838d4465461a0b7f4ae488b3b00de16183dd3214c1a6b14bf89d7fb6", size = 191555, upload-time = "2026-04-25T11:07:24.991Z" }, + { url = "https://files.pythonhosted.org/packages/76/21/b96d58568df2d01533244c3e0e5cbdd0c8b2b25c4bec4d72f19259a292d7/xxhash-3.7.0-cp313-cp313-win32.whl", hash = "sha256:d798c1e291bffb8e37b5bbe0dda77fc767cd19e89cadaf66e6ed5d0ff88c9fe6", size = 30668, upload-time = "2026-04-25T11:07:26.665Z" }, + { url = "https://files.pythonhosted.org/packages/99/57/d849a8d3afa1f8f4bc6a831cd89f49f9706fbbad94d2975d6140a171988c/xxhash-3.7.0-cp313-cp313-win_amd64.whl", hash = "sha256:875811ba23c543b1a1c3143c926e43996eb27ebb8f52d3500744aa608c275aed", size = 31524, upload-time = "2026-04-25T11:07:27.92Z" }, + { url = "https://files.pythonhosted.org/packages/81/52/bacc753e92dee78b058af8dcef0a50815f5f860986c664a92d75f965b6a5/xxhash-3.7.0-cp313-cp313-win_arm64.whl", hash = "sha256:54a675cb300dda83d71daae2a599389d22db8021a0f8db0dd659e14626eb3ecc", size = 27768, upload-time = "2026-04-25T11:07:29.113Z" }, + { url = "https://files.pythonhosted.org/packages/1c/47/ddbd683b7fc7e592c1a8d9d65f73ce9ab513f082b3967eee2baf549b8fc6/xxhash-3.7.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:a3b19a42111c4057c1547a4a1396a53961dca576a0f6b82bfa88a2d1561764b2", size = 33576, upload-time = "2026-04-25T11:07:30.469Z" }, + { url = "https://files.pythonhosted.org/packages/07/f2/36d3310161db7f72efb4562aadde0ed429f1d0531782dd6345b12d2da527/xxhash-3.7.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:8f4608a06e4d61b7a3425665a46d00e0579122e1a2fae97a0c52953a3aad9aa3", size = 31123, upload-time = "2026-04-25T11:07:31.989Z" }, + { url = "https://files.pythonhosted.org/packages/0d/3f/75937a5c69556ed213021e43cbedd84c8e0279d0d74e7d41a255d84ba4b1/xxhash-3.7.0-cp313-cp313t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:ad37c7792479e49cf96c1ab25517d7003fe0d93687a772ba19a097d235bbe41e", size = 196491, upload-time = "2026-04-25T11:07:33.358Z" }, + { url = "https://files.pythonhosted.org/packages/22/29/f10d7ff8c7a733d4403a43b9de18c8fabc005f98cec054644f04418659ee/xxhash-3.7.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:dc026e3b89d98e30a8288c95cb696e77d150b3f0fb7a51f73dcd49ee6b5577fa", size = 215793, upload-time = "2026-04-25T11:07:34.919Z" }, + { url = "https://files.pythonhosted.org/packages/8b/fd/778f60aa295f58907938f030a8b514611f391405614a525cccd2ffc00eb5/xxhash-3.7.0-cp313-cp313t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:c9b31ab1f28b078a6a1ac1a54eb35e7d5390deddd56870d0be3a0a733d1c321c", size = 237993, upload-time = "2026-04-25T11:07:36.638Z" }, + { url = "https://files.pythonhosted.org/packages/70/f5/736db5de387b4a540e37a05b84b40dc58a1ce974bfd2b4e5754ce29b68c3/xxhash-3.7.0-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:3bb5fd680c038fd5229e44e9c493782f90df9bef632fd0499d442374688ff70b", size = 214887, upload-time = "2026-04-25T11:07:38.564Z" }, + { url = "https://files.pythonhosted.org/packages/4d/aa/09a095f22fdb9a27fbb716841fbff52119721f9ca4261952d07a912f7839/xxhash-3.7.0-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:030c0fd688fce3569fbb49a2feefd4110cbb0b650186fb4610759ecfac677548", size = 448407, upload-time = "2026-04-25T11:07:40.552Z" }, + { url = "https://files.pythonhosted.org/packages/74/8a/b745efeeca9e34a91c26fdc97ad8514c43d5a81ac78565cba80a1353870a/xxhash-3.7.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5b1bde10324f4c31812ae0d0502e92d916ae8917cad7209353f122b8b8f610c3", size = 196119, upload-time = "2026-04-25T11:07:42.101Z" }, + { url = "https://files.pythonhosted.org/packages/8a/5c/0cfceb024af90c191f665c7933b1f318ee234f4797858383bebd1881d52f/xxhash-3.7.0-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:503722d52a615f2604f5e7611de7d43878df010dc0053094ef91cb9a9ac3d987", size = 286751, upload-time = "2026-04-25T11:07:43.568Z" }, + { url = "https://files.pythonhosted.org/packages/0b/0a/0793e405dc3cf8f4ebe2c1acec1e4e4608cd9e7e50ea691dabbc2a95ccbb/xxhash-3.7.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:c72500a3b6d6c30ebfc135035bcace9eb5884f2dc220804efcaaba43e9f611dd", size = 212961, upload-time = "2026-04-25T11:07:45.388Z" }, + { url = "https://files.pythonhosted.org/packages/0c/7e/721118ffc63bfff94aa565bcf2555a820f9f4bdb0f001e0d609bdfad70de/xxhash-3.7.0-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:43475925a766d01ca8cd9a857fd87f3d50406983c8506a4c07c4df12adcc867f", size = 243703, upload-time = "2026-04-25T11:07:47.053Z" }, + { url = "https://files.pythonhosted.org/packages/6e/18/16f6267160488b8276fd3d449d425712512add292ba545c1b6946bfdb7dd/xxhash-3.7.0-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:8d09dfd2ab135b985daf868b594315ebe11ad86cd9fea46e6c69f19b28f7d25a", size = 200894, upload-time = "2026-04-25T11:07:48.657Z" }, + { url = "https://files.pythonhosted.org/packages/2d/94/80ba841287fd97e3e9cac1d228788c8ef623746f570404961eec748ecb5c/xxhash-3.7.0-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:c50269d0055ac1faecfd559886d2cbe4b730de236585aba0e873f9d9dadbe585", size = 213357, upload-time = "2026-04-25T11:07:50.257Z" }, + { url = "https://files.pythonhosted.org/packages/a1/7e/106d4067130c59f1e18a55ffadcd876d8c68534883a1e02685b29d3d8153/xxhash-3.7.0-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:1910df4756a5ab58cfad8744fc2d0f23926e3efcc346ee76e87b974abab922f4", size = 277600, upload-time = "2026-04-25T11:07:51.745Z" }, + { url = "https://files.pythonhosted.org/packages/c5/86/a081dd30da71d720b2612a792bfd55e45fa9a07ac76a0507f60487473c25/xxhash-3.7.0-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:d006faf3b491957efcb433489be3c149efe4787b7063d5cddb8ddaefdc60e0c1", size = 416980, upload-time = "2026-04-25T11:07:53.504Z" }, + { url = "https://files.pythonhosted.org/packages/35/29/1a95221a029a3c1293773869e1ab47b07cbbdd82444a42809e8c60156626/xxhash-3.7.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:abb65b4e947e958f7b3b0d71db3ce447d1bc5f37f5eab871ce7223bda8768a04", size = 193840, upload-time = "2026-04-25T11:07:55.103Z" }, + { url = "https://files.pythonhosted.org/packages/c5/e0/db909dd0823285de2286f67e10ee4d81e96ad35d7d8e964ecb07fccd8af9/xxhash-3.7.0-cp313-cp313t-win32.whl", hash = "sha256:178959906cb1716a1ce08e0d69c82886c70a15a6f2790fc084fdd146ca30cd49", size = 30966, upload-time = "2026-04-25T11:07:56.524Z" }, + { url = "https://files.pythonhosted.org/packages/7b/ff/d705b15b22f21ee106adce239cb65d35067a158c630b240270f09b17c2e6/xxhash-3.7.0-cp313-cp313t-win_amd64.whl", hash = "sha256:2524a1e20d4c231d13b50f7cf39e44265b055669a64a7a4b9a2a44faa03f19b6", size = 31784, upload-time = "2026-04-25T11:07:57.758Z" }, + { url = "https://files.pythonhosted.org/packages/a2/1f/b2cf83c3638fd0588e0b17f22e5a9400bdfb1a3e3755324ac0aee2250b88/xxhash-3.7.0-cp313-cp313t-win_arm64.whl", hash = "sha256:37d994d0ffe81ef087bb330d392caa809bb5853c77e22ea3f71db024a0543dba", size = 27932, upload-time = "2026-04-25T11:07:59.109Z" }, + { url = "https://files.pythonhosted.org/packages/0e/cc/431db584f6fbb9312e40a173af027644e5580d39df1f73603cbb9dca4d6b/xxhash-3.7.0-cp314-cp314-android_24_arm64_v8a.whl", hash = "sha256:8c5fcfd806c335bfa2adf1cd0b3110a44fc7b6995c3a648c27489bae85801465", size = 36644, upload-time = "2026-04-25T11:08:00.658Z" }, + { url = "https://files.pythonhosted.org/packages/bc/01/255ec513e0a705d1f9a61413e78dfce4e3235203f0ed525a24c2b4b56345/xxhash-3.7.0-cp314-cp314-android_24_x86_64.whl", hash = "sha256:506a0b488f190f0a06769575e30caf71615c898ed93ab18b0dbcb6dec5c3713c", size = 35003, upload-time = "2026-04-25T11:08:02.338Z" }, + { url = "https://files.pythonhosted.org/packages/68/70/c55fc33c93445b44d8fc5a17b41ed99e3cebe92bcf8396809e63fc9a1165/xxhash-3.7.0-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:ec68dbba21532c0173a9872298e65c89749f7c9d21538c3a78b5bb6105871568", size = 29655, upload-time = "2026-04-25T11:08:03.701Z" }, + { url = "https://files.pythonhosted.org/packages/c2/72/ff8de73df000d74467d12a59ce6d6e2b2a368b978d41ab7b1fba5ed442be/xxhash-3.7.0-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:fa77e7ec1450d415d20129961814787c9abd9a07f98872f070b1fe96c5084611", size = 30664, upload-time = "2026-04-25T11:08:05.011Z" }, + { url = "https://files.pythonhosted.org/packages/b6/91/08416d9bd9bc3bf39d831abe8a5631ac2db5141dfd6fe81c3fe59a1f9264/xxhash-3.7.0-cp314-cp314-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:fe32736295ea38e43e7d9424053c8c47c9f64fecfc7c895fb3da9b30b131c9ee", size = 33317, upload-time = "2026-04-25T11:08:06.413Z" }, + { url = "https://files.pythonhosted.org/packages/0e/3b/86b1caa4dee10a99f4bf9521e623359341c5e50d05158fa10c275b2bd079/xxhash-3.7.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:ab9dd2c83c4bbd63e422181a76f13502d049d3ddcac9a1bdc29196263d692bb8", size = 33457, upload-time = "2026-04-25T11:08:08.099Z" }, + { url = "https://files.pythonhosted.org/packages/ed/38/98ea14ad1517e1461292a65906951458d520689782bfbae111050145bdba/xxhash-3.7.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:3afec3a336a2286601a437cb07562ab0227685e6fbb9ec17e8c18457ff348ecf", size = 30894, upload-time = "2026-04-25T11:08:09.429Z" }, + { url = "https://files.pythonhosted.org/packages/61/a2/074654d0b893606541199993c7db70067d9fc63b748e0d60020a52a1bd36/xxhash-3.7.0-cp314-cp314-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:565df64437a9390f84465dcca33e7377114c7ede8d05cd2cf20081f831ea788e", size = 194409, upload-time = "2026-04-25T11:08:10.91Z" }, + { url = "https://files.pythonhosted.org/packages/e2/26/6d2a1afc468189f77ca28c32e1c83e1b9da1178231e05641dbc1b350e332/xxhash-3.7.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:12eca820a5d558633d423bf8bb78ce72a55394823f64089247f788a7e0ae691e", size = 213135, upload-time = "2026-04-25T11:08:12.575Z" }, + { url = "https://files.pythonhosted.org/packages/8e/0e/d8aecf95e09c42547453137be74d2f7b8b14e08f5177fa2fab6144a19061/xxhash-3.7.0-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:f262b8f7599516567e070abf607b9af649052b2c4bd6f9be02b0cb41b7024805", size = 236379, upload-time = "2026-04-25T11:08:14.206Z" }, + { url = "https://files.pythonhosted.org/packages/f2/74/8140e8210536b3dd0cc816c4faaeb5ba6e63e8125ab25af4bcddd6a037b3/xxhash-3.7.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f1598916cb197681e03e601901e4ab96a9a963de398c59d0964f8a6f44a2b361", size = 212447, upload-time = "2026-04-25T11:08:15.79Z" }, + { url = "https://files.pythonhosted.org/packages/a0/d2/462001d2903b4bee5a5689598a0a55e5e7cd1ac7f4247a5545cff10d3ebb/xxhash-3.7.0-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:322b2f0622230f526aeb1738149948a7ae357a9e2ceb1383c6fd1fdaecdafa16", size = 445660, upload-time = "2026-04-25T11:08:17.441Z" }, + { url = "https://files.pythonhosted.org/packages/23/09/2bd1ed7f8689b20e51727952cac8329d50c694dc32b2eba06ba5bc742b37/xxhash-3.7.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:24cc22070880cc57b830a65cde4e65fa884c6d9b28ae4803b5ee05911e7bafba", size = 194076, upload-time = "2026-04-25T11:08:19.134Z" }, + { url = "https://files.pythonhosted.org/packages/c9/6e/692302cd0a5f4ac4e6289f37fa888dc2e1e07750b68fe3e4bfe939b8cea3/xxhash-3.7.0-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:cb5a888a968b2434abf9ecda357b5d43f10d7b5a6da6fdbbe036208473aff0e2", size = 284990, upload-time = "2026-04-25T11:08:20.618Z" }, + { url = "https://files.pythonhosted.org/packages/05/d9/e54b159b3d9df7999d2a7c676ce7b323d1b5588a64f8f51ed8172567bd87/xxhash-3.7.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:a999771ff97bec27d18341be4f3a36b163bb1ac41ec17bef6d2dabd84acd33c7", size = 210590, upload-time = "2026-04-25T11:08:22.24Z" }, + { url = "https://files.pythonhosted.org/packages/50/93/0e0df1a3a196ced4ca71de76d65ead25d8e87bbfb87b64306ea47a40c00d/xxhash-3.7.0-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:ed4a6efe2dee1655adb73e7ad40c6aa955a6892422b1e3b95de6a34de56e3cbb", size = 241442, upload-time = "2026-04-25T11:08:23.844Z" }, + { url = "https://files.pythonhosted.org/packages/9a/a9/d917a7a814e90b218f8a0d37967105eea91bf752c3303683c99a1f7bfc1f/xxhash-3.7.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:9fd17f14ac0faa12126c2f9ca774a8cf342957265ec3c8669c144e5e6cdb478c", size = 198356, upload-time = "2026-04-25T11:08:25.99Z" }, + { url = "https://files.pythonhosted.org/packages/89/5e/f2ba1877c39469abbefc72991d6ebdcbd4c0880db01ae8cb1f553b0c537d/xxhash-3.7.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:05fd1254268c59b5cb2a029dfc204275e9fc52de2913f1e53aa8d01442c96b4d", size = 210898, upload-time = "2026-04-25T11:08:27.608Z" }, + { url = "https://files.pythonhosted.org/packages/90/c6/be56b58e73de531f39a10de1355bb77ceb663900dc4bf2d6d3002a9c3f9e/xxhash-3.7.0-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:a2eae53197c6276d5b317f75a1be226bbf440c20b58bf525f36b5d0e1f657ca6", size = 275519, upload-time = "2026-04-25T11:08:29.301Z" }, + { url = "https://files.pythonhosted.org/packages/92/e2/17ddc85d5765b9c709f192009ed8f5a1fc876f4eb35bba7c307b5b1169f9/xxhash-3.7.0-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:bfe6f92e3522dcbe8c4281efd74fa7542a336cb00b0e3272c4ec0edabeaeaf67", size = 414191, upload-time = "2026-04-25T11:08:31.16Z" }, + { url = "https://files.pythonhosted.org/packages/9c/42/85f5b79f4bf1ec7ba052491164adfd4f4e9519f5dc7246de4fbd64a1bd56/xxhash-3.7.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:7ab9a49c410d8c6c786ab99e79c529938d894c01433130353dd0fe999111077a", size = 191604, upload-time = "2026-04-25T11:08:32.862Z" }, + { url = "https://files.pythonhosted.org/packages/b8/d0/6127b623aa4cca18d8b7743592b048d689fd6c6e37ff26a22cddf6cd9d7f/xxhash-3.7.0-cp314-cp314-win32.whl", hash = "sha256:040ea63668f9185b92bc74942df09c7e65703deed71431333678fc6e739a9955", size = 31271, upload-time = "2026-04-25T11:08:34.651Z" }, + { url = "https://files.pythonhosted.org/packages/64/4f/44fc4788568004c43921701cbc127f48218a1eede2c9aea231115323564d/xxhash-3.7.0-cp314-cp314-win_amd64.whl", hash = "sha256:2a61e2a3fb23c892496d587b470dee7fa1b58b248a187719c65ea8e94ec13257", size = 32284, upload-time = "2026-04-25T11:08:35.987Z" }, + { url = "https://files.pythonhosted.org/packages/6d/77/18bb895eb60a49453d16e17d67990e5caff557c78eafc90ad4e2eabf4570/xxhash-3.7.0-cp314-cp314-win_arm64.whl", hash = "sha256:c7741c7524961d8c0cb4d4c21b28957ff731a3fd5b5cd8b856dc80a40e9e5acc", size = 28701, upload-time = "2026-04-25T11:08:37.767Z" }, + { url = "https://files.pythonhosted.org/packages/45/a0/46f72244570c550fbbb7db1ef554183dd5ebe9136385f30e032b781ae8f6/xxhash-3.7.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:fc84bf7aa7592f31ec63a3e7b11d624f468a3f19f5238cec7282a42e838ab1d7", size = 33646, upload-time = "2026-04-25T11:08:39.109Z" }, + { url = "https://files.pythonhosted.org/packages/4a/3a/453846a7eceea11e75def361eed01ec6a0205b9822c19927ed364ccae7cc/xxhash-3.7.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:9f1563fdc8abfc389748e6932c7e4e99c89a53e4ec37d4563c24fc06f5e5644b", size = 31125, upload-time = "2026-04-25T11:08:40.467Z" }, + { url = "https://files.pythonhosted.org/packages/bd/3e/49434aba738885d512f9e486db1bdd19db28dfa40372b56da26ef7a4e738/xxhash-3.7.0-cp314-cp314t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:2d415f18becf6f153046ab6adc97da77e3643a0ee205dae61c4012604113a020", size = 196633, upload-time = "2026-04-25T11:08:41.943Z" }, + { url = "https://files.pythonhosted.org/packages/a4/e9/006cb6127baeb9f8abe6d15e62faa01349f09b34e2bfd65175b2422d026b/xxhash-3.7.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bb16aa13ed175bc9be5c2491ba031b85a9b51c4ed90e0b3d4ebe63cf3fb54f8e", size = 215899, upload-time = "2026-04-25T11:08:43.645Z" }, + { url = "https://files.pythonhosted.org/packages/27/e4/cc57d72e66df0ae29b914335f1c6dcf61e8f3746ddf0ae3c471aa4f15e00/xxhash-3.7.0-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:f9fd595f1e5941b3d7863e4774e4b30caa6731fc34b9277da032295aa5656ee5", size = 238116, upload-time = "2026-04-25T11:08:45.698Z" }, + { url = "https://files.pythonhosted.org/packages/af/78/3531d4a3fd8a0038cc6be1f265a69c1b3587f557a10b677dd736de2202c1/xxhash-3.7.0-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:1295325c5a98d552333fa53dc2b026b0ef0ec9c8e73ca3a952990b4c7d65d459", size = 215012, upload-time = "2026-04-25T11:08:47.355Z" }, + { url = "https://files.pythonhosted.org/packages/b4/f6/259fb1eaaec921f59b17203b0daee69829761226d3b980d5191d7723dd83/xxhash-3.7.0-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:3573a651d146912da9daa9e29e5fbc45994420daaa9ef1e2fa5823e1dc485513", size = 448534, upload-time = "2026-04-25T11:08:49.149Z" }, + { url = "https://files.pythonhosted.org/packages/7b/16/a66d0eaf6a7e68532c07714361ddc904c663ec940f3b028c1ae4a21a7b9d/xxhash-3.7.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5ec1e080a3d02d94ea9335bfab0e3374b877e25411422c18f51a943fa4b46381", size = 196217, upload-time = "2026-04-25T11:08:50.805Z" }, + { url = "https://files.pythonhosted.org/packages/8d/ef/d2efc7fc51756dc52509109d1a25cefc859d74bc4b19a167b12dbd8c2786/xxhash-3.7.0-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:84415265192072d8638a3afc3c1bc5995e310570cd9acb54dc46d3939e364fe0", size = 286906, upload-time = "2026-04-25T11:08:52.418Z" }, + { url = "https://files.pythonhosted.org/packages/fc/67/25decd1d4a4018582ec4db2a868a2b7e40640f4adb20dfeb19ac923aa825/xxhash-3.7.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:8d4dea659b57443989ef32f4295104fd6912c73d0bf26d1d148bb88a9f159b02", size = 213057, upload-time = "2026-04-25T11:08:54.105Z" }, + { url = "https://files.pythonhosted.org/packages/0d/5d/17651eb29d06786cdc40c60ae3d27d645aa5d61d2eca6237a7ba0b94789b/xxhash-3.7.0-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:05ece0fe4d9c9c2728912d1981ae1566cfc83a011571b24732cbf76e1fb70dca", size = 243886, upload-time = "2026-04-25T11:08:56.109Z" }, + { url = "https://files.pythonhosted.org/packages/8a/d4/174d9cf7502243d586e6a9ae842b1ae23026620995114f85f1380e588bc9/xxhash-3.7.0-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:fd880353cf1ffaf321bc18dd663e111976dbd0d3bbd8a66d58d2b470dfa7f396", size = 201015, upload-time = "2026-04-25T11:08:57.777Z" }, + { url = "https://files.pythonhosted.org/packages/91/8c/2254e2d06c3ac5e6fe22eaf3da791b87ea823ae9f2c17b4af66755c5752d/xxhash-3.7.0-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:4e15cc9e2817f6481160f930c62842b3ff419e20e13072bcbab12230943092bc", size = 213457, upload-time = "2026-04-25T11:08:59.826Z" }, + { url = "https://files.pythonhosted.org/packages/79/a2/e3daa762545921173e3360f3b4ff7fc63c2d27359f7230ec1a7a74e117f6/xxhash-3.7.0-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:90b9d1a8bd37d768ffc92a1f651ec69afc532a96fa1ac2ea7abbed5d630b3237", size = 277738, upload-time = "2026-04-25T11:09:01.423Z" }, + { url = "https://files.pythonhosted.org/packages/e1/4c/e186da2c46b87f5204640e008d42730bf3c1ee9f0efb71ae1ebcdfeac681/xxhash-3.7.0-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:157c49475b34ecea8809e51123d9769a534e139d1247942f7a4bc67710bb2533", size = 417127, upload-time = "2026-04-25T11:09:03.592Z" }, + { url = "https://files.pythonhosted.org/packages/17/28/3798e15007a3712d0da3d3fe70f8e11916569858b5cc371053bc26270832/xxhash-3.7.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:5a6ddec83325685e729ca119d1f5c518ec39294212ecd770e60693cdc5f7eb79", size = 193962, upload-time = "2026-04-25T11:09:06.228Z" }, + { url = "https://files.pythonhosted.org/packages/ad/95/a26baa93b5241fd7630998816a4ec47a5a0bad193b3f8fc8f3593e1a4a67/xxhash-3.7.0-cp314-cp314t-win32.whl", hash = "sha256:a04a6cab47e2166435aaf5b9e5ee41d1532cc8300efdef87f2a4d0acb7db19ed", size = 31643, upload-time = "2026-04-25T11:09:08.153Z" }, + { url = "https://files.pythonhosted.org/packages/44/36/5454f13c447e395f9b06a3e91274c59f503d31fad84e1836efe3bdb71f6a/xxhash-3.7.0-cp314-cp314t-win_amd64.whl", hash = "sha256:8653dd7c2eda020545bb2c71c7f7039b53fe7434d0fc1a0a9deb79ab3f1a4fc1", size = 32522, upload-time = "2026-04-25T11:09:09.534Z" }, + { url = "https://files.pythonhosted.org/packages/74/35/698e7e3ff38e22992ea24870a511d8762474fb6783627a2910ff22a185c2/xxhash-3.7.0-cp314-cp314t-win_arm64.whl", hash = "sha256:468f0fc114faaa4b36699f8e328bbc3bb11dc418ba94ac52c26dd736d4b6c637", size = 28807, upload-time = "2026-04-25T11:09:11.234Z" }, +] + +[[package]] +name = "yarl" +version = "1.24.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "idna" }, + { name = "multidict" }, + { name = "propcache" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/79/12/1e8f37460ea0f7eb59c221fdaf0ed75e7ac43e97f8093b9c6f411df50a78/yarl-1.24.2.tar.gz", hash = "sha256:9ac374123c6fd7abf64d1fec93962b0bd4ee2c19751755a762a72dd96c0378f8", size = 210798, upload-time = "2026-05-19T21:31:05.599Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f0/da/866bcb01076ba49d2b42b309867bed3826421f1c479655eb7a607b44f20b/yarl-1.24.2-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:b975866c184564c827e0877380f0dae57dcca7e52782128381b72feff6dfceb8", size = 129957, upload-time = "2026-05-19T21:28:51.695Z" }, + { url = "https://files.pythonhosted.org/packages/bf/1d/fcefb70922ea2268a8971d8e5874d9a8218644200fb8465f1dcad55e6851/yarl-1.24.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:3b075301a2836a0e297b1b658cb6d6135df535d62efefdd60366bd589c2c82f2", size = 92164, upload-time = "2026-05-19T21:28:53.242Z" }, + { url = "https://files.pythonhosted.org/packages/29/b6/170e2b8d4e3bc30e6bfdcca53556537f5bf595e938632dfcb059311f3ff6/yarl-1.24.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:8ae44649b00947634ab0dab2a374a638f52923a6e67083f2c156cd5cbd1a881d", size = 91688, upload-time = "2026-05-19T21:28:54.865Z" }, + { url = "https://files.pythonhosted.org/packages/fe/a5/c9f655d5553ea0b99fdac9d6a99ad3f9b3e73b8e5758bb46f58c9831f74c/yarl-1.24.2-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:507cc19f0b45454e2d6dcd62ff7d062b9f77a2812404e62dbdaec05b50faa035", size = 102902, upload-time = "2026-05-19T21:28:56.963Z" }, + { url = "https://files.pythonhosted.org/packages/5d/bc/6b9664d815d79af4ee553337f9d606c56bbf269186ada9172de45f1b5f60/yarl-1.24.2-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:c4c17bad5a530912d2111825d3f05e89bab2dd376aaa8cbc77e449e6db63e576", size = 97931, upload-time = "2026-05-19T21:28:58.56Z" }, + { url = "https://files.pythonhosted.org/packages/98/ec/32ba48acae30fecd60928f5791188b80a9d6ee3840507ffda29fecd37b71/yarl-1.24.2-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f5f0cbb112838a4a293985b6ed73948a547dadcc1ba6d2089938e7abdedceef8", size = 111030, upload-time = "2026-05-19T21:29:00.148Z" }, + { url = "https://files.pythonhosted.org/packages/82/5a/6f4cd081e5f4934d2ae3a8ef4abe3afacc010d26f0035ee91b35cd7d7c37/yarl-1.24.2-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5ec8356b8a6afcf81fc7aeeef13b1ff7a49dec00f313394bbb9e83830d32ccd7", size = 110392, upload-time = "2026-05-19T21:29:02.155Z" }, + { url = "https://files.pythonhosted.org/packages/7a/da/323a01c349bd5fb01bb6652e314d9bb218cee630a736bdb810ad50e4013f/yarl-1.24.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7e7ebcdef69dec6c6451e616f32b622a6d4a2e92b445c992f7c8e5274a6bbc4c", size = 105612, upload-time = "2026-05-19T21:29:04.247Z" }, + { url = "https://files.pythonhosted.org/packages/7c/80/264ab684f181e1a876389374519ff05d10248725535ae2ac4e8ac4e563d6/yarl-1.24.2-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:47a55d6cf6db2f401017a9e96e5288844e5051911fb4e0c8311a3980f5e59a7d", size = 104487, upload-time = "2026-05-19T21:29:06.491Z" }, + { url = "https://files.pythonhosted.org/packages/41/07/efabe5df87e96d7ad5959760b888344be48cd6884db127b407c6b5503adc/yarl-1.24.2-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:3065657c80a2321225e804048597ad55658a7e76b32d6f5ee4074d04c50401db", size = 102333, upload-time = "2026-05-19T21:29:08.267Z" }, + { url = "https://files.pythonhosted.org/packages/44/0c/bcf7c42603e1009295f586d8890f2ba032c8b53310e815adf0a202c73d9f/yarl-1.24.2-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:cb84b80d88e19ede158619b80813968713d8d008b0e2497a576e6a0557d50712", size = 99025, upload-time = "2026-05-19T21:29:10.682Z" }, + { url = "https://files.pythonhosted.org/packages/4f/82/84482ab1a57a0f21a08afe6a7004c61d741f8f2ecc3b05c321577c612164/yarl-1.24.2-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:990de4f680b1c217e77ff0d6aa0029f9eb79889c11fb3e9a3942c7eba29c1996", size = 110507, upload-time = "2026-05-19T21:29:12.954Z" }, + { url = "https://files.pythonhosted.org/packages/c4/8d/a546ba1dfe1b0f290e05fef145cd07614c0f15df1a707195e512d1e39d1d/yarl-1.24.2-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:abb8ec0323b80161e3802da3150ef660b41d0e9be2048b76a363d93eee992c2b", size = 103719, upload-time = "2026-05-19T21:29:14.893Z" }, + { url = "https://files.pythonhosted.org/packages/1a/b6/267f2a09213138473adfce6b8a6e17791d7fee70bd4d9003218e4dec58b0/yarl-1.24.2-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:e7977781f83638a4c73e0f88425563d70173e0dfd90ac006a45c65036293ee3c", size = 110438, upload-time = "2026-05-19T21:29:16.485Z" }, + { url = "https://files.pythonhosted.org/packages/48/2d/1c8d89c7c5f9cad9fb2902445d94e2ab1d7aa35de029afbb8ae95c42d00f/yarl-1.24.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:e30dd55825dc554ec5b66a94953b8eda8745926514c5089dfcacecb9c99b5bd1", size = 105719, upload-time = "2026-05-19T21:29:18.367Z" }, + { url = "https://files.pythonhosted.org/packages/a7/25/722e3b93bd687009afb2d59a35e13d30ddd8f80571445bb0c4e4ce26ec66/yarl-1.24.2-cp312-cp312-win_amd64.whl", hash = "sha256:7dafe10c12ddd4d120d528c4b5599c953bd7b12845347d507b95451195bb6cad", size = 92901, upload-time = "2026-05-19T21:29:20.014Z" }, + { url = "https://files.pythonhosted.org/packages/39/47/4486ccfb674c04854a1ef8aa77868b6a6f765feaf69633409d7ca4f02cb8/yarl-1.24.2-cp312-cp312-win_arm64.whl", hash = "sha256:044a09d8401fcf8681977faef6d286b8ade1e2d2e9dceda175d1cfa5ca496f30", size = 87229, upload-time = "2026-05-19T21:29:22.1Z" }, + { url = "https://files.pythonhosted.org/packages/82/62/fcf0ce677f17e5c471c06311dd25964be38a4c586993632910d2e75278bc/yarl-1.24.2-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:491ac9141decf49ee8030199e1ee251cdff0e131f25678817ff6aa5f837a3536", size = 128978, upload-time = "2026-05-19T21:29:23.83Z" }, + { url = "https://files.pythonhosted.org/packages/d3/58/8e63299bb71ed61a834121d9d3fe6c9fcf2a6a5d09754ff4f20f2d20baf5/yarl-1.24.2-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:e89418f65eda18f99030386305bd44d7d504e328a7945db1ead514fbe03a0607", size = 91733, upload-time = "2026-05-19T21:29:25.375Z" }, + { url = "https://files.pythonhosted.org/packages/c1/24/16748d5dab6daec8b0ed81ccec639a1cded0f18dcc62a4f696b4fe366c37/yarl-1.24.2-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:cdfcce633b4a4bb8281913c57fcafd4b5933fbc19111a5e3930bbd299d6102f1", size = 91113, upload-time = "2026-05-19T21:29:26.928Z" }, + { url = "https://files.pythonhosted.org/packages/1b/66/b63fff7b71211e866624b21432d5943cbb633eb0c2872d9ee3070648f22c/yarl-1.24.2-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:863297ddede92ee49024e9a9b11ecb59f310ca85b60d8537f56bed9bbb5b1986", size = 103899, upload-time = "2026-05-19T21:29:28.842Z" }, + { url = "https://files.pythonhosted.org/packages/9d/ac/ba1974b8533909636f7733fe86cf677e3619527c3c2fa913e0ea89c48757/yarl-1.24.2-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:374423f70754a2c96942ede36a29d37dc6b0cb8f92f8d009ddf3ed78d3da5488", size = 97862, upload-time = "2026-05-19T21:29:31.086Z" }, + { url = "https://files.pythonhosted.org/packages/1b/a5/123ac993b5c2ba6f554a140305620cb8f150fa543711bbc49be3ec0a65a4/yarl-1.24.2-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:33a29b5d00ccbf3219bb3e351d7875739c19481e030779f48cc46a7a71681a9b", size = 111060, upload-time = "2026-05-19T21:29:32.657Z" }, + { url = "https://files.pythonhosted.org/packages/23/37/c472d3af3509688392134a88a825276770a187f1daa4de3f6dc0a327a751/yarl-1.24.2-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a9532c57211730c515341af11fef6e9b61d157487272a096d0c04da445642592", size = 110613, upload-time = "2026-05-19T21:29:34.379Z" }, + { url = "https://files.pythonhosted.org/packages/df/88/09c28dad91e662ccfaa1b78f1c57badde74fc9d0b23e74aef644750ecd73/yarl-1.24.2-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:91e72cf093fd833483a97ee648e0c053c7c629f51ff4a0e7edd84f806b0c5617", size = 107012, upload-time = "2026-05-19T21:29:36.216Z" }, + { url = "https://files.pythonhosted.org/packages/07/ab/9d4f69d571a94f4d112fa7e2e007200f5a54d319f58c82ac7b7baa61f5c6/yarl-1.24.2-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:b3177bc0a768ef3bacceb4f272632990b7bea352f1b2f1eee9d6d6ff16516f92", size = 105887, upload-time = "2026-05-19T21:29:38.746Z" }, + { url = "https://files.pythonhosted.org/packages/8e/9a/000b2b66c0d772a499fc531d21dab92dfeb73b640a12eed6ba89f49bb2d0/yarl-1.24.2-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:e196952aacaf3b232e265ff02980b64d483dc0972bd49bcb061171ff22ac203a", size = 103620, upload-time = "2026-05-19T21:29:40.368Z" }, + { url = "https://files.pythonhosted.org/packages/41/7c/7c1050f73450fbdaa3f0c72017059f00ce5e13366692f3dba25275a1083d/yarl-1.24.2-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:204e7a61ce99919c0de1bf904ab5d7aa188a129ea8f690a8f76cfb6e2844dc44", size = 100599, upload-time = "2026-05-19T21:29:42.66Z" }, + { url = "https://files.pythonhosted.org/packages/ec/b1/29e5756b3926705f5f6089bd5b9f50a56eaac550da6e260bf713ead44d04/yarl-1.24.2-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:4b156914620f0b9d78dc1adb3751141daee561cfec796088abb89ed49d220f1a", size = 110604, upload-time = "2026-05-19T21:29:44.632Z" }, + { url = "https://files.pythonhosted.org/packages/a3/4b/8415bc96e9b150cde942fbac9a8182985e58f40ce5c54c34ed015407d3ee/yarl-1.24.2-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:8372a2b976cf70654b2be6619ab6068acabb35f724c0fda7b277fbf53d66a5cf", size = 105161, upload-time = "2026-05-19T21:29:46.755Z" }, + { url = "https://files.pythonhosted.org/packages/8b/d4/cde059abfa229553b7298a2eadde2752e723d50aeedaef86ce59da2718ee/yarl-1.24.2-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:f9a1e9b622ca284143aab5d885848686dcd85453bb1ca9abcdb7503e64dc0056", size = 110619, upload-time = "2026-05-19T21:29:48.972Z" }, + { url = "https://files.pythonhosted.org/packages/e7/2c/d6a6c9a61549f7b6c7e6dc6937d195bcf069582b47b7200dcd0e7b256acf/yarl-1.24.2-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:810e19b685c8c3c5862f6a38160a1f4e4c0916c9390024ec347b6157a45a0992", size = 107362, upload-time = "2026-05-19T21:29:51Z" }, + { url = "https://files.pythonhosted.org/packages/92/dd/3ae5fe417e9d1c353a548553326eb9935e76b6b727161563b424cc296df3/yarl-1.24.2-cp313-cp313-win_amd64.whl", hash = "sha256:7d37fb7c38f2b6edab0f845c4f85148d4c44204f52bc127021bd2bc9fdbf1656", size = 92667, upload-time = "2026-05-19T21:29:52.743Z" }, + { url = "https://files.pythonhosted.org/packages/10/cc/a7beb239f78f27fca1b053c8e8595e4179c02e62249b4687ec218c370c50/yarl-1.24.2-cp313-cp313-win_arm64.whl", hash = "sha256:1e831894be7c2954240e49791fa4b50c05a0dc881de2552cfe3ffd8631c7f461", size = 87069, upload-time = "2026-05-19T21:29:54.442Z" }, + { url = "https://files.pythonhosted.org/packages/40/0e/e08087695fc12789263821c5dc0f8dc52b5b17efd0887cacf419f8a43ba3/yarl-1.24.2-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:f9312b3c02d9b3d23840f67952913c9c8721d7f1b7db305289faefa878f364c2", size = 129670, upload-time = "2026-05-19T21:29:56.631Z" }, + { url = "https://files.pythonhosted.org/packages/3a/98/ab4b5ed1b1b5cd973c8a3eb994c3a6aefb6ce6d399e21bb5f0316c33815c/yarl-1.24.2-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:a4f4d6cd615823bfc7fb7e9b5987c3f41666371d870d51058f77e2680fbe9630", size = 91916, upload-time = "2026-05-19T21:29:58.645Z" }, + { url = "https://files.pythonhosted.org/packages/ba/b1/5297bb6a7df4782f7605bffc43b31f5044070935fbbcaa6c705a07e6ac65/yarl-1.24.2-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:0c3063e5c0a8e8e62fae6c2596fa01da1561e4cd1da6fec5789f5cf99a8aefd8", size = 91625, upload-time = "2026-05-19T21:30:00.412Z" }, + { url = "https://files.pythonhosted.org/packages/02/a7/45baabfff76829264e623b185cff0c340d7e11bf3e1cd9ea37e7d17934bd/yarl-1.24.2-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:fecd17873a096036c1c87ab3486f1aef7f269ada7f23f7f856f93b1cc7744f14", size = 104574, upload-time = "2026-05-19T21:30:02.544Z" }, + { url = "https://files.pythonhosted.org/packages/f3/40/3a5ab144d3d650ca37d4f4b57e56169be8af3ca34c448793e064b30baaed/yarl-1.24.2-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:a46d1ab4ba4d32e6dc80daf8a28ce0bd83d08df52fbc32f3e288663427734535", size = 97534, upload-time = "2026-05-19T21:30:04.319Z" }, + { url = "https://files.pythonhosted.org/packages/9c/b5/5658fef3681fb5776b4513b052bec750009f47b3a592251c705d75375798/yarl-1.24.2-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:73e68edf6dfd5f73f9ca127d84e2a6f9213c65bdffb736bda19524c0564fcd14", size = 111481, upload-time = "2026-05-19T21:30:05.988Z" }, + { url = "https://files.pythonhosted.org/packages/4c/06/fdcd7dde037f00866dce123ed4ba23dba94beb56fc4cf561668d27be37f2/yarl-1.24.2-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a296ca617f2d25fbceafb962b88750d627e5984e75732c712154d058ae8d79a3", size = 111529, upload-time = "2026-05-19T21:30:07.738Z" }, + { url = "https://files.pythonhosted.org/packages/c2/53/d81269aaafccea0d33396c03035de997b743f11e648e6e27a0df99c72980/yarl-1.24.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e51b2cf5ec89a8b8470177641ed62a3ba22d74e1e898e06ad53aa77972487208", size = 107338, upload-time = "2026-05-19T21:30:09.713Z" }, + { url = "https://files.pythonhosted.org/packages/ae/04/23049463f729bd899df203a7960505a75333edd499cda8aa1d5a82b64df5/yarl-1.24.2-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:310fc687f7b2044ec54e372c8cbe923bb88f5c37bded0d3079e5791c2fc3cf50", size = 106147, upload-time = "2026-05-19T21:30:11.365Z" }, + { url = "https://files.pythonhosted.org/packages/14/18/04a4b5830b43ed5e4c5015b40e9f6241ad91487d71611061b4e111d6ac80/yarl-1.24.2-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:297a2fe352ecf858b30a98f87948746ec16f001d279f84aebdbd3bd965e2f1bd", size = 104272, upload-time = "2026-05-19T21:30:12.978Z" }, + { url = "https://files.pythonhosted.org/packages/5a/f7/8cffdf319aee7a7c1dbd07b61d91c3e3fda460c7a93b5f93e445f3806c4c/yarl-1.24.2-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:2a263e76b97bc42bdcd7c5f4953dec1f7cd62a1112fa7f869e57255229390d67", size = 99962, upload-time = "2026-05-19T21:30:15.001Z" }, + { url = "https://files.pythonhosted.org/packages/d7/39/b3cce3b7dbef64ac700ad4cea156a207d01bede0f507587616c364b5468e/yarl-1.24.2-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:822519b64cf0b474f1a0aaef1dc621438ea46bb77c94df97a5b4d213a7d8a8b1", size = 111063, upload-time = "2026-05-19T21:30:16.683Z" }, + { url = "https://files.pythonhosted.org/packages/a1/ea/100818505e7ebf165c7242ff17fdf7d9fee79e27234aeca871c1082920d7/yarl-1.24.2-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:b6067060d9dc594899ba83e6db6c48c68d1e494a6dab158156ed86977ca7bcb1", size = 105438, upload-time = "2026-05-19T21:30:18.769Z" }, + { url = "https://files.pythonhosted.org/packages/8f/d2/e075a0b32aa6625087de9e653087df0759fed5de4a435fef594181102a77/yarl-1.24.2-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:0063adad533e57171b79db3943b229d40dfafeeee579767f96541f106bac5f1b", size = 111458, upload-time = "2026-05-19T21:30:21.024Z" }, + { url = "https://files.pythonhosted.org/packages/e6/5c/ceea7ba98b65c8eb8d947fdc52f9bedfcd43c6a57c9e3c90c17be8f324a3/yarl-1.24.2-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:ee8e3fb34513e8dc082b586ef4910c98335d43a6fab688cd44d4851bacfce3e8", size = 107589, upload-time = "2026-05-19T21:30:23.412Z" }, + { url = "https://files.pythonhosted.org/packages/fa/d9/5582d57e2b2db9b85eb6663a22efdd78e08805f3f5389566e9fcad254d1b/yarl-1.24.2-cp314-cp314-win_amd64.whl", hash = "sha256:afb00d7fd8e0f285ca29a44cc50df2d622ff2f7a6d933fa641577b5f9d5f3db0", size = 94424, upload-time = "2026-05-19T21:30:25.425Z" }, + { url = "https://files.pythonhosted.org/packages/92/10/7dc07a0e22806a9280f42a57361395506e800c64e22737cd7b0886feab42/yarl-1.24.2-cp314-cp314-win_arm64.whl", hash = "sha256:68cf6eacd6028ef1142bc4b48376b81566385ca6f9e7dde3b0fa91be08ffcb57", size = 88690, upload-time = "2026-05-19T21:30:27.623Z" }, + { url = "https://files.pythonhosted.org/packages/9e/13/d5b8e2c8667db955bcb3de233f18798fefe7edf1d7429c2c9d4f9c401114/yarl-1.24.2-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:221ce1dd921ac4f603957f17d7c18c5cc0797fbb52f156941f92e04605d1d67b", size = 136248, upload-time = "2026-05-19T21:30:29.297Z" }, + { url = "https://files.pythonhosted.org/packages/de/46/a4a97c05c9c9b8fd266bb2a0df12992c7fbd02391eb9640583411b6dab32/yarl-1.24.2-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:5f3224db28173a00d7afacdee07045cc4673dfab2b15492c7ae10deddbece761", size = 95084, upload-time = "2026-05-19T21:30:31.031Z" }, + { url = "https://files.pythonhosted.org/packages/95/b2/845cf2074a015e6fe0d0808cf1a2d9e868386c4220d657ebd8302b199043/yarl-1.24.2-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:c557165320d6244ebe3a02431b2a201a20080e02f41f0cfa0ccc47a183765da8", size = 95272, upload-time = "2026-05-19T21:30:33.062Z" }, + { url = "https://files.pythonhosted.org/packages/fe/16/e69d4aa244aef45235ddfebc0e04036a6829842bc5a6a795aedc6c998d23/yarl-1.24.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:904065e6e85b1fa54d0d87438bd58c14c0bad97aad654ad1077fd9d87e8478ed", size = 101497, upload-time = "2026-05-19T21:30:34.842Z" }, + { url = "https://files.pythonhosted.org/packages/15/94/c07107715d621076863ee88b3ddf183fa5e9d4aba5769623c9979828410a/yarl-1.24.2-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:8cec2a38d70edc10e0e856ceda886af5327a017ccbde8e1de1bd44d300357543", size = 94002, upload-time = "2026-05-19T21:30:37.724Z" }, + { url = "https://files.pythonhosted.org/packages/a9/35/fc1bbdd895b5e4010b8fdd037f7ed3aa289d3863e08231b30231ca9a0815/yarl-1.24.2-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e7484b9361ed222ee1ca5b4337aa4cbdcc4618ce5aff57d9ef1582fd95893fc0", size = 106524, upload-time = "2026-05-19T21:30:40.196Z" }, + { url = "https://files.pythonhosted.org/packages/1f/f2/32b66d0a4ba47c296cf86d03e2c67bff58399fe6d6d84d5205c04c66cc6d/yarl-1.24.2-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:84f9670b89f34db07f81e53aee83e0b938a3412329d51c8f922488be7fcc4024", size = 106165, upload-time = "2026-05-19T21:30:41.888Z" }, + { url = "https://files.pythonhosted.org/packages/95/47/37cb5ff50c5e825d4d38e81bb04d1b7e96bf960f7ab89f9850b162f3f114/yarl-1.24.2-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:abb2759733d63a28b4956500a5dd57140f26486c92b2caedfb964ab7d9b79dbf", size = 103010, upload-time = "2026-05-19T21:30:43.985Z" }, + { url = "https://files.pythonhosted.org/packages/6f/d2/4597912315096f7bb359e46e13bf8b60994fcbb2db29b804c0902ef4eff5/yarl-1.24.2-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:081c2bf54efe03774d0311172bc04fedf9ca01e644d4cd8c805688e527209bdc", size = 101128, upload-time = "2026-05-19T21:30:46.291Z" }, + { url = "https://files.pythonhosted.org/packages/b9/d5/c8e86e120521e646013d02a8e3b8884392e28494be8f392366e50d208efc/yarl-1.24.2-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:86746bef442aa479107fe28132e1277237f9c24c2f00b0b0cf22b3ee0904f2bb", size = 101382, upload-time = "2026-05-19T21:30:48.085Z" }, + { url = "https://files.pythonhosted.org/packages/fa/98/70b229236118f89dbeb739b76f10225bbf53b5497725502594c9a01d699a/yarl-1.24.2-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:2d07d21d0bc4b17558e8de0b02fbfdf1e347d3bb3699edd00bb92e7c57925420", size = 95964, upload-time = "2026-05-19T21:30:49.785Z" }, + { url = "https://files.pythonhosted.org/packages/87/f8/56c386981e3c8648d279fdef2397ffec577e8320fd5649745e34d54faeb7/yarl-1.24.2-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:4fb1ac3fc5fecd8ae7453ea237e4d22b49befa70266dfe1629924245c21a0c7f", size = 106204, upload-time = "2026-05-19T21:30:51.862Z" }, + { url = "https://files.pythonhosted.org/packages/1a/1e/765afe97811ca35933e2a7de70ac57b1997ea2e4ee895719ee7a231fb7e5/yarl-1.24.2-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:4da31a5512ed1729ca8d8aacde3f7faeb8843cde3165d6bcf7f88f74f17bb8aa", size = 101510, upload-time = "2026-05-19T21:30:53.62Z" }, + { url = "https://files.pythonhosted.org/packages/ee/78/393913f4b9039e1edd09ae8a9bbb9d539be909a8abf6d8a2084585bed4b7/yarl-1.24.2-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:533ded4dceb5f1f3da7906244f4e82cf46cfd40d84c69a1faf5ac506aa65ecbe", size = 105584, upload-time = "2026-05-19T21:30:55.962Z" }, + { url = "https://files.pythonhosted.org/packages/78/87/deb17b7049bbe74ea11a713b86f8f27800cc1c8648b0b797243ebb4830ba/yarl-1.24.2-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:7b3a85525f6e7eeabcfdd372862b21ee1915db1b498a04e8bf0e389b607ff0bd", size = 103410, upload-time = "2026-05-19T21:30:57.962Z" }, + { url = "https://files.pythonhosted.org/packages/8f/be/f9f7594e23b5b93affff0318e4593c1920331bcaefda326cabcad94296a1/yarl-1.24.2-cp314-cp314t-win_amd64.whl", hash = "sha256:a7624b1ca46ca5d7b864ef0d2f8efe3091454085ee1855b4e992314529972215", size = 102980, upload-time = "2026-05-19T21:30:59.735Z" }, + { url = "https://files.pythonhosted.org/packages/65/a4/ba80dccd3593ff1f01051a818694d07b58cb8232677ee9a22a5a1f93a9fc/yarl-1.24.2-cp314-cp314t-win_arm64.whl", hash = "sha256:e434a45ce2e7a947f951fc5a8944c8cc080b7e59f9c50ae80fd39107cf88126d", size = 91219, upload-time = "2026-05-19T21:31:01.934Z" }, + { url = "https://files.pythonhosted.org/packages/fd/4d/4b880086bd0d3e034d25647be1d830afc3e3f610e98c4ab3490af6b1b6d5/yarl-1.24.2-py3-none-any.whl", hash = "sha256:2783d9226db8797636cd6896e4de81feed252d1db72265686c9558d97a4d94b9", size = 53576, upload-time = "2026-05-19T21:31:03.909Z" }, +]