Spaces:
Running
Running
Commit History
-
feat: run the eval worker inside the Space (CPU, isolated venv)
Make the Space self-contained — it now runs the eval engine itself on CPU, no external machine needed: - Dockerfile: build an isolated /opt/worker-venv (CPU torch + open-eeg-bench), kept OUT of the web app's pinned stack to avoid version conflicts; launch the worker as a background '--watch' loop alongside uvicorn + the frontend, gated by RUN_EVAL_WORKER (set =0 in Space settings to disable without a rebuild). Built before the code COPY (cached across code rebuilds) and non-fatal (a failed worker-deps install never breaks the web app). - run_eval_worker.py: import-light (only huggingface_hub + pure app modules, so it runs in the minimal venv), default --device cpu, add process_queue() and a --watch/--interval service loop. - app/services/__init__.py: stop eagerly importing the heavy submodules so the pure results_mapping can be imported without dragging in datasets/transformers/ aiohttp. All call sites use direct submodule imports, so nothing breaks. Verified: the w…c54d8a2 -
feat: complete the eval engine — real worker + OEB-aware queue reader
The arena's submission loop was open: nothing produced leaderboard results. - run_eval_worker.py is now a complete one-shot worker (run on external GPU/SLURM infra, not the Space): PENDING -> mark RUNNING -> oeb.benchmark() -> map_oeb_results_to_contents -> append to public braindecode/contents (single train.parquet, deduped by model+adapter) -> mark FINISHED/FAILED. Supports --device, --dry-run, --slurm, --infra-folder, --limit. Writes parquet via pandas + upload_file to stay robust across datasets/pyarrow. - _refresh_models_cache now parses the OpenEEGBench request schema (model_name/submitter/benchmark_kwargs) instead of the legacy LLM fields (model/revision/sender), which previously KeyError'd on every entry so submissions never appeared in the queue UI. Legacy fields kept as fallback. - worker-requirements.txt pins open-eeg-bench for the worker env. Verified end-to-end against throwaway HF repos: publish() round-trips through load_dataset(...)['train'] (what the …356d76f -
feat: OpenEEGBench submission flow (arena runs the eval)
Replace the leftover LLM-leaderboard submission with a first-class OpenEEGBench flow. Users submit a model reference (model_cls + checkpoint + finetuning strategy/head/datasets); the backend validates it, enforces a 5-per-month per-user cap, and queues a request to braindecode/requests. A worker (stub) runs oeb.benchmark() and maps results to braindecode/contents. Backend: - submission_config: OEB enums/defaults (single source of truth) - submissions: pure validation, request building, monthly cap - results_mapping: OEB DataFrame -> contents schema (balanced_accuracy) - model_validation: EEG/OEB weights-source + license checks - OEB-shaped submit endpoint/service; eval worker stub (run_eval_worker.py) Frontend: - OEB submission form (model_cls, weights source, strategies, heads, PEFT modules, datasets, seeds, contact/visibility) with conditional fields - rewritten submission guide (5/month, help -> OpenEEGBench issues, citation) - queue/limit copy aligned; fixed duplicate style key…4db3be0
-
Add HF eval-results benchmark registry and API endpoint
- Add canonical EEG_BENCHMARKS registry (8 datasets) in config/benchmarks.py - Add GET /api/benchmarks endpoint for benchmark metadata - Refactor leaderboard transform_data to loop over registry instead of hardcoding - Extract _build_entry_id helper for consistent unique ID generation - Add push_eval_yaml.py script to register benchmarks on HF Hub - Add pyyaml dependency6f5cc9b