ltuncay commited on
Commit
29be9f8
·
verified ·
1 Parent(s): 5937b11

Add the project X-ARES comparison table

Browse files
Files changed (2) hide show
  1. README.md +23 -0
  2. export_manifest.json +1 -1
README.md CHANGED
@@ -20,6 +20,29 @@ This repository contains the trained encoder, its preprocessing configuration,
20
  and the custom Transformers implementation. No installation of the research
21
  repository is needed.
22
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
23
  ## Model and training
24
 
25
  | Property | Value |
 
20
  and the custom Transformers implementation. No installation of the research
21
  repository is needed.
22
 
23
+ ## X-ARES results
24
+
25
+ Scores on [X-ARES](https://arxiv.org/abs/2505.16369) (0–100, higher is better).
26
+ BEST-RQ (Conformer), BEST-RQ (ViT),
27
+ and all BEST-RQ-2 variants reported below are trained on the **same AudioSet
28
+ split for 200,000 steps**. The pretrained baselines are shown for comparison.
29
+
30
+ | Model | Speech | Music | Environment | Global Mean | Mean of Means | Training recipe |
31
+ | --- | ---: | ---: | ---: | ---: | ---: | --- |
32
+ | data2vec | 50.62 | 23.24 | 15.41 | 37.83 | 29.76 | Pretrained baseline |
33
+ | wav2vec 2.0 | 41.79 | 34.94 | 29.52 | 37.84 | 35.42 | Pretrained baseline |
34
+ | Whisper | 49.19 | 38.67 | 28.61 | 42.75 | 38.82 | Pretrained baseline |
35
+ | BEST-RQ (Conformer) | 40.43 | 35.58 | 30.81 | 37.43 | 35.60 | Separate codebase (recipe unavailable) |
36
+ | BEST-RQ (ViT) | 32.87 | 41.62 | 34.50 | 34.88 | 36.33 | [best_rq/audioset/default.yaml](https://github.com/LudovicTuncay/audio-embeddings/blob/bb88bf790b1dcf8251c6b38e7a4766534adf33d3/configs/experiment/best_rq/audioset/default.yaml) |
37
+ | BEST-RQ-2 (Interspeech 2026) | 38.49 | 54.40 | 46.39 | 43.21 | 46.43 | [best_rq_2/default.yaml](https://github.com/LudovicTuncay/audio-embeddings/blob/bb88bf790b1dcf8251c6b38e7a4766534adf33d3/configs/experiment/best_rq_2/default.yaml) |
38
+ | BEST-RQ-2.1 | 52.60 | 62.23 | 53.38 | 54.59 | 56.07 | [best_rq_2_1/masking/80.yaml](https://github.com/LudovicTuncay/audio-embeddings/blob/bb88bf790b1dcf8251c6b38e7a4766534adf33d3/configs/experiment/best_rq_2_1/masking/80.yaml) |
39
+ | BEST-RQ-2.2 | **53.78** | **63.90** | **55.71** | **56.11** | **57.80** | [best_rq_2_2/masking/80.yaml](https://github.com/LudovicTuncay/audio-embeddings/blob/bb88bf790b1dcf8251c6b38e7a4766534adf33d3/configs/experiment/best_rq_2_2/masking/80.yaml) |
40
+
41
+ Global Mean averages all benchmark task scores. Mean of Means gives equal
42
+ weight to the Speech, Music, and Environment category means.
43
+
44
+ These are the research results reported in the [project README](https://github.com/LudovicTuncay/audio-embeddings/blob/bb88bf790b1dcf8251c6b38e7a4766534adf33d3/README.md), not a new benchmark run of the Transformers exports.
45
+
46
  ## Model and training
47
 
48
  | Property | Value |
export_manifest.json CHANGED
@@ -27,7 +27,7 @@
27
  },
28
  "files": {
29
  "CODE_LICENSE": "8fe9e8b749cd4abedabcb3100df445db899e72192394d14cc2dbf24a40811af6",
30
- "README.md": "0db5dc79d19a5c6cf1fecc484e3545c191bc84492607c6cc91cd3f0de2b2c898",
31
  "adapters.py": "ca0f26826763c0e48b7508da732242bb83adfeb4814e389ed3fde93ad648c728",
32
  "config.json": "93452b1dfb678bafd2737b29cc80586e1ccd8368ad0183ed3b88ffd8f12ed6f2",
33
  "configuration_audio.py": "59f3a0b8db0df5af85e677ac33a4431595e9b2eff6d34c1e6a1778dd75a4568d",
 
27
  },
28
  "files": {
29
  "CODE_LICENSE": "8fe9e8b749cd4abedabcb3100df445db899e72192394d14cc2dbf24a40811af6",
30
+ "README.md": "5665db6962229a28aa88e64673b0fe4bbc4017f8b71881ac5d800fdfb09d54b1",
31
  "adapters.py": "ca0f26826763c0e48b7508da732242bb83adfeb4814e389ed3fde93ad648c728",
32
  "config.json": "93452b1dfb678bafd2737b29cc80586e1ccd8368ad0183ed3b88ffd8f12ed6f2",
33
  "configuration_audio.py": "59f3a0b8db0df5af85e677ac33a4431595e9b2eff6d34c1e6a1778dd75a4568d",