README.md

---
license: bsd-3-clause
tags:
- executorch
- xnnpack
- pte
- on-device
- audio-classification
base_model:
- MIT/ast-finetuned-audioset-10-10-0.4593
base_model_relation: quantized
---
# AST audio event classification — ExecuTorch

Ten seconds of sound in, 527 AudioSet labels out: speech, music, a dog, a door, an engine.
The shelf had speech recognition and speaker identity; this is the one that says what a
sound *is*.

- **Source**: MIT/ast-finetuned-audioset-10-10-0.4593 — 86M parameters, Audio Spectrogram Transformer
- **License**: bsd-3-clause
- **Input**: `input_values` `[1, 1024, 128]` fp32 — log-mel filterbank [1, 1024, 128] fp32 — `torchaudio.compliance.kaldi.fbank` at 16000 Hz with 128 mel bins, padded or trimmed to 1024 frames (10.24 s), then normalised with mean=-4.2677393 std=4.5689974. `ASTFeatureExtractor` does exactly this
- **Output**: logits [1, 527] — AudioSet labels, multi-label: apply sigmoid, not softmax

## Variants

| build | file | size (MB) | Mac median (ms)* | top-1 vs eager | worst probability shift |
|---|---|---|---|---|---|
| fp32 | `audiocls_audioset_xnnpack_fp32.pte` | 346.6 | 285.0 | 8 of 8 | 0.0000 |
| fp16 | `audiocls_audioset_xnnpack_fp16.pte` | 173.9 | 545.1 | 8 of 8 | 0.0014 |
| int8 (dynamic) | `audiocls_audioset_xnnpack_int8.pte` | 90.9 | 267.3 | 8 of 8 | 0.0160 |
| Core ML (fp16, iOS) | `audiocls_audioset_coreml_all.pte` | 173.7 | 74.6 | 8 of 8 | 0.0033 |

\*Mac arm64, single process, median of 10, one 10.24 s clip. PyTorch eager fp32 on the same
machine is **122.7 ms**. Core ML at 74.6 ms is 1.6x that; int8 at 267.3 ms
is the fastest portable build and a quarter of the fp32 file. fp16 is **slower than fp32**
here (545.1 ms) — XNNPACK emulates it — and only earns its place by halving the file.

## What the classifier actually says

The test clips are speech, and every build puts `Speech` in the top five on all
8 of them. The top-1 label matches eager on 8 of 8, the top-five sets overlap
40 of 40, and no sigmoid probability moves by more than the figure in the table.

The distance the error has to cover is printed too: the gap between the winning logit and
the runner-up is at least **3.09** on these clips, which every build's shift is far
inside. Agreement alone would not show this — a build returning a constant vector would
agree with a broken reference on every clip — so the label check is there as well.

## The features are the caller's job, and the recipe is exact

AST's front end is `torchaudio.compliance.kaldi.fbank`: a Kaldi-compatible filterbank with
its own windowing and edge handling. Reimplementing it inside the graph would be a second
model's worth of work for a transform `transformers` runs in two lines, so the graph starts
at the spectrogram. The recipe is read off the model's own preprocessor rather than written
from memory:

```python
from transformers import AutoFeatureExtractor
extractor = AutoFeatureExtractor.from_pretrained("MIT/ast-finetuned-audioset-10-10-0.4593")
inputs = extractor(waveform, sampling_rate=16000, return_tensors="pt")["input_values"]
```

Getting it wrong does not throw. It shifts every probability.

**The output is multi-label**: apply `sigmoid`, not `softmax`. A ten-second clip can be
speech *and* music *and* a car at once, which is the point of AudioSet.

## Conversion

```bash
python convert/export_audiocls.py audioset
python convert/check_audiocls.py audioset int8
```

(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))