README.md

---
tags:
- speech
- pronunciation-assessment
- streaming
- whisper
- charsiu
language:
- en
license: other
---

# StreamPA

Frozen StreamPA pronunciation-assessment inference: fine-tuned Whisper Base
N-best -> G2P / prefix-local Charsiu / PCN / prosody -> slot and utterance scores.
This is a downloadable, multi-component Python system, not a hosted endpoint
or a standard Transformers `AutoModel`/`pipeline` implementation.

## Published identity

* Experiment: `M_stress_scalar_gate_capacity64` (Ours-M), primary seed **1337**.
* Original checkpoint SHA256:
  `870df8dbe3c787487f094e0c4d4ade6bed8fcc5d86b578c4edc57581e81023b4`.
* All original tensors are retained losslessly in `scorer/model.safetensors`.
  Strict loading retains the historical inactive `utt_head`; the active
  utterance path is the frozen `gru_visible` head.
* 64 hidden dimensions, three local-attention blocks, four heads, scalar
  reliability gate, detached stress branch, 64-dimensional recurrent state.
  Inputs: 40 phone columns including epsilon, 14 global prosody features,
  17 slot prosody features, and at most **72 PCN slots**.
* The included Whisper weights are the actual fine-tuned Base checkpoint,
  SHA256 `8917c9612b44f73bb5ef2ffec48cf44a4ee0dfa5a94af25e9e6710b9fd66a17f`.
  They must not be replaced with generic Whisper Base. Beam size 8, N-best 5,
  maximum 128 decoded tokens; original hypothesis-score computation retained.
* The paper's four-seed aggregate (1337, 202601, 202602, 202603) is not the
  performance of this single checkpoint. No unverified paper metric is copied
  into this card. Interface tests do not establish assessment PCC.

## Download and install

Verified platform: **Python 3.10 on Ubuntu in local Windows WSL2, CPU FP32**.
Native Windows and GPU inference are not part of the verified platform.
Use a new Python 3.10 environment. The fixed release tag below is created only
after fresh Hub-download inference verification succeeds.

```bash
python3.10 -m venv .venv-streampa
source .venv-streampa/bin/activate
python -m pip install huggingface_hub==0.36.0
python -c "from huggingface_hub import snapshot_download; snapshot_download('faeea/StreamPA', revision='streampa-v1.0.0', local_dir='./StreamPA')"
cd StreamPA
python -m pip install --extra-index-url https://download.pytorch.org/whl/cpu -r requirements-lock.txt
python -m pip install --no-deps --no-build-isolation .
python scripts/setup_assets.py
python scripts/verify_bundle.py
```

The public bundle contains the scorer and fine-tuned Whisper weights. Setup
downloads Charsiu and G2P resources from official, **fixed revisions**, checks
their SHA256s, and installs them inside this bundle. There is no runtime
reference to a training checkout or private server. See `THIRD_PARTY_NOTICES.md`
for licenses and third-party weight terms. Internet is needed for initial
installation/setup, not for subsequent inference. No author token is required
for the public release or its official runtime resources.

## One WAV and streaming CLI

Input: your own English **mono 16,000-Hz WAV**, no transcript or labels.
The frontend supports at most 30 seconds per recording and 72 current PCN
slots; longer inputs return an explicit error/status, never hidden truncation.
Output files must not already exist.

```bash
python -m streampa.infer --audio /path/to/demo.wav --bundle-dir . --device cpu --output scores.json
python -m streampa.stream --audio /path/to/demo.wav --bundle-dir . --device cpu --output prefixes.jsonl
```

The single-file CLI returns all prefixes plus the EOF result. The streaming
CLI actually reads and appends waveform blocks, then signals EOF. Both run
the same causal implementation. CLI block size does not change the fixed
0.64-second schedule and 0.16-second lookahead.

For an appropriate CUDA installation, select one device externally and use
`--device cuda`. Do not use a broken/busy device. The inherited frontend uses
FP16 Whisper on CUDA, FP32 Charsiu/scorer; CPU is FP32 throughout. CUDA is
**untested in this release validation**, and changing precision can change ASR
decoding. The CPU lockfile deliberately selects CPU PyTorch.

## Python sessions

```python
import soundfile as sf
from streampa import StreamPAPipeline

pipeline = StreamPAPipeline('./StreamPA', device='cpu', threads=2)
session = pipeline.create_session()
with sf.SoundFile('demo.wav') as f:
    assert f.samplerate == 16000 and f.channels == 1
    while True:
        block = f.read(1600, dtype='float32')  # 100 ms received audio
        if len(block) == 0:
            break
        for result in session.append(block):
            print(result['audio_end'], result['status'], result['scores'])
print(session.finish())  # process the exact EOF, including a short final block
session.reset()          # required before a new recording in this session
assert session.current() is None
```

The pipeline owns frozen models; sessions have separate audio and recurrent
state. Use sessions sequentially (concurrent GPU service scheduling is not
provided). `StreamPAPipeline.from_pretrained(repo_id, revision=...)` is also
implemented, but its downloaded directory must have `setup_assets.py` run
before loading; use the explicit local-bundle example for first setup.

## JSON and state semantics

* `commit_time`: scheduling boundary; `audio_end`: exact prefix waveform end
  supplied to **every** frontend; `received_audio_end`: audio received by the
  caller so far, which may be later when a block spans multiple boundaries.
  A normal prefix is processed only after `commit_time + 0.16` is received.
  EOF alone defines the final endpoint. No reference boundary is consulted.
* `slots`: raw independent phone-head and word-head predictions per current
  PCN slot, ASR word assignment, slot time, validity and commitment masks.
  Epsilon/gap slots are retained; `valid_phone=false` means their phone score
  is not a localized phone assessment. `asr_word_id=-1` has no word assignment.
* `words`: mean **word-head** predictions over visible slots assigned to each
  current ASR word. This is a presentation aggregation, not mean phone accuracy.
  It is not the paper's offline reference-word-identity evaluation. ASR words
  and scores may be revised at later prefixes, including committed words.
* `new_commit_mask`: the frozen legacy boundary-aware state-update rule.
  Only newly committed word representations are consumed by the GRU. With
  no new words the persistent state remains unchanged, but visible features
  still enter the sentence head and may change current utterance scores.
  This heuristic has **no formal exactly-once guarantee** for every possible
  mixed predecessor merge. It is not silently replaced with a corrected rule.
* `scores.utterance_network_raw` retains native model outputs, including the
  untrained completeness dimension for audit only. Displayed utterance scores
  omit completeness. ASR confidence is not a probability of correct pronunciation.
* Historical target normalization divides all source scores by 5: native
  word/utterance source scores span 0--10; phone source scores span 0--2.
  Thus displayed 0--5 word/utterance = network output x 2.5, and phone = network
  output x 12.5. Outputs are **not clipped**; out-of-range estimates remain
  visible. Stress has a trained head but is not a calibrated correctness probability.
* `status != 'ok'` returns `scores=null`, never a false zero grade. Statuses
  include empty/too-short/silent audio, no usable hypothesis, G2P/alignment/
  acoustic failures, nonfinite values and unsupported slot length. Failed
  prefixes remain in the output; previously successful state is preserved.
* `stages` and `wall_time_sec` measure actual local computation, not capture
  waiting or a universal real-time guarantee. Model loading is separate.

## Offline operation and checks

```bash
python scripts/setup_assets.py --offline
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python -m streampa.infer --audio /path/to/demo.wav --bundle-dir . --device cpu --output offline_scores.json
python -m pytest tests -q
```

Keep the entire prepared bundle, including `charsiu/` and `assets/nltk_data/`.
Do not delete those setup-generated directories when preparing an offline copy.
For hash mismatch, stop and re-download the specified revision rather than
replacing a resource with another model. Missing G2P/tagger resources require
`setup_assets.py`; inference does not intentionally download them. Proxy
configuration may be needed during setup in restricted networks. Use a fresh
output filename for each command. Convert sample rate/channels before opening
a session; the inference entry point does not resample a hidden full recording.

## Scope and limitations

Inference accepts audio only. Training used human annotations and teacher
supervision. Historical evaluation data were selected by preprocessing filters
and predominantly contain read speech. Deployment removes the historical
reference-phone-count ratio filter and uses received-audio/EOF scheduling;
it does not claim to reproduce historical test membership or PCC on arbitrary
WAVs. The frozen dictionary is a corpus-derived pronunciation vocabulary, not
an utterance-specific reference transcript. Original PCN gap semantics,
alignment, normalization, repetition filtering and legacy commitment are retained.

No claim of validated spontaneous/open-ended speech generalization, formal
exactly-once commitment, GRU-induced stability improvement, or end-to-end
real-time performance is made. Synthetic validation clips test computation
and interfaces, not pronunciation-assessment accuracy. See
`validation_summary.json` for actually completed and untested checks.

Observed interface-test limitation: several short prefixes produce repeated
Whisper hypotheses and are explicitly unscorable. A synthetic "Good morning"
clip was unscorable at all three prefixes; three separately selected licensed
read-speech clips all produced valid EOF scores, with 9 valid prefixes out of
15 attempted. These failures were not hidden or repaired by changing decoding.
CPU frontend computation was much slower than the audio duration in these tests.

The old v6 47-feature system remains historical only at
`faeea/StreamPA`, revision `legacy-v6-before-streampa`. It is not this checkpoint
and must not be mixed with these configurations or entry points.