Confucius4-R2T2: A Low Latency and High Accuracy Real-Time Speech Recognition Model

Real Real-Time Transcription

Confucius4-R2T2 is a low-latency and high-accuracy true streaming Automatic Speech Recognition (ASR) model that features fine-grained and configurable decoding chunks from 80 ms to 2 s. The model operates in append-only output mode: committing transcript text permanently without revising previous words, which is critical for applications where text must be processed or acted upon instantly. This results in a smoother user experience, avoiding disruptive text revisions and visual flickering in real-time applications, such as Real-Time Live Captioning & Subtitling, Downstream NLP Pipelines & LLM Agents, Simultaneous Speech Translation, etc.

R2T2, short for Real Real-Time Transcription, is built upon the Qwen3-ASR model. And it is trained with a unique set of data construction techniques including stable-prefix data, forced time-alignment data, and token-level audio segmentation. Combined with a Longest Stable Prefix (LSP) learning paradigm (tech report will be released soon), R2T2 can dynamically determine when a stable prefix can be safely emitted and when additional audio context is needed. By exposing only stable prefixes, the model provides high-quality context that conditions subsequent predictions while guaranteeing that previously emitted text remains unchanged. Despite its streaming design, R2T2 maintains strong accuracy in offline recognition.

Low-latency and high accuracy streaming recognition — The model achieves accuracy close to that of offline recognition, with only 200 to 600 milliseconds average latency.
Stable streaming output — Emitted text is committed as it arrives and remains unchanged.
Configurable low-latency chunking - Supports decoding chunks from 80 ms to 2 s for different latency/accuracy trade-offs.
No loss in offline accuracy — Adding streaming support does not degrade offline recognition accuracy.
vLLM backend — Provides high-throughput inference. A Hugging Face transformers backend is also available.
Context and hotword prompts — Natively supported.
Multilingual support — Optimized for Chinese and English, while also supporting a broad range of additional languages.

Experimental results show that R2T2 achieves state-of-the-art (SOTA) performance in both latency and recognition quality among a range of open-source models, while remaining competitive with leading closed-source systems. The GitHub repository provides inference code, a minimal usage example, and a vLLM-based backend supporting both offline and real-time streaming inference.

Overview
Demo
- Side-by-side comparison with GPT-Live-Transcribe
- Additional resources
Evaluation
- Streaming performance
- Accuracy
  - English
  - Chinese
Installation
Docker (recommended)
Quick Start
- Configuration
Python API
- Offline transcription (vLLM backend)
- Streaming transcription (vLLM backend)
WebSocket Server
Supported Languages
Community & Contact
Acknowledgements
Citation
License

Overview

Figure 1. Overall framework of R2T2.

Demo

Side-by-side comparison with GPT-Live-Transcribe

Watch the comparison video

Figure 2. GPT-Live-Transcribe and R2T2 processing the same audio, shown together in real time — a side-by-side comparison.

Additional resources

More demonstrations, comparisons, and supporting resources will be added here.

Evaluation

If you are an author or maintainer of a model included in these comparisons and have questions or concerns about the results, please feel free to contact us through the GitHub issue tracker. We are happy to share evaluation details and work with you to verify or correct them.

Streaming performance

The streaming API supports decoding chunks from 80 ms to 2 s; the figures below show representative WER/latency trade-offs at 160 ms.

English WER and retrospective chunk-wise latency comparison across ASR models and configurations

Figure 3. English WER and retrospective chunk-wise latency across model and configuration settings.

Chinese CER and retrospective chunk-wise latency comparison across ASR models and configurations

Figure 4. Chinese CER and retrospective chunk-wise latency across model and configuration settings.

Figure 5. Accuracy-latency Pareto frontier. Lower-left is better; the frontier uses retrospective chunk-wise mean fuzzy latency.

Accuracy

English results use WER (%), and Chinese results use CER (%); lower is better.

※ Pseudo-streaming model: its partial transcript may revise previously emitted text; unmarked models use true streaming, append-only output.

English

Dataset	Qwen		R2T2 (Ours) _160ms	Open-source				Proprietary
Dataset	Qwen3-ASR※ _2s/u2/t5	Qwen3-ASR base _160ms	R2T2 (Ours) _160ms	X-ASR _160ms	WhisperRT※ _200ms	Nemotron _160ms	Voxtral _160ms	AssemblyAI※ _{min_latency}	Commercial A※	Commercial B※
AMI	9.25	24.79	11.37	14.41	24.19	18.11	15.94	12.00	13.27	8.44
Giga-clean	8.61	24.37	9.60	10.26	13.81	12.67	11.13	9.21	8.84	9.46
LS-clean	1.67	22.30	2.13	3.86	4.70	3.71	2.49	1.89	1.73	1.25
LS-other	3.54	25.74	4.88	9.64	9.86	8.27	7.15	3.37	3.57	2.48
SPGI	2.90	22.25	3.00	5.14	8.66	3.93	3.06	2.14	3.06	1.74
VoxPopuli	3.02	20.71	3.07	5.68	8.28	5.69	6.30	4.75	3.17	3.14
Earnings22	6.68	29.72	9.36	15.95	35.08	17.22	11.66	7.47	10.32	8.96
TED-LIUM	2.33	19.18	3.34	3.75	6.67	5.11	4.60	3.23	3.08	3.30
EN-RealSI	6.54	13.75	8.40	8.97	35.36	10.69	14.75	9.73	8.73	17.05

Chinese

Dataset	Qwen		R2T2 (Ours) _160ms	Open-source				Proprietary
Dataset	Qwen3-ASR※ _2s/u2/t5	Qwen3-ASR base _160ms	R2T2 (Ours) _160ms	X-ASR _160ms	WhisperRT※ _200ms	Nemotron _160ms	Voxtral _160ms	AssemblyAI※ _{min_latency}	Commercial A※	Commercial B※
Wenet-net	4.94	19.79	5.87	8.81	U	24.70	23.53	12.91	5.13	4.79
Wenet-meeting	5.97	20.38	7.27	11.33	U	20.18	60.54	11.84	7.07	3.75
SPEECHIO-06	6.10	24.50	7.30	7.86	U	22.52	32.16	15.08	5.67	5.34
SPEECHIO-07	6.19	21.16	8.20	11.22	U	24.28	22.97	10.84	6.45	6.46
CN-RealSI	3.34	39.72	3.48	4.92	U	11.52	8.74	5.15	3.99	3.64

Installation

We recommend using a fresh, isolated environment. For local development and source installation, use the Conda or uv environment below. Docker is recommended for quickly running the project with a preconfigured CUDA and runtime environment — see Docker.

Clone the repository

git clone https://github.com/netease-youdao/Confucius4-R2T2.git
cd Confucius4-R2T2

Option 1: Conda

conda create -n confucius4-r2t2 python=3.12 -y
conda activate confucius4-r2t2

# Install the package with the vLLM backend
pip install -e .

Option 2: uv

uv venv --python 3.12
source .venv/bin/activate

# Install the package with the vLLM backend
uv pip install -e .

Python 3.10+ is supported. Python 3.12 is the version we test against.

vLLM has strict CUDA / PyTorch compatibility requirements. If the install fails to resolve, check the version matrix on the vLLM website and pin a combination that matches your CUDA runtime.

Docker (recommended)

R2T2 runs out of the box on the official Qwen3-ASR Docker image, which already ships every runtime library we need.

Pre-built image: qwenllm/qwen3-asr.

Before you begin, install the NVIDIA Container Toolkit to enable GPU access from Docker. If Docker Hub access is slow or unreliable in your region, you may need to configure a registry mirror.

1. Start a container

LOCAL_WORKDIR=/path/to/your/workspace   # host path that will be mounted into the container
HOST_PORT=8000
CONTAINER_PORT=80

docker run --gpus all --name confucius4-r2t2 \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -p $HOST_PORT:$CONTAINER_PORT \
    --mount type=bind,source=$LOCAL_WORKDIR,target=/data/shared/confucius4-r2t2 \
    --shm-size=4gb \
    -it qwenllm/qwen3-asr:latest

Your local workspace ($LOCAL_WORKDIR) — including a checkout of this repository and the R2T2 checkpoint — will be mounted inside the container at /data/shared/confucius4-r2t2. Host port 8000 is mapped to container port 80; services running inside the container must bind to 0.0.0.0 (not 127.0.0.1) for port forwarding to work.

2. Run the example inside the container

Once inside the container's shell:

cd /data/shared/confucius4-r2t2/Confucius4-R2T2
MODEL_PATH=/data/shared/confucius4-r2t2/Confucius4-R2T2 \
    ./run_example.sh /path/to/audio.wav

3. Manage the container

# re-enter after exiting
docker start confucius4-r2t2
docker exec -it confucius4-r2t2 bash

# remove completely
docker rm -f confucius4-r2t2

Quick Start

Grab any audio file (mono or stereo, any sample rate — it is resampled to 16 kHz internally) and run:

./run_example.sh /path/to/audio.wav \
    --model_path /path/to/Confucius4-R2T2 \
    --infer_mode stream_vllm \
    --language Chinese \
    --chunk_size_ms 160

Logs are written to run_example.log by default. Run ./run_example.sh --help to see the full flag list.

Configuration

run_example.sh reads the following environment variables (all optional):

Variable	Default	Description
`MODEL_PATH`	(required)	Path or HF repo id of the R2T2 checkpoint
`AUDIO`	first CLI argument	Path to the input audio file
`INFER_MODE`	`stream_vllm`	`stream_vllm` or `onetime_vllm`
`LANGUAGE`	`Chinese`	Language hint (e.g. `Chinese`, `English`, …)
`CHUNK_SIZE_MS`	`160`	Streaming chunk size (80 ms–2 s supported)
`UNFIXED_TOKEN_NUM`	`1`	Number of unfixed trailing tokens (rollback window)
`CONTEXT`	`""`	Context / hotword hint prepended to the prompt
`CUDA_VISIBLE_DEVICES`	`0`	GPU id(s) to expose
`LOG_FILE`	`run_example.log`	Where to write logs

You can also call example.py directly and pass any of these as flags (--audio, --model_path, --infer_mode, --language, --chunk_size_ms, --lookahead_ms, --unfixed_token_num, --context).

Python API

Audio inputs can be passed as a local path, a URL, base64 data, or a (np.ndarray, sr) tuple. Batched inference is supported. Remember to wrap vLLM code under if __name__ == '__main__': to avoid the spawn error described in vLLM Troubleshooting.

Offline transcription (vLLM backend)

import librosa
from qwen_asr import Qwen3ASRModel

if __name__ == "__main__":
    asr = Qwen3ASRModel.LLM(
        model="/path/to/Confucius4-R2T2",
        gpu_memory_utilization=0.5,
        max_inference_batch_size=32,
        max_new_tokens=4096,
    )

    wav, sr = librosa.load("path/to/audio.wav", sr=16000, mono=True)

    results = asr.transcribe(
        audio=[(wav, 16000)],
        language=["Chinese"],       # or [None]
        return_time_stamps=False,
    )
    print(results[0].language, results[0].text)

Streaming transcription (vLLM backend)

import librosa
from qwen_asr import Qwen3ASRModel

if __name__ == "__main__":
    asr = Qwen3ASRModel.LLM(
        model="/path/to/Confucius4-R2T2",
        gpu_memory_utilization=0.4,
        max_new_tokens=4,           # keep small for low-latency streaming
    )

    wav, sr = librosa.load("path/to/audio.wav", sr=16000, mono=True)

    state = asr.init_streaming_state(
        context="",                 # optional hotword / topic hint
        language="Chinese",         # or None
        unfixed_chunk_num=0,
        unfixed_token_num=1,
        chunk_size_sec=0.16,
    )

    step = int(0.16 * 16000)
    for pos in range(0, len(wav), step):
        seg = wav[pos : pos + step]
        _, text = asr.streaming_transcribe(seg, state, max_new_tokens=2)
        print("text:", text)

    asr.finish_streaming_transcribe(state)
    print("final:", state.text)

For a complete streaming example with adaptive max_new_tokens and initial-chunk lookahead handling, see example.py.

WebSocket Server

For real-time, multi-client streaming ASR, the GitHub repository ships a ready-to-run WebSocket server (ws_server.py), a launcher script (run_start_server.sh), and a reference Python client (ws_client.py).

Start and stop the server

# Start with a VAD model
./run_start_server.sh start \
    --model_path /path/to/Confucius4-R2T2 \
    --vad_model_path /path/to/Stream-VAD \
    --port 8272 \
    --gpu 0

# Stop
./run_start_server.sh kill

# Restart in one step
./run_start_server.sh restart \
    --model_path /path/to/Confucius4-R2T2 \
    --vad_model_path /path/to/Stream-VAD \
    --port 8272 \
    --gpu 0

Flag	Env var	Default	Description
`-m`, `--model_path`	`ASR_MODEL_PATH`	(required)	Path or HF repo id of the R2T2 checkpoint
`-v`,`--vad_model_path`	`VAD_MODEL_PATH`	`checkpoints/vad/Stream-VAD`	Path to the FireRedVAD Stream-VAD model
`-p`, `--port`	`PORT`	`8272`	Port the WebSocket server binds to
`-g`, `--gpu`	`CUDA_VISIBLE_DEVICES`	`0`	GPU id(s) exposed to the server process
`-h`, `--host`	`HOST_TAG`	`localhost`	Host tag used only in the log file name

The launcher resolves its own directory, so it can be invoked from anywhere. Logs are written to nohup_service_ws_<host_tag>_<port>.log in the current directory. The FireRedVAD model is available from Hugging Face. We recommend downloading the model files into this repository's checkpoints directory:

# The FireRedVAD repo ships several detectors, but only the streaming one is
# needed. Both commands below keep the `Stream-VAD/` folder name, so the files
# land in checkpoints/vad/Stream-VAD with no extra nesting.

# Option A — hf CLI (pip install -U "huggingface_hub[cli]")
hf download FireRedTeam/FireRedVAD \
    --include "Stream-VAD/*" \
    --local-dir checkpoints/vad

# Option B — git clone
git clone https://huggingface.co/FireRedTeam/FireRedVAD
cp -r FireRedVAD/Stream-VAD checkpoints/vad/

Either command leaves the model at checkpoints/vad/Stream-VAD, which is exactly what --vad_model_path defaults to — so you can drop the flag entirely.

WebSocket endpoint

Path	Behavior
`/asr_stream_api_v1`	Streaming ASR. Each message's `text` is the new (incremental) chunk.

Message format

Client → Server:

Send raw 16 kHz mono PCM as int16 binary frames (the reference client uses ≈160 ms per frame, i.e. 2560 samples × 2 bytes).
Send the string "YOUDAO_ONETIME_ASR_STREAM_EOS" to signal end-of-audio; the server will emit any final text and close.

Server → Client: JSON messages of the form

{
  "status": "success",
  "requestId": "<uuid>",
  "msg": {
    "text": "hello",
    "reset": false,
    "asr_cost_ms": 35.4,
    "total_cost_ms": 42.0
  }
}

text is the newly recognized (incremental) segment since the previous message. Concatenate them client-side to get the full transcript.

Example client

ws_client.py is a minimal example that streams a WAV file to the server and prints the responses.

# Uses the default URI (ws://localhost:8272/asr_stream_api_v1) and built-in sample audio
python ws_client.py

# Point at a custom endpoint and audio file
python ws_client.py \
    --uri wss://your.host/asr_stream_api_v1 \
    --audio resources/test.wav \
    --save service_ws_test \
    --audio-id test.wav

Command-line options:

Flag	Env var	Default	Description
`--uri` / `-u`	`ASR_WS_URI`	`ws://localhost:8272/asr_stream_api_v1`	WebSocket endpoint to connect to.
`--audio` / `-a`	—	built-in sample path	Input audio file (WAV, 16 kHz mono recommended).
`--save` / `-s`	—	`service_ws_test`	File to append the final transcript to.
`--audio-id`	—	basename of `--audio`	Identifier written next to the result in `--save`.

Supported Languages

R2T2 is optimized for streaming recognition in Chinese and English. Beyond these primary languages, it retains useful cross-lingual streaming capability on languages such as French, German, Italian, Japanese, Korean, Portuguese, Russian, Spanish, Arabic, etc.

Community & Contact

Join our community to ask questions, share ideas, and connect with other users and developers.

WeChat Group

Scan the QR code below to join our WeChat group:

Discord Server

Join our Discord server

Business contact

For high-concurrency, production-grade, domestically deployable, or private deployment solutions, as well as business inquiries and partnership opportunities, please feel free to contact us through the channels below.

Phone: +86 010-82558901
Email: AIcloud_Business@corp.youdao.com

GitHub Issues

We also welcome discussions in this repository’s Issues section. Feel free to ask questions, report bugs, or suggest improvements!

Acknowledgements

We sincerely thank the Alibaba Qwen team for open-sourcing the Qwen3-ASR modeling code, which provides the architectural foundation for R2T2.

Citation

If you use this repository or the R2T2 checkpoint in your research, please cite Confucius4-R2T2 (this project):

@misc{Confucius4-R2T2,
  title        = {Confucius4-R2T2: A Low Latency and High Accuracy Real-Time Speech Recognition Model},
  author       = {NetEase Youdao},
  year         = {2026},
  howpublished = {https://github.com/netease-youdao/Confucius4-R2T2}
}

License

R2T2 uses dual licensing to distinguish the source code from the model weights:

Code in the accompanying GitHub repository is released under the Apache License 2.0 and is free to use, modify, and redistribute (including commercially) under the terms of that license.
Model weights are released under the NetEase Model Use License Agreement.

Downloads last month: 73

Safetensors

Model size

2B params

Tensor type

BF16

Model tree for netease-youdao/Confucius4-R2T2

Base model

Qwen/Qwen3-ASR-1.7B

Finetuned

(103)

this model

Quantizations

1 model