aoiandroid/StreamGemma-2.2b-TranslateBlue

StreamGemma-2.2B TranslateBlue

StreamGemma-2.2B TranslateBlue is a specialized, end-to-end streaming translation model built on the Gemma-2B (Gemma-2) architecture, containing 2.21B parameters. It is specifically customized and finetuned for the TranslateBlue translation ecosystem, providing low-latency, bidirectional, on-device translation capabilities.

The model supports three input modalities: raw PCM streaming audio, tokenized text, and vision patch embeddings. It is optimized for on-device deployment (iOS, macOS, Android) via dynamic range INT8 quantization and LiteRT (TFLite) compilation.

---

Key Features

• End-to-End Streaming Multi-Modality: Supports raw 16kHz PCM audio streaming, tokenized text input, and vision patch tokens (via SigLIP-Nano vision encoder).
• Quadruple Conditioning: Uses Parameter-conditioned Language Embedding (PLE) to condition input representation on Modality, Task, Language, and Translation Direction, allowing seamless language switching and target control.
• Dynamic Range INT8 Quantization: Optimised for mobile neural accelerators and GPUs, achieving sub-second latency on edge hardware.
• Multi-Signature Weight Sharing: Consolidates Audio, Text, Vision, and Draft execution paths into a single shared-weight .tflite model file, saving up to 66% of storage and memory bandwidth.
• Speculative Decoding with MTP: Features Multi-Token Prediction (MTP) draft models (x2 verification speedup for Pair-AST, x4 for Any2Any).

---

Variant Configuration Matrix

TranslateBlue defines 4 specialized target configurations depending on the platform's memory budget (~6GB unified memory target for iPhone 15 Pro, with ~4.5GB ML budget):

Variant Name	Target Use Case	TTS Head	Memory Footprint (INT8)	Target Platform	Acceleration Delegate
pair-ast	Seamless Pair / Bidirectional Interpreter (Tier S)	❌	1.2 GB	iOS, macOS, Android	CoreML / GPU
pair-ast-tts	Speech-to-Speech Translation (Tier S)		1.5 GB	iOS, macOS, Android	CoreML / GPU
any2any	Global Translation / Auto-bridging	❌	2.2 GB	iOS, macOS, Android	CoreML / GPU
any2any-tts	Speech-to-Speech Global Translation		2.5 GB	iOS, macOS, Android	CoreML / GPU

---

Language Tier Coverage

Tier S (Highest Quality, Lowest Latency)

Optimized for zero-lag bidirectional speech translation:

• English (eng)
• Japanese (jpn)
• Mandarin Chinese (cmn)

Tier A (Strong Multilingual Coverage)

Extended high-fidelity languages:

• Korean (kor)
• Spanish (spa)
• German (deu)
• French (fra)
• Portuguese (por)

Global Tier

Over 100+ languages supported for Auto-bridging / Any2Any text and audio-to-text rescue paths.

---

Architectural Details (Unreduced Backbone)

• Parameter Count: 2.2 Billion
• Layers: 18
• Model Dimension (d_model): 1024
• Attention Heads: 8 (Query), 4 (Key/Value) — Grouped Query Attention (GQA)
• Head Dimension: 128
• FFN Hidden Dimension: 4096 (SwiGLU)
• Vocabulary Size: 262,144 (Gemma 4 compatible SentencePiece tokenizer)
• Streaming Geometry: 8 frames per chunk (320ms chunk duration at 40ms/frame, 16kHz sample rate)
• KV Cache size: Up to 64 rolling chunks

---

Deployment & Platform Support

1. iOS / macOS

• Minimum OS: iOS 17.0+ / macOS 14.0+
• Recommended Delegate: CoreML (Apple Neural Engine / ANE)
• Optimization: Compiled to .tflite with CoreML delegate compatibility for ANE offloading, respecting the strict memory budgets of modern Apple Silicon devices.

2. Android

• Minimum OS: Android 14 (API 34)+
• Recommended Delegate: GPU Delegate (OpenCL/Vulkan)
• Optimization: Direct NN acceleration on Tensor G3 (Pixel 8 Pro) and Snapdragon 8 Gen 3 (Galaxy S24).

---

Python Usage Example

The following is a basic example using the PairASTSession from the streamgemma streaming execution engine to run bidirectional translation between English and Japanese:

import numpy as np
from streamgemma.model import StreamGemmaNano
from streamgemma.config import StreamGemmaConfig
from streamgemma.streaming.pair_ast_session import PairASTSession

# 1. Load configuration and model weights
config = StreamGemmaConfig.pair_ast()
model = StreamGemmaNano(config=config)
params = load_params("path/to/params.msgpack") # Or load from TFLite/LiteRT path

# 2. Initialize a bidirectional Pair AST session (English <-> Japanese)
session = PairASTSession(
    model=model,
    params=params,
    config=config,
    languages=["ja", "en"],
    direction_lock_threshold=0.85, # Prevent rapid direction flipping
    direction_hold_chunks=3,       # Min chunks to hold direction
)

# 3. Simulate streaming raw PCM audio chunks (16kHz, mono, 320ms chunks)
# Each chunk consists of 5120 audio samples (320ms * 16000Hz)
for audio_chunk in stream_microphone():
    result = session.feed_chunk(audio_chunk)
    
    # Check output
    if result["detected_language"]:
        print(f"[LID] Detected: {result['detected_language']}")
        print(f"[ASR] Speech Text: {result['asr_tokens']}")
        print(f"[AST] Translated: {result['translation_tokens']}")

---

References & Citations

If you use StreamGemma in your work, please cite the Gemma paper:

@article{gemma_2024,
    title={Gemma: Open Models Based on Gemini Research and Technology},
    author={Gemma Team, Google},
    journal={arXiv preprint arXiv:2408.00768},
    year={2024}
}

---

License

This model is released under the Apache 2.0 License.