Hugging Face's logo

Transformers documentation

GraniteSpeech5

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.15.1).
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

This model was contributed to Hugging Face Transformers on 2026-08-25.

GraniteSpeech5

Overview

Granite Speech 5.0 Turbo CTC is a lightweight (~470M parameters) conformer encoder for automatic speech recognition, trained with Connectionist Temporal Classification (CTC) on BPE targets. It is a fast, encoder-only member of the Granite Speech family: transcription requires a single forward pass followed by greedy CTC decoding, with no autoregressive decoder.

Architecturally, it extends the Granite Speech conformer CTC encoder with:

  1. Frame stacking + block-wise time subsampling: the feature extractor stacks pairs of log-mel(+delta) frames (2x), and the first two conformer blocks each subsample time by 2 through a stride-2 depthwise convolution (with a mean-pooled residual), for a total 8x time reduction at 10 ms mel hop.

  2. Block attention with Shaw’s relative positional embeddings: attention is computed over fixed-size blocks (the sequence is right-padded to a whole number of blocks, with padded frames masked out), using separate bias-free query/key/value projections.

  3. Self-conditioned CTC: the CTC posteriors of the middle layer are projected and fed back into the hidden states, and the CTC head is shared between this mid-layer self-conditioning and the final prediction.

This model was contributed by Eustache Le Bihan.

Usage

GraniteSpeech5ForCTC usage

Pipeline
AutoModel
from transformers import pipeline


pipe = pipeline("automatic-speech-recognition", model="ibm-granite/granite-speech-5.0-470m-turboctc")
out = pipe("https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/bcn_weather.mp3")
print(out)
# {'text': 'yesterday it was 35 degrees in barcelona but today the temperature will go down to -20 degrees'}

GraniteSpeech5CTCConfig

class transformers.GraniteSpeech5CTCConfig

< >

( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: typing.Union[str, ForwardRef('torch.dtype'), NoneType] = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: typing.Optional[typing.Literal['regression', 'single_label_classification', 'multi_label_classification']] = Nonevocab_size: int = 16384ctc_loss_reduction: str = 'mean'ctc_zero_infinity: bool = Trueencoder_config: dict | transformers.configuration_utils.PreTrainedConfig | None = Nonepad_token_id: int | None = 0tie_word_embeddings: bool = True )

Parameters

  • vocab_size (int, optional, defaults to 16384) — Vocabulary size of the model. Defines the number of different tokens that can be represented by the input_ids.
  • ctc_loss_reduction (str, optional, defaults to "mean") — Specifies the reduction to apply to the output of torch.nn.CTCLoss. Only relevant when training an instance of GraniteSpeech5ForCTC.
  • ctc_zero_infinity (bool, optional, defaults to True) — Whether to zero infinite losses and the associated gradients of torch.nn.CTCLoss. Infinite losses mainly occur when the inputs are too short to be aligned to the targets. Only relevant when training an instance of GraniteSpeech5ForCTC.
  • encoder_config (Union[dict, GraniteSpeech5EncoderConfig], optional) — The config object or dictionary of the encoder.
  • pad_token_id (int, optional, defaults to 0) — Token id used for padding in the vocabulary.
  • tie_word_embeddings (bool, optional, defaults to True) — Whether to tie weight embeddings according to model’s tied_weights_keys mapping.

This is the configuration class to store the configuration of a Granite Speech5Model. It is used to instantiate a Granite Speech5 model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the ibm-granite/granite-speech-5.0-470m-turboctc

Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.

Example:

>>> from transformers import GraniteSpeech5ForCTC, GraniteSpeech5CTCConfig
>>> # Initializing a GraniteSpeech5 configuration
>>> configuration = GraniteSpeech5CTCConfig()
>>> # Initializing a model from the configuration
>>> model = GraniteSpeech5ForCTC(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config

GraniteSpeech5EncoderConfig

class transformers.GraniteSpeech5EncoderConfig

< >

( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: typing.Union[str, ForwardRef('torch.dtype'), NoneType] = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: typing.Optional[typing.Literal['regression', 'single_label_classification', 'multi_label_classification']] = Nonevocab_size: int = 16384hidden_size: int = 1024intermediate_size: int = 4096num_hidden_layers: int = 16num_attention_heads: int = 8num_key_value_heads: int | None = Nonenum_mel_bins: int = 80head_dim: int | None = Nonehidden_act: str = 'silu'max_position_embeddings: int = 512context_size: int = 128conv_kernel_size: int = 7conv_expansion_factor: int = 2subsample_layers: list[int] | None = Noneattention_bias: bool = Trueattention_dropout: float | int = 0.0activation_dropout: float | int = 0.0initializer_range: float = 0.02 )

Parameters

  • vocab_size (int, optional, defaults to 16384) — Vocabulary size of the model. Defines the number of different tokens that can be represented by the input_ids.
  • hidden_size (int, optional, defaults to 1024) — Dimension of the hidden representations.
  • intermediate_size (int, optional, defaults to 4096) — Dimension of the MLP representations.
  • num_hidden_layers (int, optional, defaults to 16) — Number of hidden layers in the Transformer decoder.
  • num_attention_heads (int, optional, defaults to 8) — Number of attention heads for each attention layer in the Transformer decoder.
  • num_key_value_heads (int, optional) — This is the number of key_value heads that should be used to implement Grouped Query Attention. If num_key_value_heads=num_attention_heads, the model will use Multi Head Attention (MHA), if num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out this paper. If it is not specified, will default to num_attention_heads.
  • num_mel_bins (int, optional, defaults to 80) — Number of mel features used per input frame. Should correspond to the value used in the AutoFeatureExtractor class.
  • head_dim (int, optional) — The attention head dimension. If None, it will default to hidden_size // num_attention_heads
  • hidden_act (str, optional, defaults to silu) — The non-linear activation function (function or string) in the decoder. For example, "gelu", "relu", "silu", etc.
  • max_position_embeddings (int, optional, defaults to 512) — Maximum relative position index of Shaw’s relative positional encoding; the embedding table holds 2 * max_position_embeddings + 1 entries.
  • context_size (int, optional, defaults to 128) — Context size for block-wise conformer attention.
  • conv_kernel_size (int, optional, defaults to 7) — Kernel size of the depthwise convolution in the conformer convolution module.
  • conv_expansion_factor (int, optional, defaults to 2) — Expansion factor for the conformer convolution module.
  • subsample_layers (list[int], optional, defaults to [0, 1]) — Indices of the conformer blocks that subsample time by 2 (stride-2 depthwise convolution with a mean-pooled residual).
  • attention_bias (bool, optional, defaults to True) — Whether to use a bias in the query, key, value and output projection layers during self-attention.
  • attention_dropout (Union[float, int], optional, defaults to 0.0) — The dropout ratio for the attention probabilities.
  • activation_dropout (Union[float, int], optional, defaults to 0.0) — The dropout ratio for activations inside the fully connected layer.
  • initializer_range (float, optional, defaults to 0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices.

This is the configuration class to store the configuration of a Granite Speech5Model. It is used to instantiate a Granite Speech5 model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the ibm-granite/granite-speech-5.0-470m-turboctc

Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.

Example:

>>> from transformers import GraniteSpeech5EncoderConfig, GraniteSpeech5Encoder

>>> # Initializing a GraniteSpeech5EncoderConfig
>>> configuration = GraniteSpeech5EncoderConfig()

>>> # Initializing a GraniteSpeech5Encoder (with random weights)
>>> model = GraniteSpeech5Encoder(configuration)

>>> # Accessing the model configuration
>>> configuration = model.config

GraniteSpeech5FeatureExtractor

class transformers.GraniteSpeech5FeatureExtractor

< >

( num_mel_bins: int = 80sampling_rate: int = 16000n_fft: int = 512win_length: int = 400hop_length: int = 160delta_win_length: int = 3logmel_floor_db: float = 8.0padding_value: float = 0.0**kwargs )

Parameters

  • num_mel_bins (int, optional, defaults to 80) — Number of mel filter banks.
  • sampling_rate (int, optional, defaults to 16000) — The sampling rate at which the audio files should be digitalized expressed in hertz (Hz).
  • n_fft (int, optional, defaults to 512) — Size of the Fourier transform.
  • win_length (int, optional, defaults to 400) — Window length in samples.
  • hop_length (int, optional, defaults to 160) — Length of the overlapping windows for the STFT used to obtain the mel spectrogram, in samples.
  • delta_win_length (int, optional, defaults to 3) — Window length used to compute the delta features.
  • logmel_floor_db (float, optional, defaults to 8.0) — The log-mel features are floored at this many dB below the per-sample maximum.
  • padding_value (float, optional, defaults to 0.0) — Padding value used to pad the audio.

Constructs a Granite Speech 5.0 feature extractor.

This feature extractor inherits from SequenceFeatureExtractor which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

GraniteSpeech5Processor

class transformers.GraniteSpeech5Processor

< >

( feature_extractortokenizer )

Parameters

  • feature_extractor (feature_extractor_class) — The feature extractor is a required input.
  • tokenizer (tokenizer_class) — The tokenizer is a required input.

Constructs a GraniteSpeech5Processor which wraps a feature extractor and a tokenizer into a single processor.

GraniteSpeech5Processor offers all the functionalities of feature_extractor_class and tokenizer_class. See the ~feature_extractor_class and ~tokenizer_class for more information.

GraniteSpeech5Encoder

class transformers.GraniteSpeech5Encoder

< >

( config: GraniteSpeech5EncoderConfig )

Parameters

  • config (GraniteSpeech5EncoderConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The Granite Speech 5.0 conformer encoder, adapted from the Granite Speech CTC encoder with block-wise time subsampling and self-conditioned CTC from the middle layer.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

forward

< >

( input_features: Tensorattention_mask: typing.Optional[torch.Tensor] = Noneoutput_attention_mask: bool = True**kwargs: Unpack ) → GraniteSpeech5EncoderModelOutput or tuple(torch.FloatTensor)

Parameters

  • input_features (torch.Tensor of shape (batch_size, sequence_length, feature_dim)) — The tensors corresponding to the input audio features. Audio features can be obtained using feature_extractor_class. See feature_extractor_class.__call__ for details (processor_class uses feature_extractor_class for processing audios).
  • attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:

    • 1 for tokens that are not masked,
    • 0 for tokens that are masked.

    What are attention masks?

  • output_attention_mask (bool, optional, defaults to True) — Whether to return the output attention mask. Only effective when attention_mask is provided.

Returns

GraniteSpeech5EncoderModelOutput or tuple(torch.FloatTensor)

A GraniteSpeech5EncoderModelOutput or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (None) and inputs.

The GraniteSpeech5Encoder forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

  • last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.

  • pooler_output (torch.FloatTensor of shape (batch_size, hidden_size)) — Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.

  • hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

    Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

  • attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

    Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

  • attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices after sequence compression. Returned because the sequence length may differ from the input sequence length. Mask values selected in [0, 1]:

    • 1 for tokens that are not masked,
    • 0 for tokens that are masked.

Example:

>>> from transformers import AutoProcessor, GraniteSpeech5Encoder
>>> from datasets import load_dataset, Audio

>>> model_id = "ibm-granite/granite-speech-5.0-470m-turboctc"
>>> processor = AutoProcessor.from_pretrained(model_id)
>>> encoder = GraniteSpeech5Encoder.from_pretrained(model_id)

>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))

>>> inputs = processor(ds[0]["audio"]["array"])
>>> encoder_outputs = encoder(**inputs)

>>> print(encoder_outputs.last_hidden_state.shape)

GraniteSpeech5ForCTC

class transformers.GraniteSpeech5ForCTC

< >

( config: GraniteSpeech5CTCConfig )

Parameters

  • config (GraniteSpeech5CTCConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

Granite Speech 5.0 encoder with a Connectionist Temporal Classification (CTC) head.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

forward

< >

( input_features: Tensorattention_mask: typing.Optional[torch.Tensor] = Nonelabels: typing.Optional[torch.Tensor] = None**kwargs: Unpack ) → CausalLMOutput or tuple(torch.FloatTensor)

Parameters

  • input_features (torch.Tensor of shape (batch_size, sequence_length, feature_dim)) — The tensors corresponding to the input audio features. Audio features can be obtained using feature_extractor_class. See feature_extractor_class.__call__ for details (processor_class uses feature_extractor_class for processing audios).
  • attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:

    • 1 for tokens that are not masked,
    • 0 for tokens that are masked.

    What are attention masks?

  • labels (torch.Tensor of shape (batch_size, sequence_length), optional) — Labels for computing the masked language modeling loss. Indices should either be in [0, ..., config.vocab_size] or -100 (see input_ids docstring). Tokens with indices set to -100 are ignored (masked), the loss is only computed for the tokens with labels in [0, ..., config.vocab_size].

Returns

CausalLMOutput or tuple(torch.FloatTensor)

A CausalLMOutput or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (None) and inputs.

The GraniteSpeech5ForCTC forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

  • loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Language modeling loss (for next-token prediction).

  • logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).

  • hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

    Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

  • attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

    Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

Example:

>>> from transformers import AutoProcessor, GraniteSpeech5ForCTC
>>> from datasets import load_dataset, Audio

>>> model_id = "nvidia/granite_speech5-ctc-1.1b"
>>> processor = AutoProcessor.from_pretrained(model_id)
>>> model = GraniteSpeech5ForCTC.from_pretrained(model_id)

>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))

>>> inputs = processor(ds[0]["audio"]["array"], text=ds[0]["text"])
>>> outputs = model(**inputs)

>>> print(outputs.loss)

generate

< >

( input_features: Tensorattention_mask: typing.Optional[torch.Tensor] = Nonereturn_dict_in_generate: bool = Falsecompile_config: transformers.generation.configuration_utils.CompileConfig | None = None**kwargs: Unpack )

compile_config (CompileConfig, optional): If provided, torch.compile will be applied to the forward calls in the decoding loop.

Example:

>>> from transformers import AutoProcessor, GraniteSpeech5ForCTC
>>> from datasets import load_dataset, Audio

>>> model_id = "ibm-granite/granite-speech-5.0-470m-turboctc"
>>> processor = AutoProcessor.from_pretrained(model_id)
>>> model = GraniteSpeech5ForCTC.from_pretrained(model_id)

>>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
>>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))

>>> inputs = processor(ds[0]["audio"]["array"])
>>> predicted_ids = model.generate(**inputs)
>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)

>>> print(transcription)
Update on GitHub

←GraniteSpeech GraniteSpeechPlus→