README.md

metadata

license: mit
language:
  - en
pipeline_tag: fill-mask
library_name: transformers
tags:
  - RoPE
  - PreLN
  - MLM
  - base
  - SwiGLU

RoPERT base model (cased)

Pretrained model on English language using a masked language modeling (MLM) objective. This model is case-sensitive: it makes a difference between english and English.

RoPERT is a modernized BERT-style bidirectional encoder. It keeps the encoder-only, fill-in-the-blank pretraining recipe of the original BERT, but replaces several architectural components with designs from recent transformer research (rotary position embeddings, SwiGLU, pre-layer normalization), and extends the context window from 512 to 2048 tokens.

Model description

RoPERT is a transformer encoder pretrained on a large corpus of English text in a self-supervised fashion, with no human labelling. During pretraining, a fraction of the tokens in each input is masked and the model learns to predict the original tokens from both left and right context. This bidirectional objective makes the model well suited for producing representations of whole sentences and documents, which can then be fine-tuned for downstream tasks.

Compared to the original BERT-base architecture, RoPERT makes the following changes:

Component	BERT base	RoPERT base
Position encoding	Learned absolute embeddings	Rotary position embeddings (RoPE), θ = 10,000
Context length	512	2048
Layer normalization	Post-LN	Pre-LN + final LayerNorm
Feed-forward network	GELU, 3072 intermediate	SwiGLU, 2048 intermediate (parameter-neutral)
Linear biases	Yes	No (bias-free encoder)
Segment (token type) embeddings	Yes, with NSP objective	None (MLM only, no NSP)
Attention	Eager	SDPA / FlashAttention-compatible

Other specifications:

• Layers / hidden size / heads: 12 / 768 / 12
• Parameters: 108M total (~22M embeddings), input and output embeddings tied
• Tokenizer: WordPiece, google-bert/bert-base-cased vocabulary (28,996 tokens), cased
• Precision: trained in bfloat16, weights published in float32 Because RoPERT uses rotary embeddings, there is no position embedding table: all 2048 positions are handled by rotating the query/key vectors, and every position within the context window was seen during pretraining.

How to use

The architecture is implemented as custom code shipped with this repository, so trust_remote_code=True is required. PyTorch only.

You can use this model directly with a pipeline for masked language modeling:

from transformers import pipeline
 
pipe = pipeline("fill-mask", model="kd13/RoPERT-MLM-base", trust_remote_code=True)
pipe("I went to the [MASK] to buy some milk.")

[{'score': 0.137, 'token_str': 'market',      'sequence': 'I went to the market to buy some milk.'},
 {'score': 0.082, 'token_str': 'supermarket', 'sequence': 'I went to the supermarket to buy some milk.'},
 {'score': 0.070, 'token_str': 'shop',        'sequence': 'I went to the shop to buy some milk.'},
 ...]

And here is how to get the hidden states of a given text, e.g. for feature extraction or as the backbone for fine-tuning:

import torch
from transformers import AutoTokenizer, AutoModel
 
tokenizer = AutoTokenizer.from_pretrained("kd13/RoPERT-MLM-base", trust_remote_code=True)
model = AutoModel.from_pretrained("kd13/RoPERT-MLM-base", trust_remote_code=True)
 
inputs = tokenizer("Hello, world!", return_tensors="pt")
with torch.no_grad():
    outputs = model(input_ids=inputs["input_ids"], attention_mask=inputs["attention_mask"])
 
outputs.last_hidden_state  # [batch, seq_len, 768]

Note that the model has no segment embeddings, so token_type_ids are accepted but ignored. Sentence pairs can still be encoded in the standard [CLS] A [SEP] B [SEP] format.

Training data

RoPERT was pretrained on roughly 10B+ tokens of English text drawn from three sources:

• English Wikipedia (WikiText-style encyclopedic text)
• Web text (filtered general web documents)
• BookCorpus (4,085 long-form book documents) Documents were tokenized with the bert-base-cased WordPiece tokenizer and packed into full 2048-token sequences, so the model was trained on long contexts throughout pretraining rather than being extended afterwards.

Intended uses & fine-tuning

The raw model can be used as-is for masked token prediction, but it is primarily intended to be fine-tuned on downstream tasks that use the whole sentence or document to make decisions, such as:

• Text classification — sentiment analysis, topic classification, spam / toxicity detection, natural language inference
• Token classification — named entity recognition (NER), part-of-speech (POS) tagging, chunking
• Extractive question answering — SQuAD-style span prediction
• Sentence and document embeddings — semantic similarity, clustering, dense retrieval
• Reranking — scoring query–document pairs for search
• Long-document tasks — the native 2048-token window covers most articles, reviews, and reports without truncation or sliding windows The repository ships the encoder (AutoModel) and the MLM head (AutoModelForMaskedLM). For other tasks, attach a task head on top of the encoder outputs (e.g. a linear classifier over the [CLS] position or over per-token hidden states) and fine-tune end-to-end.

This model is a bidirectional encoder and is not intended for text generation; for generative tasks, use a decoder model such as GPT or Llama.

Limitations and bias

• English only, and case-sensitive by design.
• Maximum input length is 2048 tokens; longer inputs must be truncated or chunked.
• Loading requires trust_remote_code=True, since the architecture is not part of the core transformers library.
• The training data includes unfiltered text from the web and books, so the model may reproduce societal biases present in those sources. Predictions and fine-tuned models built on top of it should be evaluated for bias before deployment in sensitive applications.