Instructions to use apple/OpenELM-1_1B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use apple/OpenELM-1_1B-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="apple/OpenELM-1_1B-Instruct", trust_remote_code=True)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("apple/OpenELM-1_1B-Instruct", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use apple/OpenELM-1_1B-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "apple/OpenELM-1_1B-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "apple/OpenELM-1_1B-Instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/apple/OpenELM-1_1B-Instruct

SGLang

How to use apple/OpenELM-1_1B-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "apple/OpenELM-1_1B-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "apple/OpenELM-1_1B-Instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "apple/OpenELM-1_1B-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "apple/OpenELM-1_1B-Instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use apple/OpenELM-1_1B-Instruct with Docker Model Runner:
```
docker model run hf.co/apple/OpenELM-1_1B-Instruct
```

OpenELM remote code is incompatible with transformers 5.x: config load fails with TypeError: OpenELMConfig.__post_init__() got an unexpected keyword argument 'use_cache'

by jbernloehr - opened 22 days ago

Discussion

jbernloehr

22 days ago

Summary

The trust_remote_code implementation shipped with apple/OpenELM-1_1B-Instruct cannot be loaded under transformers 5.x. AutoConfig.from_pretrained raises before any model is constructed, so the model is unusable on current transformers.

Reproduction

pip install "transformers==5.15.0"
python -c "
from transformers import AutoConfig
AutoConfig.from_pretrained('apple/OpenELM-1_1B-Instruct', trust_remote_code=True)
"

Expected behavior

The configuration loads and the model can be instantiated, as it does on transformers 4.57.1.

Actual behavior

TypeError: OpenELMConfig.__post_init__() got an unexpected keyword argument 'use_cache'

Root cause

transformers 5.x re-based PreTrainedConfig on a huggingface_hub validated dataclass. The generated __init__ (wrap_init_to_accept_kwargs, huggingface_hub/dataclasses.py:190) forwards every undeclared keyword argument into self.__post_init__(**additional_kwargs).

configuration_openelm.py defines __post_init__(self) as its own private zero-argument helper that derives the per-layer num_query_heads / num_kv_heads from qkv_multipliers. Because the shipped config.json contains "use_cache": true, which was never a declared base field, transformers 5.x forwards it into that zero-argument method and the load dies. The name is now reserved by the base class, so the private helper collides with the framework hook.

Layered follow-on incompatibilities

Working around the first error does not make the model usable — each fix exposes the next 4.x-era assumption. Observed empirically, in order:

TypeError: OpenELMConfig.__post_init__() got an unexpected keyword argument 'use_cache'
AttributeError: 'OpenELMConfig' object has no attribute '_output_attentions' (the private base attributes are never set once the base __post_init__ is shadowed)
AttributeError: 'OpenELMConfig' object has no attribute 'use_cache'

Still present in the shipped code and not yet reached at runtime:

config.cache_implementation is read but is never set by transformers 5.x.
modeling_openelm.py calls three cache APIs removed in transformers 5.x: from_legacy_cache (line 665), to_legacy_cache (line 731), and seen_tokens (line 923).

Environment

transformers 5.15.0 (fails); 4.57.1 (works)
huggingface_hub 1.27.0
Model revision: effd796da2a77361d7360e45e54c7cc14bc5df2a
Hardware/GPU count: not relevant — the failure occurs during configuration loading, on CPU, before any device is used.

Suggested fix

Rename the private __post_init__ helper (for example to _derive_head_counts) and call it explicitly from __init__, so it no longer shadows the framework hook. Declaring use_cache and cache_implementation as real config fields and migrating off the three removed cache APIs would restore full 5.x compatibility.

This issue was drafted with assistance from the opus AI model.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment