Lightricks

/

LTX-2.5

like 1.89k

Follow

LTX.io 4.8k

Hi again!

Seeing and testing the changes in https://huggingface.co/Lightricks/LTX-2.5/commit/1b92891cedc4a823c35a3b23588d7a57a3a23c65 I'm starting to think the text encoder is indeed the semantic bottleneck in this model and is strongly responsible for core behavior.

As the encoder doesn't influence the actual diffusion generation speed, it is launched once and then cached, it shouldn't add a lot of overhead.

There are already experimental works on swapping text encoders with regression https://github.com/nicolab28/ComfyUI-ClipProj and they show the impact of the encoder on the world knowledge.

Gemma 4 26B A4B seems to be an ideal candidate: it's a MoE and activates only 4B parameters at once, yet has the world knowledge of 26B model, making it very efficient for training and inference.

Last note, please, do not train the text encoder. The encoder already has all the knowledge in the world and it perfectly understands the intent (if it's sufficiently large). It only adds unneeded gradients, instability and simply is an overhead.

Hope you can consider the text encoder improvements.

hi did you notice any difference in their latest change in text encoder .Didnt understand why they uploaded new text encoder and didnt gave us any details .

They uploaded an entire wrong checkpoint, which was not synchronized with the video model, resulting in artifacts and really weird prompt following

They uploaded an entire wrong checkpoint, which was not synchronized with the video model, resulting in artifacts and really weird prompt following

So shall i download the newest text encoder ?

@kabachuha ,
Can you please elaborate on your comment? What are the issues you are experiencing with the prompt adherence and which artefacts do you see?
Can you please provide the prompt and your setup?
Did you try to run the official LTX-2 inference pipeline:

Clone the repo:

git clone https://github.com/Lightricks/LTX-2.git
cd LTX-2

Install the dependencies. The natten extra is the fastest backend for the diffusion video VAE below, and is Linux + CUDA only -- on Windows and macOS it is skipped automatically and decoding falls back to a Triton or eager implementation, so the same command works everywhere (see neighborhood attention backends)

uv sync --extra natten

Download the models or use the Hugging Face CLI

hf auth login
hf download Lightricks/LTX-2.5 \
    diffusion_models/ltx-2.5-22b-distilled-transformer-bf16.safetensors \
    text_encoders/gemma4-12b-with-proj-ltx-2.5-bf16.safetensors \
    vae/ltx-2.5-video-vae-bf16.safetensors \
    vae/ltx-2.5-audio-vae-bf16.safetensors \
    latent_upscale_models/ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0.safetensors \
    --local-dir models/ltx-2.5

Generate

uv run python -m ltx_pipelines.distilled \
    --transformer-path       models/ltx-2.5/diffusion_models/ltx-2.5-22b-distilled-transformer-bf16.safetensors \
    --text-encoder-path      models/ltx-2.5/text_encoders/gemma4-12b-with-proj-ltx-2.5-bf16.safetensors \
    --video-vae-path         models/ltx-2.5/vae/ltx-2.5-video-vae-bf16.safetensors \
    --audio-vae-path         models/ltx-2.5/vae/ltx-2.5-audio-vae-bf16.safetensors \
    --spatial-upsampler-path models/ltx-2.5/latent_upscale_models/ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0.safetensors \
    --num-frames 121 \
    --seed 42 \
    --output-path output.mp4 \
    --prompt "A medium close-up shot features a Caucasian man with a beard, wearing a green and white baseball cap without any letters on the front, and a light blue shirt over a white t-shirt. He is positioned in the center of the frame, looking intently directly at the camera, his eyes focused on camera. His facial expression is one of deep concentration, with his brow slightly raised. As he looks straight at the camera, a quick sniff sound is heard, and then he speaks with a deep male voice and a satisfied tone, saying, 'I think it's so good.' The camera remains static throughout, maintaining a shallow depth of field, which keeps the man in sharp focus while the background is softly blurred, showing a beige wall behind him. After a brief pause, another short, audible sniff is heard. The man then continues to speak, his voice maintaining the same quality, as he states, 'So good. So good.' He elaborates further, emphasizing his point with a final statement, 'This got to be, it's got to be the best tool I've ever seen.'"

Hi! There are extremely huge problems with object permanence, both with the old and with the new text encoders.

This is the official ComfyUI workflow "LTX 2.5 Image2Video" from the template selection, no changes to the subgraph. I'm using ComfyUI because it supports dynamic VRAM offload and convrot quantization and I think the majority of the userbase uses ComfyUI too. If you think the official workflow is broken compared to LTX pipelines from Github, I think you need to look into it and fix it, because currently it is the "face" of LTX 2.5 to any newcomer.

Prompt:

sound-driven video, audio-reactive motion, continuous visual flow

A cinematic top-down overhead shot of the black Lada car from the image. Suddenly, dark brown fizzy cola starts pouring out of all the windows and doors like a waterfall. An anthropomorphic fluffy brown bear wearing a t-shirt quickly opens the door and climbs inside. The car then violently accelerates, performing a sharp, aggressive drift on the gravel road, leaving smoke and tire tracks, and finally crashes head-on into the green bushes on the side of the road. High dynamic range, realistic physics, splashes of liquid, flying debris from the bushes.

I also made a comparison for prompt following in a previous issue, where the prompt following suffers. But that workflow is custom (for audio-video synchromization) and I attach it as a Github gist https://gist.github.com/kabachuha/b7ed9ab78310052bfd8f04ad48328196.

https://huggingface.co/Lightricks/LTX-2.5/discussions/38

The video in the first message is with the old encoder, the second with the newer encoder.

The main point of this message (not the second one about Gemma) is that prompt following usually depends on the text encoder size, as the diffusion model can extrapolate concepts, even some of the missing ones.

For example, Bytedance's Waver https://arxiv.org/pdf/2508.15761 and Google's Imagen https://arxiv.org/abs/2205.11487 have shown that the scaling of the text encoder (the "brain" of the model) often gives more results in prompt following than the scaling of the sole diffusion model

Like, I think, 80% of the issues LTX is currently experiencing, if not searching for new data that is hard, could be fixed by

1. Scaling the text encoder greatly. From 12B to 31B or at least 26B to have world knowledge.
2. Ditching pure flow-matching objective and making heavy use of representational alignment (CREPA, easiest), Waver's semantic alignment of LLM embeddings (hard) or borrowing Flux 3's Self-Flow algorithm. Semantic alignment is shown to like extremely improve the prompt following, convergence speed and generalization.
3. Ditching the "mirror-audio-video-branch" entirely. Flux (including Flux 1!) and Minimax show that you absolutely need the text tokens to "evolve" as they pass through the transformer layers and not attend to the fixed representation. Concatenation into the single stream DiT provides that. This bias will probably hurt the compute efficiency, but in the world where you starve of data, the semantic improvement is the only thing you can do with the transformer.

Hi again!

Seeing and testing the changes in https://huggingface.co/Lightricks/LTX-2.5/commit/1b92891cedc4a823c35a3b23588d7a57a3a23c65 I'm starting to think the text encoder is indeed the semantic bottleneck in this model and is strongly responsible for core behavior.

As the encoder doesn't influence the actual diffusion generation speed, it is launched once and then cached, it shouldn't add a lot of overhead.

There are already experimental works on swapping text encoders with regression https://github.com/nicolab28/ComfyUI-ClipProj and they show the impact of the encoder on the world knowledge.

Gemma 4 26B A4B seems to be an ideal candidate: it's a MoE and activates only 4B parameters at once, yet has the world knowledge of 26B model, making it very efficient for training and inference.

Last note, please, do not train the text encoder. The encoder already has all the knowledge in the world and it perfectly understands the intent (if it's sufficiently large). It only adds unneeded gradients, instability and simply is an overhead.

Hope you can consider the text encoder improvements.

hi please make model with trnsformable encoer like if you want 12b or if you want 26b cuz we dont have infinite vram/ram

@AsThirtyThree

Good question. Actually, people have done this already! There is an addon that does alignment between the text embeddings of a bigger model of the same family (same tokenizer). If the next version of LTX will be trained with a bigger text encoder (e.g. 31B or 26B), they can employ some techniques to "downscale" the brain on inference.

For Minimax H3 it's https://github.com/nicolab28/ComfyUI-ClipProj. It works by performing ridge regression on the embeddings on Qwen 30B and Qwen 8B/Qwen 4B. But it doesn't work in the reverse direction, as the model must be trained with a bigger brain to understand the fine subtleties of the text encoder in the first place

@kabachuha
The prompt that you provided is indeed tricky and doesn't produce good results as it is.
But with very minor changes (even without prompt enhancement) on a default ComfyUI I2V template the model produced a decent and accurate result.

The car is peacefully parked at the side of the road when suddenly, dark brown fizzy cola starts pouring out of all the car's windows and doors. 

An anthropomorphic fluffy brown bear wearing a t-shirt quickly approaches the car from the left, opens the front door and climbs inside the driver seat.
The car violently accelerates, performing a sharp, aggressive drift on the gravel road, leaving smoke and tire tracks.
After a short drive the car looses control and crashes head-on into the green bushes on the left side of the road.

@art-alex Thank you! Although it's a bit cheating as the car turns on the side where there are no other vehicles. I think if it circled and turned to the other side of the road they would vanish

@kabachuha
The prompt that you provided is indeed tricky and doesn't produce good results as it is.
But with very minor changes (even without prompt enhancement) on a default ComfyUI I2V template the model produced a decent and accurate result.

The car is peacefully parked at the side of the road when suddenly, dark brown fizzy cola starts pouring out of all the car's windows and doors. 

An anthropomorphic fluffy brown bear wearing a t-shirt quickly approaches the car from the left, opens the front door and climbs inside the driver seat.
The car violently accelerates, performing a sharp, aggressive drift on the gravel road, leaving smoke and tire tracks.
After a short drive the car looses control and crashes head-on into the green bushes on the left side of the road.

prompt: The car is peacefully parked at the side of the road when suddenly, dark brown fizzy cola starts pouring out of all the car's windows and doors.

An anthropomorphic fluffy brown bear wearing a t-shirt quickly approaches the car from the left, opens the front door and climbs inside the driver seat.
The car violently accelerates, performing a sharp, aggressive drift on the gravel road, leaving smoke and tire tracks.
After a short drive the car looses control and crashes head-on into the green bushes on the left side of the road.
after some time the car catches fire and explodes like bomb,camera and ground shakes
(it took 26 minutes btw)

@kabachuha
The prompt that you provided is indeed tricky and doesn't produce good results as it is.
But with very minor changes (even without prompt enhancement) on a default ComfyUI I2V template the model produced a decent and accurate result.

The car is peacefully parked at the side of the road when suddenly, dark brown fizzy cola starts pouring out of all the car's windows and doors. 

An anthropomorphic fluffy brown bear wearing a t-shirt quickly approaches the car from the left, opens the front door and climbs inside the driver seat.
The car violently accelerates, performing a sharp, aggressive drift on the gravel road, leaving smoke and tire tracks.
After a short drive the car looses control and crashes head-on into the green bushes on the left side of the road.

that so good，make over compilcated prompt is not good，instead simple and accuracy is perfect，please do you work forward，I like LTX model than every other local model，please keep efford

make over compilcated prompt is not good

A modern model MUST be able to process overcomplicated prompts

Coupled with general, soundscape and shot-description, a model must be able to handle at least 5 paragraphs of text

This is where the text encoder comes into the equation

it doesn't have to be simple. Or better, it can be multiple simple things after another, separated by lines. It works very well to make those things happen in that order. It did even in ltx2.0.
2.3 and 2.5 It can do quite long videos (up to 40s in my experience) with a proper step by step prompt (created by a llm). The trick is to give it enough things to do, enough steps to fill that time. If it has not enough to do for the video length, it freaks out completely. It is MUCH worse at filling the gaps by itself than H3 or Wan2.2.

You just have to FORGET! the idea everything has to be in "one flowing paragraph". This isn't z-image and a video has things that happen after each other. If the text encoder has to extract what you want from a cumbersome "flowing paragraph" it very quickly fails (there size matters, but you can avoid that with a clearer prompt.
What you do is press enter after every section of the video. You can probably do "shot 1: shot 2:" and so on but its not even needed.

It needs bit more micromanagement of how people or things move than h3 or wan2.2 because LTX2.5's world understanding is still fairly bad (if better). That's likely not the te but the model itself. so
" a person walks ,
another person joins. Left person looks sad, right person looks happy,
person 1 does that, then bends down, then does that..".
meanwhile person 2 does this."

works in h3 and Wan, it does not work in ltx with these simple phrases BUT in general. you have to micromanage the motion and emotion details for every of these steps. BUT with that, step by step still works well. And twice as long as H3 (i found 22secs is the H3 limit) and faster.

One thing it still struggles with it keeping track of who says what. You really have to babysit that. That may also be influenced by a weak llm like g4 12b.
Also In that aspect H3 is actually worse because it just invents sound and gibberish talk, that make no sense, unless YOU give it something to say.

btw in general use, i find Gemma 4 12b, e2b and e4b catastrophically bad with vision stuff. Whatever fancy new way google tried there, just doesn't work. It misses stuff, it makes stuff up, it seems basically blind. Try captioning images with them, it's is a disaster.
26b is the first gemma 4 that does it very well again. And i actually use it a lot for image2prompt stuff, because it is good, fast and smaller then q3.6 35b but not as small as the weaker q3.5 9b. although even q3.5 9b wipes the floor with the smaller gemma4's.

it doesn't have to be simple. Or better, it can be multiple simple things after another, separated by lines. It works very well to make those things happen in that order. It did even in ltx2.0.
2.3 and 2.5 It can do quite long videos (up to 40s in my experience) with a proper step by step prompt (created by a llm). The trick is to give it enough things to do, enough steps to fill that time. If it has not enough to do for the video length, it freaks out completely. It is MUCH worse at filling the gaps by itself than H3 or Wan2.2.

You just have to FORGET! the idea everything has to be in "one flowing paragraph". This isn't z-image and a video has things that happen after each other. If the text encoder has to extract what you want from a cumbersome "flowing paragraph" it very quickly fails (there size matters, but you can avoid that with a clearer prompt.
What you do is press enter after every section of the video. You can probably do "shot 1: shot 2:" and so on but its not even needed.

It needs bit more micromanagement of how people or things move than h3 or wan2.2 because LTX2.5's world understanding is still fairly bad (if better). That's likely not the te but the model itself. so
" a person walks ,
another person joins. Left person looks sad, right person looks happy,
person 1 does that, then bends down, then does that..".
meanwhile person 2 does this."

works in h3 and Wan, it does not work in ltx with these simple phrases BUT in general. you have to micromanage the motion and emotion details for every of these steps. BUT with that, step by step still works well. And twice as long as H3 (i found 22secs is the H3 limit) and faster.

One thing it still struggles with it keeping track of who says what. You really have to babysit that. That may also be influenced by a weak llm like g4 12b.
Also In that aspect H3 is actually worse because it just invents sound and gibberish talk, that make no sense, unless YOU give it something to say.

btw in general use, i find Gemma 4 12b, e2b and e4b catastrophically bad with vision stuff. Whatever fancy new way google tried there, just doesn't work. It misses stuff, it makes stuff up, it seems basically blind. Try captioning images with them, it's is a disaster.
26b is the first gemma 4 that does it very well again. And i actually use it a lot for image2prompt stuff, because it is good, fast and smaller then q3.6 35b but not as small as the weaker q3.5 9b. although even q3.5 9b wipes the floor with the smaller gemma4's.

Wouldn't a simple "prompt-enhance," paired with a good LLM and an adequate system prompt to format the prompt exactly as LTX was formatted, solve most of these shortcomings?

yes, if you have the right system prompt. that does not create a convoluted mega paragraph. A prompt enhance is only as good as it's system prompt. If it's told to create a step by step timeline (no timestamps, just one after the other) with enhancing details on the specific movements a bit, LTX2.5 is fairly good. I never managed "up to a minute" but 1025 frames or about 41 seconds does work.
Scenes rarely just explode anymore like in 2.0. Unless you ha a short prompt for a long video.

That: " you have to micromanage the motion and emotion details for every of these steps. BUT with that, step by step still works well. " is where and why you need an llm for. Describing how a muscle or fabric moves, when someone walks, or how water splashes (so it's not rendered as solid glass), is just way too tedious by hand.
That's where it is MILES behind h3 and even ye olde wan2.2 who just know the context for what to do (mostly).

i remember the official guide for ltx2.0: Don't prompt "abstract" concepts like "sad", prompt every face muscle movement instead... ltx2.5 is a LOT better but still far from h3 or even wan2.2 in world understanding.
but all the way back at ltx2.0 i found this official guide, separate from the non-long video guide for ltx2: : https://ltx.io/blog/prompting-long-shots-with-ltx-2-how-to-build-20-second-cinematic-moments
and it worked, even in ltx2.0 if you were very lucky with matching things to do vs frames to fill.

But if you enhance prompt with a ltx gpt , or the ltx official prompt enhancer, likely made for a one shot, 5-10 second video (which is a waste!), you'd probably not get a working 10, 20 or 40 second video.
The model has simply no idea what to do in all the frames... the prompt has to be different for that. and longer. . That's my point.

But for the og post: gemma4 12b seems to manage a long prompt with multiple steps, just fine as text encoder.
But it may be out of it's depth as the actual prompt enhancing model, since the system prompt is fairly long and complex and g4 12b is rubbish at vision ( prompt working off a start image in i2v) .

btw the opposite happens with Minimax H3. The clipproj project successfully replaces the ludicrous 32b text encoder model with a qwen3 vl 8b or even 4b. Works a charm. At least 8b.