Transformers runs Mistral Large 4 and trains LoRA on FP8 checkpoints

Transformers now runs Mistral Large 4, the largest model Mistral has made. It's a mixture of experts with about a trillion parameters, 49 billion of them active per token, and it takes text and images as input.

Slide text

Transformers

01 02 03 04 New models

huggingface/transformers · release v5.22.0 · PR #50121

Transformers now runs Mistral Large 4

About 1T parameters 49B active per token Text and image in

Mistral Large 4 reads up to a million tokens of context, works in more than 160 languages, and takes a reasoning effort on each request, from a quick reply to a long agent run.

The checkpoint ships in FP8 and needs about a terabyte of GPU memory, spread across several GPUs. Its sparse attention runs as a mask over dense attention: the outputs match, but long sequences don't get faster.

The weights come under the Mistral Ecosystem Open-Weight License, a modified MIT license for commercial and non-commercial use. Running it as a hosted model service needs a separate license from Mistral.

NVIDIA's Nemotron 3.5 Transcribe also lands. It transcribes many languages, tags who is speaking, translates speech, answers questions about audio, and times every word. Word timestamps need torchaudio and SciPy installed.

Part two: training on FP8 checkpoints.

You can now train LoRA adapters with PEFT on top of an FP8 checkpoint. Before, the FP8 layers silently dropped gradients to earlier layers, and grouped experts crashed in backward. FP4 weights still can't train.

Training those experts got cheaper too. The backward pass now dequantizes experts in small chunks, so a DeepSeek V3 sized expert layer peaks under 3 gigabytes instead of nearly 29.

Part three: what stopped breaking.

The last release changed how configs set rope scaling, and that broke remote-code models in vLLM. Phi 3.5 vision, InternVL2, EAGLE3 speculators and others load again in the patch release.

Passing a token to from_pretrained now opens private and gated repos. The config step dropped it, so loading failed with a 401 error unless the HF_TOKEN environment variable was also set, which hit clean CI machines.

The default data collator turned NumPy integer labels into floats, which can break a cross-entropy loss. Labels typed as int32 or int64 now stay integers.

np.int64 label → float32 in default_data_collator

Pin a remote-code revision in a pipeline, and before, only the config honored it. Now the model, tokenizer and processor load that revision too, and saved tokenizers stop writing the pin into their config file.

Janus stops splitting a string prompt into one character per row. ESMFold2 Fast stops running a randomly initialized MSA encoder, so its structures no longer change between loads. And TimesFM runs in half precision without overflowing.

Part four: one breaking change.

Expert-parallel code moved into its own file, with no change in behavior. Custom expert-parallel styles are affected. Old imports keep working, with a warning, until version 5.23.

If you fine-tune mixture-of-experts models, I'd take this release today for FP8 adapter training alone. Everything else is in the Transformers release notes on GitHub.