Transformers v5.20.0 runs Meta's Muse Spark Open, and here is the hardware it needs

Transformers now runs Muse Spark Open, the full-size open-weight model in Meta's Muse series. It takes text, images, audio and video, and answers in text, with up to a million tokens of context.

Slide text

Transformers

01 02 03 04 Muse Spark Open

huggingface/transformers · GitHub release notes · huggingface.co/blog · Welcome Muse Spark Open

Text, images, audio, video in Text out Up to 1M tokens of context

Meta trained it for agentic coding, which means editing across many files, debugging, and calling tools. It can also watch a screen recording or hear a voice note while it works on your code.

About 42 billion of its 816 billion parameters work on each token, but every expert stays in memory. In Transformers, the 16-bit checkpoint spans 32 H200 GPUs. The four-bit checkpoint fits on one B200 node.

If you train models, you can fine-tune the full checkpoint with TRL. The TRL recipe trains LoRA adapters on the attention layers and keeps the experts frozen, spread across 64 H100 GPUs.

More new models.

InclusionAI's Ling 3.0 Flash is now native in Transformers. It's a hybrid mixture-of-experts model that loads without the trust_remote_code flag, so loading it doesn't run code from the model's repository.

UnlimitedOcr is a 3-billion-parameter OCR model from Baidu. Its attention keeps the KV cache the same size the whole time it decodes, so it can transcribe dozens of pages in a single forward pass.

MOSS-Transcribe-Diarize handles long recordings with several speakers. One pass gives you the transcript, which speaker said each line, and timestamps, in more than 50 languages, from a model under a billion parameters.

Kolibri 1 is Aleph Alpha's reasoning model for German and English. UltraBERT encoders handle classification and retrieval on inputs up to 32 thousand tokens. And MiniCPM-V 4.7 places image tiles and video frames on a 2D canvas, so they keep their layout and order.

Exporting and scaling.

Transformers exporters can now quantize a model during export to Dynamo, ONNX, ExecuTorch and OpenVINO. That covers static and dynamic quantization, plus eight-bit or four-bit weight-only compression, with calibration you can configure.

Model and optimizer checkpoints made with FSDP and tensor parallelism can now be saved under one parallel setup and loaded under a different one. Fixes also landed for expert-parallel dispatch, tensor-parallel plans and static-cache generation.

On AMD's ROCm, Flash Attention 2 now falls back to the AITER kernel when flash attention isn't installed. T5-style models use fused kernels again. And the rotary kernel from the Hub is now limited to CUDA inference, because its backward pass gave incorrect gradients.

Before you upgrade.

Inference with expert parallelism has speed regressions in this release, and a fix is in progress. That hits mixture-of-experts models like Muse Spark Open hardest, since they split their experts across GPUs.

Transformers now needs a newer kernels package, which lets a failed kernel load fall back instead of raising an error. Deprecated cache methods are removed. And the LUKE tokenizers now treat entity tokens as special tokens, so their output can change.

The Transformers release notes on GitHub list every breaking change and what replaces it. The Muse Spark Open post on the Hugging Face blog has the recipes to run and fine-tune it.