Muse Spark Open runs in Transformers
Transformers
Transformers now runs Muse Spark Open, the full-size open-weight model in Meta's Muse series. It takes text, images, audio and video, and answers in text, with up to a million tokens of context.
Trained for agentic coding
TransformersMuse Spark Open is trained for agentic coding
It can also take a screenshot, a voice note or a screen recording as input.
Meta trained it for agentic coding, which means editing across many files, debugging, and calling tools. It can also watch a screen recording or hear a voice note while it works on your code.
The hardware it needs
Transformers816B parameters, 42B active per token
About 42 billion of its 816 billion parameters work on each token, but every expert stays in memory. In Transformers, the 16-bit checkpoint spans 32 H200 GPUs. The four-bit checkpoint fits on one B200 node.
Fine-tune it with TRL
TransformersFine-tune the full 816B model with TRL
If you train models, you can fine-tune the full checkpoint with TRL. The TRL recipe trains LoRA adapters on the attention layers and keeps the experts frozen, spread across 64 H100 GPUs.
More new models
TransformersMore new models.
Ling 3.0 Flash, native
TransformersLing 3.0 Flash loads without trust_remote_code
InclusionAI's hybrid mixture-of-experts language model, now native as BailingHybridForCausalLM
InclusionAI's Ling 3.0 Flash is now native in Transformers. It's a hybrid mixture-of-experts model that loads without the trust_remote_code flag, so loading it doesn't run code from the model's repository.
UnlimitedOcr
TransformersUnlimitedOcr transcribes dozens of pages in one pass
UnlimitedOcr is a 3-billion-parameter OCR model from Baidu. Its attention keeps the KV cache the same size the whole time it decodes, so it can transcribe dozens of pages in a single forward pass.
MOSS-Transcribe-Diarize
TransformersMOSS-Transcribe-Diarize says who spoke, and when
[start][S01]text[end]Transcript, speaker and timestamps in one pass · 50+ languages · 0.9B parameters
MOSS-Transcribe-Diarize handles long recordings with several speakers. One pass gives you the transcript, which speaker said each line, and timestamps, in more than 50 languages, from a model under a billion parameters.
Kolibri 1, UltraBERT, MiniCPM-V 4.7
TransformersKolibri 1, UltraBERT and MiniCPM-V 4.7 also land
Kolibri 1 is Aleph Alpha's reasoning model for German and English. UltraBERT encoders handle classification and retrieval on inputs up to 32 thousand tokens. And MiniCPM-V 4.7 places image tiles and video frames on a 2D canvas, so they keep their layout and order.
Exporting and scaling
TransformersExporting and scaling.
Quantize while you export
Transformers| Quantizer | Dynamo | ONNX | ExecuTorch | OpenVINO |
|---|---|---|---|---|
| PT2E | yes | yes | yes | yes |
| NNCF, FX graph | yes | yes | weight-only | yes |
| ONNX Runtime | – | yes | – | – |
| NNCF, ONNX | – | yes | – | – |
| NNCF, OpenVINO | – | – | – | yes |
Transformers exporters can now quantize a model during export to Dynamo, ONNX, ExecuTorch and OpenVINO. That covers static and dynamic quantization, plus eight-bit or four-bit weight-only compression, with calibration you can configure.
Checkpoints across parallel layouts
TransformersSave on one parallel layout, load on another
FSDP × tensor-parallel model and optimizer checkpoints now move between parallel configurations.
Also fixed: expert-parallel dispatch, tensor-parallel plans, static-cache generate()
Model and optimizer checkpoints made with FSDP and tensor parallelism can now be saved under one parallel setup and loaded under a different one. Fixes also landed for expert-parallel dispatch, tensor-parallel plans and static-cache generation.
Kernel fixes
TransformersKernel fixes for ROCm, T5 and training
On AMD's ROCm, Flash Attention 2 now falls back to the AITER kernel when flash attention isn't installed. T5-style models use fused kernels again. And the rotary kernel from the Hub is now limited to CUDA inference, because its backward pass gave incorrect gradients.
Before you upgrade
TransformersBefore you upgrade.
Expert-parallel inference is slower
TransformersExpert-parallel inference is slower in this release
A fix is in progress.
Inference with expert parallelism has speed regressions in this release, and a fix is in progress. That hits mixture-of-experts models like Muse Spark Open hardest, since they split their experts across GPUs.
Changes that can break code
TransformersThree changes that can break code
kernels requireda failed kernel load falls back instead of raising<ent> are now special tokensTransformers now needs a newer kernels package, which lets a failed kernel load fall back instead of raising an error. Deprecated cache methods are removed. And the LUKE tokenizers now treat entity tokens as special tokens, so their output can change.
Where to go
TransformersThe release notes list every breaking change
github.com/huggingface/transformers › Releases
Run and fine-tune Muse Spark Open: huggingface.co/blog/meta-muse-spark-open
The Transformers release notes on GitHub list every breaking change and what replaces it. The Muse Spark Open post on the Hugging Face blog has the recipes to run and fine-tune it.
