Transformers now exports to ONNX and OpenVINO, and transcribes 1,600 languages
1 day ago
Transformers v5.22.2 to v5.24.0: the new Exporters API runs models on ONNX Runtime, OpenVINO and ExecuTorch and shares exports on the Hub; Meta's Omnilingual ASR transcribes more than 1,600 languages; Inkling 2 Small and Mini, Apertus 1.5, LFM3, Dory and EXAONE arrive; the KV cache gets one name, cache; and transformers[audio] shrinks to about 1.4 MiB.
Links
Ask about this presentation
Chapters
0:00Exporters
0:10Faster than torch.compile
0:20Share an export
0:33New models
0:35Omnilingual ASR
0:47Inkling 2
0:58Apertus 1.5
1:09More models
1:19One name for the cache
1:21The cache rename
1:32Assisted decoding
1:40A lighter audio install
1:43Audio extra
1:52Who it affects
2:02What stopped breaking
2:04Jinja sandbox
2:14Generation fixes
2:24Training fixes
2:35Should you upgrade?
Show transcriptHide transcript
Transformers can now export a model, share the export on the Hub, and run it on ONNX Runtime, OpenVINO, ExecuTorch or Dynamo. You call the export just like the original model.
On EmbeddingGemma 2, an ONNX Runtime export with TensorRT beats torch.compile on an A100. On an Intel Xeon, OpenVINO runs twice as fast as compiled PyTorch.
An export now lives on the Hub like a model. ONNX, OpenVINO and ExecuTorch exports load without trusting remote code, while a pickled Dynamo program asks first. Older export scripts need updating.
Part two. New models.
Meta's Omnilingual ASR transcribes speech in more than 1,600 languages. It comes in four sizes up to 7 billion parameters, as a fast CTC model or with a Llama decoder that takes a language hint.
Thinking Machines' Inkling 2 arrives in two sizes. Small has 594 billion parameters, 17 billion active. Mini has half the layers, and runs its middle twelve three times over.
The Swiss AI Initiative's Apertus 1.5 reads text, images and audio as tokens in one vocabulary. It answers in text only, and its audio tokenizer, WavTokenizer, arrives too.
Liquid AI's LFM3 arrives, with a vision model beside it. NVIDIA's Dory lets you trade speed for quality by setting its loop count. And LG's EXAONE loads without remote code.
Part three. One name for the cache.
The cache now has one name, cache, in place of past key values and cache params. The old names keep working, with a warning, until version 5.40. PEFT must be 0.22 or newer.
Assisted decoding now accepts a static cache, so it compiles. Llama 3.1 8B with a small assistant generates about 40 percent more tokens per second.
Part four. A much lighter audio install.
The audio extra now installs only torchaudio, soundfile and soxr. That's about 1.4 megabytes on top of PyTorch instead of 370, and NumPy stays on version two.
Existing environments keep what they have. New ones need librosa or phonemizer installed separately. Speech checkpoints with a language model fall back to greedy decoding, so transcripts can change.
Part five. What stopped breaking.
A CPython bug let a chat template overflow an array from inside the Jinja sandbox. Transformers now hands templates a stripped-down dictionary, since older Pythons may never get the fix.
The text streamer keeps its spaces, and streams Chinese as it's generated. Continuous batching in inference mode stops returning empty answers. Pipelines land on the device you asked for.
FSDP2 loading a 30 billion parameter model peaks at 60 gigabytes of CPU memory again, down from 480. A failed attention kernel load is now retried. And Trainium runs start again.
Should you upgrade? I would say yes. Check that PEFT is at 0.22 or newer, and that fresh audio environments install librosa themselves. The old cache names stop working in version 5.40.
Up next
3:39Transformers runs Mistral Large 4 and trains LoRA on FP8 checkpoints
@hugging-face
3 weeks ago
2:43huggingface_hub v2.4.0 copies files into Buckets and flags outdated CLI extensions
@hugging-face
3 weeks ago
3:05huggingface_hub v2.3.0 lists an 800,000-file dataset in 20 seconds, down from 13 minutes
@hugging-face
1 month ago
2:27Transformers adds Reflection AI’s Beam and loads models on PyTorch 2.7 again
@hugging-face
1 month ago
3:46Transformers v5.20.0 runs Meta's Muse Spark Open, and here is the hardware it needs
@hugging-face
1 month ago
3:18huggingface_hub 2.2.0: hf jobs stats prints a snapshot, and parallel downloads always use hf_xet
@hugging-face
1 month ago
3:12Transformers v5.19.0: EmbeddingGemma 2 and token-dispatch expert parallelism
@hugging-face
1 month ago
2:53huggingface_hub's HfFileSystem now downloads a 2 GB bucket file in seconds
@hugging-face
1 month ago
3:12Transformers v5.18.0: streaming speaker diarization, GGUF mixture-of-experts, FSDP2 with expert parallelism
@hugging-face
1 month ago
2:12What huggingface_hub 2.0 changes, and why transformers keeps you on 1.x for now
@hugging-face
1 month ago
