Transformers now exports to ONNX and OpenVINO, and transcribes 1,600 languages

1 day ago

Transformers v5.22.2 to v5.24.0: the new Exporters API runs models on ONNX Runtime, OpenVINO and ExecuTorch and shares exports on the Hub; Meta's Omnilingual ASR transcribes more than 1,600 languages; Inkling 2 Small and Mini, Apertus 1.5, LFM3, Dory and EXAONE arrive; the KV cache gets one name, cache; and transformers[audio] shrinks to about 1.4 MiB.

Ask about this presentation

Chapters

  1. 0:00Exporters
  2. 0:10Faster than torch.compile
  3. 0:20Share an export
  4. 0:33New models
  5. 0:35Omnilingual ASR
  6. 0:47Inkling 2
  7. 0:58Apertus 1.5
  8. 1:09More models
  9. 1:19One name for the cache
  10. 1:21The cache rename
  11. 1:32Assisted decoding
  12. 1:40A lighter audio install
  13. 1:43Audio extra
  14. 1:52Who it affects
  15. 2:02What stopped breaking
  16. 2:04Jinja sandbox
  17. 2:14Generation fixes
  18. 2:24Training fixes
  19. 2:35Should you upgrade?
Show transcript
  1. Transformers can now export a model, share the export on the Hub, and run it on ONNX Runtime, OpenVINO, ExecuTorch or Dynamo. You call the export just like the original model.

    Slide text
    Transformers
    01 02 03 04 05 Exporters
    huggingface/transformers · release v5.23.0 · PRs #50029, #50032

    Transformers now exports models and runs them on their own runtime

    ONNX Runtime OpenVINO ExecuTorch Dynamo
  2. On EmbeddingGemma 2, an ONNX Runtime export with TensorRT beats torch.compile on an A100. On an Intel Xeon, OpenVINO runs twice as fast as compiled PyTorch.

    Slide text
    Transformers
    01 02 03 04 05 Exporters
    huggingface/transformers · release v5.23.0 · Exporters

    On EmbeddingGemma 2, exports beat torch.compile

    NVIDIA A100

    torch.compile
    1.0×
    ONNX Runtime + TensorRT
    1.4×

    Intel Xeon Platinum 8570

    torch.compile
    1.0×
    OpenVINO
    2.2×

    Speed against torch.compile, 8 × 128 tokens

  3. An export now lives on the Hub like a model. ONNX, OpenVINO and ExecuTorch exports load without trusting remote code, while a pickled Dynamo program asks first. Older export scripts need updating.

    Slide text
    Transformers
    01 02 03 04 05 Exporters
    huggingface/transformers · release v5.23.0 · PR #50029 · huggingface/transformers · release v5.24.0 · PRs #50813, #50886

    An export saves and loads like a model

    On the Hub its own repo, or a subfolder of the model's repo
    Safe to load ONNX, OpenVINO and ExecuTorch need no trust_remote_code; pickled .pt2 programs do
    Breaking export() returns an ExportedModel; output_path is gone
  4. Part two. New models.

    Slide text
    Transformers
    01 02 03 04 05 New models
    02
  5. Meta's Omnilingual ASR transcribes speech in more than 1,600 languages. It comes in four sizes up to 7 billion parameters, as a fast CTC model or with a Llama decoder that takes a language hint.

    Slide text
    Transformers
    01 02 03 04 05 New models
    huggingface/transformers · release v5.24.0 · PR #43265

    Meta's Omnilingual ASR transcribes more than 1,600 languages

    300M 1B 3B 7B
    CTC a linear head, decoded in one greedy pass
    Audio language model a Llama decoder that can take a language code
  6. Thinking Machines' Inkling 2 arrives in two sizes. Small has 594 billion parameters, 17 billion active. Mini has half the layers, and runs its middle twelve three times over.

    Slide text
    Transformers
    01 02 03 04 05 New models
    huggingface/transformers · release v5.23.0 · PR #50682 · huggingface/transformers · release v5.24.0 · PR #51050

    Thinking Machines' Inkling 2 arrives in two sizes

    Small 594B total, 17B active; text and images in
    Mini 24 layers; layers 6 to 17 run three times per token
    Inkling 2 Mini architecture diagram from the release notes
  7. The Swiss AI Initiative's Apertus 1.5 reads text, images and audio as tokens in one vocabulary. It answers in text only, and its audio tokenizer, WavTokenizer, arrives too.

    Slide text
    Transformers
    01 02 03 04 05 New models
    huggingface/transformers · release v5.24.0 · PR #47662

    Apertus 1.5 reads text, images and audio

    Text Images Audio

    Answers in text only. Its audio tokenizer, WavTokenizer, arrives as a model of its own.

  8. Liquid AI's LFM3 arrives, with a vision model beside it. NVIDIA's Dory lets you trade speed for quality by setting its loop count. And LG's EXAONE loads without remote code.

    Slide text
    Transformers
    01 02 03 04 05 New models
    huggingface/transformers · release v5.23.0 · PR #50322 · huggingface/transformers · release v5.24.0 · PRs #50099, #50966

    LFM3, Dory and EXAONE load too

    LFM3 Liquid AI's convolution and attention hybrid, with LFM3-VL for images and video
    Dory NVIDIA's looped model: set the loop count at run time, speed against quality
    EXAONE LG's 3.0, 3.5 and Deep load without trust_remote_code
  9. Part three. One name for the cache.

    Slide text
    Transformers
    01 02 03 04 05 The cache
    03
  10. The cache now has one name, cache, in place of past key values and cache params. The old names keep working, with a warning, until version 5.40. PEFT must be 0.22 or newer.

    Slide text
    Transformers
    01 02 03 04 05 The cache
    huggingface/transformers · release v5.24.0 · PRs #50585, #50730

    past_key_values and cache_params are now cache

    Old names still work with a warning: models until v5.40, generate until v6
    PEFT 0.22 or newer is now required
  11. Assisted decoding now accepts a static cache, so it compiles. Llama 3.1 8B with a small assistant generates about 40 percent more tokens per second.

    Slide text
    Transformers
    01 02 03 04 05 The cache
    huggingface/transformers · release v5.24.0 · PR #49415

    Assisted decoding now compiles with a static cache

    Before
    34.5 tok/s
    Now
    48.9 tok/s

    Llama 3.1 8B with a Llama 3.2 1B assistant, A100

  12. Part four. A much lighter audio install.

    Slide text
    Transformers
    01 02 03 04 05 Audio install
    04
  13. The audio extra now installs only torchaudio, soundfile and soxr. That's about 1.4 megabytes on top of PyTorch instead of 370, and NumPy stays on version two.

    Slide text
    Transformers
    01 02 03 04 05 Audio install
    huggingface/transformers · release v5.23.0 · PR #50610

    transformers[audio] shrinks to 1.4 MiB

    Before
    ~371 MiB
    Now
    ~1.4 MiB

    torchaudio, soundfile and soxr only, and NumPy stays on 2.x

  14. Existing environments keep what they have. New ones need librosa or phonemizer installed separately. Speech checkpoints with a language model fall back to greedy decoding, so transcripts can change.

    Slide text
    Transformers
    01 02 03 04 05 Audio install
    huggingface/transformers · release v5.23.0 · PRs #50610, #50408

    Fresh installs lose librosa and LM decoding

    Existing environments keep every package already installed
    New environments install librosa or phonemizer yourself
    WithLM speech checkpoints fall back to greedy CTC with a warning; LM decoding leaves in v6
  15. Part five. What stopped breaking.

    Slide text
    Transformers
    01 02 03 04 05 Fixed
    05
  16. A CPython bug let a chat template overflow an array from inside the Jinja sandbox. Transformers now hands templates a stripped-down dictionary, since older Pythons may never get the fix.

    Slide text
    Transformers
    01 02 03 04 05 Fixed
    huggingface/transformers · release v5.22.2 · PR #50431

    Chat templates run in a harder sandbox

    A CPython dict bug could overflow an array inside the Jinja sandbox. Python 3.13 to 3.15 carry the fix; older versions may not get it.

  17. The text streamer keeps its spaces, and streams Chinese as it's generated. Continuous batching in inference mode stops returning empty answers. Pipelines land on the device you asked for.

    Slide text
    Transformers
    01 02 03 04 05 Fixed
    huggingface/transformers · release v5.24.0 · PRs #51036, #36708, #50863, #50645

    Streaming, batching and pipelines behave

    TextStreamer keeps its spaces, and streams Chinese or Japanese as it's generated
    Continuous batching works under torch.inference_mode(); it used to return 0 tokens
    pipeline lands on the single device its device_map names
  18. FSDP2 loading a 30 billion parameter model peaks at 60 gigabytes of CPU memory again, down from 480. A failed attention kernel load is now retried. And Trainium runs start again.

    Slide text
    Transformers
    01 02 03 04 05 Fixed
    huggingface/transformers · release v5.23.0 · PRs #45649, #50391 · release v5.22.2 · PR #50281

    Fixes for FSDP2, attention kernels and Trainium

    FSDP2 loading 30B model, 8 ranks: CPU peak ~60 GB again, down from ~480 GB
    Attention kernels a failed kernel load is retried instead of breaking the process
    Trainium process groups start: device_id is set only for NCCL
  19. Should you upgrade? I would say yes. Check that PEFT is at 0.22 or newer, and that fresh audio environments install librosa themselves. The old cache names stop working in version 5.40.

    Slide text
    Transformers
    01 02 03 04 05 Fixed
    huggingface/transformers · release v5.24.0 · huggingface/transformers · release v5.23.0 · Transformers docs: Caching, Exporters

    Upgrade? Yes, with two checks

    PEFT 0.22 or newer
    Fresh audio environments install librosa yourself
    The limit old cache names stop working in v5.40

    Upgrade steps: the v5.23.0 and v5.24.0 release notes, and the Caching and Exporters guides

What happened in AI today, in your inbox

One short email each evening. Read it and you’re caught up.

Built with ❤️ in New York