Transformers 5.20.2 keeps your input types intact inside generate

Transformers 5.20.2 is a patch release. Its main fix is in generate, where some inputs could lose their original type inside prepare inputs for generation, which caused problems with multimodal models. It also fixes zero expert gradients in bfloat16 training with expert parallelism, declares image input on nine vision-language models that reported text only, and removes a host-to-device copy from the eager attention mask.

Slide text

huggingface / transformers v5.20.2

Transformers 5.20.2 keeps your input types intact inside generate

Fix inputs dict casting in generate #49731

Fix zero expert gradients in bfloat16 with expert parallelism and ep_size < fsdp_size #49688

Declare the input modalities of the VLMs that report text only #49677

Avoid a host-to-device copy in eager_mask #49684

The cause was one line in the PyTorch utilities. When generate moves inputs onto the model's device, the move to device helper rebuilt every dictionary as a plain dict. Now it calls the input's own type, the same way it already handled lists and tuples. The new test sends a BaseModelOutput through prepare inputs for generation on a mismatched device, and checks that it comes back as a BaseModelOutput, with its hidden states still a tuple.

Slide text

huggingface / transformers v5.20.2

A BaseModelOutput you pass to generate now reaches the model as a BaseModelOutput

src/transformers/pytorch_utils.py · move_to_device

if isinstance(value, dict):
-   return {k: move_to_device(v, device) for k, v in value.items()}
+   return type(value)({k: move_to_device(v, device) for k, v in value.items()})
if isinstance(value, (list, tuple)):
    return type(value)(move_to_device(v, device) for v in value)

tests/generation/test_utils.py · test_generation_device_placement_respect_type

prepared_inputs = model.prepare_inputs_for_generation(ids, prev_outputs=out)
self.assertIsInstance(prepared_inputs["prev_outputs"], BaseModelOutput)
self.assertIsInstance(prepared_inputs["prev_outputs"].hidden_states, tuple)

The second fix is for training mixture-of-experts models with FSDP2 and expert parallelism, when the expert-parallel size is smaller than the FSDP size. Each expert's gradient is then divided by the FSDP size. FSDP2 applies that division through a premultiplied sum, and NCCL zeroes it in bfloat16 up to PyTorch 2.13. The fix forces a plain sum on every backend and divides locally. A new four-process test checks the loss and every gradient norm against the same batch on one device, within 2 percent.

Slide text

huggingface / transformers v5.20.2

Expert gradients stop coming out as zero in bfloat16 when ep_size < fsdp_size

FSDP2 expert parallelism bfloat16 NCCL ep_size < fsdp_size

src/transformers/distributed/fsdp.py

- if torch.distributed.get_backend(expert_mesh.get_group()) != "nccl":
-     # Non-NCCL backends need to sum first, then apply the division otherwise it runtime error.
-     module.set_force_sum_reduction_for_comms(True)
+ # FSDP2 applies a divide factor other than the group size through PREMUL_SUM, which non-NCCL backends
+ # reject and NCCL zeroes in bfloat16 up to torch 2.13 (pytorch/pytorch#190747): sum, divide locally.
+ module.set_force_sum_reduction_for_comms(True)

≤ 2% Loss and every gradient norm against the same batch on one device, 4 processes, ep_size 2 · test_fsdp2_expert_parallel_bf16_grads

Nine vision-language model families reported text as their only input, even though they take images. Each now declares its modalities on the model classes: image, video and text for eight of them, and image and text for Step3p7. These are class attributes, so your calls to the models stay the same.

Slide text

huggingface / transformers v5.20.2

Nine vision-language models no longer report text as their only input

exaone4_5

image · video · text

glm5_next

image · video · text

hyperclovax_vision_v2

image · video · text

muse_glimmer

image · video · text

muse_spark

image · video · text

qwen3_5

image · video · text

qwen3_5_moe

image · video · text

qwen4_exp

image · video · text

step3p7

image · text

added to the model classes of the first eight; step3p7 gets ("image", "text")

+   input_modalities = ("image", "video", "text")

A smaller change sits on the eager attention path. The mask builder created its zero as a new tensor from a Python number, which copies that value from the host to the device. It now allocates the zero directly on the device instead, and the mask it returns is the same.

Slide text

huggingface / transformers v5.20.2

The eager attention mask builds its zero on the device, skipping a host-to-device copy

5.20.1

mask = torch.where(mask, torch.tensor(0.0, device=mask.device, dtype=dtype), min_dtype)

5.20.2

mask = torch.where(mask, torch.zeros((), device=mask.device, dtype=dtype), min_dtype)

src/transformers/masking_utils.py · eager_mask

Apart from the version string, the diff carries only these fixes and their tests, so this is a drop-in upgrade from 5.20.1. One behavior does change. On NCCL, expert gradients under FSDP2 are now summed and then divided locally in every precision, not only in bfloat16. To upgrade, install it with pip, pinned to 5.20.2. Transformers 5.20.2 keeps your input types intact inside generate.

Slide text

huggingface / transformers v5.20.2

Transformers 5.20.2 keeps your input types intact inside generate

Breaking changes

None. Beyond the version bump, the diff holds only these four fixes and their tests.

Behavior change

Expert gradients under FSDP2 on NCCL are now summed, then divided locally, in every dtype.

Install

pip install -U transformers==5.20.2