Recursive's Flint now runs natively in Transformers

Recursive's Flint is a mixture-of-experts language model, and this change gives it native support in Transformers. The checkpoint now loads through the usual auto classes, backed by its own configuration and model classes, automatic registration, checkpoint conversion, documentation and tests. It was ported from an earlier private integration.

Slide text

huggingface / transformers Add Flint model support

Recursive's Flint now runs
natively in Transformers

A mixture-of-experts language model with hybrid attention, attention sinks and sigmoid routing.

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("RecursiveSI/flint", dtype="auto", device_map="auto")

The new model page describes Flint against a standard mixture-of-experts transformer like Qwen3-MoE. Most layers use sliding-window attention over the current token and the 512 before it, and the rest use full attention with no positional encoding. Every attention head has a learned sink, as in GPT-OSS. And each sparse layer sends a token to 6 of 256 experts by sigmoid score, then adds a shared expert with no gate.

Slide text

huggingface / transformers Add Flint model support

Flint is a Qwen3-MoE-style model with three changes

+ Hybrid attention Sliding window over the current token and the 512 before it, with RoPE. Full-attention layers have no positional encoding.

+ Attention sinks One learned sink logit per head, as in GPT-OSS. Queries and keys are RMS-normalized per head.

+ Sigmoid routing Top 6 of 256 experts by sigmoid score, plus an ungated shared expert. The first two layers are dense.

docs/source/en/model_doc/flint.md

Flint is written as a modular file of about 270 lines, and the generator expands it into a modeling file of nearly 700 lines. Most of its classes inherit straight from Qwen2-MoE, Qwen3-MoE and GPT-OSS. Besides the config, four classes carry new code: the attention, the router, the sparse block, and the base class, which turns SDPA off and initializes the sinks. To change Flint, you edit the modular file and regenerate.

Slide text

huggingface / transformers Add Flint model support

Flint is 265 lines of modular code on top of Qwen and GPT-OSS classes

modular_flint.py · 265 lines generates modeling_flint.py · 683 lines

FlintConfig ← Qwen2MoeConfig new defaults, layer_types FlintRMSNorm ← GptOssRMSNorm FlintAttention ← Qwen3MoeAttention sinks, RoPE on sliding layers only FlintTopKRouter ← Qwen3MoeTopKRouter sigmoid scores FlintSparseMoeBlock ← Qwen3MoeSparseMoeBlock ungated shared expert FlintPreTrainedModel ← Qwen3MoePreTrainedModel SDPA off, sink init FlintModel ← Qwen2MoeModel FlintForCausalLM ← Qwen3MoeForCausalLM

Here are Flint's 49 default layers. The config sets full attention on the first two layers, on every fourth layer after that, and on the last one. That makes 14 full-attention layers. The other 35 use a sliding window of 513 tokens: the current token and the 512 before it. Only the sliding layers apply rotary embeddings. And the first two layers use a dense MLP instead of experts.

Slide text

huggingface / transformers Add Flint model support

14 of Flint's 49 layers use full attention; the other 35 use a 513-token window

0 1 5 9 13 17 21 25 29 33 37 41 45 48

Full attention, no positional encoding Sliding window, rotary Dense MLP

FlintConfig.__post_init__

"full_attention" if i < 2 or i % 4 == 1 or i == self.num_hidden_layers - 1 else "sliding_attention"

The sink lives in the eager attention function. First, the scores are computed in float32, even under autocast, because with query and key normalization the logits reach the tens. Then each head's learned sink is appended as one more logit, the softmax runs over it, and that column is dropped. So the sink takes a share of the probability without attending to any value.

Slide text

huggingface / transformers Add Flint model support

Each head's sink is one more softmax logit that attends to nothing

modular_flint.py · eager_attention_forward

# QK-normed attention logits reach tens, so the scores and the softmax stay in float32, even under autocast
with maybe_autocast(device_type=query.device.type, enabled=False):
    attn_weights = torch.matmul(query.float(), key_states.float().transpose(2, 3)) * scaling
if attention_mask is not None:
    attn_weights = attn_weights + attention_mask

# Each head's learned sink is an extra softmax logit that attends to nothing
sinks = module.sinks.reshape(1, -1, 1, 1).expand(*attn_weights.shape[:-1], 1)
attn_weights = F.softmax(torch.cat([attn_weights, sinks], dim=-1), dim=-1)[..., :-1].to(value_states.dtype)

The router computes its logits in float32, with autocast turned off. It takes a sigmoid over them, keeps the top 6 experts out of 256, and renormalizes the chosen scores so they sum to one. Then the sparse block adds the shared expert. Unlike Qwen2-MoE, Flint adds it to every token in full, with no gate.

Slide text

huggingface / transformers Add Flint model support

Flint routes by sigmoid score and adds the shared expert in full

modular_flint.py · FlintTopKRouter and FlintSparseMoeBlock

# FlintTopKRouter.forward
with maybe_autocast(device_type=device_type, enabled=False):  # Force float32
    router_logits = F.linear(hidden_states.float(), self.weight.float())
# Sigmoid scores without expert bias; the selected scores are renormalized
routing_weights, selected_experts = torch.topk(router_logits.sigmoid(), self.top_k, dim=-1)
if self.norm_topk_prob:
    routing_weights = routing_weights / routing_weights.sum(dim=-1, keepdim=True)

# FlintSparseMoeBlock.forward
# Unlike Qwen2-MoE, the shared expert is always added in full, without a gate
final_hidden_states = self.experts(hidden_states, selected_experts, routing_weights)
final_hidden_states = final_hidden_states + self.shared_expert(hidden_states)

SDPA cannot apply attention sinks, so Flint switches it off and defaults to eager attention, computed in float32. For long inputs, the docs point to sink-aware FlashAttention kernels: the vLLM flash attention 3 build on Hopper, or flash attention 4 on Blackwell. Asking for flash attention 2 picks one of those instead. And under tensor parallelism, attention stays replicated, because the per-head sinks can't be sharded with the heads yet. Only the experts and the MLPs are split.

Slide text

huggingface / transformers Add Flint model support

Flint can't use SDPA, so it runs eager attention unless you pick a sink-aware kernel

sdpa	Off: it cannot apply the attention sinks
eager (default)	Scores and softmax in float32
kernels-community/vllm-flash-attn3	Sink-aware, on Hopper
flash_attention_4	Sink-aware, on Blackwell
flash_attention_2	Selects one of the two above

Tensor parallelism: attention is replicated, because it "cannot shard the per-head sinks along with the heads yet". Experts and MLPs are sharded.

On disk, a Flint checkpoint stores one tensor per expert, and it stores the sinks under the name softmax offset. The conversion mapping copies the Qwen2-MoE entry, which fuses the experts on load, and adds one rename from softmax offset to sinks. Saving runs both in reverse, so the model writes back the same format it read.

Slide text

huggingface / transformers Add Flint model support

Flint checkpoints keep one tensor per expert and load through the Qwen2-MoE conversion

On disk

layers.1.mlp.experts.0.gate_proj.weight
layers.1.mlp.experts.0.up_proj.weight
layers.0.self_attn.softmax_offset

load → ← save

In memory

layers.1.mlp.experts.gate_up_proj
(all experts, fused)
layers.0.self_attn.sinks

src/transformers/conversion_mapping.py

+ mapping["flint"] = mapping["qwen2_moe"].copy()
+ mapping["flint"] += [WeightRenaming("self_attn.softmax_offset", "self_attn.sinks")]

The tests check each of those claims directly. The sink test rebuilds attention by hand in float64, adding each sink's exponent to the softmax denominator, and matches it within one part in a hundred thousand. The positional test makes every layer full attention, doubles the position ids, and expects identical output. The round trip saves, reloads and saves again, with exact weights and the same keys. And a slow accelerator test runs the real checkpoint greedily. Asked for 17 times 23, it thinks briefly and answers 391.

Slide text

huggingface / transformers Add Flint model support

New tests check the sinks, the position-free layers and the checkpoint round trip

test_attention_sinks Matches a hand-built float64 softmax with exp(sink) in the denominator, rtol 1e-5

test_full_attention_layers_have_no_positional_encoding With every layer set to full attention, doubling every position id leaves the output unchanged

test_checkpoint_format_round_trip Save, load, save: weights equal at atol 0, same keys on disk

test_greedy_chat@slow, accelerator, real checkpoint "What is 17 * 23? Answer briefly." returns a short <think> block, then "391"

tests/models/flint/test_modeling_flint.py · FlintModelTest also runs the shared CausalLMModelTest suite

The limit to keep in mind is attention: eager by default, FlashAttention only through the sink-aware kernels, and replicated under tensor parallelism. The rest of the model inherits from the Qwen classes, so to change Flint, edit the modular file and regenerate. To try it, load Recursive's checkpoint with the auto classes or the text generation pipeline. Recursive's Flint now runs natively in Transformers.

Slide text

huggingface / transformers Add Flint model support

Recursive's Flint now runs natively in Transformers

Watch for

Eager attention by default, FlashAttention only through the sink-aware kernels, and replicated attention under tensor parallelism.

Edit

models/flint/modular_flint.py, then regenerate the modeling file.

Try

RecursiveSI/flint with the auto classes or pipeline("text-generation").