huggingface_hub 2.2.0: hf jobs stats prints a snapshot, and parallel downloads always use hf_xet
1 month ago
What changed in huggingface_hub 2.2.0: hf jobs stats prints one snapshot and exits, parallel downloads always use hf_xet, HfFileSystem caching fixes, Windows fixes for the hf command, and a new exception from load_torch_model. https://github.com/huggingface/huggingface_hub/releases/tag/v2.2.0 https://github.com/huggingface/huggingface_hub/compare/v2.1.1...v2.2.0
Ask about this presentation
Chapters
0:00hf jobs stats prints a snapshot
0:16The live view stays behind follow
0:27Snapshots for scripts and agents
0:39Logs and stats stop hanging
0:52Downloads
0:53Parallel downloads always use hf_xet
1:06A filter that matches zero files warns
1:19Offline, redirects, scheduled pushes
1:31HfFileSystem
1:33A new repo shows up
1:45Listings are replaced
1:58Empty buckets list as empty
2:07The hf command
2:08Windows passes patterns as typed
2:24Env files, uploads, cache cleanup
2:39hf_xet on Windows ARM64
2:51Exceptions
2:53load_torch_model raises FileNotFoundError
3:09Release notes
Show transcriptHide transcript
huggingface_hub 2.2.0 changes hf jobs stats. Run it on a job and you get one reading of CPU, memory and GPU, then your prompt back. It used to stream until the job finished, so a script calling it on a running job hung until it timed out.
Hugging Face lists this as a breaking change, and scripts that relied on the stream can pass the follow option. The Python call fetch_job_metrics changed the same way.
The snapshot also comes in machine-readable formats. There's a JSON mode, an agent mode and a quiet mode, and they print the raw metrics that the Jobs API returns, instead of the table.
Logs without the follow option used to hang while a job kept printing, and now they return. Stats works when a listed job has finished, and its JSON output is an empty list when zero jobs are running.
Downloads.
In a fresh process, the first parallel downloads could see a placeholder version for hf_xet and treat it as missing, so some files in a snapshot_download skipped Xet. Every download now sees the real version.
snapshot_download, and the hf download command, now print a warning when your allow or ignore patterns match zero files. Until now, a typo in a pattern looked exactly like a successful download.
An offline snapshot_download with a pinned revision now works. Following a resolve redirect keeps the port number. And CommitScheduler had a race that skipped its first scheduled push, which is fixed.
HfFileSystem.
HfFileSystem cached "not found" results. A repo, branch or bucket you created in the same process stayed missing until you cleared the whole cache. Now a refresh on that one path clears it.
Directory listings were appended to the cache, so a listing after a find call could show duplicate entries. A refresh kept deleted files, too. Each new listing now replaces the cached one.
Listing a bucket you had just created, with zero files in it, used to raise FileNotFoundError. In 2.2.0 it returns an empty list.
The hf command.
On Windows, the hf command expanded quoted patterns into local file names. So a download filtered to JSON files, run in a folder that held a JSON file, quietly fetched zero files. Arguments now arrive as typed.
Env and secrets files are now read as UTF-8, with the byte order mark stripped. Upload takes an existing path with glob characters literally. And planning a cache cleanup no longer slows quadratically with the number of files in a revision.
On Windows ARM64 machines, Python reports the processor name in capitals, so the install rule missed it. Installing huggingface_hub there now pulls in hf_xet as well.
Two exceptions change.
Hugging Face lists this as breaking too. load_torch_model used to raise a value error when the checkpoint path didn't exist. Code that catches it is affected. And model_index_to_eval_results now raises a value error when you pass an empty list.
The 2.2.0 release notes on GitHub link every pull request behind these changes, and they point to the Jobs guide for the new snapshot output.
Up next
3:39Transformers runs Mistral Large 4 and trains LoRA on FP8 checkpoints
@hugging-face
1 week ago
2:43huggingface_hub v2.4.0 copies files into Buckets and flags outdated CLI extensions
@hugging-face
1 week ago
3:05huggingface_hub v2.3.0 lists an 800,000-file dataset in 20 seconds, down from 13 minutes
@hugging-face
3 weeks ago
2:27Transformers adds Reflection AI’s Beam and loads models on PyTorch 2.7 again
@hugging-face
3 weeks ago
3:46Transformers v5.20.0 runs Meta's Muse Spark Open, and here is the hardware it needs
@hugging-face
1 month ago
3:12Transformers v5.19.0: EmbeddingGemma 2 and token-dispatch expert parallelism
@hugging-face
1 month ago
2:53huggingface_hub's HfFileSystem now downloads a 2 GB bucket file in seconds
@hugging-face
1 month ago
3:12Transformers v5.18.0: streaming speaker diarization, GGUF mixture-of-experts, FSDP2 with expert parallelism
@hugging-face
1 month ago
2:12What huggingface_hub 2.0 changes, and why transformers keeps you on 1.x for now
@hugging-face
1 month ago