Large repos list in parallel
huggingface_hub v2.3.0 lists large repos in parallel
huggingface_hub 2.3 lists large repos in parallel. Listing every file in the objaverse dataset, almost 800,000 of them, took about 13 minutes. Now it takes about 20 seconds.
Shallow listings stop early
huggingface_hub v2.3.0A two-level listing now stops at two levels
| datasets/allenai/objaverse | before | after |
|---|---|---|
find(), 798,924 files | 801 requests, 776 s | 807 requests, 19.7 s |
find(maxdepth=2), 328 entries | 801 requests, 776 s | 6 requests, 1.5 s |
The Hub sends a file tree a thousand entries at a time, each page after the last, so the library now lists folders side by side. And a listing two levels deep stops at two levels, in six requests.
Bucket listings stay fresh
huggingface_hub v2.3.0Bucket listings update while another process writes
HfFileSystem caches folder listings, so files another process added to a Bucket stayed hidden until you cleared the cache. With live follow, the Hub sends each change and the stale listing is dropped. It works from pandas and pyarrow too.
Live follow limits
huggingface_hub v2.3.0Live follow is off by default, and Buckets only
It's off by default, because it keeps a connection open for every Bucket it follows. It only refreshes listings, since reading a file always fetches it from the Hub. Model, dataset and Space repos don't get it.
02: Discussions and models
huggingface_hub v2.3.0Discussions, models and gated repos.
Attachments in discussions
huggingface_hub v2.3.0Attach images, video and audio to discussions and PRs
Discussions and pull requests on the Hub can now carry images, video and audio, from Python or from the hf command line. Before, that meant calling an undocumented Hub endpoint by hand.
Shard headers in one request
huggingface_hub v2.3.0Every safetensors shard header in one request
get_sharded_safetensors_metadata reads the main revision only. get_safetensors_metadata is unchanged.
Reading the safetensors headers of a sharded model took one request per shard. A new function gets them all in one. It only reads the main revision, and the old function is unchanged.
Gated repos that hide requesters
huggingface_hub v2.3.0Access requests work on repos that hide who asked
KeyErrorrequest_id to accept or reject it byusername, fullnamenow typed str | NoneGated repos can now hide the name and email of people asking for access. Listing their requests used to crash. Now you accept each request by its own identifier. Type checkers may flag code that expects a username.
03: Fixes
huggingface_hub v2.3.0Fixes you may have hit.
create_commit keeps re-added files
huggingface_hub v2.3.0create_commit keeps files you re-add after a delete
CommitOperationDelete("x/", is_folder=True),
# unchanged on the Hub:
CommitOperationAdd("x/a.txt", b"keep me\n"),
# before: x/a.txt is deleted
# after: x/a.txt is keptSince version 0.24, deleting a folder and pushing the same unchanged files back in one commit removed those files from the repo, without any error. Now the files are kept.
hf jobs run keeps streaming
huggingface_hub v2.3.0hf jobs run keeps streaming logs
hf jobs run used to stop printing logs when the stream ended before the Job did. It now reconnects until the Job finishes, and Control C exits right away with a hint on how to follow or cancel it.
Queued Jobs show their status
huggingface_hub v2.3.0A Job waiting for hardware says so
While a Job waited for hardware, the CLI printed nothing, so it looked stuck. Now it shows the Hub's status message and how long the Job has been waiting, in interactive terminals only.
InferenceClient fixes
huggingface_hub v2.3.0InferenceClient frees memory and unblocks async
AsyncInferenceClientqueue-based providers such as fal-ai no longer freeze the event looptimeoutraises InferenceTimeoutError, as documentedA reused Inference Client stops holding every finished response in memory. The async client stops freezing your event loop while queue-based providers like fal work. And a timeout raises the documented timeout error.
Endpoint, offline and Bucket fixes
huggingface_hub v2.3.0Your token stays on your endpoint
HfApi(endpoint=...)20 methods went to huggingface.co with your tokenHF_HUB_OFFLINE=1a pinned download could delete the file in local_dirpush_to_hub into a new Bucketfailed with FileNotFoundError since v2.2.0With a custom endpoint, 20 methods were sent to huggingface dot co with your token. In offline mode, a pinned download could delete your local copy. And pushing a dataset into a new Bucket works again.
The biggest repo you list
huggingface_hub v2.3.0How many files are in the biggest repo you list?
Release notes: github.com/huggingface/huggingface_hub › Releases › v2.3.0
The full list is in the 2.3 release notes on GitHub. So, how many files are in the biggest repo you list with HfFileSystem?
