huggingface_hub v2.3.0 lists an 800,000-file dataset in 20 seconds, down from 13 minutes

huggingface_hub 2.3 lists large repos in parallel. Listing every file in the objaverse dataset, almost 800,000 of them, took about 13 minutes. Now it takes about 20 seconds.

Slide text

01 02 03 HfFileSystem

huggingface/huggingface_hub · GitHub release notes v2.3.0 · #5153

huggingface_hub v2.3.0 lists large repos in parallel

allenai/objaverse · 798,924 files 776 s → 19.7 s

The Hub sends a file tree a thousand entries at a time, each page after the last, so the library now lists folders side by side. And a listing two levels deep stops at two levels, in six requests.

HfFileSystem caches folder listings, so files another process added to a Bucket stayed hidden until you cleared the cache. With live follow, the Hub sends each change and the stale listing is dropped. It works from pandas and pyarrow too.

It's off by default, because it keeps a connection open for every Bucket it follows. It only refreshes listings, since reading a file always fetches it from the Hub. Model, dataset and Space repos don't get it.

Discussions, models and gated repos.

Discussions and pull requests on the Hub can now carry images, video and audio, from Python or from the hf command line. Before, that meant calling an undocumented Hub endpoint by hand.

Reading the safetensors headers of a sharded model took one request per shard. A new function gets them all in one. It only reads the main revision, and the old function is unchanged.

Gated repos can now hide the name and email of people asking for access. Listing their requests used to crash. Now you accept each request by its own identifier. Type checkers may flag code that expects a username.

Fixes you may have hit.

Since version 0.24, deleting a folder and pushing the same unchanged files back in one commit removed those files from the repo, without any error. Now the files are kept.

CommitOperationDelete("x/", is_folder=True),
# unchanged on the Hub:
CommitOperationAdd("x/a.txt", b"keep me\n"),

# before: x/a.txt is deleted
# after:  x/a.txt is kept

hf jobs run used to stop printing logs when the stream ended before the Job did. It now reconnects until the Job finishes, and Control C exits right away with a hint on how to follow or cancel it.

Slide text

huggingface_hub v2.3.0

01 02 03 Fixes

huggingface/huggingface_hub · GitHub release notes v2.3.0 · #5162

hf jobs run keeps streaming logs

While a Job waited for hardware, the CLI printed nothing, so it looked stuck. Now it shows the Hub's status message and how long the Job has been waiting, in interactive terminals only.

Slide text

huggingface_hub v2.3.0

01 02 03 Fixes

huggingface/huggingface_hub · GitHub release notes v2.3.0 · #5163

A Job waiting for hardware says so

A reused Inference Client stops holding every finished response in memory. The async client stops freezing your event loop while queue-based providers like fal work. And a timeout raises the documented timeout error.

With a custom endpoint, 20 methods were sent to huggingface dot co with your token. In offline mode, a pinned download could delete your local copy. And pushing a dataset into a new Bucket works again.

The full list is in the 2.3 release notes on GitHub. So, how many files are in the biggest repo you list with HfFileSystem?