huggingface_hub v2.4.0 copies files into Buckets and flags outdated CLI extensions

In huggingface_hub 2.4, HfFileSystem copies files into a Bucket. Until now, any copy that touched a Bucket stopped with a not implemented error.

Slide text

01 02 03 Buckets

huggingface/huggingface_hub · GitHub release notes v2.4.0 · #5080

huggingface_hub v2.4.0 copies files into Buckets

before: NotImplementedError now: copied

The copy can start inside the same Bucket, in a different Bucket, or in a model, dataset or Space repo, so a dataset can land in a Bucket in one call.

Files stored with Xet are copied on the Hub by their hash, so nothing comes down to your machine. Other repo files are downloaded first, then uploaded to the Bucket.

fsspec moves a file by copying it and then deleting the source. So with copies working, moving and renaming files inside a Bucket work too.

hffs.mv(
  "buckets/me/my-bucket/data/tmp.csv",
  "buckets/me/my-bucket/data/final.csv",
)

Copies only go one way for now. Copying from a Bucket back into a model, dataset or Space repo is not supported yet.

Extensions for the hf command.

The extensions list only showed when each extension was installed. Now it adds the installed version, the latest one, and a hint naming every extension that's out of date.

Slide text

huggingface_hub v2.4.0

01 02 03 CLI extensions

huggingface/huggingface_hub · GitHub release notes v2.4.0 · #5269

hf extensions list shows what's outdated

Running an outdated extension prints a hint, at most once a day. The latest version is checked once a day too, to stay within GitHub's rate limits. One environment variable turns the checks off.

When a Homebrew upgrade removed the Python an extension was installed with, the extension stopped starting, and update said it was current. Now update rebuilds its virtual environment, even with no newer commit.

$ hf extensions update speedtest
Creating virtualenv at: ~/.local/share/hf/extensions/hf-speedtest/venv
✓ Extension updated

Fixes you may have hit.

Opening a private repo through HfFileSystem without a token gave the same not found error as a typo in its name. The error now also tells you to check that you're logged in.

Creating a webhook from the command line failed unless you named the repo domain, and discussions could not be watched at all. Now a new webhook watches both by default.

A filename that pointed at a parent folder, or an absolute path, could make a cache lookup resolve outside the cache folder. Those names are now rejected, the way downloads already rejected them.

Finding a single file with details returned nothing, which broke datatrove readers. Cache verify flagged its own metadata as extra files. And on Windows, it crashed when its output was piped.

If you keep data in Buckets, I'd upgrade for the copy alone: files stored with Xet land there without a download. The full list is in the 2.4 release notes on GitHub.