Hugging Face Storage Buckets: Revolutionizing ML Artifact Management

In the rapidly evolving world of machine learning, managing the constant stream of intermediate files generated during model training, data processing, and agent development has always presented a challenge. Traditional version control systems like Git were not designed for this type of dynamic, often redundant data. Hugging Face has addressed this very issue with the introduction of Storage Buckets – a groundbreaking new repository type on the Hugging Face Hub. These mutable, S3-like object storage containers are specifically designed to handle the volatile, high-throughput data that defines modern ML workflows, offering significant benefits in terms of storage costs, speed, and efficiency. This comprehensive guide will delve into the intricacies of Storage Buckets, exploring their functionality, benefits, pricing, and practical use cases. We’ll also cover how to integrate them into your existing ML pipelines using the command-line interface (CLI) and Python SDK. Whether you’re a seasoned machine learning engineer or just starting out, understanding Storage Buckets is crucial for streamlining your projects and optimizing your resources.

This blog post will provide a deep dive into everything you need to know about Hugging Face Storage Buckets, from the underlying technology to how they compare to traditional storage solutions like AWS S3 and Google Cloud Storage. We’ll explore the power of Xet deduplication, the advantages of pre-warming, and the seamless integration with popular tools like Python and JavaScript. By the end of this article, you’ll have a solid understanding of how Storage Buckets can revolutionize the way you manage your ML artifacts.

The Challenge with Traditional Version Control for ML Workflows

For years, Git has been the de facto standard for version controlling code and static assets. However, the nature of ML projects often creates a problem that Git isn’t well-suited to solve. Consider a typical training run: numerous checkpoints are saved throughout the process, optimizer states are periodically recorded, and intermediate data shards are generated. Each of these artifacts is constantly changing and often shares significant portions of data with previous versions. Git struggles with this constant flux, leading to large repository sizes, slow commit times, and an inefficient use of storage.

Furthermore, these files are frequently overwritten. A new training run might discard older checkpoints or update data shards. Git’s history-centric approach, while beneficial for code, becomes cumbersome when dealing with mutable, rapidly evolving data. Real-time collaboration, frequent updates, and the sheer volume of data generated during experimentation quickly overwhelm Git’s capabilities. The overhead of committing every minor change adds up, leading to significant storage bloat. This is where Storage Buckets step in to fill the gap.

Introducing Storage Buckets: A New Paradigm for ML Artifacts

Hugging Face Storage Buckets offer a fundamentally different approach to managing ML artifacts. They are mutable, non-versioned containers designed for the specific needs of modern ML workflows. Think of them as highly optimized, cloud-based storage solutions tailored for the ephemeral nature of checkpoints, datasets, logs, and agent traces. Buckets are not meant to track a history of changes in the same way Git does; instead, they provide a fast, cheap, and efficient way to store and retrieve these artifacts. The core principle is to prioritize speed and cost-effectiveness over auditability.

Buckets leverage Xet, Hugging Face’s proprietary chunk-based storage backend, to achieve these goals. Xet breaks files into smaller, manageable chunks and then uses a sophisticated deduplication algorithm to identify and eliminate redundant data. This is particularly beneficial for ML projects where successive checkpoints often share a significant portion of their content. By storing only unique chunks, Buckets can dramatically reduce storage costs and improve transfer speeds. This approach is a game-changer for organizations dealing with large datasets and computationally intensive training runs.

Key Features of Hugging Face Storage Buckets

• Mutable Storage: Buckets are designed for writing, overwriting, and deleting objects freely, without the constraints of version control.
• Xet Deduplication: Content-defined chunking and deduplication algorithms optimize storage and transfer efficiency.
• Scalable and Cost-Effective: Tiered pricing with significant discounts for large volumes (500 TB+) makes storage cost-effective.
• Easy Integration: Seamless integration with the Hugging Face CLI and Python SDK. Support for JavaScript via `@huggingface/hub` is also available.
• Pre-warming: Ability to bring data closer to compute regions for faster access.
• Permissions and Access Control: Standard Hugging Face permission model (public, private, user, organization).

How Xet Deduplication Works: The Engine Behind Efficiency

At the heart of Storage Buckets lies Xet, Hugging Face’s innovative chunk-based storage backend. Unlike traditional object storage systems that treat files as monolithic entities, Xet divides data into smaller, independent chunks. Each chunk is then assigned a unique identifier, and the system maintains a database of these chunks. When a new file is uploaded, Xet identifies any existing chunks that match the new data. If a match is found, the system simply references the existing chunk instead of storing a duplicate. This significantly reduces storage space and improves transfer speeds.

The deduplication process is transparent to the user and automatic. It works seamlessly in the background, ensuring that storage is used efficiently without requiring any manual intervention. This approach is particularly advantageous for large-scale ML projects where the amount of redundant data can be substantial. The benefits extend beyond storage savings, as smaller chunks can be transferred more quickly, accelerating training and deployment processes.

Pricing and Tiered Plans

Hugging Face Storage Buckets offer a competitive pricing structure that is designed to be cost-effective for both small and large-scale projects. The pricing model is tiered, with discounts available for larger storage volumes. Here’s a breakdown of the pricing tiers:

Storage Volume	Public	Private	Organization
Up to 50 TB	$12/TB/month	$18/TB/month	$10/TB/month
500 TB – 1 TB	$10/TB/month	$16/TB/month	$8/TB/month
1 TB – 5 TB	$9/TB/month	$14/TB/month	$8/TB/month
5 TB – 10 TB	$8/TB/month	$12/TB/month	$9/TB/month
10 TB+	$8/TB/month	$12/TB/month	$8/TB/month

Key Takeaway: The 500 TB+ tier offers a significantly more cost-effective solution than traditional cloud object storage providers like AWS S3, which currently charge $23/TB/month. The combination of Xet deduplication and tiered pricing makes Storage Buckets an attractive option for organizations seeking to reduce their cloud storage costs without compromising performance.

Getting Started: A Step-by-Step Guide

Using Hugging Face Storage Buckets is straightforward. Here’s a quick guide to getting started:

1. Install the CLI and Authenticate

First, install the Hugging Face CLI:

curl -LsSf https://hf.co/cli/install.sh | bash

Then, log in to your Hugging Face account:

hf auth login

2. Create a Bucket

Create a new bucket for your project. Specify whether the bucket should be public or private.

hf buckets create my-training-bucket --private

3. Sync Data to the Bucket

Sync your local data directory to the bucket using the `hf buckets sync` command.

hf buckets sync ./checkpoints hf://buckets/your_username/my-training-bucket/checkpoints

For a dry run (to see what will happen without actually moving any data), use the `–dry-run` flag.

hf buckets sync ./checkpoints hf://buckets/your_username/my-training-bucket/checkpoints --dry-run

Safer operation: `–plan sync-plan.jsonl hf buckets sync –apply sync-plan.jsonl`

4. Inspect the Bucket

Check the bucket contents in the CLI or directly through the Hugging Face Hub.

hf buckets list your_username/my-training-bucket -h

Alternatively, browse the bucket at: https://huggingface.co/buckets/your_username/my-training-bucket

Using Storage Buckets with Python

The `huggingface_hub` library (version 1.5.0 and above) provides a convenient Python SDK for interacting with Storage Buckets. The API follows a familiar pattern: create, sync, and inspect. Here’s a basic example:

from huggingface_hub import create_bucket, list_bucket_tree, sync_bucket

bucket = create_bucket("my-training-bucket", private=True, exist_ok=True)
sync_bucket("./checkpoints", "hf://buckets/your_username/my-training-bucket/checkpoints")
list_bucket_tree("hf://buckets/your_username/my-training-bucket/checkpoints")

This simplifies the integration of Storage Buckets into your existing Python-based ML pipelines, allowing you to effortlessly manage your intermediate files and artifacts alongside your models and datasets.

Pre-warming for Optimized Performance

For distributed training and large-scale pipelines, the location of your data can significantly impact performance. Pre-warming allows you to bring your data closer to the compute regions where your jobs are running. By specifying the desired regions during bucket creation or sync, you ensure that the necessary data is already present when your training runs begin, avoiding costly data transfer delays.

The Future of Hugging Face Storage Buckets

Hugging Face is committed to expanding the capabilities and integration of Storage Buckets. Future plans include:

• Direct Promotion from Buckets to Repositories: A seamless workflow from working storage in Buckets to final models and datasets in repositories.
• More Cloud Provider Support: Expanding pre-warming capabilities to more cloud providers, including Google Cloud Platform.
• Enhanced Security Features: Increased control over data access and security settings.

The integration of Buckets, Repos, and other tools will streamline the entire ML lifecycle, from development to deployment.

Conclusion: Embrace the Power of Storage Buckets

Hugging Face Storage Buckets represent a significant advancement in the way machine learning artifacts are managed. By providing a fast, cost-effective, and scalable storage solution specifically designed for the needs of modern ML workflows, Buckets address a critical gap in the ecosystem. The combination of Xet deduplication, flexible pricing, and seamless integration with the Hugging Face CLI and Python SDK make Storage Buckets a compelling option for organizations of all sizes. Whether you’re a solo researcher or a large enterprise, embracing Storage Buckets will streamline your projects, reduce your costs, and accelerate your time to market. They are poised to consolidate both the storage and version control aspects of ML workflows, leading to greater efficiency and productivity.

FAQ

1. What are Hugging Face Storage Buckets?

   Hugging Face Storage Buckets are mutable, S3-like object storage containers designed for managing ML artifacts, such as checkpoints, datasets, and logs. They are a new repository type on the Hugging Face Hub.

2. How do Storage Buckets differ from Hugging Face Repositories?

   Repositories are ideal for publishing finished models and datasets, with built-in version control. Buckets are designed for the dynamic and mutable artifacts generated during model training and data processing, where version control isn’t necessary or efficient.

3. What is Xet deduplication and how does it benefit me?

   Xet is Hugging Face’s chunk-based storage backend. It identifies and eliminates redundant data, allowing you to store fewer bytes and transfer data faster. This significantly reduces storage costs and improves overall performance.

4. How does pricing work for Storage Buckets?

   Pricing is tiered based on storage volume. Higher volumes qualify for discounted rates, making it cost-effective for large-scale projects. As of the latest pricing information, pricing starts at $8/TB/month for 500 TB+ with further discounts for even larger volumes.

5. Can I use Storage Buckets with Python?

   Yes! The `huggingface_hub` library (version 1.5.0 and above) provides a Python SDK that allows you to create, sync to, and inspect Buckets programmatically.

6. What is pre-warming and why is it useful?

   Pre-warming brings data closer to compute regions, reducing data transfer latency during training and deployment. This is particularly beneficial for distributed training and multi-region setups.

7. Is Storage Buckets secure?

   Yes, Storage Buckets inherit the standard Hugging Face permission model, allowing you to control access and ensure the security of your data. Buckets can be set to public, private, or restricted to specific users or organizations.

8. Can I sync files from a local directory to a Bucket?

   Yes. You can use the `hf buckets sync` command to copy files and directories from your local machine to a Bucket.

9. What is the recommended approach for checking the changes before syncing data?

   Use the `–dry-run` flag with the `hf buckets sync` command. This will show you a plan of the changes that would be made without actually transferring any data.

10. What are the supported cloud providers for Storage Buckets?

    Currently, Support is available for AWS and GCP, With plans to expand to other cloud providers like Azure in the future.