dagster-hf-datasets
Project Overview
dagster-hf-datasets is a Dagster integration library that brings Hugging Face Datasets into Dagster’s asset-based orchestration model.
The library enables Hugging Face datasets to become first-class Dagster assets with automatic lineage, metadata tracking, reproducible materializations and seamless publishing back to the Hugging Face Hub.
Built following the milestone of over one million datasets on the Hugging Face Hub, the project bridges modern data orchestration with one of the largest machine learning data ecosystems.
Core Features
- Native
DatasetandDatasetDictDagster assets - Split-aware multi-asset pipelines
- Streaming (
IterableDataset) support - Automatic metadata extraction
- Asset lineage and observability
- Hugging Face Hub publishing
- Parquet-backed IO Manager
- Dataset card generation
Technical Highlights
Native Asset Integration
Create Hugging Face datasets directly as Dagster assets using dedicated decorators while preserving dependency graphs, scheduling, partitioning and lineage.
Production Dataset Pipelines
Supports complete dataset engineering workflows:
- Data ingestion
- Cleaning
- Feature engineering
- Validation
- Versioning
- Publishing to the Hugging Face Hub
Streaming Dataset Support
Runtime-safe handling of streaming datasets enables processing of datasets that cannot fit into memory while preventing invalid persistence assumptions.
Rich Metadata
Every materialization records useful metadata including:
- Schema
- Features
- Fingerprints
- Dataset splits
- Execution mode
- Hub metadata
- Reproducibility information
Custom IO Manager
Includes a dedicated Parquet IO Manager optimized for Hugging Face Datasets while correctly handling both materialized and streaming datasets.
Resources
- 📦 Examples — https://huggingface.co/buckets/the-hf-stack/dagster-hf-datasets-examples
- 📖 Documentation — https://docs.dagster.io/integrations/libraries/hf-datasets
- 📦 PyPI Package — https://pypi.org/project/dagster-hf-datasets/
- 🎤 Dagster Community Interview — https://dagster.io/blog/community-showcase-part-2#parag-ekbote
- 📰 Release Article — https://huggingface.co/blog/AINovice2005/dagster-hf-datasets
Impact
The project enables reproducible, observable and production-ready dataset engineering pipelines by combining Dagster’s orchestration capabilities with the Hugging Face Datasets ecosystem, reducing the effort required to operationalize large-scale machine learning datasets.