🔀lakeFS

NEW

An open-source system that brings Git-like branching and commits to data lakes on object storage.

DESCRIPTION

lakeFS provides repository-style version control for data on object stores such as S3, GCS, and Azure Blob. It enables branching, committing, and merging datasets, improving reproducibility and safe experimentation. Data engineers can isolate changes, run pipelines against branches, and promote tested snapshots to production. lakeFS integrates with Spark, Hive, and Presto/Trino, and supports hooks for policy enforcement. Typical use cases include ML experiments, backfills, and data quality remediation. It complements table formats by controlling datasets at object level. Competing tools include DVC and Pachyderm.

ALTERNATIVE DEFINITIONS

lakeFS

lakeFS is an open-source data version control for your data lake, with Git-like operations for object stores.

Emphasizes Git-like operations on object storage.

Source

lakeFS Documentation

lakeFS provides Git-like versioning for data stored in object stores, enabling branching, committing, and merging of data.

Defines core operations and object-store scope.

Source

RELATED TERMS

Data Versioning Git Object Storage Amazon S3 Data Lake Branching Commit Reproduction Experiment Tracking Extract-Transform-Load

COMPETING TERMS

Data Version Control Pachyderm

KNOWLEDGE AREAS

Data Engineering Platform & Tooling Data Storage & Operations

TARGET AUDIENCE

Data Engineer MLOps Engineer AI Engineer Analytics Engineer Data Architect

EXTERNAL SOURCES

• Wikipedia - LakeFS
• lakeFS Documentation - Repository concepts in lakeFS
• lakeFS Documentation - Spark integration guide

TOP QUESTIONS

Why adopt lakeFS for data versioning?

It enables reproducible pipelines and safe experimentation with Git-like workflows on data lakes.

Category: Benefits

How does lakeFS compare to DVC?

lakeFS focuses on object-store datasets and branchable repositories, while DVC integrates tightly with Git for ML pipelines.

Category: Comparison

What storage systems does lakeFS support?

It supports Amazon S3, Google Cloud Storage, and Azure Blob Storage via native integrations.

Category: Capabilities

How can lakeFS improve data quality workflows?

Create branches for remediation, validate changes, and merge only clean snapshots into production.

Category: Best practices

What are common mistakes with lakeFS?

Ignoring branch isolation and merge policies can lead to inconsistent datasets across environments.

Category: Mistakes

What risks exist when using lakeFS?

Poor governance over branches and retention may inflate storage costs or expose stale data.

Category: Risks

How does lakeFS integrate with Spark?

It offers connectors and catalogs allowing Spark jobs to read/write against branch-specific paths and commits.

Category: Integration

Can lakeFS be used for ML experiment tracking?

Yes, branch datasets per experiment, run training, and merge successful results to production.

Category: Use cases

Where should you start with lakeFS?

Try the quickstart with an S3-backed repository and integrate with a small Spark job.

Category: Getting started