dagster

From the distributed monolith of CRON jobs to lazy coupling

The data orchestrator is at the heart of the data pipelines. We start by exploring how a modern data orchestrator drastically eases the development of pipelines. Then we will see how govanance can be conducted efficiently in a MDS-based setup.

The fragmented modern data stack has emerged as the unbundling of Airflow. Various tools operate in silos. Dagster as a next-generation data orchestrator allows you to clearly see the data dependencies of the individual pipelines on your data factory floor.

Comparing established and up-and-coming streaming approaches for an integrated real-time data model

Way too many data pipelines still work with SFTP file transfer. Even a modern data orchestrator needs to interface here well.

Simplify data ingestion with the plentiful connectors of Airbyte without compromising on data lineage

Getting started with simple dagster pipelines.

Include jupyter notebooks into reliable data pipelines.

A full example E2E

Flip the data pipeline to get a better notion about the things we actually care about: data assets not transformations.