L-Mds

Cost efficient alternative to databricks lock-in
Spark-based data PaaS solutions are convenient. But they come with their own set of challenges such as a high vendor lock-in and obscured costs. We show how to use a dedicated orchestrator (dagster-pipes). It can not only make Databricks an implementation detail but also save cost. Also, it improves developer productivity. It allows you to take back control.
Sep 2, 2024 12:00 AM — Sep 12, 2024 12:45 PM
Georg Heiler
Slides
Cost efficient alternative to databricks lock-in
Cost efficient alternative to databricks lock-in
Save money 💰 and increase developer productivity 👩‍💻👨‍💻 by limiting scope-creep of Spark-based data PaaS solutions: 🌐 turn them into an implementation detail 🔧.
Georg Heiler, Hernan Picatto
Jun 21, 2024 22 min read
Cost efficient alternative to databricks lock-in
Dagster, dbt, duckdb as new local MDS
Lean and efficient MDS experience: Delivers better software engineering practices to the data ecosystem with the new local MDS stack comprised of Dagster, dbt and DuckDB which offers better developer productivity by enhancing testability of the E2E pipeline.
Aleksandar Milicevic, Georg Heiler
Dec 11, 2023 19 min read
Dagster, dbt, duckdb as new local MDS