Cost efficient alternative to databricks lock-in
Spark-based data PaaS solutions are convenient. But they come with their own set of challenges such as a high vendor lock-in and obscured costs. We show how to use a dedicated orchestrator ([dagster-pipes](https://docs.dagster.io/guides/dagster-pipes)). It can not only make Databricks an implementation detail but also save cost. Also, it improves developer productivity. It allows you to take back control.
Sep 12, 2024
Cloud arbitrage for spark pipelines
Spark-based data PaaS solutions are convenient. But they come with their own set of challenges such as a high vendor lock-in and obscured costs. We show how to use a dedicated orchestrator ([dagster-pipes](https://docs.dagster.io/guides/dagster-pipes)). It can not only make Databricks an implementation detail but also save cost. Also, it improves developer productivity. It allows you to take back control.
Jun 21, 2024
Introduction to Geostatistics
Nov 10, 2023
Data Engineering in the DBT ecosystem
Resolving the pain of the distributed data monolith with governance and orchestration. Code examples of a similar previous talk: https://github.com/geoHeil/dataengineering-meetup-vienna-2023-05 Date: 2023-09-06
Sep 6, 2023
Modern data stack in the enterprise
Resolving the pain of the distributed data monolith with governance and orchestration. Code examples: https://github.com/geoHeil/dataengineering-meetup-vienna-2023-05 Date: 2023-05-10
May 10, 2023
Governance and pipelines in the modern data stack
The data orchestrator is at the heart of the data pipelines. We start by exploring how a modern data orchestrator drastically eases the development of pipelines. Then we will see how govanance can be conducted efficiently in a MDS-based setup.
Dec 8, 2022
Efficient Temporal Graph Analytics
Oct 19, 2022
AI basierte Root Cause Analyse von CPD Störquellen in Docsis Netzen
Good quality network connectivity is ever more important. For hybrid fiber coaxial (HFC) networks, searching for upstream \emph{high noise} in the past was cumbersome and time-consuming. Even with machine learning due to the heterogeneity of the network and its topological structure, the task remains challenging. We present the automation of a simple business rule (largest change of a specific value) and compare its performance with state-of-the-art machine-learning methods and conclude that the precision@1 can be improved by 2.3 times. As it is best when a fault does not occur in the first place, we secondly evaluate multiple approaches to forecast network faults, which would allow performing predictive maintenance on the network.
May 10, 2022
Orchestrating data in the mesh of the fragmented modern data stack
The fragmented modern data stack has emerged as the unbundling of Airflow. Various tools operate in silos. Dagster as a next-generation data orchestrator allows you to clearly see the data dependencies of the individual pipelines on your data factory floor.
Apr 27, 2022
Mobility analytics
Nov 16, 2020