python

📊 Unleash the power of metadata extraction in your data engineering pipelines with the new DBT API in Dagster! 🚀 Learn how to seamlessly integrate and leverage DBT transformations, while enriching your data catalog with advanced metadata. Elevate your data governance and collaboration to new heights!

The data orchestrator is at the heart of the data pipelines. We start by exploring how a modern data orchestrator drastically eases the development of pipelines. Then we will see how govanance can be conducted efficiently in a MDS-based setup.

Good quality network connectivity is ever more important. For hybrid fiber coaxial (HFC) networks, searching for upstream \emph{high noise} in the past was cumbersome and time-consuming. Even with machine learning due to the heterogeneity of the network and its topological structure, the task remains challenging. We present the automation of a simple business rule (largest change of a specific value) and compare its performance with state-of-the-art machine-learning methods and conclude that the precision@1 can be improved by 2.3 times. As it is best when a fault does not occur in the first place, we secondly evaluate multiple approaches to forecast network faults, which would allow performing predictive maintenance on the network.

The fragmented modern data stack has emerged as the unbundling of Airflow. Various tools operate in silos. Dagster as a next-generation data orchestrator allows you to clearly see the data dependencies of the individual pipelines on your data factory floor.

Towards simpler and perhaps more energy efficient data platforms with increased developer productivity.

Comparing established and up-and-coming streaming approaches for an integrated real-time data model

Good quality network connectivity is ever more important. For hybrid fiber coaxial (HFC) networks, searching for upstream high noise in …

Way too many data pipelines still work with SFTP file transfer. Even a modern data orchestrator needs to interface here well.

Simplify data ingestion with the plentiful connectors of Airbyte without compromising on data lineage

Getting started with simple dagster pipelines.