Posts

Local data stack template
Local data stack template

Jumpstart your data processing with this local modern data stack template

Oct 25, 2024

Cost efficient alternative to databricks lock-in
Cost efficient alternative to databricks lock-in

Save money 💰 and increase developer productivity 👩‍💻👨‍💻 by limiting scope-creep of Spark-based data PaaS solutions: 🌐 turn them into an implementation detail 🔧.

Jun 21, 2024

Dagster, dbt, duckdb as new local MDS
Dagster, dbt, duckdb as new local MDS

Lean and efficient MDS experience: Delivers better software engineering practices to the data ecosystem with the new local MDS stack comprised of Dagster, dbt and DuckDB which offers better developer productivity by enhancing testability of the E2E pipeline.

Dec 11, 2023

Securing Secrets with Mozilla SopS and AGE: A Powerful Combo
Securing Secrets with Mozilla SopS and AGE: A Powerful Combo

🔐 Exploring the power duo of Mozilla's sops & AGE for secret management! Dive deep into their benefits: simplicity, version control compatibility & robust encryption. Secure your data the modern way! 💻🛡️ #ITSecurity #Encryption #DataProtection

Dec 1, 2023

Unlocking Advanced Metadata Extraction with the New DBT API in Dagster
Unlocking Advanced Metadata Extraction with the New DBT API in Dagster

📊 Unleash the power of metadata extraction in your data engineering pipelines with the new DBT API in Dagster! 🚀 Learn how to seamlessly integrate and leverage DBT transformations, while enriching your data catalog with advanced metadata. Elevate your data governance and collaboration to new heights!

Jun 13, 2023

Making BigData small again (and green)
Making BigData small again (and green)

Towards simpler and perhaps more energy efficient data platforms with increased developer productivity.

Apr 2, 2022

Comparing SQL-based streaming approaches
Comparing SQL-based streaming approaches

Comparing established and up-and-coming streaming approaches for an integrated real-time data model

Apr 1, 2022

SFTP sensor
SFTP sensor

Way too many data pipelines still work with SFTP file transfer. Even a modern data orchestrator needs to interface here well.

Mar 4, 2022

Connector goodness from Airbyte E2E lineage
Connector goodness from Airbyte E2E lineage

Simplify data ingestion with the plentiful connectors of Airbyte without compromising on data lineage

Mar 4, 2022

Scalable data pipelines from dagster with pyspark
Scalable data pipelines from dagster with pyspark

Getting started with simple dagster pipelines.

Mar 4, 2022