Deep Dive

Terraform vs Asset Bundles for Databricks Workflows

Sharing the experience of moving from Terraform to Asset Bundle for Databricks workflow deployment, covering challenges and benefits.

Oct 28, 2024 •

Challenges: From Databricks to Open Source Spark & Delta

Sharing the challenges to save hours when doing migration from Databricks to Open Source.

Sep 25, 2024 •

Data Modelling Using Complex Data Types

Complex data types like struct, array, map in modern warehouses are game changer, learn the useful aspects from a Data Engineer.

Jul 6, 2024 •

Messaging Systems: Queue Based vs Log Based

Learn the key differences and important properties of queue and log based messaging systems.

Jun 15, 2024 •

Handling Duplicates In Streaming Pipeline

Three ways to handle duplicate data in streaming pipelines. Learn the benefits, use cases and more in this article.

May 22, 2024 •

Data Pipeline - Incremental vs Full Load

Learn the pros, cons and use cases about the data pipeline design patterns, full load and incremental commonly used across the industry.

Apr 13, 2024 •

Data Processing in 21st Century

Timeline of Data Processing technologies covering from MapReduce to Polars.

Mar 20, 2024 •

How I saved $70k a month in BigQuery

Learn the simple yet powerful optimization techniques that helped me reduce BigQuery spend by $70,000 a month.

Mar 9, 2024 •

My Data Pipeline Orchestrators Journey

Sharing my journey of different data pipeline orchestrators used for large scale batch pipelines, discussing migrations, reasoning and much more.

Feb 10, 2024 •

5 Real Time Pipeline Architecture on Cloud

Data Engineering Streaming Pipeline Architectures from my experiences working in multiple companies leveraging AWS, GCP and open source services.

Jan 10, 2024 •

Testing Data in Apache Spark

How to efficiently test large scale Spark jobs, using open source libraries.

Sep 9, 2023 •

Real Time CDC Pipeline with AWS DMS and Snowflake

Build a real time CDC pipeline using DMS and Snowpipe using Terraform.

Aug 12, 2023 •

#nojs-banner { position: fixed; bottom: 0; left: 0; padding: 16px 16px 16px 32px; width: 100%; box-sizing: border-box; background: red; color: white; font-family: -apple-system, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 13px; line-height: 13px; } #nojs-banner a { color: inherit; text-decoration: underline; } This site requires JavaScript to run correctly. Please turn on JavaScript or unblock scripts