Metrics

• 28.7M active riders in Q3 2025, completing ~2.7M rides per day.

• Apache Kafka processes millions of real-time events per second for streaming analytics.

• Thousands of Airflow + Flyte pipelines orchestrate ETL and ML workflows.

• Data warehouse exceeds 100+ PB stored in S3 with Hive Metastore.

• Trino ETL executes ~250K queries/day, reading ~10 PB/day and writing ~100 TB/day.

Content is based on multiple sources including Lyft Blog, AWS Blog and other public articles etc. You will find references to dive deep as you read.

Platform

AWS

Lyft’s data and infrastructure are fully hosted on AWS, leveraging managed services and elastic scaling to support real-time transportation and logistics workloads. AWS powers everything from API services to large-scale data analytics, enabling Lyft to handle millions of rider-driver interactions efficiently.

📖Further Reading: Lyft Case Study

Messaging System

Kafka

Lyft adopted Kafka relatively recently to address scaling challenges, as explained in the video below. Kafka facilitates real-time data streaming, processing millions of events per minute, and supports various use cases such as trip updates, driver location tracking, and telemetry ingestion.

Kinesis

Alongside Kafka, Lyft also leverages AWS Kinesis which was introduced at Lyft long before Kafka. There is not enough public information about the plans if Kinesis will be replaced by Kafka.

Processing

Flink & Beam

Lyft adopted Apache Flink as the core real time and streaming engine to support wide variety of use cases. They leverage Apache Beam on top of the Flink runner due to portability and multi language capabilities.

Below is the video from the summit where you can learn more on how these tools are used at Lyft.

Spark

Lyft leverages Spark mostly on the Machine Learning side, e.g. LyftLearn, a ML platform which streamline Spark-on-Kubernetes for improved resource allocation, SQL adaptability, and integration with ML libraries. This seamlessly integrates with the orchestration tool Flyte.

📖 Recommended Reading: Spark At Lyft

Trino

Lyft uses Trino it to power large-scale ETL workloads that read 10 PB and write 100 TB of data daily from its Hive warehouse. Trino also serves as a fast, user-friendly query layer for teams, with over 90% of queries completing in under three minutes.

📖 Recommended Reading: Trino for large scale ETL at Lyft

Orchestrator

Airflow & Flyte

Lyft provides two orchestration platforms Airflow and Flyte, each having its pros and cons. Historically, Airflow has been widely used across Lyft for building data pipelines, while Lyft developed Flyte focusing on high intensive tasks like Machine Learning.

📖 Read more: Comparing Flyte and Airflow at Lyft

Warehouse

S3 & Hive

Lyft’s core data warehouse containing 100s of petabytes runs on top of Amazon S3, with Hive used as the metastore. While Lyft hasn’t publicly stated using modern open table formats like Iceberg or Delta, their S3 + Hive setup still powers large-scale historical storage.

Catalog

Amundsen

Developed in-house and open-sourced, Amundsen is Lyft’s metadata and data discovery platform. It provides a centralized catalog for datasets, dashboards, and pipelines, integrating with Hive Metastore, Trino, and Airflow to surface ownership, lineage, and documentation. Amundsen has since become a leading open metadata framework adopted across the industry.

📖 Learn more: Amundsen: Lyft’s Data Discovery Engine

Data Store

ClickHouse

Lyft faced key challenges like data deduplication using Apache Druid, which prompted their migration to ClickHouse. This switch not only resolved those issues but also delivered improved performance and operational efficiency for real-time analytics.

📖 Recommended Reading: Druid Deprecation and ClickHouse Adoption at Lyft

Dashboard

Superset

For visualization, Lyft primarily uses Apache Superset, which connects to Trino as its query engine, to deliver both analytics and operational dashboards.

As per this article from 2019, Lyft supported multiple dashboarding tools, there is no recent public information whether the tools are consolidated or still supported.

Related Content:

💬 Overall, Lyft blends AWS with cloud-native and open-source systems to support massive real-time data processing, analytics, and machine learning across its platform.

How does it work

Traditionally, if you want to move data from one format to another, you need to read as source format and write as target format, which generates a copy of data while consuming resources. With Uniform, interoperability is possible as all the open table formats are built on top of Parquet files, the difference is how they store metadata.

💡One data copy with multiple format support through metadata files.

UniForm is part of Delta Lake so it only supports Delta to Iceberg/Hudi conversion. To enable Uniform for Iceberg on a Delta Table:

CREATE TABLE main.default.UniForm_demo_table (msg STRING)
TBLPROPERTIES(’delta.universalFormat.enabledFormats’ = ‘iceberg’);

Once the property is enabled, it generates and incrementally maintains the metadata files for the target version.

📖 Recommended Reading: How Delta Lake Works

Since source table is Delta, it supports Delta optimizations which also reflect in the target metadata files to give similar read performance. However, target specific optimizations are not possible today.

Write Flow

• User submits a write query with Iceberg enabled through UniForm.

• Native Delta Table logs and data are updated. Read detailed Delta write flow here.

• UniForm translates logical actions (insert/update/delete/merge) into target format commits.

• Target metadata is incrementally & asynchronously updated i.e. Iceberg metadata files. This makes sure Iceberg metadata is always in sync giving a unified view of the same data.

Impact: Slight overhead due to metadata translation which could degrade write performance upto 5%.

Read Flow

Read is same as like reading a native table as shared for Delta Table here.

• User submits a query to read from an Iceberg Table.

• Query fetches the required metadata, in our case Iceberg from the below options.

  • Delta → _delta_log/

  • Iceberg → metadata/

• Construct the table snapshot and read the Parquet Data files using the native reader. Note: These are the same files that were generated by Delta during write operation.

• Results returned via the compute engine to the user.

Impact: The table is read directly using Iceberg metadata so there is no impact on read performance.

Use Case

A simple scenario for using UniForm is when other systems require your Delta table to be accessible in Iceberg or Hudi format. In such cases, UniForm can seamlessly generate the necessary metadata files with almost negligible impact on performance.

Below is the architecture showing where does UniForm fits in the picture.

In the above image, Databricks generates Delta Table with Iceberg enabled through Uniform. Delta is for Databricks and Iceberg is for Snowflake.

Alternative

Apache XTable is another option that supports interoperability across all major table formats, enabling conversions from any source to any target. In contrast, Delta UniForm only supports Delta as the source format.

📖 UniForm initially supported interoperability from Delta to Iceberg. To extend this capability to Hudi, Delta UniForm partnered with XTable.

Conclusion

By bridging the gap between modern lakehouse formats, Delta UniForm simplifies data sharing and format compatibility across platforms, positioning itself as a key enabler of open, multi-engine lakehouse interoperability alongside emerging alternatives like Apache XTable.

References

• Open Source Delta UniForm: https://delta.io/blog/unifying-open-table/

• Databricks Delta UniForm: https://www.databricks.com/blog/delta-uniform-universal-format-lakehouse-interoperability

• AWS Delta to Iceberg through UniForm: https://aws.amazon.com/blogs/big-data/expand-data-access-through-apache-iceberg-using-delta-lake-uniform-on-aws/

• Apache XTable overview: https://xtable.apache.org/

Metrics

• 120+ million verified users worldwide, source.

• 8.7+ million monthly transacting users (MTU), source.

• $400+ billion in assets under custody, source.

• Billions of events processed daily across user activity, blockchain data, and market feeds.

• 30 Kafka brokers with ~17TB storage per broker.

Content is based on multiple sources including Coinbase Blog, AWS Blog and other public articles etc. You will find references to dive deep as you read.

Platform

AWS

Coinbase leverages several AWS cloud services to solve their complex large scale challenges. The company also partnered with AWS to modernize and optimize its cloud infrastructure, migrating legacy workloads to Amazon EC2 instances powered by AWS Graviton processors and adopting Amazon EKS for automated scaling and resource management. As a result of this partnership, they have saved cost by roughly 62% and infra scaling time by 50%.

📖 Read More: Coinbase Boosts Efficiency and Accelerates Development by Collaborating with AWS

Messaging System

Kafka

For centralized messaging service, Coinbase uses Kafka through AWS managed offering, known as Managed Streaming for Apache Kafka (MSK). Kafka ingests billions of events everyday from user actions, applications, crypto feeds, and database change data capture (CDC).

With MSK, Coinbase reduced operational burden, achieved very low end-to-end latency (<10ms) for many pipelines (versus ~200ms with previous systems), improved reliability across AZs, and made scaling more seamless.

It takes few steps to provision a new MSK cluster.

📖Recommended Reading: How we scaled data streaming at Coinbase using AWS MSK

Processing

Spark (SOON)

Coinbase built SOON (Spark cOntinuOus iNgestion) on Databricks to replace slow, siloed Airflow <> Kafka <> Snowflake ETLs with a unified, low-latency streaming framework. Using Spark Structured Streaming and Delta Lake, SOON supports both append-only and merge (upsert/delete) ingestion, enabling scalable real-time data processing.

They also use Spark outside of SOON framework for batch processing.

📖 More on SOON: Spark cOntinuOus iNgestion for near real-time data

Orchestrator

Airflow

Coinbase adopted Airflow in 2017 when it was still gaining popularity. They made Airflow as their centralize orchestrator for data pipelines used by hundreds of data engineers and scientists.

With their adoption of Databricks, they are most likely leveraging Databricks Workflows, however there is no public information available.

📖 Recommended Reading: Revamping the Apache Airflow

Warehouse

Snowflake

Coinbase is also a customer of Snowflake, they have migrated the real time pipelines to Databricks, but other workflows still rely heavily on Snowflake as per this source. Furthermore, their BI team that leverages Looker which is most likely fetching data from Snowflake.

Lakehouse

Delta Lake

Delta is used through Databricks as their open table format. One of the usecase is for the Streaming pipeline that is built using SOON on Databricks, see image in Spark section.

S3

S3 is the object storage under Delta, however this is managed through Databricks. It is also used to dump full data snapshots from various Databases like PostgresSQL and DynamoDB.

Data Store

StarRocks

Coinbase uses StarRocks via CelerData to enable real-time analytics directly on their data lakehouse, avoiding complex ETL. This setup delivers sub-second query latency, supports high concurrency, and scales with growing data volumes, improving performance for analytics workloads.

Dashboard

Looker

As per this source from 2022, they onboarded Looker as their Business Intelligence (BI) Platform mainly due to its technical capabilities.

Related Content:

💬 Coinbase is a modern tech company that heavily relies on commercial and cloud offerings such as MSK, Databricks, Snowflake, and CelerData, reflecting a tech culture that prefers buying solutions rather than building them in-house.

To recap: the series follows a Q&A format, featuring professionals who share their journeys, insights, and challenges.

What to Expect:

• Practical insights – Get a clear view of what data engineers do in their day-to-day work.

• Emerging trends – Stay informed about new technologies and evolving best practices.

• Real-world challenges – Understand the obstacles data engineers face and how they overcome them.

• Myth-busting – Uncover common misconceptions about data engineering and its true impact.

Let’s dive into Inside Data Engineering:

How would you describe Data Engineering?

Data engineering is about understanding what different teams need from data, aligning on definitions, and providing the systems and infrastructure that address those needs. It’s a mix of technical and social skills, because good data engineering often comes down to;

• Clear communication

• Effective collaboration between teams

• Near-optimal architectural choices

• Systems people can trust

How did you end up being a Data Engineer?

I started out in Industrial Engineering, exploring different paths through internships. While Industrial Engineering didn’t excite me much, I got into data mining and machine learning during grad school, which led to my first role as a Data Scientist. I was the third person to join an AI startup, and without a dedicated Data Engineer, Architect, or Cloud Engineer, I had to build the entire data foundation myself. That experience made me realise I enjoyed the Data Engineering parts the most. Therefore, I began a PhD on distributed stream processing, and that marked my full transition into Data Engineering almost 3 years ago.

What's your day-to-day look like?

My average day-to-day probably looks like this:

• 20% Meetings

• 60% Development/Coding

• 10% Documentation (Aiming to make this 20% and Coding 50%)

• 10% Learning/Reading

It’s hard to put it into a fixed structure, since something unexpected almost always pops up, but I think these percentages reflect the general distribution quite accurately.

What are some stakeholders that you work with?

I work with a full range of stakeholders, it’s really the full package. I work with:

• Analytics

• Product teams

• AI/ML teams

• HR

• C-suite.

What kind of projects do you work on?

The projects I work on vary quite a bit. Most frequently I build data pipelines that ingest data from APIs or databases, transforming and modelling it in the semantic layer, and orchestrating the entire process with workflow tools.

I often dive deep into semantic modelling to create metrics that meet criteria of different domains. From time to time, I also build dashboards on top of the pipelines I’ve created to support stakeholders directly.

On top of that, I handle DevOps-related tasks like implementing CI checks to maintain standards and manage our cloud infrastructure using Infrastructure as Code.

What kind of data do you work with?

I mostly work with tabular data and event data. Tabular data typically comes from APIs or transactional databases and event data from streaming tools which captures frontend or backend events.

What data size do you work with?

It is relatively small. I’d say a couple TBs the most.

What tech stack do you use?

What tools do you leverage for GenAI?

I mostly use ChatGPT and Claude for brainstorming and sometimes quick prototyping. I also use Warp, an AI-powered terminal, which I think is a great productivity boost for data engineers who spend a lot of time in the shell.

What is your favorite area of Data Engineering?

I’m not sure I have a single favourite area, but what I really enjoy about Data Engineering is the mixture it offers of learning, research, and engineering. For someone like me who loves deep reading and also building real systems to test and implement new ideas, it’s the perfect mix.

What is the next big thing according to you in Data Engineering?

I’d say a “real” self-serve analytics layer. Most tools today that claim to offer self-serve analytics still require a lot of dependency on data teams. But with the emergence of semantic models and the integration of GenAI, I believe we’re getting closer.

What advice would you give your past self as a beginner Data Engineer?

• Don’t hesitate to ask more questions!

• Listen to advice, but always think critically and filter what aligns with your goals and understanding.

• No one knows everything.

• Optimal is the enemy of good.

What are some challenging aspects of Data Engineering?

The first challenge I’d point out is how overwhelming the field can be for beginners. There are so many concepts, tools, and stakeholder dynamics involved that people can get stuck just trying to figure out where to start.

The second one is that many companies still lack data-literate managers or executives. This can lead to unrealistic expectations, poor prioritisation, and unnecessary pressure on data engineers.

I hope this article was helpful for the readers. Thanks to Hasan for sharing his experience with my audience. Stay tuned for more!

Please reach out if you like:

• To be the guest and share your experiences & journey.

• To provide feedback and suggestions on how we can improve the quality of questions.

• To suggest guests for the future articles.

This article dives into what the small file problem is, why it happens, and how to solve it; with strategies that can deliver up to 100x faster reads especially for streaming jobs.

💡1000 files → compacted to 10 files will give 100x faster reads.

What Is the Small File Problem?

When Spark writes data, it does so in parallel. Each task typically outputs one file. With high parallelism (e.g., 100 tasks), you often end up with hundreds of small files rather than fewer large ones.

These small files make writes fast (parallelized, no need to coordinate much), but reads slow, because:

1. Spark has to open and scan thousands of files.

2. Each file triggers task scheduling overhead.

3. Metadata operations dominate actual data scanning.

Why It Hurts Read and Write Performance

• Writes: Small files are easy to generate, but repartitioning to reduce them adds shuffle cost.

• Reads: File explosion causes Spark to spend more time on metadata + scheduling than actual compute

💡 Reads happen more often than writes, just like code is read far more often than it is written.

Impact on Batch and Streaming Jobs

• Batch workloads: Small files accumulate, degrading performance of downstream queries and compaction jobs.

• Streaming workloads: Streaming prioritizes low-latency writes, which naturally produce small files. If compaction is forced inline, it adds latency; often unacceptable for SLA-driven pipelines.

Traditional Solutions in Spark

Before open table formats, the common fix was to repartition/coalesce before write or reduce the number of cores/tasks when starting the job.

Before Write:

df.repartition(10).write.format("parquet").save("...")
OR
df.coalesce(10).write.format("parquet").save("...")

Setting Cores at Spark Job (Gives 10 in total tasks):

spark.executor.cores 2 
spark.executor.instances 5

• Repartition triggers a full shuffle (expensive).

• Coalesce reduces files but can cause data skew.

• With N cores, you typically get N output files (10 cores = 10 files).

In all the above scenarios, we set the file count to 10, which requires you to manually determine the right number; unlike in open table format solutions.

Open Table Format Solutions

Modern table formats provide smarter, managed ways to handle small files. Since open table formats support versioning, they can perform these optimizations efficiently and smartly without disrupting the user experience.

Offline Optimization (Table & Partition Level)

This feature is commonly known as compaction which periodically merges many small files into fewer large ones per table or partition.

• This is asynchronous process and conceptually similar to coalesce; fewer output files but efficient. It makes data available asap but makes it read efficient later.

• Example: Merge 100 files into 10 larger ones. Number of files depend on the file size which can be configured through spark config if needed.

• Still requires scanning files, so the more small files you start with, the more expensive compaction becomes.

• Great for batch data where compaction can run right after the job is done.

Online Optimization (Task Level)

Instead of blindly writing one file per task, Spark can group writes more intelligently before write. Delta calls it optimized write.

• This is synchronous process and conceptually similar to coalesce; fewer output files but efficient. It adds latency but once data is available its read efficient.

• Example: Instead of 100 small files, Spark writes 50 medium-sized files.

• Makes later compaction faster, since it only needs to process 50 files instead of 100.

• Streaming jobs that writes small files can take advantages of this along with compaction when the SLA permits.

Decision Flow

Conclusion

The small file problem is a silent killer of Spark performance. While small files make writes faster, they heavily penalize reads. Open table formats like Delta, Iceberg, and Hudi provide optimization features that strike the right balance. Solving this problem can unlock up to a 100x boost in read performance, crucial for truly scalable Lakehouse analytics.

Metrics

• 284 million peak requests per minute to Shopify’s app servers during peak sales events, powering $11.5billion in sales over four days in 2024, supported by 99.9% uptime.

• $4.6million in peak sales per minute and up to 967,000 requests per second during Black Friday and Cyber Monday.

• Shopify’s infrastructure is 100% powered by Google Cloud, leveraging the same dependable network as Gmail, Search, and YouTube.

• Kafka has handled 66 million messages per second at peak.

• 76k Spark jobs with 300 TB processed per day.

• Airflow has 10,000 DAGs with 400 tasks running at a given moment and over 150,000 runs executed per day.

• DBT has 100+ models with 400+ unit tests running on average under ~3min.

Content is based on multiple sources including Shopify Blog, GCP Blog and other public articles etc. You will find references to dive deep as you read.

Platform

Google Cloud Platform

Shopify operates fully on Google Cloud Platform, benefiting from GCP’s reliability, scalability, and security. The partnership enables Shopify to deliver seamless merchant experiences during surges like BFCM and holiday sales, with near-instant scaling to accommodate billions of app server requests. GCP’s advanced infrastructure is integral for Shopify’s uptime, performance, and integration of AI-powered solutions

📖Integration details: Shopify Partner Spotlight: Google Cloud

Messaging System

Kafka

Kafka is the backbone for Shopify’s messaging, providing real-time streaming and event-driven architecture vital for ecommerce scale. It supports order processing, inventory updates, and notifications, enabling high availability and reliable communications across distributed systems.

📖More on tech choices: Shopify Tech Stack - ByteByteGo

Processing

Beam/DataFlow

Shopify leverages Apache Beam running on DataFlow to orchestrate large data processing jobs in both batch and streaming modes. This enables scalable ETL workflows and near real-time analytics for customer events and operational metrics.

📖 Recommended Reading: How Shopify Improved Consumer Search Intent with ML

Spark (Starscream)

Shopify built Starscream, an internal PySpark-based data pipeline platform, to efficiently run tens of thousands of jobs processing hundreds of terabytes of data daily. It abstracts common complex patterns into reusable components.

It was designed to accelerate translation from SQL prototypes to scalable PySpark jobs, empowering teams to build robust data models quickly and at scale.

📖 More on Starscream: The Complex Data Models Behind Shopify's Tax Insights Feature

Trino

Trino serves as a query engine for interactive analytics, federating data across storage systems such as BigQuery. It excels at large-scale SQL workloads, offering low-latency access to Lakehouse for analysts.

📖 Read More: Shopify's Path to a Faster Trino Query Execution: Infrastructure

Orchestrator

Airflow

Shopify’s adoption of Apache Airflow delivers robust DAG-driven automation for ETL, model training, and data pipeline management. The team has shared scaling lessons and customizations that support thousands of workflows and seamless production deployments, essential for complex scheduling requirements as data volumes grow.

📖 Recommended Reading: Lessons Learned Scaling Apache Airflow

Warehouse

BigQuery

BigQuery is Shopify’s central data warehouse, powering analytics, reporting, and dashboarding for business teams and merchants. Its seamless scalability and rapid-query response underpin key features such as real-time sales insights, optimization reports, and recommendation engines, with production-grade data managed via DBT.

🎙️Recommended Podcast: Data Warehouse at Shopify

Transformation

DBT

Shopify relies heavily on DBT for scaling their transformation and unit testing used by 200+ Data Scientists across the company.

Along with Starscream, they also have a tool called Seamster, that allows users to skip the conversion to PySpark and build scalable pipelines using DBT and BigQuery.

📖More on Seamster: Production Grade Workflow with SQL Modelling

Below is a detailed video on how they perform unit testing at scale.

Lakehouse

Google Cloud Storage & Iceberg

Raw and staged data lands in GCS, forming the foundation for Shopify’s lake architecture. GCS works seamlessly with Iceberg to provide a Lakehouse architecture that is easily accessible by hundreds of downstream users and consumers.

🎥Recommended Video: Iceberg at Shopify

Data Store

Druid

Druid is adopted for high-performance, low-latency OLAP analytics. It serves operational dashboards and provides fast filtering, drill-down capabilities, and aggregated metrics for merchants and internal teams, supporting millions of queries daily, read more at Apache Druid.

Dashboard

Polaris-Viz

Polaris-viz is Shopify’s custom React visualization library, designed initially for both internal and external dashboards. While open source is deprecated but internally is still widely adopted.

📖 Background: React Library for Consistent Visualization

Related Content:

💬 Shopify’s tech stack relies on GCP and other core technologies for building a modern scalable architecture that can scale well under extreme demands for both online and offline processes.

To recap: the series follows a Q&A format, featuring professionals who share their journeys, insights, and challenges.

What to Expect:

• Behind the Scenes – Get a close-up view of the real work, rhythms, and responsibilities of data engineers in action.

• Getting Started – Dive into the essential skills, tools, and entry points that open doors to a data engineering career.

• Industry Watch – Stay informed on emerging trends, evolving tech stacks, and shifts driving the future of data engineering.

• The Real Work – Go beyond the theory to explore the gritty, unexpected challenges engineers solve in the wild.

• Debunking the Hype – Clear up common myths and misconceptions about what data engineers actually do.

• From the Trenches – Learn from the experiences, lessons, and advice of seasoned professionals working in the field.

Let’s dive into Inside Data Engineering:

How would you describe Data Engineering?

In one sentence, data engineering means making data accessible for stakeholders to create business value. Although it sounds simple, there are many factors involved: why the data matters, what data is needed, and how it should be handled. The how is especially important, which is why we need to understand how to design systems, architect solutions, and model data in a way that maximises business value while keeping costs optimised.

How did you end up being a Data Engineer?

I have a Bachelor's degree in IT Engineering and worked for about two years as a programmer. Later, I decided to pursue a Master's in Business Analytics with a specialisation in Data Science. After graduating, I started working as a Data Analyst. One of the companies I worked for needed me to maintain and improve their existing data pipelines and implement a data quality framework to address data quality challenges, that’s what sparked my interest. I realised that the work I was doing closely resembled that of a Data Engineer, and it motivated me to dive deeper. I started learning from the data engineers in the company, observing how they source data, the technologies they use, and even asking to explore their code repositories. Over time, I kept learning more and gradually began contributing like a Data Engineer.

Here you can read the full story of My Journey from Data Analyst to Data Engineer, along with my advice and recommendations for others looking to make the same transition. Also, I highly recommend checking out the below article:

Pipeline To Insights

How to Transition from Data Analytics to Data Engineering

In our previous post, one of the co-founders of Pipeline To Insights shared his reason for transitioning into data engineering. We also conducted a poll, and based on the results, we've decided to write this post specifically for data analysts and professionals in similar roles looking to switch to data engineering…

Read more

a year ago · 19 likes · 2 comments · Erfan Hesami

What's your day-to-day look like?

My day-to-day typically involves gathering requirements to understand business problems, identifying whether the necessary data sources exist, and determining if any new automation is needed. I also maintain and improve existing pipelines and help address issues related to data quality, governance, or infrastructure. I regularly attend meetings with different stakeholders to understand their needs and explore how our team can support them.

Who do you typically work with across teams?

• Software Engineers: We work together to understand how internal/external applications function and explore ways to improve data capture, especially for storing historical data and enhancing data quality at the source. I also support them in identifying what information needs to be collected to solve specific business problems and assist with data modelling for transactional databases.

• Vendors: External data engineers or teams who share data with us from outside the company. I collaborate with them to ensure the data is reliable, well-documented, and integrated smoothly into our systems.

• Engineering Executives: I consult with them on architectural decisions, technology choices, and long-term solutions to align with strategic goals.

• BI Developers: I collaborate with them to provide the necessary data in the right format and at the right frequency for reporting and analytics.

• ML/AI Engineers: I work with them to understand what data they need, in what format, and how often. Their focus is on building machine learning models and AI applications to solve business problems.

What real-world business problems do you solve through data?

One of the projects I'm most proud of was creating a 360-degree view of customer data as a single source of truth. Previously, data was scattered across different systems, and teams were generating reports based on inconsistent sources. There was no clear data dictionary, no data lineage, and very limited governance; many people didn’t even know where certain data lived. I took the initiative to centralise and model this data, creating a unified customer view that could be used company-wide. This helped teams build their own data marts more confidently and consistently.

What kind of projects do you work on?

I work on building and optimising data infrastructure, ensuring reliable data pipelines and high data quality, and designing systems that support analytics and AI. This includes integrating data from various sources, managing complex transformations, and improving data accessibility for business teams. Some examples include sourcing and modelling data for an AI Copilot project to support account managers, and helping stakeholders detect unprofitable contracts so they can take timely action.

What kind of data do you work with?

I work with both structured and semi-structured data, including traditional fields and columns, as well as open text.

What data size do you work with?

I have experience working with data at terabyte and even petabyte scale, though in my current role, the data volume is typically a few terabytes.

What tech stack do you use?

• Azure: All open-source components are hosted and managed within the Azure cloud environment.

• Azure storage: Used to store raw and processed data for analytics and AI use cases.

• Postgres: Used as a storage layer for analytical purposes.

• Data load tool (dlt): Used to ingest data from various sources efficiently.

• Dagster: Automate and schedule data workflows

• Data build tool (dbt): Used for transforming and modelling data.

• Great Expectations: Used for data quality.

What programming languages do you use?

Python and SQL are primary languages for transformation, tooling, etc.

What tools do you leverage for GenAI?

I use Claude for programming assistance, as well as Ollama and Llama models for running and experimenting with generative AI locally.

What is your favorite area of Data Engineering?

My favourite areas of data engineering are data quality and data modelling because they are fundamental to building reliable, trustworthy, and scalable data systems.

How can Data Engineering benefit from GenAI?

I use generative AI models to create test cases and generate documentation. Some open-source tools have model-powered documentation that allows you to quickly search and find relevant information, making development and troubleshooting much more efficient.

What advice would you give your past self as a beginner Data Engineer?

For beginners, I’d suggest focusing on the fundamentals first and not getting lost chasing certifications alone. Early on, I thought that collecting certifications for specific platforms would make me stand out. But I soon realised that not all companies use or require those platforms. More importantly, it’s crucial to understand how these technologies work and to explore the open-source ecosystem. Working on side projects can really help build practical skills. I’m not saying certifications aren’t valuable, but always ask yourself why you’d choose one technology over another; understanding your options is key.

What are some challenging aspects of Data Engineering?

One of the biggest challenges is making decisions across many areas, especially when it comes to architecture. It’s critical to think carefully and always consider the overall system design. Choosing the right tools can also be difficult because new tools constantly emerge, and deciding which one fits best requires experience and judgment. Additionally, there are many ways to implement solutions, and every engineer or team tends to find their own approach, which adds complexity and variability.

What are some common misconceptions about data engineering?

• Data engineering is just about creating data pipelines.

• Data engineering is only writing SQL.

• Data engineers should only focus on engineering tasks.

Reality:

• Data engineers should also understand analytics and be good analysts, especially in smaller teams.

• When stakeholders bring data problems, data engineers often analyze feasibility and provide insights.

• Creating dashboards, charts, and telling data-driven stories can also be part of the role.

• Data governance and data quality are critical responsibilities that data engineers must manage to ensure trustworthy and compliant data.

What advice do I have for new beginners?

• Don’t jump into trending technologies just for the sake of it, some tools are powerful, but not every company uses them, and not every data problem requires them.

• Focus on mastering the fundamentals first; don’t sacrifice core understanding just to follow hype.

• Learn from others by reading articles, attending meetups and conferences, and reading books. A book like Fundamentals of Data Engineering by and Matt Housley really helped me connect the dots and build a strong foundation.

• Most importantly, apply what you learn through hands-on projects to build real understanding and confidence.

I hope this article was helpful for the readers. Thanks to Erfan for sharing his experience with my audience. Stay tuned for more!

Please reach out if you like:

• To be the guest and share your experiences & journey.

• To provide feedback and suggestions on how we can improve the quality of questions.

• To suggest guests for the future articles.

Delta allows compute engines like Apache Spark, Trino, and Flink to access the same data reliably via a transaction log that records every change made to a table.

Architecture & Components

Delta Lake’s architecture centers around a small set of well-defined components:

• Parquet Data Files: The actual table data is stored in columnar Parquet format within a directory.

• _delta_log/: A transaction log directory containing a series of JSON and Parquet files that track all changes to the table. This log ensures ACID guarantees.

  • Table Metadata: Stored in the JSON files, includes schema, partitioning, and table properties (example shared later).

  • Checkpoint: Stored in the Parquet file, includes the optimized snapshot of table state.

  • Commit Protocol: Ensures atomic commits using optimistic concurrency control. Writers coordinate through exclusive commit file naming (e.g. 00001.json).

• Readers/Writers: Clients read consistent snapshots by replaying the log, and writers create new log entries to record changes. Each transaction creates a new JSON file.

Internal Mechanics

Write Flow

1. Plan: Read latest snapshot (if needed) to identify affected files.

2. Stage: Write new Parquet files for all changes.

3. Log: Create JSON "add" (and optional "remove"/"metadata") actions for each change.

   {
     "add": {
       "path": "part-00001-1234abcd.snappy.parquet",
       "partitionValues": {
         "year": "2024",
         "month": "06"
       },
       "size": 345678,
       "modificationTime": 1686780000000,
       "dataChange": true,
       "stats": "{\"numRecords\": 50000, \"minValues\": {\"id\": 1}, \"maxValues\": {\"id\": 50000}}"
     }
   }

4. Validate: Check for conflicting concurrent commits using optimistic concurrency.

5. Commit: If no conflict, write a new log file (e.g., 00000123.json) to _delta_log/.

6. Checkpoint: Periodically write Parquet checkpoints to speed up reads.

Read Flow

1. Load: Load the latest checkpoint (if available) and all subsequent JSON logs.

2. Reconstruct: Reconstruct the current table state (active files, schema, metadata).

3. Plan: Apply query filters and partition pruning based on metadata.

4. Read: Read the relevant Parquet files from storage.

5. Return: Return the final filtered and projected result to the user.

References

• Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores: https://www.vldb.org/pvldb/vol13/p3411-armbrust.pdf

• Delta Lake Docs: https://docs.delta.io/latest/

• GitHub: https://github.com/delta-io/delta

• Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics: https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf

I hope this article was helpful in understanding how the Delta Transaction log works. Please share your feedback and suggestions

Stay tuned for deep dives into Z-Ordering, Liquid Clustering, and advanced concurrency control in future posts.

Metrics

• 1.4+ trillion events processed daily.

• 670+ million monthly active users.

• 38,000+ Data Pipelines active in production environment.

• Spotify runs the largest Hadoop cluster in Europe.

• 1800+ different event types representing interactions from Spotify users.

• ~5k dashboards serving to ~6k users.

Content is based on multiple sources including Spotify Blog, Open Source websites, Job descriptions and other public articles etc. You will find references to dive deep as you read.

Platform

Google Cloud Platform (GCP)

GCP is Spotify’s core cloud provider, supporting both compute and advanced analytics. Spotify migrated from AWS in the mid-2010s to leverage GCP's scalable infrastructure, big data, and machine learning services.

📖 Recommended Reading: Spotify Case Study

Messaging System

PubSub

Spotify moved from Kafka to GCP Pubsub for ingesting their massive amount of event driven data back in 2016.

As per recent article, their data platform supports automatic deployment of PubSub, anonymization pipelines and streaming jobs.

Processing

Apache Beam

Apache Beam (GCP Dataflow) is the primary processing tool used at Spotify for handling real time and batch workloads. Spotify has their open source Scala API implementation called Scio.

Apache Flink

While most pipelines leverage Scio (Beam), Data Platform also supports Apache Flink. There is not enough public information on how exactly they leverage Flink.

Orchestrator

Flyte

Spotify migrated from Luigi and Flo to Flyte starting in 2019 to address challenges like fragmented orchestration logic, limited visibility, and lack of extensibility. Flyte offered a centralized service with a thin SDK, better workflow visibility, caching, and multi-language support.

Today, Spotify uses Flyte to manage and introspect data workflows at scale (38k+ jobs), while execution remains on Kubernetes via their existing Styx scheduler.

📖 Recommended Reading: Why We Switched Our Data Orchestration Service

Warehouse

BigQuery

With their migration to GCP, they also moved to BigQuery as the centralized warehouse, processing SQL based workflows through DBT while storing all the analytical data served through dashboards tools e.g. Looker.

Storage

HDFS / GCS

Spotify maintains the largest Hadoop cluster in Europe, with the on going migration to Cloud, they serve and store data on both on premise HDFS and Google Cloud Storage.

There’s no public information confirming whether Spotify uses a lakehouse architecture.

Management

Spotify has in house tooling for data management as part of their Data Platform, solving the problems from the following common areas.

• Metadata

• Lineage

• Retention

• Access Control

📖Read more: Data Management & Data Processing

Dashboard

Looker / Tableau

Spotify provides both Looker and Tableau as the dashboarding platforms. As per 2023, Spotify had 4900+ dashboards serving to 6000+ users across the company.

• Tableau is used for complex, highly customized dashboards; so all design flexibility is available for deep-dive internal products with specific user needs.

• Looker Studio is preferred for fast, lightweight dashboards; especially among engineering and product teams, thanks to its tight integration with BigQuery and ease of SQL‑to‑visualization workflows.

Related Content:

💬 Spotify’s cloud journey has been unique; starting with on-premise then leveraging AWS and then migrating to GCP in 2016, while still operating some on-prem systems like Hadoop. Today, they rely heavily on GCP-native tools alongside in-house platforms that empower internal teams. I may have missed details, feel free to share in the comments!

To recap: the series follows a Q&A format, featuring professionals who share their journeys, insights, and challenges.

What to Expect:

• Inside the Role – Get a real-world look at what data engineers do day in and day out.

• Getting Started – Discover the essential skills, tools, and career routes to break into the field.

• Tech Trends – Stay in the loop with evolving technologies and shifts shaping data engineering.

• Debunking Myths – Clear up common misconceptions about the data engineering profession.

• Voices from the Field – Hear firsthand insights and experiences from seasoned data engineers.

Let’s dive into Inside Data Engineering:

How would you describe Data Engineering?

Data engineering is the art of moving data from point A to point B as efficiently as possible. “Efficient” means choosing the tools, designs and processes that best fit the context, your company’s scale, the nature of the data, and the latency required.

How did you end up being a Data Engineer?

I started out in 2015 as a data scientist. Like many in that role, I spent about 90 % of my time cleaning, reshaping and piping data for my models. At some point I realised I was already doing data engineering work, just without the title. From there, I leaned fully into that specialty and transitioned step-by-step into a dedicated data engineering position, which is where I am today.

What's your day-to-day look like?

I work as a freelance data engineer. I spend most mornings on client projects, and in the afternoons, I focus on sales, marketing, or content creation.

What are some stakeholders that you work with?

When I engage with a client, my first contacts are usually the Head of Engineering, Head of Data, or Data Team Lead. After the initial discussions, I work mainly with the data engineering team.

What kind of projects do you work on?

I support startups by defining their data strategy, selecting the right tools, and designing data architecture. I also work with larger companies in a more hands-on role, helping them build data platforms.

What kind of data do you work with?

Most of my projects involve tabular data:

• Time-series

• Events

• User data/profiles

What data size do you work with?

It largely depends on the company and its data volume, which can range anywhere from tens of GB to 100+ TB.

What tech stack do you use?

Tech Stack depends on client/projects, typically:

What is your favorite area of Data Engineering?

Data Platform Design

How can Data Engineering benefit from GenAI?

V0 of a data pipeline: With a solid foundation in place, GenAI can generate the v0 of a pipeline. However, GenAI struggles with anything beyond ~20 lines of code unless it’s tightly constrained.

That’s exactly why I launched boringdata.io. The templates provide 80% of the pipeline code out of the box; GenAI fine-tunes another 10%, and the engineer refines the final 10%.

What advice would you give your past self as a beginner Data Engineer?

If you hate AWS IAM, that's normal — you're not alone. 😄

What are some challenging aspects of Data Engineering?

Design your pipeline with maintenance in mind. Building a pipeline's v0 is easy; making it robust is not.

What is the next big thing according to you in Data Engineering?

Open table formats are transforming data platform design: We can keep all data in a single storage layer while choosing the best compute engine for each task.

What are some common misconceptions about data engineering?

Moving data from A to B may sound boring, but there are countless ways to design the same pipeline, which makes it fascinating. Technology evolves so quickly that you must keep learning new tools and approaches.

Reach out if you like:

• To be the guest and share your experiences & journey.

• To provide feedback and suggestions on how we can improve the quality of questions.

• To suggest guests for the future articles.

I wanted to share my findings to help you decide which option to choose if you're in a similar situation.

The article covers:

• Benchmarking a single batch job across Spark Operator, EMR on EC2, EMR on EKS, and EMR Serverless.

• Key considerations for selecting the right option and when to use each.

Job

To benchmark these options, I used a heavy-shuffling Spark batch job written in Scala. Since it was run on a proprietary dataset, I can't share specific details.

Inputs

Dataset A

Size: ~270 GB 
File Count: 500 
Row Count: ~350 Million

Dataset B

Size: ~1 TB 
File Count: 500
Row Count: ~3 Billion

Resource

I used same resources across each approach:

• Job runs in us-east-1 region.

• Using On-Demand because with Spot the cost and performance will vary depending on how often the job lose the spot.

• Ignoring the Driver cost to simplify calculation.

• Not enabling or considering the autoscaling feature when calculating the cost, it makes easy to compare and calculate cost.

• Using 150 executors, with:

  • 50 GB memory

  • 8 cores

Result

Some Important points to keep in mind while looking at results:

• Spark Operator uses the 3.2 version with no custom optimization. It is the existing environment we use, so its not apples to apples, but gives enough idea.

• AWS says EMR is 4x faster than open source Spark.

• Cost is not the actual bill, calculation shared in the next section.

• On paper EMR Serverless might look like the most expensive option, but in reality due to better resource consumption, it is actually the fastest.

• Benchmarking can give idea in terms of cost and performance, but there are more factors to consider which are shared later.

Based on just benchmarking results for the specific job, EMR Serverless is the right choice in terms of both cost and performance, but it may not be true in all cases.

Cost Calculation

Calculation is done through AWS calculator based on r5.2xlarge (8 cores, 64GB memory). The actual bill may differ.

Spark Operator

r5.2xlarge Instance fee = $0.504
EKS cost per hour = $0.1

Cost Per Hour: 
(Instance fee) * (Number of pods * hours) + (EKS cost * hours)
$0.504 * (150 * 1) + (0.1 * 1) = $75.70

Cost Per Minute: 
cost_per_hour * (mins / 60)
$75.70 * (22 / 60) = $27.75

EMR on EKS

r5.2xlarge Instance fee = $0.504
EMR fee per hour = $0.15208 (based on 8 cores and 64GB memory)
EKS cost per hour = $0.1

Cost Per Hour: 
(Instance fee + emr fee) * (Number of instances * hours) + (EKS cost * hours)
($0.504 + $0.15208) * (150 * 1) + (0.1 * 1) = $98.51

Cost Per Minute: 
cost_per_hour * (mins / 60)
$98.51 * (18 / 60) = $31.19

EMR on EC2

r5.2xlarge Instance fee = $0.504
EMR fee per hour = $0.126

Cost Per Hour: 
(Instance + emr fee) * (Number of instances * hours)
($0.504 + $0.126) * (150 * 1) = $94.5

Cost Per Minute: 
cost_per_hour * (mins / 60)
$94.5 * (17 / 60) = $26.7

EMR Serverless

EMR fee per hour = $0.7908

Cost Per Hour: 
(emr fee) * (resources * hours)
$0.7908 * (150 * 1) = $118.62

Cost Per Minute: 
cost_per_hour * (mins / 60)
$118.62 * (11 / 60) = $21.75

Which one to Pick?

The biggest challenge as a Team is to pick the right compute infra for your use case. Benchmarking can help, but there are lot of other factors that needs to considered, e.g. complexity, operational overhead, workflow requirements, team dynamics, etc.

The following key points will help you decide which one to pick and will save time by narrowing down the benchmarking options.

Spark Operator

• Complex setup requiring heavy investment upfront with high operational overhead.

  • Great for dedicated teams which has the expertise of running open source tool and Kubernetes.

• Fully customizable, however requires additional setup from Spark UI to performance optimization, e.g. setting up a magic committer for write performance.

• For large scale, this will be the cheapest option in the long run if implemented correctly.

• Can support both Spot and On Demand EC2 Instances.

EMR on EKS

• Bit easier than Spark Operator but still a complex setup requiring similar investment upfront but only on the Kubernetes part.

• Spark is fully managed and optimized through EMR giving additional features:

  • EMR virtual containers console: providing restricted view only mode for Spark jobs with monitoring, logging and Spark UI.

• Depending on job type, EMR could be expensive due to additional EMR Service Fee. EMR fee is per resource (memory & cores).

• Can support both Spot and On Demand EC2 Instances.

EMR on EC2

• Easier than the previous one as no Kubernetes is required, runs directly on EC2. However, requires the expertise of EC2 instances, types, pricing, etc.

• Spark is fully managed and optimized through EMR giving additional features:

  • EMR console: providing boarder control of resources, applications, monitoring and logging, Spark UI.

• Depending on job type, EMR could be expensive due to additional EMR Service Fee. EMR Fee is per instance type.

• Can support both Spot and On Demand EC2 Instances.

EMR Serverless

• Easiest in the list, with no complex infra setup required, no EC2 or EKS.

  • Go with serverless if small team with small amount of data processing.

• Spark is fully managed and optimized through EMR giving additional features:

  • EMR Studio console: providing control of applications, monitoring and logging, Spark UI.

• It is a pay per use model. Depending on job type this could be the most expensive option. EMR Fee is per resource (memory & cores).

  • Serverless seems to better suited for small size jobs with strict SLAs, which we saw earlier in the benchmarking results.

• Only supports On Demand option at the moment.

To simplify in the image form, the following can be used at a high level to narrow down the option to get the most cost effective results.

After narrowing down the options, you can perform similar benchmarking on multiple jobs to get an idea.

💬 Let me know what else would you add.

Metrics

• Ingesting 4+ TB of data into BQ every day, source.

• Ingesting 1.8 trillion events per day at peak, source.

• Datawarehouse contains more than 200 PB of data in 30k GCS bucket, source.

• Snapchat receives 5 billions Snaps per day, source.

• Snapchat has 3,000 Airflow DAGS with 330,000 tasks, source.

Content is based on multiple sources including Snap Blog, Open Source websites, Job descriptions and other public articles etc. You will find references to dive deep as you read.

Platform

GCP

Snapchat leverages GCP as their platform for all data infra, processing and analytical needs. They have been a big user of variety of GCP services.

💡Snapchat’s multi-cloud architecture includes AWS; however, there is no publicly available information about its use for offline data processing.

Messaging System

PubSub

Real time events are ingested through GCP native PubSub service. The fully managed services scales seamlessly during peak hours and works well with the GCP Dataflow service.

Processing

Beam

Snapchat relies on GCP Dataflow, a fully managed Apache Beam service, to process data in both streaming and batch modes.

Spark

Snapchat extensively uses Spark (GCP DataProc) for both batch and stream processing, making it a core component of the feature generation architecture that powers its recommendation systems.

📖 Recommended Reading: Speed Up Feature Engineering

Orchestrator

Airflow

Snapchat has been using Airflow since 2016 and has faced numerous challenges over the years, which were addressed by upgrading to Airflow 2+.

As per 2024, Snap runs over 3,000 DAGs that execute more than 330,000 task instances daily, covering ETL, reporting/analytics, machine learning workloads, and more. Snap relies on 200+ custom operators to support a wide range of services and use cases, serving over 1,000 active Airflow users.

Warehouse

BigQuery

Snapchat heavily relies on BigQuery for its data warehousing needs. All event-driven data from various sources ends up in the warehouse.

BigQuery is used by many teams across different organizations for both ad hoc and scheduled workflows, orchestrated through Airflow.

Lakehouse

GCS & Iceberg

Alongside centralized data warehousing, Snapchat uses a lakehouse architecture by combining Google Cloud Storage (GCS) with Apache Iceberg. This allows them to efficiently access large datasets without duplication in BigQuery.

With BigLake integration, GCS data becomes directly accessible in BigQuery, giving users a native-like experience.

📖 Recommended Reading: Apache Iceberg support for BigLake

ML teams are among the primary users of Iceberg and BigQuery tables in their Bento Platform.

Governance

Dataplex

No surprise, as Snapchat is a GCP-heavy user, they also use GCP's Dataplex service for data management. It enables the creation of a logical data organization layer spanning more than 1,500 GCP projects, allowing them to manage and make decisions without the need to move data physically.

Dashboard

Looker

Looker plays a central role in enabling data-driven decisions across the company. Integrated with BigQuery, it empowers teams to self-serve their data needs.

Snapchat has made significant investments in Looker, partnering with GCP to upskill teams worldwide, as per this source.

Related Content:

Interesting Read:

Architecture Notes — System Design & Software Development

The Cost of AI in Code: Goodbye, Fingerspitzengefühl?

AI coding tools such as Cursor, Claude Code, and OpenAI’s Codex are transforming software development. They can generate, refactor, and even test code autonomously, significantly boosting productivity. For instance, Claude Code operates as a command-line interface, allowing developers to interact with their codebase through natural language prompts. Cur…

Read more

a year ago · 18 likes · Mahdi Yusuf

💬 Snapchat primarily use GCP for Data Analytics and Platform, but with their recent adoption of AWS, there is likely some information that I am unaware of. Please let me know in the comments.

To recap: the series follows a Q&A format, featuring professionals who share their journeys, insights, and challenges.

What to Expect:

• Inside the Day-to-Day – See what life as a data engineer really looks like on the ground.

• Breaking In – Explore the skills, tools, and career paths that can get you started.

• Tech Pulse – Keep up with the latest trends, tools, and industry shifts shaping the field.

• Real Challenges – Uncover the obstacles engineers tackle beyond the textbook.

• Myth-Busting – Set the record straight on common data engineering misunderstandings.

• Voices from the Field – Get inspired by stories and insights from experienced pros.

Let’s dive into Inside Data Engineering:

How would you describe Data Engineering?

Prior to the rise of AI, Data Engineering has become about being the best at Python, SQL, or whatever new hot tool was released. AI coding assistants, like it or not, have lowered the bar and made Data Engineering less about coding and more about providing business value from data. More than ever Data Engineering is about …

• High-level architectural and data platform designs and maintenance.

• Reducing data processing costs and complexity

• Communication with non-engineering groups

• Leading and upskilling others

• Project planning and implementation

How did you end up being a Data Engineer?

I came from a non-traditional background, I never took a computer science class in my life. I taught myself during college to write PHP, Perl, MySQL, etc. After a few years of working as an engineer, I decided I wanted to move into tech. I taught myself SQL, got SQL Server certified, and got a job as a Data Analyst on a Business Intelligence team.

This was when Data Engineering was just becoming a thing, so I continued to hone my programming skills and taught myself things like Spark before it was popular. The rest is history.

What's your day to day look like?

I work at a small startup, so it can vary greatly, but it is typically made up of the following different tasks.

• Project planning (planning new features, writing docs, implementation plans, etc., creating JIRA tasks etc.)

• Answering questions from others and helping unblock others as needed.

• Focused coding time.

  • Could be AWS infra, Spark, Databricks, Postgres, Docker, etc.

What are some stakeholders that you work with?

I work with:

• Data Science

• Product (the business)

• Data Analysts

• C-Suite (CTO, etc.)

What real-world business problems do you solve through data?

We work on Machine Learning pipelines at scale that can predict debit card/credit card fraud. This requires building reliable systems that scale and ingest large quantities of data with a small team.

What kind of projects do you work on?

A very large variety of projects, working at startups requires a wide range of skills. AWS infrastructure, lots of Databricks/Spark pipelines, even LLM fine-tuning and RAGs. I enjoy working on a wide variety of technologies, it keeps me engaged and always learning something new.

What kind of data do you work with?

Mostly tabular data.

What data size do you work with?

300TBish.

What technologies do you use?

What is your favorite area of Data Engineering?

Rust-based Data Engineering tools like Polars and Daft, I think it’s the future of Data Engineering.

How can Data Engineering benefit from GenAI?

Increased efficiency in producing results, let GenAI assist in writing tests, finding bugs, and bouncing ideas off it. It can shorten the time horizon in many areas of the Software Development Lifecycle.

It’s fair that some people are skeptical and worried about losing fundamental skills, but if you are a continuous learner before AI, the chances that AI will change who you are are low.

What advice would you give your past self as a beginner Data Engineer?

Work on soft skills as hard as you work on programming and technical skills. Writing, speaking, communicating, project planning, etc.

Never stop learning, always push yourself to do things you don’t understand. Find people smarter than yourself and then work closely with them.

What are some challenging aspects of Data Engineering?

Working with the business in a way that will build relationships while at the same time being strict about best practices and approaching problems in a reasonable manner. Also, in the fast-paced, changing landscape of tech and Data Engineering, it is important to stay focused on the basics and provide reliable and scalable solutions that simply work well.

What is the next big thing in Data Engineering?

DuckDB, Polars etc., they will become distributed in nature and start to eat Spark’s market share (over a long period of time).

Data Engineering Central

DuckDB vs Polars - Thunderdome.

If you know me, you know I like to stir the pot, the big boiling and smoldering cauldron of Data Tools pot. Yes, that’s the one, blackened and burned pot from years of conjurers pouring myriads of Modern Data Stack tools into it, which have since bubbled and encrusted us all with the refuse of a thousand promises to be the cure-all for our ailments…

Read more

2 years ago · 19 likes · 2 comments · Daniel Beach

What are some common misconceptions about data engineering?

You can simply learn SQL and Python and be a good Data Engineer. Now, with GenAI that is even less true than it was before. We need Engineers who can work well on teams, stay focused, make good tradeoffs, communicate well, and can do more than just write code.

Reach out if you like:

• To be the guest and share your experiences & journey.

• To provide feedback and suggestions on how we can improve the quality of questions.

• To suggest guests for the future articles.

The goal of this article is to provide a high-level overview of Lakehouse governance using open-source tools. In future articles, I will dive deeper into these technologies.

Access Control & Security

Goal: Control who can access what data and under what conditions.

• Apache Ranger: Centralized policy management for fine-grained access control at various levels (database, table, column, row) in Hive Metastore.

  • Ranger can work with Lakehouse through Hive Metastore.

  • Works well with Apache Atlas.

• Keycloak: Not specifically for data, but for data services and applications. It integrates with tools like Trino, Airflow, and Superset to manage access to UIs and APIs through SSO and role-based access control.

• Open Policy Agent (OPA): General-purpose policy engine for access control enforcement, especially when integrated with metadata platforms like DataHub.

Data Lineage & Tracking

Goal: Understand how data moves, transforms, and is consumed across systems.

• Amundsen: Basic lineage tracking through metadata relationships and a user-friendly UI, emphasizing search rather than detailed lineage.

• Apache Atlas: Rich lineage capabilities, tracking data flow across systems like Hive, HDFS, Kafka, Lakehouse (through Hive Metastore) providing insights into data transformations and processing workflows.

• OpenLineage + Marquez: OpenLineage defines lineage metadata for pipelines; Marquez provides metadata service with UI for search and graph-based exploration.

  • OpenLineage is also used by Amundsen and DataHub under the hood.

• Spline: Captures runtime data lineage from Apache Spark applications with minimal code changes.

Metadata Management & Discovery

Goal: Make data assets easily discoverable, understandable, and governed.

• Amundsen: Metadata and search engine with user-friendly UI, showing dataset schemas and ownership. Built on Neo4j and Elasticsearch.

• Apache Atlas: Metadata catalog with support for taxonomies, classifications, and integration with security tools.

• DataHub: Comprehensive metadata platform supporting powerful metadata discovery, faceted search, schema versioning, data ownership, and impact analysis.

• Metacat: Metadata and dataset catalog from Netflix, supporting multiple storage systems and integration with Hive and Presto.

Data Quality & Observability

Goal: Ensure data meets business expectations through validation and monitoring.

• Great Expectations: Framework for defining "expectations" on datasets (e.g., no nulls, unique keys). Integrates into pipelines for runtime validations.

• Soda Core: CLI-based tool for rule-based data profiling and monitoring, detecting freshness, duplicates, and other issues.

• Deequ: Library for setting constraints on datasets (completeness, uniqueness) and working at scale with Spark Dataframes.

• DQX: Framework for orchestrating data quality checks across engines, integrated with Spark, Delta Lake, and Lakehouse ecosystems.

Data Versioning & Auditing

Goal: Enable reproducibility, rollback, and audit trails for datasets.

• Delta Lake: Provides ACID-compliant versioning, intuitive time travel by version or timestamp, rollback to previous states, and full audit logs via a transaction log.

• Apache Hudi: Enables versioned data with commit timelines, supports time travel through instant queries, allows rollback of failed writes, and maintains an auditable history.

• Apache Iceberg: Tracks data through immutable snapshots, supports time travel and rollback via snapshot IDs, and logs detailed metadata for auditing.

• LakeFS: Git-like version control for data lakes, enabling branching, committing, and merging of data.

Data Classification & Tagging

Goal: Categorize and protect sensitive or regulated datasets appropriately.

• Amundsen: Offers lightweight tagging and ownership attribution, helping teams document and search datasets, though with limited support for hierarchical glossaries or complex policies.

• Apache Atlas: Provides advanced data classification with custom tags, business glossaries, and automated lineage-based tag propagation for enterprise-grade metadata governance.

  • Commonly used with Apache Ranger.

• DataHub: Supports dynamic classification through tags and glossary terms, with policy enforcement and access control via seamless integration with Open Policy Agent (OPA).

Users:

Some of these tools are used by large companies on a large scale.

Final Thoughts

Governance doesn’t have to be expensive or complex. By stitching together the right open-source tools, you can build a highly governed, secure, and observable Lakehouse architecture without locking into proprietary platforms.

Next Steps for Readers

If you're interested in learning more, I encourage you to follow future deep dives.

If you are looking to adopt, consider the following:

• Identify governance gaps in your current architecture.

• Pilot lightweight tools (e.g., Soda Core, Great Expectations) before scaling.

• Gradually build a centralized metadata and lineage system.

• Automate access controls and classification tagging as your data grows.

💬 Let me know in the comments if I missed something.

Their Data Platform is powered mainly by open-source tools like Kafka, Flink, and Delta, alongside commercial products like Sigma, which we will explore today.

Content is based on multiple sources including DoorDash Engineering Blog, Careers’ Page and third party articles. You will find references as you read.

Platform

AWS

DoorDash leverage AWS as their cloud platform and utilizes several AWS services. One of the use case has been shared by AWS here.

Storage

S3

S3 is the core component of their Lakehouse, storing Petabytes of data for offline processes, heavily used in Flink and Spark processing.

Delta

DoorDash primarily provide Delta table format on S3 to build a strong Lakehouse architecture. However, according to this job description they also use Iceberg.

Last year, they showcased a Flink + Delta real-time architecture at the Databricks Summit.

Pinot

DoorDash uses Apache Pinot for real-time analytics and low-latency queries. It powers two use cases:

• Mx Portal Ads Campaign Reporting by tracking impressions, clicks, and orders.

• Risk Platform dashboards built with Superset on top of Pinot Tables.

📹Recommended Video: Supporting Multiple Pinot Use Cases at Scale

Snowflake

Snowflake is the Data Warehouse primarily to support reporting and metrics in Sigma to drive data-driven decision-making. Data is inserted from S3 using Snowpipe.

📖 Recommended Reading: DoorDash Logs a 30% Increase in Queries while Keeping the Snowflake Cost Constant with Sigma

Processing

Kafka

DoorDash uses Apache Kafka as a distributed event streaming platform to handle billions of real-time events. They leverage Kafka for message queuing, real-time event processing, and have implemented multi-tenancy awareness for both producers and consumes.

📖 Recommended Reading: Real time event processing with Kafka and Flink

Flink

Flink is used as a primary stream processing engine, processing 220 TB per day into their data lake using the Flink Delta Sink.

Airflow

DoorDash use Airflow as its data platform orchestration tool, allowing users to seamlessly orchestrate complex pipelines with Spark while accessing data from their Lakehouse.

Spark

Spark is the core Batch Processing Engine, enabling users to perform large scale data transformations and aggregations.

Trino

Trino allows DoorDash’s data team to seamlessly query data from the Lakehouse using SQL. It supports both batch jobs and ad-hoc analysis, enabling efficient exploration and processing of large datasets.

📹Recommended Video: How DoorDash is Realizing a Unified Query Engine

Dashboard

Sigma

Sigma is DoorDash's primary Business Intelligence tool, empowering ~12,000 internal users and 5,000 dashboards with real-time data, self-service capabilities, and advanced analytical insights.

Superset

DoorDash use Apache Superset for their Risk Platform, enabling real-time analytics on top of Apache Pinot, according to this source.

Related Content: Tech Stack Series

💬 Let me know in the comments if I missed something.

To recap: the series follows a Q&A format, featuring professionals who share their journeys, insights, and challenges.

What to Expect:

• Real-world insights – Learn what data engineers actually do on a daily basis.

• Industry trends – Stay updated on evolving technologies and best practices.

• Challenges – Discover what real-world challenges engineers face.

• Common misconceptions – Debunk myths about data engineering and clarify its role.

Let’s dive into Inside Data Engineering:

How would you describe Data Engineering?

To me, data engineering is all about designing, building, and maintaining the foundation that enables efficient data storage, retrieval, and processing, ensuring that data is organized, and ready for analysis.

How did you end up being a Data Engineer?

Honestly, it’s a bit of an unexpected journey.

Back in 2019, I graduated in Electrical and Telecommunications but quickly realized I had no interest in working in that field. So, like many fresh grads, I went online searching for high-paying jobs, and “Data X” (whether scientist, engineer, or analyst) sounded cool.

Then, a company offered me a “data” job, even though they weren’t sure what the exact role was. (To this day, I’m not sure how I got that offer!)

But that job introduced me to Docker, Spark, HDFS, and Airflow, though mostly for building POCs. Eventually, the company shut down, forcing me to look for a new job.

That’s when I really took the time to research the differences between the “Data X” roles. And it finally clicked: I wanted to be a data engineer.

What's your day to day look like?

I start my day by checking Slack—just in case someone is complaining about a data bug. (Kidding… mostly!)

A typical day involves:

• Coding data pipelines

• Fixing bugs

• Helping data analysts optimize queries

Lately, I’ve also been working on an internal data app focused on the semantic layer, which has been an interesting challenge.

What are some stakeholders that you work with?

Primary owners that I closely work with are Data Analysts and Project Owners.

What kind of data do you work with?

I work for a music game company with mobile games played globally.

Most of the data comes from:

• In-app event tracking

• Third-party services that track user acquisition

• Revenue collection processes

This means handling large volumes of player interactions, engagement metrics, and monetization data to drive insights and optimize the game's performance.

What data size do you work with?

My day to day work involves a couple of TBs.

What tech stack do you use?

We primarily use BigQuery, DBT and Airflow. (I’m a fan of BigQuery)

What programming languages do you use?

We are a Python and SQL shop.

What tools do you leverage for GenAI?

I primarily use ChatGPT, but I also leverage Perplexity and NotebookLM when I need deeper research.

What is your favorite area of Data Engineering?

My favorite thing about Data Engineering is that you never run out of things to learn. The field is constantly evolving, with new tools, architectures, and best practices emerging all the time.

What advice would you give your past self as a beginner Data Engineer?

Spend time understanding the real value a data engineer provides. This awareness will give you clarity on your contributions and help you avoid roles that expect you to train ML models instead. It will also guide you in prioritizing which skills to learn and which ones to skip.

Next, I’d tell myself to build strong data engineering fundamentals at all costs, especially data modeling, even if it feels tedious at first. It pays off in the long run.

What are some challenging aspects of Data Engineering?

There are two challenges that I can talk about:

• First, in Data Engineering there is the sheer amount of things to learn. That’s why understanding the true value a data engineer brings is crucial, it helps you focus on what’s worth mastering instead of getting lost in an endless tech stack.

• Second, I observe is that many companies know they need a data team but have no clear direction on how to build or utilize it effectively.

What is the next big thing in Data Engineering?

I believe Data Lake Houses will continue to grow, with table formats and query engines becoming even more efficient. The adoption of object storage as the primary storage layer will keep expanding, driven by its scalability and cost-effectiveness.

What are some common misconceptions about data engineering?

A common misconception is that data engineering is always about big data. In reality, the core goal is to build a solid data foundation, ensuring data is stored, retrieved, and delivered efficiently for insights.

Sometimes, this means working with massive datasets, but often, you must work with "not-so-big" data.

Reach out if you like:

• To be the guest and share your experiences & journey.

• To provide feedback and suggestions on how we can improve the quality of questions.

• To suggest guests for the future articles.

Today, we’ll explore how to set up and secure an SFTP server using AWS Transfer Family. We’ll cover:

• Deploying an SFTP server

• Setting up users with read & write restricted permissions

• Securing onboarding with SSH (MFA approach shared)

• Leveraging Transfer Family Workflows for automation

By the end, you’ll have a secure, scalable, and automated solution for managing file transfers efficiently.

Prerequisites

Before we dive into AWS Transfer Family, following must be created:

• Create S3 Bucket for storing the files

• Create Lambda Function for downstream processing

• Create IAM role with access to S3 and Lambda for the Workflow

• Create IAM role with restricted access to S3 for SFTP user

  • Restricting at the prefix level e.g. <bucket>/username/

Creating SFTP Server

Creating a managed server with Public endpoint and workflow details.

resource "aws_transfer_server" "sftp_server" {
  domain                 = "S3"
  identity_provider_type = "SERVICE_MANAGED"
  endpoint_type          = "PUBLIC"
  protocols              = ["SFTP"]

  workflow_details {
    on_upload {
      execution_role = <workflow_iam_role_arm>
      workflow_id    = aws_transfer_workflow.sftp_workflow.id
    }
  }
}

• <workflow_iam_role_arn>: Provide the role arn that has S3 and Lambda access

Creating SFTP User with SSH Access

Creating a user with S3 directory mapping, this allows user to only have the required permission to that prefix.

resource "aws_transfer_user" "sftp_user" {
  server_id = aws_transfer_server.sftp_server.id
  user_name = "<username>"
  role      = <sftp_user_iam_role_arn>
  home_directory_type = "LOGICAL"

  home_directory_mappings {
    entry  = "/test"
    target = "/<bucket_name>/<username>/"
  }
}

resource "aws_transfer_ssh_key" "sftp_ssh" {
  server_id = aws_transfer_server.sftp_server.id
  user_name = aws_transfer_user.sftp_user.user_name
  body      = <ssh_public_key>
}

• <username>: Provide the user name

• <sftp_user_iam_role_arn>: Provide the role arn that has S3 access restricted access

• <bucket_name>: Provide bucket name

• <ssh_public_key>: Provide the public key (generate using this guide)

  • In real world, vendor can generate key pair on their end and send over the public key.

For advance authentication mechanisms like Multi Factor (MFA), you can leverage a Lambda function as recommended by AWS.

📖 Recommended Reading: Implement multi-factor authentication based managed file transfer using AWS Transfer Family and AWS Secrets Manager

Test connection

Using SSH based approach it is pretty straight forward to log in to sftp server.

ssh -i <ssh_private_key> <username>@<host_address>

• <ssh_private_key>: Provide the generated key from previous section.

• <username>: Provide the user name from the previous section.

• <host_address>: Provide the host address that can be found from the AWS sftp transfer family console.

For non technical users, Transfer Family offers a managed web based interface for file transfers. Explore more here.

Setting up Workflows to trigger downstream jobs

Workflows are great pre configured series of step that are part of the Transfer Family ecosystem. It allows you to perform several fully managed functionality on the files that you receive from SFTP right away.

In our case, we do three steps:

• Copy to internal S3 bucket with limited access, its recommended since the SFTP backed S3 has external access through SFTP.

• Tag S3 objects depending on your usecase

• Trigger a Lambda function for downstream processing and custom logic. e.g. sending email notification.

📖 Recommended Reading: How to return the response to workflow from Lambda Function

resource "aws_transfer_workflow" "sftp_workflow" {

  steps {
    type = "COPY"

    copy_step_details {
      destination_file_location {
        s3_file_location {
          bucket  = <bucket_name>
          key     = <prefix_path>
        }
      }
      overwrite_existing = "TRUE"
    }
  }

  steps {
    type = "TAG"

    tag_step_details {
      tags {
        key   = <key>
        value = <value>
      }
    }
  }

  steps {
    type = "CUSTOM"
    
    lambda_step_details {
      target = <lambda_function_arn>
    }
  }
}

• <bucket_name>: Provide the destination bucket name

• <prefix_path>: Provide the destination prefix path for the object

• <key> & <value>: Provide the key and value for tagging

• <lambda_function_arn>: Provide arn for the Lambda function

First step reference the original file while the corresponding steps automatically reference the file from previous step, it can be overwritten if needed.

Alternatively, you can skip the workflow setup by handling copy and tagging directly in your Lambda function, triggering it via SNS on an S3 drop. However, this approach falls outside the Transfer Family.

I hope you found this article useful. With it, you’ve now learned how to develop a secure, scalable, and automated solution for efficiently managing file transfers in the AWS cloud environment.

Meta has open-sourced several tools like Hive and Presto, while others remain internal—some of which we will discuss in today’s article.

Content is based on multiple sources including Meta Engineering Blog, Meta Research Papers, third party articles. You will find references as you read.

Today’s post is brought to you by Multiplayer:

Multiplayer's makes debugging distributed systems easier with deep session replays. From frontend screens to backend traces, metrics, and logs, you have every detail you need to find and fix a bug in one place.

Platform

On Premise

Since its inception in 2004, Meta has operated out of its on-premise data centers spread across the globe, including the United States, Europe, and Asia-Pacific.

💡Meta has 35 data centers as per this source.

Storage

Hive

Hive was created at Meta and open-sourced in 2008. It is an exabyte-scale data warehouse storing millions of tables across multiple data centers. Hive leverages the ORC columnar format to efficiently store massive datasets.

Scuba

Scuba is an in-house tool designed for real-time data analysis. It provides multiple ways to access data at high speed through a UI or programmatic interfaces. Scuba excels at handling ad-hoc queries, with most responses returning in under a second.

💡Scuba is like Apache Druid at a very high level, read more here.

Laser

Laser is a high-throughput, low-latency key-value storage service built on RocksDB. It reads data in real-time from Scribe streams and daily from Hive tables. It powers the Facebook product and integrates with apps like Puma and Stylus.

📖 Recommended reading: Realtime Data Processing at Facebook

Processing

Scribe

Scribe processes over 2.5 TB per second of input from millions of producers and outputs 7+ TB per second to hundreds of thousands of consumers, primarily sending data to Scuba and Hive. Apache Kafka is an open-source alternative to Scribe.

📖 Recommended Reading: Scribe: Transporting petabytes per hour via a distributed, buffered queueing system

Puma / Swift / Stylus

Meta Engineering has built three in-house tools to read and write data in real-time to and from Scribe.

The following help you understand where these components sit in the system.

Puma:

• Puma is a stream processing system where apps are written in a SQL-like language with Java UDFs.

• Puma processes Scribe streams with a few seconds' delay, outputting to another stream, real-time processor, or data store.

• It's optimized for compiled queries, not ad-hoc analysis.

Swift:

• Swift is a basic stream processing engine providing checkpointing for Scribe.

• It offers a simple API to read streams with checkpoints, allowing apps to restart from the latest checkpoint.

• Swift is ideal for low-throughput, stateless processing, with client apps often written in scripting languages like Python.

Stylus:

• Stylus is a low-level stream processing framework written in C++.

• It supports both stateless and stateful processors.

• Its processing API is similar to other procedural stream processing systems.

Presto

Presto, open sourced by Meta in 2013, is a successful big data product widely used for SQL-based data processing. At Meta, it enables fast data access, ranging from seconds to minutes, for billions of records.

Spark

Spark is another query engine offering an alternative to Presto. Unlike Presto, it provides greater flexibility, allowing users to leverage Java, Scala, and Python APIs for complex transformations.

Dataswarm

Dataswarm is an in-house orchestration tool similar to Apache Airflow, built in Python. It enables job orchestration and scheduling in a DAG-based pipeline and supports Presto and Spark jobs.

📹 Recommended video: Dataswarm

Dashboard

UniDash

UniDash is an in-house visualization tool for creating dashboards, accessible via a web interface or Python API. It is powered by Presto and includes an additional caching layer through RaptorX.

📖 Read More: High-Level Overview of the internal tech stack

Related Content: Tech Stack Series

💬 Let me know in the comments if I missed something.

This series follows a Q&A format, featuring professionals from around the world.

What to Expect:

• Real-world insights – Learn what data engineers actually do on a daily basis.

• Career advice – Get guidance on skills, tools, and pathways to break into the field.

• Industry trends – Stay updated on evolving technologies and best practices.

• Challenges – Discover what real-world challenges engineers face.

• Common misconceptions – Debunk myths about data engineering and clarify its role.

• Inspiration from experts – Hear personal stories from seasoned professionals.

Today, we have , who has been in the Data Engineering space for many years. He also writes about Data Engineering at .

Let’s dive into Inside Data Engineering:

How would you describe Data Engineering?

Data engineering is like any engineering discipline. It’s about designing, building, and maintaining systems. Instead of roads or bridges, we build the infrastructure that moves, processes, and organises data so businesses can make informed decisions. It’s a mix of software engineering, systems architecture, and business understanding.

How did you end up being a Data Engineer?

I started as a software engineer, but I was always drawn to the bigger picture—how systems fit together and how businesses use technology to drive decisions. Over time, I moved into data engineering because it combined the technical challenge of software with the strategic impact of data. The truth is that I spent more than 1.5 years in data engineering before realising I am not a software engineer anymore.

Junaid Effendi | Sharing knowledge for Engineers

Transition: Software Engineer to Data Engineer

Last month, I shared article on Types of Data Engineers with the goal to start a series that will cover the transition from various roles to Data Engineering. Today I am kicking off the series with Software to Data Engineering Transition…

Read more

2 years ago · 31 likes · 4 comments · Junaid Effendi

What excites you about Data Engineering?

The mix of problem-solving, business impact, and continuous learning. Data engineering isn’t just about writing code—it’s about designing systems that power decision-making. I enjoy bridging the gap between tech and business, figuring out the best architecture, and helping companies make better use of their data.

What is your day to day look like?

A mix of technical work, problem-solving, and collaboration. I spend time designing and reviewing architecture, writing or reviewing code, and troubleshooting issues. But I also work closely with stakeholders—understanding their needs, aligning on priorities, and making sure our solutions fit the business. As a leader, I also focus on strategy, mentoring, and making sure the team is working effectively.

Unfortunately, I rarely spend over 20% of my time writing code myself, nowadays.

What are some stakeholders that you work with?

Analysts, data scientists, finance teams, and executives. A big part of data engineering is not just building pipelines but ensuring that the right people get the right data in a way that makes sense for their needs.

What kind of projects do you work on?

I work on building and optimising data infrastructure, ensuring reliable data pipelines, and designing systems that support analytics and reporting. This includes integrating data from different sources, managing transformations, and improving data accessibility for business teams.

Some of the big projects I am involved in at the moment are:

• Helping with the migration and data remodeling of our billing tooling. It means more flexibility for the business to build packages that work better for our customers.

• Restructuring the internal tooling and processes for the Data and Analytics team. With this one, I aim for simplicity and better alignment within the team.

📖 Related Reading: Large Scale Migration Best Practices

What kind of data do you work with?

Mostly structured and semi-structured data—things like financial transactions, user behaviour, and operational data. Since I work at a fintech company, the data is often tied to business performance, financial metrics, and customer interactions.

Aside from that, my side gig allows me to work on different niches like crypto and marketing.

What size of data do you work with?

It varies, but typically in the terabytes range. Overall, my team and I are responsible for over 100 terabytes of data in Snowflake. But it’s not about the size. The focus is more on efficiency, quality, and usability rather than just handling extreme scale.

What tech stack do you use?

Meltano for data ingestion, dbt for transformations, Airflow for orchestration, Synq for Data Quality, Looker for data visualisation and Snowflake as the data warehouse. I also work with Python, SQL, Git for development. My stack is focused on modern, scalable, and maintainable data engineering practices.

What tools do you leverage for GenAI?

I use most of them. Perplexity was my default search engine for a long time. I also have some custom GPTs on ChatGPT and use Claude for coding.

Two tools I highly recommend are Ollama and LM Studio for anybody who wants to run models locally and not expose any data.

What is your favourite area of Data Engineering?

Architecture and systems design. I enjoy thinking about how different components fit together and how to build scalable, maintainable systems. I also like the intersection of data engineering and business—understanding how data can provide value rather than just focusing on the technical side.

How can Data Engineering benefit from GenAI?

The sky's the limit. As a Head of Data Engineering, I use AI every day to:

• Write code

• Draw architecture diagrams

• Brainstorm architecture ideas

• Summarise meetings and whitepapers

• Write technical and non-technical documentation

Today’s post is brought to you by Multiplayer:

With Multiplayer you can build better distributed systems: automatically document your system architecture, effortlessly debug with deep session replays, and collaboratively design your system. Why are you still wasting time drawing manual diagrams?

What is the next big thing in Data Engineering?

AI will automate repetitive tasks, allowing engineers to focus on solving complex problems and designing efficient systems. However, this shift will raise the skill bar for entry-level roles, demanding stronger capabilities from aspiring engineers.

At the same time, companies are adopting flexible hiring models, favouring consultants and freelancers for their expertise. This creates new opportunities for engineers who can adapt quickly and deliver value.

What advice would you give your past self as a beginner Data Engineer?

Focus on the fundamentals first—SQL, Python, and understanding how data moves. But don’t just stay technical. Learn how businesses use data and how to communicate with stakeholders.

Also, don’t wait too long to build a personal brand. Data engineers often get stuck in the background, but being visible can open a lot of doors.

💡What was enough 5 years ago won’t cut it today.

What are some challenging aspects of Data Engineering?

• Aligning with business needs – Data engineers often get stuck building things without clear business impact.

• Managing complexity – As systems grow, keeping pipelines reliable and maintainable is a constant challenge.

• Tech hype vs. reality – Many tools promise to solve all problems but introduce new complexities. Knowing what to adopt and when is crucial.

• Trust and data quality – Data is only useful if people trust it. Ensuring accuracy and explaining how data is derived is a key challenge.

What are some common misconceptions about data engineering?

• It’s just ETL – Data engineering is much more than moving data. It’s about designing systems, managing infrastructure, and ensuring data is usable.

• Self-service BI will replace data engineers – Most stakeholders need guidance, and self-service tools rarely eliminate the need for proper data engineering.

• Big data is everywhere – Not every company needs a petabyte-scale solution. Many real-world problems can be solved with simpler, well-designed systems.

• More tools = better data – Tools don’t fix bad processes. Good data engineering is about understanding business needs, not just stacking technologies.

Reach out if you like:

• To be the guest and share your experiences & journey.

• To provide feedback and suggestions on how we can improve the quality of questions.

• To suggest guests for the future articles.

If you don’t have the culture of knowledge sharing you can end up with variety of problems as shared in his book Leading Effective Engineering Teams (Page 156), listed below:

Lets dive into the 7 ways to prevent above issues from happening.

1-on-1 Sessions

1-on-1 sessions are dedicated opportunities for personalized interactions with various members, such as managers or cross-team engineers. These sessions can occur during onboarding, routine check-ins, or on-demand for specific projects like pair programming.

How to Benefit:

• Facilitate focused and tailored knowledge exchange, such as exploring a system, function, or codebase in depth.

• Leverage these sessions for mentorship, constructive feedback, or transferring specialized expertise.

• Foster open dialogue to address challenges, brainstorm solutions, and enhance collaboration.

Group Chats

Group chat is another common way in most companies to share knowledge, usually done through a asynchronous communication tool like Slack.

How to Benefit:

• Enable async/non urgent sharing of ideas, solutions, and best practices through dedicated groups.

• Use topic-specific channels to centralize knowledge and keep it organized, e.g. Project based channels.

• Encourage peer-to-peer learning by creating an open forum for questions and answers.

Tech Talks

The idea is to share knowledge among different teams by showing what your team has been doing on a higher level from design to architecture that can be beneficial to other.

How to Benefit:

• Provide a platform for team members to share their expertise on relevant topics.

• Create opportunities to learn from diverse perspectives and experiences through Q/A sessions.

• Build a repository of recorded sessions for on-demand knowledge access.

Junaid Effendi | Sharing knowledge for Engineers

Why Tech Talks are Important

Today’s article I will be sharing why presenting at tech talks is important for you and how it plays an important role in your career growth. Lets first see what tech talks are: Tech Talks are deep technical discussion forums within the organization or company, the purpose is to share knowledge across teams so they can better work together in the long r…

Read more

2 years ago · 13 likes · 2 comments · Junaid Effendi

Newsletter

Newsletter is an effective way to share a range of information with a wide audience on a regular schedule. They often include subscribe/unsubscribe options, making it easy to manage the volume of newsletters you receive.

How to Benefit:

• Consolidate and share key knowledge updates, tips, and resources regularly.

• Highlight achievements or learnings that inspire further knowledge sharing.

• Keep the team aligned and informed about relevant developments and tools.

Documentation

Documentation can be of many types:

Design Document

This focuses on design-related documents created during the planning phase, allowing peers to review and resolve design issues, fostering mutual learning.

How to Benefit:

• Gain insights into design decisions and identify potential issues early.

• Receive feedback to improve design quality.

• Establish a shared understanding across the team.

Process Document

A centralized repository (e.g. Multiplayer, Notion or Confluence) for existing processes, such as final design documents, system architectures, workflows, and data flows, written for a broader audience.

Today’s post is brought to you by:

Multiplayer auto-discovers, tracks, and documents your entire system architecture—from its components to APIs, dependencies, and environments. Gain real-time, comprehensive visibility into your system, all at a glance.

How to Benefit:

• Understand the system and how components interact and integrate.

• Reuse components and processes efficiently.

• Ensure knowledge is accessible to a wider audience beyond the immediate team.

Codebase

Codebase can be a great place to learn and share knowledge, it can be provide several different options; code comments, unit testing and tutorials.

How to Benefit:

• Code Comments: Quickly understand code functionality through comments and examples.

• Unit Testing: Learn function usage directly from unit tests.

• Tutorials: Onboard new contributors efficiently with tutorials and setup guides.

Code Review

Code reviews are among the most effective ways to share knowledge, benefiting both authors and reviewers by enabling mutual learning. They can be conducted asynchronously via a Version Control System or synchronously through Pair or Mob Programming sessions.

How to Benefit:

• Enable knowledge transfer through in-depth feedback and collaborative discussions.

• Identify areas for improvement while promoting consistent coding standards across the team.

• Offer a structured method for learning from each other’s coding practices.

Junaid Effendi | Sharing knowledge for Engineers

Why Code Review?

Code reviews are great in the Software Engineering world, it is a great way to maintain consistency and quality of codebase based on some standards defined by the team, company or industry with some great free benefits like Knowledge Sharing. Code reviews have evolved in the past few decades. It started mainly from human review and now machines help hum…

Read more

4 years ago · 3 likes · Junaid Effendi

Code Search

Code search is invaluable for navigating large codebases. It serves as a one-stop solution for code discovery.

How to Benefit:

• Quickly locate and understand existing implementations, patterns, and best practices.

• Identify duplicate efforts to improve collaboration and optimize workflows.

• Enable independent learning and knowledge exploration within the codebase.

By understanding the importance of these seven ways, you'll gain valuable learning experiences that will help you become a better team player as well as a better engineer.

💬Let me know if you have more ways in mind.