Inside Data Engineering with Yordan Ivanov
Join Yordan Ivanov as he demystifies data engineering, clarifying misconceptions about the role.
Today, I’m kicking off a new series where we dive into the world of Data Engineering, straight from the perspective of experienced data engineers.
This series follows a Q&A format, featuring professionals from around the world.
What to Expect:
Real-world insights – Learn what data engineers actually do on a daily basis.
Career advice – Get guidance on skills, tools, and pathways to break into the field.
Industry trends – Stay updated on evolving technologies and best practices.
Challenges – Discover what real-world challenges engineers face.
Common misconceptions – Debunk myths about data engineering and clarify its role.
Inspiration from experts – Hear personal stories from seasoned professionals.
⭐ If you're curious about data engineering or considering it as a career, this series is for you!
Today, we have
, who has been in the Data Engineering space for many years. He also writes about Data Engineering at .
Let’s dive into Inside Data Engineering:
How would you describe Data Engineering?
Data engineering is like any engineering discipline. It’s about designing, building, and maintaining systems. Instead of roads or bridges, we build the infrastructure that moves, processes, and organises data so businesses can make informed decisions. It’s a mix of software engineering, systems architecture, and business understanding.
How did you end up being a Data Engineer?
I started as a software engineer, but I was always drawn to the bigger picture—how systems fit together and how businesses use technology to drive decisions. Over time, I moved into data engineering because it combined the technical challenge of software with the strategic impact of data. The truth is that I spent more than 1.5 years in data engineering before realising I am not a software engineer anymore.
What excites you about Data Engineering?
The mix of problem-solving, business impact, and continuous learning. Data engineering isn’t just about writing code—it’s about designing systems that power decision-making. I enjoy bridging the gap between tech and business, figuring out the best architecture, and helping companies make better use of their data.
What is your day to day look like?
A mix of technical work, problem-solving, and collaboration. I spend time designing and reviewing architecture, writing or reviewing code, and troubleshooting issues. But I also work closely with stakeholders—understanding their needs, aligning on priorities, and making sure our solutions fit the business. As a leader, I also focus on strategy, mentoring, and making sure the team is working effectively.
Unfortunately, I rarely spend over 20% of my time writing code myself, nowadays.
What are some stakeholders that you work with?
Analysts, data scientists, finance teams, and executives. A big part of data engineering is not just building pipelines but ensuring that the right people get the right data in a way that makes sense for their needs.
What kind of projects do you work on?
I work on building and optimising data infrastructure, ensuring reliable data pipelines, and designing systems that support analytics and reporting. This includes integrating data from different sources, managing transformations, and improving data accessibility for business teams.
Some of the big projects I am involved in at the moment are:
Helping with the migration and data remodeling of our billing tooling. It means more flexibility for the business to build packages that work better for our customers.
Restructuring the internal tooling and processes for the Data and Analytics team. With this one, I aim for simplicity and better alignment within the team.
📖 Related Reading: Large Scale Migration Best Practices
What kind of data do you work with?
Mostly structured and semi-structured data—things like financial transactions, user behaviour, and operational data. Since I work at a fintech company, the data is often tied to business performance, financial metrics, and customer interactions.
Aside from that, my side gig allows me to work on different niches like crypto and marketing.
What size of data do you work with?
It varies, but typically in the terabytes range. Overall, my team and I are responsible for over 100 terabytes of data in Snowflake. But it’s not about the size. The focus is more on efficiency, quality, and usability rather than just handling extreme scale.
What tech stack do you use?
Meltano for data ingestion, dbt for transformations, Airflow for orchestration, Synq for Data Quality, Looker for data visualisation and Snowflake as the data warehouse. I also work with Python, SQL, Git for development. My stack is focused on modern, scalable, and maintainable data engineering practices.
What tools do you leverage for GenAI?
I use most of them. Perplexity was my default search engine for a long time. I also have some custom GPTs on ChatGPT and use Claude for coding.
Two tools I highly recommend are Ollama and LM Studio for anybody who wants to run models locally and not expose any data.
What is your favourite area of Data Engineering?
Architecture and systems design. I enjoy thinking about how different components fit together and how to build scalable, maintainable systems. I also like the intersection of data engineering and business—understanding how data can provide value rather than just focusing on the technical side.
How can Data Engineering benefit from GenAI?
The sky's the limit. As a Head of Data Engineering, I use AI every day to:
Write code
Draw architecture diagrams
Brainstorm architecture ideas
Summarise meetings and whitepapers
Write technical and non-technical documentation
Today’s post is brought to you by Multiplayer:
With Multiplayer you can build better distributed systems: automatically document your system architecture, effortlessly debug with deep session replays, and collaboratively design your system. Why are you still wasting time drawing manual diagrams?
What is the next big thing in Data Engineering?
AI will automate repetitive tasks, allowing engineers to focus on solving complex problems and designing efficient systems. However, this shift will raise the skill bar for entry-level roles, demanding stronger capabilities from aspiring engineers.
At the same time, companies are adopting flexible hiring models, favouring consultants and freelancers for their expertise. This creates new opportunities for engineers who can adapt quickly and deliver value.
What advice would you give your past self as a beginner Data Engineer?
Focus on the fundamentals first—SQL, Python, and understanding how data moves. But don’t just stay technical. Learn how businesses use data and how to communicate with stakeholders.
Also, don’t wait too long to build a personal brand. Data engineers often get stuck in the background, but being visible can open a lot of doors.
💡What was enough 5 years ago won’t cut it today.
What are some challenging aspects of Data Engineering?
Aligning with business needs – Data engineers often get stuck building things without clear business impact.
Managing complexity – As systems grow, keeping pipelines reliable and maintainable is a constant challenge.
Tech hype vs. reality – Many tools promise to solve all problems but introduce new complexities. Knowing what to adopt and when is crucial.
Trust and data quality – Data is only useful if people trust it. Ensuring accuracy and explaining how data is derived is a key challenge.
What are some common misconceptions about data engineering?
It’s just ETL – Data engineering is much more than moving data. It’s about designing systems, managing infrastructure, and ensuring data is usable.
Self-service BI will replace data engineers – Most stakeholders need guidance, and self-service tools rarely eliminate the need for proper data engineering.
Big data is everywhere – Not every company needs a petabyte-scale solution. Many real-world problems can be solved with simpler, well-designed systems.
More tools = better data – Tools don’t fix bad processes. Good data engineering is about understanding business needs, not just stacking technologies.
Reach out if you like:
To be the guest and share your experiences & journey.
To provide feedback and suggestions on how we can improve the quality of questions.
To suggest guests for the future articles.
 | A guest post by
|
Thank you this is very insightful. I love this and learning a lot
great read