Master data engineering and AI like a big tech engineer

Real Time Formula 1 Analytics

Big Bag Data

Chess.com Analytics

Trading Strategy for Crypto Currency - Analytics Engineering Capstone Submission

BetFlow - Real Time Sports Betting App

Showcase of student projects

Capstone Projects

Our students have gone on to work at companies like Meta, Airbnb and Amazon. As well as achieve 100% raises!

Fast Track Your Career

Immediate free cloud access to Databricks, AWS, Snowflake, Astronomer, and more!

Free Cloud Access with tons of hands on exercises

Weekly Guest Speaker Sessions

Why Choose DataExpert.io Academy?

I have led teams of data engineers and software engineers at Airbnb, Facebook, and Netflix. My next goal is to upskill as many data knowledge workers as I can!

Zach Wilson

Data engineering is more than Spark, SQL, and Airflow! "
""
"Robust data modeling that answers immediate and anticipated questions is a make or break skill of a good data engineer. "
""
" #dataengineering

Breaking into #dataengineering can be very confusing! "
""
"Should I learn Spark or Snowflake? Python or Scala? Airflow or Argo? Flink or Spark Streaming? AWS or GCP? Superset or Tableau?"
""
"Fundamentals are more important than technologies: "
""
"- understanding distributed computing is more important than Spark vs Snowflake "
""
"- understanding when to use batch or streaming is more important than Flink vs Spark Streaming "
""
"- understanding functional and object-oriented programming is more important than Scala vs Python. "
""
"- understanding how pipelines are deployed in the cloud is more important than AWS vs GCP"
""
"- understanding how to make comprehensible data viz is more important than Tableau vs Superset. "
""
"- understanding pipeline schedule and SLAs is more important than Airflow vs Argo.  "
""
"If you understand the underlying concepts, when the technology inevitably changes, it’ll be much easier to pick up the new thing.

Building an Airflow DAG is complicated. Building a beautiful one really reduces tech debt. "
""
"Here's some tips: "
""
"- Don't write directly to production. The data quality risk is too high. Use the stage-check-exchange pattern instead. "
""
"- Use clear task names. Don't call your task "load_data." Call it "load_user_data" or something more clear"
""
" - Declare all your partitions sensors near the top of the DAG. Declaring them in the middle or as you go is very confusing."
""
"- Don't write SQL directly in the DAG. Move your SQL into a separate file so the DAG is clean and concise. "
""
"- Don't hardcode table names. Abstracting them to a constants file where they can be reused for easy renaming is better. "
""
""
"#dataengineering

Many people have asked me, “how did you get into #dataengineering?”"
""
"In 2014, I read a stat, 90% of the worlds data had been created in the last 18 months. "
""
"I thought, “that’s going to be a hard problem to manage” "
""
"Started my Hortonworks Hadoop Java Dev certification the next week. "
""
"Cracked a junior DE role at Teradata a few weeks after getting it. "
""
"Do I recommend this path in 2022? No, not really. Hadoop is becoming the COBOL of data engineering in some ways.  "
""
"This is the tooling that’s important/relevant for a wide swathe of DE roles in 2022: "
""
"Storage: S3/Postgres"
"Compute: Spark/Flink/Snowflake/BigQuery"
"Viz: Tableau/Superset/Power BI/D3.js"
"Scheduler: Airflow/Prefect"
"APIs: Spring Boot/NodeJS"
"Build: Gradle/pip/Maven"
"CI/CD: Jenkins/CircleCI

I went on Preset  and Maxime Beauchemin's podcast a few weeks ago! We talked about #dataengineering, building data products and much more! Max is the original creator of Apache Airflow! "
""
"This was a very fun podcast to be on. Thanks for having me Preset !

I'm going to be doing a live podcast episode with Maxime, the creator of Apache Airflow, and Preset  on Monday, April 4th at 11 AM Pacific. We're going to be talking about all sort of things related to #dataengineering, #datascience and building data products! "
"Definitely register for this, it's an event you won't want to miss!

Had a really great talk with Wanjohi and his teammates in Kenya today! We talked about how to build a good #dataengineering project to get a job! "
""
"Things that I recommended:"
"1. Pick a data source that you're interested in. Some examples that you might be interested in are sports data, web scraping your favorite website, Twitter data, the stock market, etc. The key thing when picking the source for the project is that it is data that you're interested in. "
""
"2. After you've picked a source. Create an ETL for it. You can pick either a very heavy tool like Airflow. Or a very lightweight one like CRON. The key thing is that this ETL runs at a cadence, usually daily. "
""
"3. Your ETL needs a place to write data. There are a couple of options here. You can easily spin up a new Postgres database for free with tools like Heroku. Or you can write to Amazon S3 and store the data very cheaply there. "
""
"4. Once you have your ETL writing data, then go ahead and connect some sort of visualization software to it. The options here are Tableau or Power BI if you want to do it quickly. Or D3.js if you want to really wow people. Here's an example of my D3 visualization I used to create a really cool visualization about how stocks are mentioned together in the news. https://lnkd.in/eGg73yPA"
""
"Now if you complete these 4 steps, you now have a very strong portfolio project that you can talk about in interviews. This is critical in convincing interviewers that you know what you're doing.

When you're writing a pipeline in #dataengineering, how big of a DAG is too big? "
""
"In my career, I've worked with DAGs that have hundreds of steps and tons of tiny DAGs that have 3 steps each."
""
"There are pros and cons to doing it both ways. "
""
"Keeping everything in a big ol' DAG:"
""
"Pros: "
"You can backfill everything with 1 command"
"You can see the entire DAG tree in 1 view "
"You can use task dependencies instead of data dependencies which makes backfilling cleaner"
""
"Cons:"
"Airflow UI gets laggy and sometimes won't open if the DAG is too big"
"Backfilling just a piece of the DAG can be painful"
""
""
"Breaking them up into a family of bite-sized DAGs:"
""
"Pros:"
"For each DAG, it's obvious what it is doing and clear"
"Backfilling pieces is extremely easy "
""
"Cons:"
"Keeping track of all the tiny DAGs can be cumbersome"
"Backfilling the family of DAGs can't be done with 1 command

Data engineering != just building data pipelines! "
""
"Data engineering has much more to it than writing Spark jobs scheduled by Airflow. "
""
"You need to have a good business sense. "
"You need to have empathy for your downstream users whether they are data scientists, app users, or other data engineers. "
"You need to understand trust on a deep level and how/when trust is broken. "
"You need to communicate timelines effectively and prioritize the highest value work first. "
"You need care about costs and consider the impact you're having on the environment. "
"You need to be a good team member and contribute to your teams culture. "
"You need to be confident and say no when you're getting too many requests at once. "
""
"#dataengineering

Adhoc SQL queries and SQL queries running in production will generally look pretty different. Copy and pasting the data scientist’s query into Airflow isn’t quite enough to be considered “putting into production” "
""
"Some high-level things to look for in ad-hoc queries that should be changed before moving to production. "
""
"1. GROUP BY 1,2,3… / ORDER BY 1,2,3"
"This is used to speed up writing ad-hoc queries. Please spell them out in production. "
""
"2. SELECT * "
"This grabs all the columns quickly for ad-hoc queries. Please spell out * in production. "
""
"3. LIMIT 1000"
"Limits should generally be removed when moving to production since you want the entire data set. "
""
"4. Sub queries "
"Sub queries should almost always be abstracted as CTEs when running in production. "
""
"5. WHERE date >= startDate "
"This should be switched to WHERE date BETWEEN startDate AND endDate. Otherwise your pipeline won’t be idempotent and will produce different results depending on when it’s ran. "
""
"#dataengineering

I’ve worked for so many companies as a data engineer. #dataquality is something that some companies miss the mark on. "
""
"A few of my favorite things about #dataengineering at Airbnb. "
"1. They really care a lot about data quality. They have company-wide initiatives that are in the DNA of the company now to care about quality.  "
"2. They’ve transformed the data engineering landscape by open sourcing tools like Superset and Airflow. "
"3. Airbnb has been transitioning from a high-speed startup. Finding the middle ground between moving fast and maintaining high quality data is one of my favorite balancing acts.  "
""
"Join us on Wednesday, July 28th at 12pm Pacific Time to hear how we've evolved our teams and tools to ensure trustworthy, high quality data at Airbnb! Visit the link in the comments to register for this awesome virtual event!"
""
"#airbnbtechtalks #airbnbtech #insideairbnb #data #dataengineering #datascience

My favorite tooling depending on the technical job I’m working on. "
""
"1. Full Stack Web App. "
"Frontend with NextJS"
"Backend with ExpressJS"
"Database with Postgres"
"ORM with MassiveJS or Sequelize"
""
"2. Data visualization "
"In HTML? D3.js"
"Exploratory data analysis? Tableau"
"Scalable dashboarding? Apache Superset "
""
"3. Data pipelines "
"Streaming? Apache Flink"
"Batch and complex? Apache Spark"
"Batch and simple? Snowflake or Presto"
"Orchestrator? Apache Airflow  "
"Language? Python, Scala, or SQL depending on team and complexity "
""
"4. Miscellaneous "
"IDE? Anything by Jetbrains (e.g. IntelliJ, WebStorm, DataGrip, PyCharm) "
"Vim or Emacs? Vim "
"Linux Distro? Ubuntu "
"Shell? Zsh "
"JVM build system? Gradle"
"Cache? Redis "
""
"#dataengineering "
"#softwareengineering

Many people have asked me how to be successful in data engineering. I like to think of data engineering as a cake. "
"The core cake pieces are: "
""
"1. SQL, Python, Scala, and/or Java "
"2. Data modeling and data quality checks"
"3. Communication and planning skills "
"4. Dashboard and data viz "
""
"The frosting and sprinkles on the cake are the specific technologies used to actually implement the skills listed in the core cake. "
""
"In the real-time world: Flink/Spark Streaming, Kafka, SQS"
""
"In the batch world: Spark, Presto, Hive, HDFS, Airflow, RDBMS, S3"
""
"It’s better to work on the core cake skills before really jumping deep into the implementation technologies.  That way when those technologies change, you aren’t starting from square one. "
""
"#data"
"#dataengineer "
"#dataengineering "
"#learningtech  #apachespark

 Follow Technology stacks that haven't changed much since 2016:- MERN stack is still king. Postgres is seeing more love but Mongo still has a lot of adoption.- Spark + Airflow + S3 is still on fire for #dataengineering. Although Snowflake is starting to make some ground!Technology moves more slowly than you think sometimes! Especially when there's a big shift! …see more 429 22 comments 6 reposts Like Comment Repost Send in a private message Send

Breaking into data engineering can be challenging! "
"Here are the skills you need to know to break into data engineering: "
""
"- SQL"
"Basic syntax along with window functions. Understand the different types of JOINs that you can use. "
""
"- Python "
"Understand basics along with some data structures. Lists, maps, queues, and stacks will get you really far. Understand how to install packages with pip and use other people's code. "
""
"- Distributed Compute"
"Horizontal scalability is an important part of data engineering. Learn the basics of a tool like Spark, Snowflake, or BigQuery to get your feet wet here! "
""
"- Data quality"
"How do you validate a data set and know it's high quality? Learning how to check for things like duplicates, weird distributions, NULLs, etc. SQL skills will help you greatly here! "
""
"- Job Orchestration "
"How can you automate your data ingests so your pipelines run while you sleep? Maybe use something like Airflow or Mage to get the ball rolling here. "
""
""
"Join my free Discord community to get more of the support you need in acquiring all the skills needed here: https://lnkd.in/gubZFMXB"
""
"#dataengineering 

LLMs are going to make data modeling an overpowered data engineering skill! "
""
"If you can describe the input schemas and the output schemas of your pipeline, LLMs can generate the SQL details and Airflow DAG steps that are in between! "
""
"I noticed when working with ChatGPT that it does much better at generating pipelines when you supply the expected output schema as a "nudge" in the right direction. "
""
"If you just supply the input schemas and analytical pattern (slowly-changing dimension, cumulation, etc), it doesn't quite know what to output and the results are kind of garbage. "
""
"Definitely try this when writing your next pipeline! You'll be surprised how much faster you are! "
""
"#dataengineering "
"

Here is how EcZachly Inc's boot camp is different from other data engineering boot camps:"
""
"- It covers fundamentals more than technologies. You'll learn foundational skills that don't change like: "
"-- data modeling"
"-- data architecture"
"-- distributed computing (via Spark)"
"-- data quality and testing (via Airflow)"
"-- unit and integration testing of ETLs"
""
"- Homework is submitted through git and PRs to feel more like a real-world data engineering job"
""
"- The material is sourced from my experience doing data engineering at Facebook, Netflix, and Airbnb so you'll get better at scalable data engineering"
""
"- It's shorter, most DE boot camps are 3-4 months, but EcZachly Inc's is 6 weeks. I want to give you the highest value knowledge and get you on your way! "
""
"If you're interested in the next cohort, please follow EcZachly Inc and fill out this form here: https://lnkd.in/gqpn2tJW "
""
"#dataengineering "
"#softwareengineering 

Conceptual knowledge is more important than tooling! "
""
"Spark is a means of distributed compute"
"Airflow is a means of job orchestration "
"dbt is a means of data quality"
"Tableau is a means of data visualization "
"Iceberg is a means of data lake storage "
"Flink is a means of stream processing "
"Postgres is a means of consistent + available storage. "
"MongoDB is a means of consistent + partition tolerant storage "
"Cassandra is a means of available + partition tolerant storage "
"Parquet is a means of data compression and serialization "
""
"#dataengineering

Some unstoppable trends: "
""
"- tech layoffs creating a surge in flexible work and solopreneurship with platforms like SylphAI and topmate.io "
""
"- data engineering becoming both more software engineering-focused as data products become more mainstream "
""
"- simpler data engineering becoming less software focused with low-code solutions like Fivetran "
""
"- Airflow becoming the “weakest link” in the data engineering stack and replaced by popular competitors like Mage, Prefect and Dagster "
""
"- MLOps getting a lot more attention as more and more companies learn the pains of running a model in production "
""
"- MLOps, DataOps, data pipeline and ML training tooling see a unification when people realize they are strongly connected "
""
"Any MLOps companies you know that deserve a shoutout?"
""
" #dataengineering "
"#machinelearning

Technology stacks that haven't changed much since 2016:"
""
"- MERN stack is still king. Postgres is seeing more love but Mongo still has a lot of adoption."
"- Spark + Airflow + S3 is still on fire for #dataengineering. Although Snowflake is starting to make some ground!"
""
"Technology moves more slowly than you think sometimes! Especially when there's a big shift! 

Here’s my skill set that got me senior data and software engineering jobs at FAANG and Airbnb: "
""
"- JavaScript / Java / Python / Scala - level expert "
"- Spark - level expert "
"- Airflow - level expert "
"- Hadoop / Hive - level expert "
"- React - level intermediate "
"- D3 / data visualization - level intermediate"
"- Spring Boot / ExpressJS - level expert "
"- Postgres - level expert "
""
"You probably don’t need to know all of these things to land a great job. This was just the skill set that worked for me! "
""
"#dataengineering "
"#softwareengineering

The S tier data engineering stack is:"
""
"- S3 and Apache iceberg for storage "
"- Spark and Flink for compute "
"- Airflow or Mage or Prefect for orchestration"
"- Great Expectations for data quality "
"- Druid for fast columnar storage for dashboards "
"- AWS as the cloud platform "
""
"What’s missing or wrong on this stack? "
""
"#dataengineering

I’ll be emailing a signup form for my data engineering bootcamp in 10 days! "
""
"Here are some criteria I’ll be looking for: "
""
"Required: "
"- at least 2 years of SQL experience "
"- basic Python knowledge "
"- has worked with Airflow/Mage/Prefect before "
"- has a basic understanding of distributed compute "
""
"Preferred: "
"- experience programming in Scala "
"- writing skills and social media presence "
"- asks good questions "
""
"Subscribe to my newsletter at https://lnkd.in/gPQV5mZu  to receive this form! "
""
"If you don’t quite meet these requirements, please stick with me because I’ll be releasing this content as a self-paced bootcamp in the summer! "
""
"Follow EcZachly Inc to not miss any other updates about what I’m up to!"
""
"#dataengineering

This “unconventional” career path will become very common in the near future.  "
""
"self-study (with tools like Lonely Octopus) ->"
""
"portfolio building and project completion ->"
""
"First job as data analyst using Power BI or Tableau + excel + SQL -> "
""
"Data engineer using Python, BigQuery, Snowflake, Databricks, and Airflow -> "
""
"Data architect who are owners of large data warehouses and data models OR grow into ML engineer to build after the #dataengineering fundamentals.

So many people say #dataengineering is so dynamic and changing. I don’t know if I agree with that completely. "
""
"Since 2016, it’s been obvious that the state of the art is: "
""
"- Spark for compute "
"- S3 for storage layer "
"- Airflow for orchestration"
""
"That’s been the stack I’ve used for the last 7 years. The only piece here I see changing over the long run is Airflow getting replaced by better options like Mage. "
""
"There has been innovations in metadata and file storage though. Moving from Hive metastore to Apache Iceberg has been a big step up for my game. Iceberg is so much more naturally accepting of both streaming and batch! "
""
"The platforms like Snowflake and Databricks are changing data engineering outside big tech but I haven’t really used them as much!

You shouldn't try to learn all of data engineering at once! You'll get overwhelmed and feel like you aren't making any progress!"
""
"A piecemeal approach to eating the data engineering elephant is better. "
""
"Start with: "
"- SQL"
"Get good with SELECT, GROUP BY, WHERE, HAVING, JOIN, etc. DataLemur 🐒 (Ace the SQL & Data Interview) is a great resource to get into this. "
""
"Then branch into:"
"- Python"
"Get good with loops, variables, classes, dictionaries, tuples, and  arrays. LeetCode still seems like the best place to practice this. "
""
"Then branch into:"
"- job orchestration"
"Airflow is the most popular option here but the startup costs are kind of high to get going. A new option is Mage that has a very easy startup and can orchestrate things as well as Airflow. "
""
"Then branch into: "
"- distributed compute (Snowflake, Spark, BigQuery, etc)"
"Almost all of these platforms have free trials. Some key things here to learn about are partitioning, memory, broadcast joins, and caching. "
""
"Then branch into: "
"- data modeling"
"Learn about fact tables, dimension tables, slowly changing dimensions, cumulative table design, and change data capture. Reading one of Bill Inmon's books about this will get you ahead here! "
""
""
"#dataengineering 

Data engineers love and hate the following things: "
""
"- data quality checks "
"- comparing two tables to see if they match "
"- Airflow "
"- backfills "
"- finding data inaccuracies "
""
"#dataengineering

Data engineering is more than SQL, Spark and Airflow! "
""
"You need to be aware of data models! "
"Is the current ask a new model or a modification of an existing on? How can you automate validation of your assumptions about data quality?"
""
"You need to be aware of the business! "
"Should you build model A or B first? Which impacts the business more? "
""
"You need to be aware of costs! "
"Should this pipeline be deprecated or optimized? Should you sample here or is the entire data set needed? "
""
"You need to be aware of your team! "
"Who can you ask for help about what? How can you distribute maintenance costs? Who owns what? "
""
" #dataengineering

Trends that are unstoppable: "
""
"- remote work becoming a bigger percentage of roles "
"- flexible work growing where more people do shorter term projects and contracts joining platforms like SylphAI "
"- the rise of the solopreneur and personal brands that generate millions"
"- AI transforming how we do work with things like ChatGPT and #datascience "
"- people quitting their jobs to build their own stories"
"- Apache Flink becoming as mainstream as Apache Spark for #dataengineering "
"- Airflow losing market share to products like Mage, Prefect and Dagster "
"- Analytics engineering becoming a very popular role for business focused data engineers

Data engineering tooling in 2023 will be defined by a few battles. "
""
"- Airflow vs Mage vs Prefect "
"Airflow is the entrenched player that data engineers love but are annoyed by. Mage is a newer option that streamlines many aspects of ETL development including data quality and initial setup.  Prefect is loved by many data engineers and is seeing more adoption as well! "
""
"- Spark vs BigQuery vs Snowflake "
"Spark has been so huge for 5 years now. Seeing Google and Snowflake offer compelling alternatives for compute is exciting and it’ll be interesting to see where this goes! "
""
"- Hive vs Iceberg vs Delta Lake "
"The big tech companies have been married to the Hive meta store for longer than my career.  Apache Iceberg offers compelling functionality to make it worth migrating! Iceberg you can actually append new data instead of always having to INSERT OVERWRITE. "
""
"- AWS vs GCP vs Azure "
"This cloud battle will continue. AWS still has the lead here but you never know how these types of things will shake out! "
""
"- Python vs strongly typed languages "
"Is data engineering closer to data analytics or software engineering? This age-old question will continue to ebb and flow as people write pipelines to solve problems. The data analytics crowd loves Python. The software engineering crowd loves Scala/Java/Rust/Go. "
""
"- Streaming vs Batch "
"Many companies will keep pushing for real-time. Many data engineers will push back because it’s a false requirement.  Others will accept the requirement and build real-time pipelines that make the business more nimble. "
""
"Been itching to write some posts since I took a break! So excited for the future ahead!"
""
"#dataengineering "
"#softwareengineering "
"#dataanalytics

Data engineering at big tech companies is often substantially different from the wider data engineering community. "
""
"Big tech companies often roll their own frameworks for things. "
""
"At Airbnb, "
"- I don’t use dbt, I use Wall, an internal library "
"- I don’t use Snowflake, Databricks or BigQuery, I use Spark on EMR"
"- I don’t need to think much about deploy since that’s managed for me"
"- I don’t have to manage an Airflow server since that’s managed for me "
""
"In some ways, doing #dataengineering outside big tech gets you exposure to a broader set of technologies than doing it inside of big tech. "
""
"One of the main reasons I quit working at Facebook was because I almost exclusively worked with internal tools that had no application outside Facebook which made me think what I was learning wasn’t that valuable!

My favorite tech at the end of 2022: "
""
"- REST API - ExpressJS"
"- Queue - Apache Kafka "
"- Job Orchestration - Apache Airflow"
"- Batch data processing - Apache Spark "
"- Stream data processing - Apache Flink "
"- Data quality - Great Expectations"
"- Frontend - React, Svelte is close though "
"- SQL Database - Postgres "
"- NoSQL Database - MongoDB"
"- Cache - Redis, Memcached is close though"
"- Graph Database - Neo4j "
""
"#dataengineering "
"#softwareengineering

There are so many ways to schedule a job in data engineering: "
""
"- CRON"
"The most basic way to do it. The issues here are the job has no resilience, retry, or DAG dependency capabilities. CRON is great to start a schedule but not robust enough for true production #dataengineering "
""
"- Airflow "
"Airflow is the most poplar framework for job scheduling. Created by my good friend Maxime. It allows you to create operators that define a job on a schedule. It has nice retry, resilience and DAG management. Like CRON but way better. Some pains here are around deploying and maintaining the Airflow server.  Although services like Astronomer make that much easier.  "
""
"- Mage "
"Mage is similar to Airflow except more specialized for data intensive workflows where as Airflow is more general purpose. Direct integrations with things like DBT make data quality an integral part of the pipeline rather than an after thought added later in Airflow. "
""
""
"- Prefect, Dagster, Oozie "
"These are the other long-tail options that offer some advantages over airflow like passing data between tasks. They haven’t seen as much adoption yet because they’re only marginally better than Airflow and migration costs are painful.

I’ve focused on different languages and tech throughout my career.  "
""
"2011: wrote my first line of code in Java "
""
"2012 - 2014: Java, Android development, Git "
""
"2015 - 2018: SQL, Python, Hadoop, Java MapReduce, SparkSQL, JavaScript, NodeJS, ExpressJS "
""
"2018 - 2022: Scala, Rust, Bash, Spark, Airflow, TypeScript, NextJS, Serverless "
""
"I put Java, Scala, Python, SQL and JavaScript as programming languages on my resume. "
""
"You can try a bunch of stuff out and keep learning and growing! Tech keeps changing and #softwareengineering is signing up for a career of learning!

I’m happy to share that I’m starting a new position as Advisor at Mage!  I'm going to be working with Tommy Dang and his awesome engineering team to build a next-generation competitor to Airflow! "
""
"This product is going to revolutionize #dataengineering! I'm so excited to be onboard on this team!

The tools and languages that I use everyday as a data engineer at Airbnb "
""
"Scala: 60%  - mostly using Apache Spark or Apache Flink"
""
"SQL: 20% - creating dashboards with Apache Superset. Validating data with queries. "
""
"Python: 10% - creating DAGs with Airflow "
""
"Java: 5% - working with online libraries and compiling them in my data pipelines "
""
"YAML: 5% - defining metrics with Minerva and data quality checks with Wall "
""
"#dataengineering "
"#softwareengineering

Tried to start a backfill from 20217-01-01 today and Airflow wouldn't let me. How unreasonable! "
""
"#dataengineering 

Fundamentals matter more in #dataengineering than tooling! "
""
"Knowing distributed compute is more important than Spark vs BigQuery "
""
"Knowing cloud computing is more important than AWS vs Azure "
""
"Knowing data modeling is more important than S3 vs Snowflake "
""
"Knowing job orchestration is more important than Airflow vs Prefect "
""
"Knowing how to communicate clearly is more important than debating technology!

#dataengineering and #datascience has its own set of Pepsi or Coke, Android or iPhone questions. "
""
"- Bayesian or Frequentist"
"- Explainable model or Ensemble model"
"- Databricks or Snowflake "
"- Postgres or MongoDB"
"- AWS or Azure "
"- Python or R"
"- Tensorflow or PyTorch "
"- Airflow or Prefect "
"- Hive or Iceberg "
"- Spark or Flink"
"- Lambda architecture or Kappa architecture "
""
"Sure you can pick other options outside these two but you’d probably be wrong.

If I had to start learning #dataengineering all over again, I’d follow this plan, mostly in order: "
""
"- Learn SQL "
"— Aggregations with GROUP BY"
"— Joins (INNER, LEFT, FULL OUTER) "
"— Window functions "
"— Common table expressions "
""
"- Learn about data modeling"
"— read about data normalization and 3rd normal form"
"— read about fact, dimension, and aggregate tables "
"— read about efficient table designs such as cumulative "
"— learn about complex data types like MAP, ARRAY, and STRUCT "
""
"- Learn Python"
"— Learn loops, if statements"
"— try out some really solid libraries like Pandas, numpy, scikit-learn, and Great Expectations"
""
"- Learn about data quality "
"— how do you write a good data check? "
"— how can you implement the write-audit-publish pattern in your pipelines? "
""
"- Learn about distributed compute "
"— Read about MapReduce and see how it guided the design of todays distributed computing "
"— learn about partitioning, skew, and spilling to disk"
""
"- learn about job orchestration "
"— read about CRON "
"— try out a scheduler like Airflow or Prefect "
""
"- Apply distributed compute principles "
"— a free trial with Snowflake or BigQuery is a good place to start "
"— or try out Spark + S3 if you want to be adventurous "
""
"If you follow this, you’ll feel way more confident in interviews.

Apache Airflow can abused in ways that make some people not like it! Quit doing these things with Airflow please. "
""
"- creating operators / queries that don’t perform idempotent operations "
"If you run a task 1 time or 100 times you should get the same result. "
"Avoid INSERT INTO queries and opt for MERGE or INSERT OVERWRITE queries. Make sure your operators follow this paradigm too. "
""
"- overly using subdags "
"Subdags can be a huge pain to maintain since you have to click through multiple menus to find the true failed task. "
""
"- using depends_on_past=True for backfills "
"This parameter will destroy your parallelism and cause your backfills to take way longer than necessary. Cumulative table DAGs may need this set to true though. "
""
"- creating DAGs that are too complex and really should be multiple DAGs"
"My cutoff point for DAG complexity is when the Airflow UI starts freezing when you open the DAG. (>100 tasks usually)"
""
"- using cross-DAG task dependencies  "
"Between DAGs should use data dependencies instead of task dependencies. These cross-DAG task dependencies seem to always be a huge mess when implemented."
""
"#dataengineering

My favorite open-source batch #dataengineering stack is: "
""
"- Apache Airflow for scheduling and job orchestration "
"- Apache Spark for batch processing "
"- Apache Iceberg or Hive for metadata management"
"- S3/GCP/Azure for regular storage"
"- Druid for fast columnar storage "
"- Apache Superset for visualization "
""
"There’s a ton of other very valid ways to do data engineering on other stacks.  "
""
"What DE tools make your eyes light up when using them?

How do you get a job in #dataengineering  given the huge increase in job openings? "
""
"1. Know SQL"
"You’ll be grilled on SQL for sure. Know window functions, common table expressions, aggregations, temp tables, and views."
""
"2. Know data modeling "
"You’ll be asked questions on how to design a data model. Make sure to know about fact tables, dimension tables, aggregate tables, and cumulative tables. "
""
""
"3. Know data structures and algorithms "
"You’ll be asked questions where you’ll need to know the data structures: stacks, queues, trees, and heaps. You’ll need to know algorithms: depth and breadth first search, binary search, and recursion "
""
"4. Know pipeline tooling "
"You’ll probably be asked questions about Spark, BigQuery, Airflow, and/or Snowflake. Make sure you build up some knowledge on these topics.

#dataengineering has had so many inflated hype cycles in the last 7 years. "
""
"2014: Hadoop / MapReduce"
"2015: Hive/Pig"
"2016: Spark "
"2017: Airflow"
"2018: Flink "
"2019: Data Mesh"
"2020: Snowflake / BigQuery"
"2021: dbt / analytics engineering "
"2022: Modern Data Stack

In 2022, my favorite technologies to learn are: "
""
"- ReactJS "
"- NodeJS "
"- Apache Spark "
"- Snowflake"
"- BigQuery "
"- Delta Lake "
"- Apache Airflow "
"- Spring Boot "
""
"Any that I’m missing? "
""
"#softwareengineering "
"#dataengineering

Data engineering tooling has different groups of stacks depending on the company. "
""
"- the big tech company stack "
""
"- compute: Spark "
"- Orchestration: Airflow (or similar)"
"- data quality: custom built "
"- serving layer: Druid"
"- storage: Iceberg + S3 "
""
"- the mid-sized company stack "
""
"- compute: Snowflake/BigQuery "
"- orchestration: Airflow or Fivetran or Informatica"
"- data quality: Great expectations and DBT "
"- serving layer: Tableau extracts or Druid "
"- storage: Snowflake/BigQuery "
""
"- the startup stack "
""
"- compute: Postgres "
"- orchestration: CRON"
"- data quality: skipped "
"- serving layer: SQL queries in Postgres"
"- storage: Postgres"
""
"What would you change in these stacks?"
""
"#dataengineering

Data engineering has many "this or that" questions"
""
"- Python or Scala?"
"If you don't know either, start with Python. If you want to transition to the software/data engineer archetype, pick up Scala later. "
""
"- Streaming or Batch?"
"A vast majority of data engineering jobs are batch oriented. This will still be true in ten years. Streaming-oriented data engineering jobs pay better since the skillset is more niche and harder to come by. Remember there's a middle ground with things like microbatch. "
""
"- Kimball or One Big Table? "
"Kimball is better at preserving data integrity and for 80+% of the data modeling use cases out there it is going to be the preferred way to model the data. One Big Table has its place though especially if you're trying to minimize shuffle. I've seen some really big performance gains from switching from Kimbal to OBT but just because I saw them at Airbnb doesn't mean you'll see the same. "
""
"- Snowflake or Databricks?"
"I like Databricks a lot since it has the versatility of Apache Spark. That being said, it's a more technical platform that takes much longer to learn and set up. The amount of time it takes to get value out of Snowflake is very little and that's a very impressive quality of Snowflake."
""
"- AWS or GCP or Azure?"
"AWS is the clear leader in market share and I have a slight bias towards using it over GCP or Azure. That being said, there will always be great Azure and GCP data engineering jobs as well! "
""
"- Airflow or Mage or Prefect or Dagster?"
"Airflow is the 9000-pound gorilla in this fight that is looking to be dethroned. The challengers have some really great features that are making Airflow look dated. I'm teaching Airflow in my boot camp though since it has the highest adoption by far"
""
"#dataengineering 

Data architecture always revolves around pushes or pulls!"
""
"The "pull" architecture is the most common and includes the following technologies:"
""
"A batch computing engine such as Apache Spark, BigQuery, Snowflake"
"A job orchestrator such as Airflow, Mage, Prefect, or Dagster"
"A place to put the batch of data such as Apache Iceberg, Delta Lake, Snowflake, Druid"
"An API to query the data on demand such as HTTP or SQL"
""
"The "push" architecture also called the "real-time" architecture is substantially different and includes the following technologies:"
""
"A streaming computing engine such as Apache Flink, Spark Structured Streaming "
"A set of jobs that run 24/7 to process data as it arrives"
"A queue of events that are processed such as Apache Kafka or RabbitMQ "
"A place to put the streams of data such as Apache Iceberg or Apache Kafka"
"An API to expose the data in real-time such as Websockets or Kafka consumers"
""
"#dataengineering"
"#softwareengineering 

Data engineering is much more than knowing Spark and Airflow! "
""
"Data engineers need to be communicative with the business. Can you produce data sets that solve business problems? Can you produce data sets that are a delight to use? "
""
"Data engineers need to be team players. Can you work well with upstream software engineers and downstream data analysts? Can you work with your team and manage oncall well? "
""
"Data engineers need to care about quality. Do you understand the link between data trust and data quality? Can you produce reliable numbers that people can make decisions from? Do you have a process in place when anomalies occur? "
""
"What other things do data engineers need to care about? "
""
"#dataengineering

Data analysts make great data engineers. What skills should they learn to transition? This episode today is about how data analysts really have a lot of the core skills needed to do data engineering and what tools they should add to their tool belt to make that career transition!

Data analysts and data engineers generally have a shared skillset around: 

- SQL
- Data analysis
- Gathering requirements from stakeholders
- Solving business problems with data

Data analysts also often have strong storytelling and data visualization skills as well which can be strong assets in data engineering!

The key things that data analysts need to learn to really succeed at data engineering are:

- Git
   - without git, collaborating on pipeline code with your teammates it’s very challenging
- Python
   - a vast majority of data engineering jobs require knowing Python. 
- A way to transform the data at scale, such as tools like Apache Spark, Snowflake, and BigQuery 
- A scheduling library to automate your pipelines, such as Apache Airflow


If data analysts can learn these technical skills, they become unstoppable because they already know how to ask the right business questions. These skills empower them to go and get that data as well! 

What other skills do you think data analysts need to learn in order to be effective data engineers? Drop a comment and let’s discuss! 

I hoped you liked this video today. Please like and subscribe to my channel for more videos about data!

How is data engineering different between Airbnb, Facebook, and Netflix? How is data engineering different between Facebook, Netflix, and Airbnb?

So, I started my big tech data engineering career at Facebook in 2016. I worked there for about 2 years. Then I jumped to Netflix where I worked for about 2 years. I’ve currently been at Airbnb for about a year. 

Working at these companies gave me a taste of a bunch of different flavors of data engineering. They have some similarities and some differences. Let’s dive into it! 

Let’s start with when I worked at Facebook. 
At Facebook, I worked in growth analytics data engineering. There was a strong focus in my role creating metrics that could be used to measure incremental impact of experiments. 

The languages I used: 
- Python
- SQL

The execution engines I used were:
- SparkSQL
- Apache Hive
- Presto/Trino

The storage systems I used were:
- Hadoop

The file formats I used were:
- ORC (optimized row columnar)

The scheduler I used was Dataswarm (https://asiliconvalleyinsider.com/2016/05/01/data-engineering-facebook/), which is Facebook’s internal scheduler very similar in API design to Apache Airflow. 

My role at Facebook had the strongest focus on visualization out of the three companies. I used a mix of Tableau and an internal tool called Unidash (https://medium.com/analytics-and-data/i-hear-facebook-has-data-does-it-have-data-tools-a9f4c4d82ee7)

Data quality checks were generally put in after a data quality error in the data instead of proactively since Facebook really focuses on “moving fast” 

Something unique about Facebook was Presto was often used in ETL code where as at the other companies Presto is strictly used for auditing only and Spark does the heavy lifting. Also every pipeline I wrote at Facebook was SQL-based which is very different from Netflix and Airbnb. 

Next up, Netflix

At Netflix, I worked in platform and security data engineering. I did a mix of creating metrics for experiments and creating master data for the InfoSec teams to use. 

The languages I used were:
- Python
- SQL
- Scala 

The execution engines I used were:
- SparkSQL and Scala Spark
- Presto/Trino (for data audits and analysis)

The storage systems I used were:
- S3
- Apache Druid (for visualization)
- Postgres (for low-latency queries)

The file formats I used were:
- Parquet

The scheduler I used was an internal scheduler called Meson (https://netflixtechblog.com/meson-workflow-orchestration-for-netflix-recommendations-fc932625c1d9)  which took in a YAML file to generate the pipeline DAG. I prefer Python APIs for schedulers over YAML-based APIs. 

Proactive data quality was pushed heavily at Netflix. They really like the write-audit-publish pattern and want you to have all your ETLs use it. 

I did very light visualization work at Netflix. They have analytics engineers and viz specialists there who focus more deeply on it. 

Netflix also focused on unit and integration testing of ETLs, which Airbnb also does but Facebook did not since Facebook is so SQL-focused. 

Something unique about Netflix is there was a good mix of PySpark and Scala Spark due to their “freedom and responsibility” culture which allowed for more variations in ETLs.  



Next up, Airbnb

At Airbnb, I work in commercial products data engineering. I work mostly on creating high quality master data and leadership since I’m at tech lead at Airbnb. 

The languages I use are:
- Python
- SQL
- Scala 

The execution engines I use are:
- SparkSQL and Scala Spark
- Presto (for data audits and analysis)

The storage systems I use are:
- S3
- Apache Superset (for visualization)
- Postgres (for low-latency queries)

The file formats I used were:
- Parquet

The scheduler I used was of course, Apache Airflow, the popular open source tool created by Airbnb. 

Proactive data quality is pushed heavily at Airbnb. There’s a company wide process called the MIDAS process which is a very thorough process for creating high quality data (https://medium.com/airbnb-engineering/data-quality-at-airbnb-870d03080469). We use an internal framework called Wall for our data quality checks (https://medium.com/airbnb-engineering/how-airbnb-built-wall-to-prevent-data-bugs-ad1b081d6e8f).

This page allows you to search Zach Wilson's content

Search my content library!
or view categories

Search results for: Airflow

Search my content library! or view categories

Search results for: Airflow

Search my content library!
or view categories