Master data engineering and AI like a big tech engineer

OpenAI

Amazon

Netflix

Tesla

Salesforce

Robinhood

Microsoft

Real Time Formula 1 Analytics

Big Bag Data

Chess.com Analytics

Trading Strategy for Crypto Currency - Analytics Engineering Capstone Submission

BetFlow - Real Time Sports Betting App

Showcase of student projects

Capstone Projects

Book a discovery call with Zach

Discovery call

Our students have gone on to work at companies like Meta, Airbnb and Amazon. As well as achieve 100% raises!

Fast Track Your Career

Immediate free cloud access to Databricks, AWS, Snowflake, Astronomer, and more!

Free Cloud Access with tons of hands on exercises

Weekly Guest Speaker Sessions

Why Choose DataExpert.io Academy?

Anthropic

Databricks

Discord

Snowflake

GitHub

A five-round deep-dive mock interview focused entirely on data architecture design at production scale. Each round escalates in complexity: from batch warehouse design, to real-time streaming, to experimentation platforms, to ML feature stores, and finally to a 500TB migration war story. Every question requires a detailed architecture diagram with numbered requirements, trade-offs, and technology choices justified. Designed for senior data engineers and architects.

Data System Design Gauntlet

Separate staging and mart schemas, incremental loading pattern (CDC or timestamp-based), orchestration tool (Airflow/dbt/Prefect), data quality framework (Great Expectations or similar), schema registry for drift detection, S3/data lake for archive, clear separation of operational and analytical concerns, documented SLAs per data source.

Batch ELT Warehouse for Mid-Size SaaS

Kafka/Kinesis for ingestion and buffering, schema registry (Confluent/Protobuf) for evolution, stream processor (Flink/Spark Structured Streaming/Kafka Streams) for exactly-once semantics and deduplication, dual writes to OLAP store (Druid/Pinot/ClickHouse) and lakehouse, dead-letter queue for unparseable events, idempotent write pattern for exactly-once, state store for deduplication windows, event replay mechanism with reprocessing framework.

Real-Time Clickstream Pipeline: 200K Events/sec

Assignment table (immutable log of user-experiment-variant mappings), exposure table (deduplicated events per user per experiment), metric computation layer (both batch + streaming), metric definition store (with versioning), real-time aggregation layer (sliding windows for early results), guardrail alerting, statistical computation module, historical experiment archive with time-travel queries, clear separation between assignment service (write-optimized) and analytics queries (read-optimized).

Experimentation & Metrics Platform

Offline feature store (data lake + metadata layer) for training data with versioned snapshots and point-in-time joins, online feature store (Redis/DynamoDB/Feast) for serving, materialization layer (batch + streaming) that updates both stores, feature registry with computation code and metadata, lineage tracking, freshness monitoring, separate compute for training (SQL/Spark) vs serving (optimized retrieval), versioning strategy for features, clear separation of concerns between data engineering (feature creation) and ML (feature usage).

ML Feature Platform: Offline + Online Serving

Dual-write or shadow-read architecture with clear pros/cons (dual-write requires app changes, shadow-read adds latency), event-driven job flipping (metadata event in catalog, job polling mechanism or webhook), comprehensive validation layer (row counts, checksums, sampling), rollback via metadata swaps (pointers in metastore, not data copies), partition reconciliation strategy for late arrivals, cost analysis per table including S3 tiering, risk mitigation (canary jobs on Iceberg first, then wave-based migration of job groups), monitoring dashboard showing % migrated, data drift, rollback triggers.

Data System Design Gauntlet

In order. Round N+1 unlocks when Round N passes.