# DataExpert.io Academy > Master data engineering and AI like a big tech engineer Website: https://www.dataexpert.io/ Contact: support@dataexpert.io Elsewhere: https://www.linkedin.com/in/eczachly, https://www.github.com/EcZachly, https://www.instagram.com/eczachly, https://www.Twitter.com/eczachly, https://www.YouTube.com/c/datawithzach, https://blog.dataexpert.io, https://discord.com/invite/JGumAXncAK ## About ##### We will be doing a **[July 2025 live boot camp on AI](https://www.dataexpert.io/program/ai-boot-camp-starting-june-7th-8623/details)**! ##### If you are looking for the free boot camp join [here](https://learn.dataexpert.io/pricing) ##### Here are all the benefits of joining our academy! - **Community**: Join our exclusive DataExpert.io Boot Camp community, a dedicated space where you can connect with learners, get career advice and make friends! - **Guest Speakers**: Get inspired and acquire practical, real-world knowledge from industry leaders during our weekly exclusive guest speaker series. - **Homework Support**: Automated AI-driven feedback for faster learning - **Experience-based Learning**: Zach brings his 7 years of working at Facebook, Netflix, and Airbnb at petabyte scale into every lecture! ## Programs ### Snowflake and dbt Analytics Engineering Boot Camp URL: https://www.dataexpert.io/program/snowflake-dbt-analytics-engineering-boot-camp-4721 Master Snowflake, Airflow and dbt and become a powerful analytics engineer ### All-Access DataExpert.io Subscription URL: https://www.dataexpert.io/program/all-access-dataexpert-io-subscription Type: Subscription This gets you access to all self-paced content and the discord community of highly motivated learners! Includes: - Build an awesome portfolio! - Join the dedicated community - Weekly Q&A with Zach Wilson - Master the hottest AI tech: RAG, OpenClaw, Claude Code, and AI Agents - Master the hottest data engineering tech: Spark, Airflow, Delta, Iceberg, and Databricks - Master the hottest analytics engineering tech: Snowflake, dbt, BigQuery, and Iceberg ## Frequently Asked Questions ### When are live classes held? We do live classes from 6-8 PM pacific time. We record everything and the recording is available within 8 hours of the live class ### What is the refund policy? Subscription and self-paced products are non-refundable Live boot camp products have this refund policy: 25% processing fee for refund requests made within the first 7 days 50% processing fee for refund requests made within the first 14 days After 14 days, we offer no refunds ### How to create Schema? Go to the query page (https://www.dataexpert.io/query). and run the CREATE SCHEMA QUERY, which can be found under the Schema Info button. ### How to get Access to Cloud Services? Go to the Dashboard and click on the "Go to Onboarding page" button for the registered program. From there, all access can be activated. ### How to updated Discord username? Go to the page https://www.dataexpert.io/profile and update your details under Social Metadata. ### How to activate GitHub for accessing DataExpert.io repos? You need to update your GitHub username (example: https://www.github.com/[your_username]) in the dataexpert.io profile. Then you can try activating GitHub from the credentials tab. ### How to update timezone? Go to the page https://www.dataexpert.io/profile and update your details under Timezone ### How to unsubscribe DataExpert.io email notifications? You can disable the email notifications from the profile page. ### Where the live classes held? Live classes for lectures and labs take place on the Discord stages. Book club/ TA sessions happen on Google Meet. Links can be found on the Dashboard and schedule page(https://www.dataexpert.io/schedule) ## Mock Interviews Index: https://www.dataexpert.io/interviews ### Netflix Data Engineer Interview URL: https://www.dataexpert.io/interviews/netflix-data-engineer A themed six-round mock interview simulating a real Netflix Data Engineer loop. Covers SQL analytics on viewing behavior, data pipeline architecture for streaming events, dimensional modeling for the content catalog, behavioral questions around Netflix's Freedom & Responsibility culture, and a Python DSA problem built around recommendation data structures. ### Airbnb Analytics Engineer Interview URL: https://www.dataexpert.io/interviews/airbnb-analytics-engineer A themed six-round mock interview simulating a real Airbnb Analytics Engineer loop. Covers marketplace SQL analytics (funnels, cohorts, occupancy), dimensional modeling for a two-sided marketplace, experimentation platform architecture, behavioral questions around Airbnb's "Belong Anywhere" culture, and a Python problem themed around search ranking data processing. ### OpenAI AI Engineer Interview URL: https://www.dataexpert.io/interviews/openai-ai-engineer A themed six-round mock interview simulating a real OpenAI AI Engineer loop. Covers SQL for ML monitoring and usage analytics, system design for LLM-powered infrastructure, evaluation framework architecture, behavioral questions around OpenAI's mission of safe AGI, and Python problems involving token optimization and embedding search. ### Data Modeling Gauntlet URL: https://www.dataexpert.io/interviews/data-modeling-gauntlet A five-round, escalating deep-dive on dimensional modeling. Start with basic star schemas, progress through Slowly Changing Dimensions and multi-valued relationships, design petabyte-scale facts, and finish with SQL queries against your model. Calibrated for analytics engineers and data architects. ~2h 40min total. ### Data System Design Gauntlet URL: https://www.dataexpert.io/interviews/system-design-gauntlet A five-round deep-dive mock interview focused entirely on data architecture design at production scale. Each round escalates in complexity: from batch warehouse design, to real-time streaming, to experimentation platforms, to ML feature stores, and finally to a 500TB migration war story. Every question requires a detailed architecture diagram with numbered requirements, trade-offs, and technology choices justified. Designed for senior data engineers and architects. ### DSA Gauntlet URL: https://www.dataexpert.io/interviews/dsa-gauntlet A deep-dive five-question DSA interview for Senior Data Engineers. Covers sliding-window stream processing, heap-based merging with memory budgets, graph dependency traversal with topological sorting, LRU cache with TTL expiration, and external-memory deduplication algorithms. All problems are themed around real data pipeline challenges. ## SQL Practice Questions Index: https://www.dataexpert.io/questions ### Month with the Highest Total Births URL: https://www.dataexpert.io/question/highest-birth-month Difficulty: easy Determine the month with the highest total number of births in the playground.us_birth_stats table. The output should show the month and the total number of births. ### Find Viewers with Multiple Article Views in a Day URL: https://www.dataexpert.io/question/find-multiple-article-viewers Difficulty: easy Using the table playground.views, write a SQL query to identify all viewers who viewed more than one article on the same day. The table includes columns viewer_id (the ID of the viewer), article_id (the ID of the article viewed), and view_date (the date of the view). The result should contain a single column named viewer_id, listing each viewer who meets the criteria without duplicates, and should be sorted in ascending order of viewer_id. ### Find Product Prices URL: https://www.dataexpert.io/question/find-product-prices Difficulty: medium Using the table playground.product_prices, create a SQL query to find all products and their prices on 2023-08-17, assuming the initial price of all products was 10 before any price changes. Order the results in ascending order of product_id. ### Comparing State Fatal Collisions to the National Average URL: https://www.dataexpert.io/question/state-fatal-collisions Difficulty: medium Using playground.bad_drivers, write a SQL query to compare each state’s fatal collisions per billion miles to the national average. Include a column that indicates whether the state is "Above Average" or "Below Average". The resultant table should have three columns, "state", "fatal_collisions_per_billion_miles" and "comparison_to_national_avg". Show the result ordered by state name asc. ### Calculating Median Searches per User URL: https://www.dataexpert.io/question/calculate-median-searches Difficulty: hard Using the table playground.search_freq with each row representing the number of searches (searches column) made by a certain number of users (users column), write a SQL query to calculate the median number of searches per user up to one decimal place and be sure to cast it as a double. ### Who are the top 10 authors by number of reviews? URL: https://www.dataexpert.io/question/top-10-authors-by-reviews Difficulty: medium Using bootcamp.books, find the top 10 authors by reviews, no_of_reviews is a string column with bad data, try your best to get the values to parse correctly ### Finding Highest Grade and Corresponding Course per Student URL: https://www.dataexpert.io/question/highest-grade-corresponding-course Difficulty: medium Using the table playground.student_grades, write a SQL query to find the highest grade and its corresponding course for each student. In cases where there is a tie for the highest grade, select the course with the smallest course_id. The output should include student_id, course_id, and the grade, with the results sorted by increasing student_id. ### Determining Optimal Packaging for Christmas Gifts URL: https://www.dataexpert.io/question/optimal-gift-packaging Difficulty: hard Given two tables, playground.gifts and playground.packages, write a SQL query to match each gift to the smallest package it fits into based on dimensions. A gift fits in a package if its dimensions are less than or equal to those of the package. A package is considered smaller than another if its volume is smaller. Each package can hold only one gift. Produce a table with columns: package_type and number, where number indicates how many gifts are matched to each package_type. Exclude package types not used. Sort the result by package_type in ascending order. Assume every gift fits in at least one package and no two packages have the same volume. ### Total Number of Births Per Year URL: https://www.dataexpert.io/question/total-births-per-year Difficulty: easy Write a SQL query to calculate the total number of births recorded for each year in the playground.us_birth_stats table. Order the results by year. ### Countries and Their Preferences for Beer, Spirit, and Wine URL: https://www.dataexpert.io/question/countries-drink-preferences Difficulty: hard Create a SQL query to find all countries and their preference for beer, spirit, and wine, based on the highest serving type as their preference. The output should show only the countries whose preferences are wine and spirits, ordered in ascending order of the country name. ### Identifying students with the best grade URL: https://www.dataexpert.io/question/option3-students-best-mark Difficulty: medium Using the playground.marks table, write a SQL query to determine the students who achieved the highest marks when graded according to Option 3, where the final exam accounts for 100% of the grade. The other grading options are provided for context: Option 1: Midterm 1 contributes 25%, Midterm 2 contributes 25%, and the Final exam contributes 50% of the grade. Option 2: Midterm 1 contributes 50% and Midterm 2 contributes 50% of the grade. Option 3: The Final exam contributes 100% of the grade. The output should be sorted by the students' names in ascending order. Ensure the column names in the result are displayed in lowercase. The resultant table should have two columns, name and their id. ### Most Frequently Purchased Sub-Category in Each Region URL: https://www.dataexpert.io/question/subcat-per-region Difficulty: hard Write a SQL query to determine the most frequently purchased sub-category in each region based on the playground.superstore dataset. The result should include the region, sub-category, and the count of purchases, and should list only the top sub-category for each region. Display the result in ascending order of the region ### Identifying Active Businesses Based on Event Occurrences URL: https://www.dataexpert.io/question/active-businesses-identification Difficulty: medium Using the playground.business_events table, identify all active businesses. An active business is defined as one that has more than one event type with occurrences greater than the average occurrences of that event type among all businesses. The goal is to filter businesses based on their activity level, comparing individual event occurrences to the overall average for those event types. The result should be a table with one column called business_id with all the business ids that are active. ### Which states have high risk of fatal collisions involving Alcohol URL: https://www.dataexpert.io/question/high-risk-state-analysis Difficulty: medium Using playground.bad_drivers, determine the states and their percentage of alcohol impaired collisions where the risk of fatal collisions involving alcohol is significantly higher than the national average. Consider a state as high-risk if its percent_alcohol_impaired is at least 20% higher than the national average. Show the output in descending order of the percent_alcohol_impaired. ### Calculate 3-Day Rolling Average of Posts per User URL: https://www.dataexpert.io/question/calculate-3-day-rolling-average Difficulty: medium Using the table playground.posts, write a SQL query to calculate the 3-day rolling average of posts for each user. The output should include columns: user_id, post_date, and third_rolling_avg, where third_rolling_avg is the 3-day rolling average of post count for each user, rounded to 2 decimal places. The rolling average should be calculated over a specified 3-day period for each user, including the current day and the two preceding days. The result should be ordered in ascending order of post date ### Top 2 Customers with Highest Sales Every Month URL: https://www.dataexpert.io/question/top-customers-monthly-sales Difficulty: medium Using the table playground.sales, Identify the top 2 customers based on their total sales for each month. Display the result based on asc order of their rank and month. ### Cars with Above Average Engine Size URL: https://www.dataexpert.io/question/cars-above-average-engine-size Difficulty: easy Using the table playground.automobile, Create a SQL query to identify cars that have an engine size above the average across all cars in the dataset. The result should include the brand, fuel_type, and engine size, ordered by engine size in descending order and then brand_name in asc order. ### Total Server Fleet Uptime in Full Days URL: https://www.dataexpert.io/question/server-fleet-uptime-full-days Difficulty: hard Using the 'playground.sessions' table, write a SQL query to calculate the total uptime of the server fleet in full days. The table records each server's start and stop times as separate entries. The total uptime ('total_uptime_days') is defined as the sum of all periods during which any server was running, represented as a single integer value for the total number of full days. Assume each 'start' session_status has a corresponding 'stop' status, and sessions do not overlap for the same server. ### Optimizing Stock of Prime and Non-Prime Items in a Warehouse URL: https://www.dataexpert.io/question/warehouse-item-optimization Difficulty: hard Given a warehouse with a capacity of 500,000 square feet, Amazon wishes to optimize the number of items stocked, prioritizing prime items. The goal is to fill the warehouse with as many prime items as possible and then supplement the remaining space with non-prime items. However, there are new operational constraints: prime and non-prime items must be stocked in equal quantities, and there must always be non-prime items in stock. Assuming each item occupies the same amount of space, using the table playground.inventory, write a query to determine the maximum number of prime and non-prime items that can be accommodated in the warehouse. The output should list the item_type (prime_eligible or not_prime) followed by the maximum number of item_count that can be stocked, ensuring item counts are whole numbers and adhering to the given constraints and prime_eligible first. ### Check Test Answers URL: https://www.dataexpert.io/question/check-test-answers Difficulty: easy Create a SQL query to evaluate test answers stored in a table named playground.answers with columns id (unique question ID), correct_answer (string), and given_answer (which can be NULL). Return a table with columns id and checks, where checks is "no answer" if given_answer is NULL, "correct" if given_answer matches correct_answer, and "incorrect" otherwise. Order the results by id. ### Average Number of Births by Day of the Week URL: https://www.dataexpert.io/question/average-births-per-day-of-week Difficulty: easy Create a SQL query that finds the average number of births for each day of the week across all years in the playground.us_birth_stats table. Cast the average as an integer. Order the results by the day of the week. ### Check Second Sale Brand Preference Match URL: https://www.dataexpert.io/question/second-item-preferred-brand-match Difficulty: hard Using the tables playground.users, playground.items, and playground.orders, write a SQL query to determine for each user, whether the brand of the second item they sold matches their preferred brand. The output should include two columns: seller_id and has_pref_brand, where has_pref_brand is "yes" if the brand of the second item sold matches the users preferred brand, and "no" otherwise. If a user sold less than two items, report the answer for that user as "no". It is guaranteed that no seller sold more than one item on the same day. The result should be ordered by user_id in ascending order. ### Identifying Empty Departments URL: https://www.dataexpert.io/question/empty-departments-list Difficulty: easy Given two tables, playground.employees and playground.departments, with employees containing id, full_name, and department, and departments containing id (unique department ID) and dep_name (department name), write a SQL query to build a table with one column, dep_name. This table should list all the departments that currently have no employees, sorted by the department id. ### Filtering Students in Active Clubs URL: https://www.dataexpert.io/question/active-club-members Difficulty: easy Given tables clubs (id: unique club id, name: club name) and students (id: unique student id, name: student name, club_id: club's id), return a list from the students table for those who are in clubs that still exist in the clubs table. The result should have three columns (id, name, club_id) and be sorted by students' ids (id) and include only those students whose club_id matches an id in the clubs table. ### Find Top 3 Commonly Bought Product Pairs URL: https://www.dataexpert.io/question/commonly-bought-product-pairs Difficulty: hard Using the table playground.product_transactions, write a query to find the top 3 pairs of products that are most frequently bought together in the same transaction from the table playground.product_transactions. For this analysis, consider a pair to consist of two different products purchased in the same transaction. The output should list the pairs of products and the frequency of each pair (product1, product2, freq), ordered by product1 in ascending order, product2 in ascending order and frequency in descending order. ### Count Unique Conversation Threads URL: https://www.dataexpert.io/question/unique-conversation-threads Difficulty: hard Using the table playground.messenger, write a SQL query to find the number of unique conversation threads in the playground.messenger table. A conversation thread is identified by the sender_id and receiver_id columns. Note that if a thread has sender_id and receiver_id inverted in another row, it should still be counted as the same thread. For example, a conversation where sender_id=1 and receiver_id=2 should be considered the same thread as one where sender_id=2 and receiver_id=1. The result should just be a single column called count given in any order. ### Most Popular Fuel System for Each Fuel Type URL: https://www.dataexpert.io/question/popular-fuel-system-by-fuel-type Difficulty: medium Using the table playground.automobile, create a SQL query to find the most popular fuel system for each fuel type based on the number of cars. Include columns for fuel type, fuel system, and the count of cars, showing only the top fuel system for each fuel type. Order the result on the basis of fuel type in ascending order. ### Identifying the Bank Robber URL: https://www.dataexpert.io/question/bank-robber-identification Difficulty: easy Using table playground.suspect, filter out suspects who cannot be the bank robber based on the following clues: the robber is not taller than 170cm, and their name matches the pattern "B. Gre?n" where the first letter of the name is "B" or "b" and the surname is similar to "Green" but with the fourth letter being unreadable and potentially any character. The match should be case-insensitive. For each suspect that fits these criteria, select their id, name, and surname. Order the results by suspect id in ascending order. ### Identifying Multi-Component Bugs and Their Impact URL: https://www.dataexpert.io/question/multi-component-bugs-impact Difficulty: hard Analyze the playground.bug, playground.component, and playground.bug_component tables to identify bugs affecting multiple components. Bug table contains num (unique bug number) and title (bug title). Component table includes id (unique component id) and title (component title). BugComponent table links bugs to components with bug_num (references Bug.num) and component_id (references Component.id). The task is to list each bug (that affects more than one component) with the titles of affected components and the total number of bugs in these components. Output columns: bug_title, component_title, bugs_in_component, ordered by bugs_in_component (descending), then by bug_title (ascending). ### Calculate Average Lead Time for Each Shipping Option URL: https://www.dataexpert.io/question/average-lead-time-shipping-option Difficulty: medium Using the playground.superstore table, write a SQL query to calculate the average lead time from order to shipping for each shipping option. The lead time is defined as the number of days from the order date to the ship date. Additionally, count the total number of shipping services provided for each shipping option. The output should include the shipping mode (ship_mode), the total count of shipping services (total_shipping_service), and the average lead time (avg_lead_time), rounded to two decimal places. Order the results by shipping mode in descending order. ### Salary Range Difference URL: https://www.dataexpert.io/question/salary-range-difference Difficulty: medium Calculate the difference between the sum of the highest salaries and the sum of the lowest salaries in the company. The table "playground.employees_salary" contains columns "id" (unique employee ID), "name" (employee's name), and "salary" (employee's salary as a positive integer). The result should be a single column "difference" with one row representing the calculated difference. If the "playground.employees_salary" table is empty, "difference" should be 0. ### Finding Actors by Favorite Movie Genre URL: https://www.dataexpert.io/question/actors-by-favorite-genre Difficulty: medium Given three tables - playground.movies, playground.starring_actors, and playground.actor_ages - identify actors from the most common movie genre in your library. Assume each actor is listed once representing their best role. The task is to list actors with their age, sorted from oldest to youngest, who star in movies of your favorite genre (the most common genre in your list). In case of actors being the same age, sort them by name.The resultant table should have two columns, actor and age. ### Determining the Order of Succession URL: https://www.dataexpert.io/question/order-of-succession Difficulty: easy Given a table Successors with columns: name, birthday, and gender, write a SQL query to list the names of the King's children in order of their succession to the throne and their birthday("name", "birthday"). Succession is based on age seniority. Prefix the name with "King" for males and "Queen" for females. The result should be sorted by birthday in ascending order to determine the succession order. ### Top Reviewed Customers per Product URL: https://www.dataexpert.io/question/top-reviewed-customers-per-product Difficulty: easy Using the table playground.product_reviews, write a SQL query to identify, for each product, the customer who provided the highest review score. If there are ties in review score, the customer with the most helpful votes should be considered top. The output should include columns for product_id, customer_id, review_score, and helpful_votes, capturing the details of the top review for each product ordered in ascending order of product_id ### Identify Department for Reduction URL: https://www.dataexpert.io/question/department-reduction-criteria Difficulty: hard Select departments that have no more than 5 employees. From these, choose departments based on the descending order of the total salary of its employees. In case of a tie in total salary, prioritize the department with the greater number of employees. If still tied, prioritize by the smallest department id. From the resulting list, eliminate departments in even positions, leaving only those in odd positions for further consideration. The resulting table should contain columns for department name, number of employees, and total salary, sorted as described. ### Analyze Engine Efficiency by Fuel Type URL: https://www.dataexpert.io/question/analyze-engine-efficiency Difficulty: medium Using the table playground.automobile, write a SQL query to analyze engine efficiency by comparing the average city mileage to highway mileage for each fuel type. The efficiency ratio is defined as the average highway mileage divided by the average city mileage. Include fuel type, average city mileage, average highway mileage, and efficiency ratio in the results, rounded up to 2 decimal places, ordered by the highest efficiency ratio. ### Determining the Soccer Series Winner URL: https://www.dataexpert.io/question/soccer-series-winner Difficulty: hard From the "scores" table, write a SQL query to determine the series winner based on the following criteria: 1) The team with the most game wins is the winner. 2) If the wins are equal, the team with the better overall goal difference (goals scored - goals conceded) wins. 3) If goal differences are also equal, the winner is the team with more goals scored in away games (when not the host). The "scores" table includes match_id (unique match ID), first_team_score, second_team_score, and match_host (1 or 2, indicating the host team). Return a table with a single column "winner" that contains 1 (for the first team), 2 (for the second team), or 0 if no winner is determined based on these criteria. ### Find US Customers Who Rented and Streamed Videos in Early February URL: https://www.dataexpert.io/question/us-customers-rented-streamed-early-feb Difficulty: easy Write a SQL query to return the US customers who rented a video on February 1st, 2023, and then streamed the same video between February 2nd and February 8th, 2023. Use the tables playground.rental for rental data and playground.streams for streaming data. The output should include unique user IDs of these customers ordered in ascending order. ### Who are the top 10 NBA players by consecutive 20+ point seasons? URL: https://www.dataexpert.io/question/consecutive-20-point-seasons Difficulty: hard Using the table bootcamp.nba_player_seasons, find the most consecutive seasons a player has scored 20+ points and find the top 10 players! Make sure to handle ties correctly (you might have more than 10 records!). Sort the output data set by consecutive_seasons descending and player_name ascending! ### Analyze Yearly Profit Performance URL: https://www.dataexpert.io/question/yearly-profit-performance Difficulty: easy Write a SQL query to analyze the profit performance of the company throughout the years using the playground.superstore table. Ensure to convert profit from varchar to a suitable numeric type for aggregation. The output should include the year extracted from the order_date (order_year), the total annual profit (total_profit) rounded to two decimal places, and should be ordered by year in descending order. ### Filtering Dance Contest Scores URL: https://www.dataexpert.io/question/dance-contest-score-filtering Difficulty: medium Given a table playground.dance_scores (columns: arbiter_id, first_criterion, second_criterion, third_criterion) with scores from 1 to 10 awarded by judges in a dance contest, write a query to exclude scores from any judge who awarded an extreme score (either minimum or maximum) for at least two criteria. Return the filtered scores, sorted by arbiter_id. ### Summarizing Leisure Activity Resorts by Country URL: https://www.dataexpert.io/question/leisure-activity-resorts-by-country Difficulty: hard Given a table playground.country_activities with columns: id (unique identifier), country (name of the country), region (region within the country), leisure_activity_type (type of leisure activity), and number_of_places (number of resorts offering the activity). Construct a SQL query to aggregate this information into a new table structure. The resulting table should have columns for each country, with the counts of resorts offering each type of activity: adventure_park, golf, river_cruise, and kart_racing. Sort the result by country name in ascending order. ### Identify Products Sold Exclusively in January 2024 URL: https://www.dataexpert.io/question/products-exclusive-jan-sales Difficulty: medium Using the tables playground.products and playground.product_sales, write a SQL query to find products that were sold exclusively within the month of January 2024 (from "2024-01-01" to "2024-01-31", inclusive). The output should include the product_id and product_name for each qualifying product. ### Select Rows With Maximum Revenue URL: https://www.dataexpert.io/question/select-max-revenue-rows Difficulty: easy Using the table playground.revenue, write a SQL query to select rows from a given table that have the maximum revenue value for each id. The resultant table should have three columns - "id", "rev", "content". Additionally, the results should be ordered in descending order by revenue. ### Top N per group with ties URL: https://www.dataexpert.io/question/top-n-per-group-with-ties Difficulty: medium Return the top score per game_date including ties (if two players share top score, include both). Also include rank and total players that day. ### Customer Orders URL: https://www.dataexpert.io/question/customer-orders Difficulty: medium List all customers who have not placed any orders. ### Detecting Gaps in a Sequence URL: https://www.dataexpert.io/question/detecting-gaps-in-a-sequence Difficulty: medium Given a table quiz_orders(order_id, customer_id, order_date) where order_id should be consecutive daily numbers, list any missing order_ids ### Question 1 URL: https://www.dataexpert.io/question/question-1 Difficulty: easy Select 20 nba platers ### Find Overlapping Date Ranges URL: https://www.dataexpert.io/question/find-overlapping-date-ranges Difficulty: hard Given a table bookings (room_id, start_date, end_date), list all pairs of bookings that overlap for the same room. ### SQL Question 2 URL: https://www.dataexpert.io/question/sql-question Difficulty: hard Question Discription testing ### Hierarchical JSON Aggregation with Conditional Flattening URL: https://www.dataexpert.io/question/hierarchical-json-aggregation-with-conditional-flattening Difficulty: medium A table orders(id, payload VARCHAR) stores a JSON document per order, e.g.: Produce a result set with one row per order‑item, containing: | order_id | customer | sku | total_price | region | channel | total_price = price qty. Only include items where total_price > 25 USD. ### Latest price per product URL: https://www.dataexpert.io/question/latest-price-per-product Difficulty: medium Return the most recent price for a given product in the table, sorted by product in increasing order. ### Orders with missing customers URL: https://www.dataexpert.io/question/orders-with-missing-customers Difficulty: medium Return the orders with missing customers ### Running total amount URL: https://www.dataexpert.io/question/running-total-amount Difficulty: easy Write a query that returns each row with a running total of amount per salesperson ordered by sale_date. ### Sessionization and gap-based grouping URL: https://www.dataexpert.io/question/sessionization-and-gap-based-grouping Difficulty: hard Assume a session is a series of events for the same user where consecutive events are no more than 30 minutes apart. Assign session numbers per user, show session start, end, duration (in minutes), and event count. ### Calculate Monthly Average Product Rating URL: https://www.dataexpert.io/question/monthly-average-product-rating Difficulty: medium Using the table playground.product_reviews, write a SQL query to calculate the monthly average rating for each product. The resulting table should have three columns: month, product_id, and avg_review, where avg_review is the average review rating of the product for that month. The results should be ordered in ascending order of month and product_id ### Year-on-Year Growth Rate Calculation URL: https://www.dataexpert.io/question/year-on-year-growth-rate Difficulty: hard Using the table playground.dates, write a SQL query to calculate the year-on-year growth rate for the total spend of each product. The output should include the year, product ID, current years spend, previous years spend, and the year-on-year growth percentage, rounded to 2 decimal places, grouped by product ID and sorted by year in ascending order. ### Get nba player from bootcamp.nba_player_seasons table with age 25 and name is Aaron McKie URL: https://www.dataexpert.io/question/get-nba-player-from-bootcamp-nba-player-seasons-table-with-age-and-name-is-aaron-mckie Difficulty: hard Get nba player from bootcamp.nba\_player\_seasons table with age 25 and name is Aaron McKie. Just testing if this breaks anything else. ### Select all nba players with college as Connecticut and ordered by player_name desc URL: https://www.dataexpert.io/question/select-all-nba-players-with-college-as-connecticut-and-ordered-by-player-name-desc Difficulty: hard Select all nba players with college as Connecticut and ordered by player_name desc ### Testing sql question URL: https://www.dataexpert.io/question/testing-sql-question Difficulty: hard Any description for the question ### Summarizing Foreign Competitors by Country with Total Summary URL: https://www.dataexpert.io/question/foreign-competitors-summary Difficulty: medium Given the playground.foreign_competitors table with columns: competitor and country, write a SQL query to calculate the number of competitors per country. Additionally, include a final summary row that shows "Total:" and the total number of competitors. The resulting table should have two columns: country and competitors, sorted by competitors and country names in ascending order, and include the summary row at the bottom. ### Calculating Available Seats on Flights URL: https://www.dataexpert.io/question/available-seats-per-flight Difficulty: medium Calculate the number of seats not yet purchased for each flight. Use three tables: playground.flights (with columns: flight_id, plane_id), playground.planes (with columns: plane_id, number_of_seats), and playground.purchases (with columns: flight_id, seat_no). Each row in purchases is unique for (flight_id, seat_no) pairs. For each flight_id, compute the free_seats as the number of seats not purchased. Order the results by flight_id in ascending order. Ensure consistency in purchases, with no records for non-existing flight_ids or seat_nos. ### Customers with More Than 20 Orders URL: https://www.dataexpert.io/question/loyal-customers Difficulty: easy Write a SQL query to display all loyal customers from the playground.superstore table. A customer is considered loyal if they have placed more than 20 orders. The query should return the customer ID, customer name, and the total number of orders for each of these customers. Display the result in descending order of their orders and then ascending order of their names ### Transform workers_info Table to Exclude NULL Values URL: https://www.dataexpert.io/question/exclude-null-values-workers-info Difficulty: medium Given a table named playground.workers_info, write a SQL query to transform this table. The transformation should result in a table with three columns: id, column_name, and value, where column_name is the name of the original column containing a non-NULL value for each id, and value is the stringified non-NULL value from this column. Exclude rows where the original value is NULL. The result should be sorted by id in ascending order, and for rows with the same id, sorted by column_name in ascending order. ### Data Engineer Interview — SQL URL: https://www.dataexpert.io/question/data-engineer-sql Difficulty: medium SQL challenge placeholder for Data Engineer Interview. Replace with a real Trino-compatible question against seeded tables. ### Analytics Engineer Interview — SQL URL: https://www.dataexpert.io/question/analytics-engineer-sql Difficulty: medium SQL challenge placeholder for Analytics Engineer Interview. Replace with a real Trino-compatible question against seeded tables. ### AI Engineer Interview — SQL URL: https://www.dataexpert.io/question/ai-engineer-sql Difficulty: medium SQL challenge placeholder for AI Engineer Interview. Replace with a real Trino-compatible question against seeded tables. ## Course Lessons Lessons marked free are open to everyone; others require enrollment in a program. ### TA Office Hour 5 URL: https://www.dataexpert.io/lesson/ta-office-hour-5-jan2025 Enrolled students Office Hour 5 provides a platform for students to discuss their capstone projects and seek guidance on technical challenges. Participants engage in discussions about using tools like Spark, Airflow, and Databricks, while addressing project constraints and data access issues. The session also covers homework policies and offers insights into effective project design, emphasizing the importance of business value and documentation in showcasing their work. This interactive environment is beneficial for students looking to refine their projects and gain clarity on course expectations. ### RAG and LLMs Day2 Lecture URL: https://www.dataexpert.io/lesson/rag-and-llms-day-2-lecture-jan2025 Enrolled students In this lecture, Zach dives into the intricacies of full stack engineering, particularly focusing on retrieval-augmented generation (RAG) systems. He discusses the importance of chunking data effectively and the role of AI in processing assignments, emphasizing the need for fine-tuning as our platform scales. ### Q&A with Zach Week 3 (2025-07-25) URL: https://www.dataexpert.io/lesson/qa-with-zach-week3-july2025-p737-l1304 Module: Q&A with Zach · Enrolled students A live Q&A session with Zach covers a wide range of topics related to data engineering and AI. Participants ask questions about tools like Cursor, RAG versus Graph RAG, and the use of AI in business. Zach shares insights on project ideas, the importance of understanding customer needs, and the balance between foundational skills and emerging technologies. This session is beneficial for anyone looking to deepen their understanding of AI applications and data engineering practices. ### Data Quality Patterns: WAP Patterns Day 2 Lecture URL: https://www.dataexpert.io/lesson/data-quality-patterns-day-2-lecture-yt-p736-l700 Enrolled students Cardi B ruined WAP Pattern! Learn more about it here! ### Course 2 updating - test URL: https://www.dataexpert.io/lesson/course-2-updating-test-6933 Enrolled students ### RAG optimisation + Reranking Day 1 Lecture URL: https://www.dataexpert.io/lesson/rag-optimisation-reranking-day1-lecture-oct-2025-p737-l1510 Module: Week 2: Advanced RAG, Reranking & Dev Workflow · Enrolled students In this video, Zach discusses the concepts of vectors and RAG. He emphasizes the importance of identifying business problems suitable for AI, noting that AI is not a self-starter and requires specific trigger points to deliver value. Zach explores various interfaces beyond chatbots, such as CICD and cron jobs, and explains how these can enhance performance and relevance in RAG systems. Additionally, he outlines strategies for improving vector search performance and making RAG outputs more relevant. ### Real Time data with Delta Live Table Day2 Lab URL: https://www.dataexpert.io/lesson/realtimedatawithdeltalivetableday2lab-aug26 Enrolled students In this lab, Zach demonstrates building an end-to-end Delta Live Tables and Spark Streaming pipeline, emphasizing why Python UDFs perform poorly. He then walks through a Kafka-driven bronze silver gold pipeline with data quality expectations, quarantining bad rows, and adjusting a watermark initially at 10 minutes down to 5 seconds for faster testing. Finally, he adds a week-over-week row count gate for the gold table, uses a staging approach to ensure the check blocks publishing. ### How to use snowpark and Snowflake optimization Lab URL: https://www.dataexpert.io/lesson/how-to-use-snowpark-and-snowflake-optimization-lab Module: Applying Analytical Patterns · Enrolled students In this lab, Zach dives into cost management and data handling processes. He explores loading data into the Polygon API, connecting to Snowflake, and managing super secret credentials. He also touches on Capstone project deadlines and demonstrate querying data from Trino. ### Capstone Project Day 2 Class URL: https://www.dataexpert.io/lesson/capstone-project-day-2-class-april2025-p735-l1055 Module: Week 8: End-to-End Project Hacking · Enrolled students In this class, Zach walks through our recent lab work on user master DAG and how we can optimize our data management processes. He highlights the importance of parameterization and the need for a clear naming convention in our tables. He also discusses the certification process and how we can track user engagement effectively. ### Dimensional Data Modeling - Complex Data Types Arrays and Structs with Iceberg (Day 1 Lecture) URL: https://www.dataexpert.io/lesson/dimensional-data-modeling-day-1-lecture-v3 Module: Week 1: Dimensional Data Modeling in Iceberg · Enrolled students In this presentation, we explore the fundamental concepts essential for effective data modeling and database management. We dive into understanding your data consumer, the distinctions between OLTP and OLAP data modeling, the principles of Cumulative Table design, and the tradeoff between compactness and usability. We also address the challenges of temporal cardinality explosion and the potential pitfalls of run-length encoding compression. By the end of this lecture, you will have a strong foundation in the key concepts of data modeling and be prepared to apply these principles in practice. [Recorded Nov 7, 2023] ### Data Accessibility and Privacy (Day 4 Lecture) URL: https://www.dataexpert.io/lesson/fact-data-modeling-day-4-lecture-v4 Module: Week 2: Fact Data Modeling in Iceberg · Enrolled students In this lecture, Zach discusses the importance of data privacy and storage strategies. He highlights the risks of non-compliance and the need for proper handling and storage of data. He also touches on topics such as pseudonymization, data retention policies, and the impact of storage choices on cost and efficiency. [Recorded on May16th, 2024] ### Test 2 URL: https://www.dataexpert.io/lesson/890964a6-282e-4a75-921d-9b757dd788a3test2 Enrolled students Learn how to make a tangible impact as a data engineer by focusing on effective communication, robust data modeling, and stakeholder management. This course covers strategies for pushing back on low-value requests, enhancing data visualization skills, and understanding the nuances of measurable versus immeasurable impacts. Ideal for data professionals looking to elevate their influence within organizations, the lessons emphasize the importance of quality data practices and collaboration with cross-functional teams. ### AI Application Setup URL: https://www.dataexpert.io/lesson/ai-application-setup-22551-p737-l1568 Module: Kickoff & Setup · Enrolled students In this video we go over how to setup Supabase, OpenAI, Render, and PubNub. These are the free services we will be using to create amazing applications together! ### Test Course URL: https://www.dataexpert.io/lesson/631f3b55-d50a-452f-bb99-da927574e13atestcourse Enrolled students ### TA Office hour 6 URL: https://www.dataexpert.io/lesson/taofficehour6-7abba-p735-l1944 Enrolled students This office hours session focuses on addressing student queries related to multi-factor authentication (MFA) issues when using DBT with Snowflake. Participants discuss the challenges of MFA prompts, potential solutions like using RSA keys, and the implications for their capstone projects. Students also seek clarification on project requirements and deadlines, as well as technical aspects of data ingestion and transformation using tools like Airflow and Databricks. ### Book Club 5 URL: https://www.dataexpert.io/lesson/bookclub5-mar26 Enrolled students The session focuses on evaluating AI systems, covering essential topics such as designing evaluation criteria, model selection, and creating an evaluation pipeline. Participants will learn about domain-specific capabilities, generation quality, instruction-following, and cost-effectiveness, which are crucial for assessing AI models. This lesson is beneficial for data engineers and AI practitioners looking to enhance their understanding of AI evaluation methodologies and best practices. ### Q&A Week 3 URL: https://www.dataexpert.io/lesson/qa-week-3-jan2025-p736-l781 Module: Q&A with Zach · Enrolled students In this Q-A session, Zach shares his journey of moving and setting up my new place while also discussing the Capstone project for our data engineering bootcamp. He emphasizes the importance of creating a solid conceptual data model and provide guidelines for tackling the project effectively. He also touches on the significance of collaboration and mentorship during this process. This recording has been cut short, we are working on adding the second part to it. ### test 2 URL: https://www.dataexpert.io/lesson/test2-6dc1a Enrolled students ### TA Office Hour 1 (2025-10-09) URL: https://www.dataexpert.io/lesson/ta-office-hour1-oct2025-p737-l1536 Enrolled students ### Slowly Changing Dimensions Lecture URL: https://www.dataexpert.io/lesson/slowly-changing-dimensions-lecture Module: Bonus · Enrolled students In this presentation, we delve into the concept of idempotent pipelines, exploring their significance, advantages, and potential drawbacks. We provide insights into Slowly Changing Dimensions (SCDs) and discuss the various types. Discover whether SCDs are inherently idempotent or not and grasp the implications of their usage in data pipelines. Join us to gain a clear understanding of these foundational concepts in data processing. ### Advanced dbt Lecture Day 1 URL: https://www.dataexpert.io/lesson/advanced-dbt-lecture-day-1-april2025 Enrolled students The first day of the DBT Advanced course provided an in-depth exploration of Jinja, macros, variables, and incremental models. The session emphasized the importance of efficient SQL coding practices and the powerful capabilities of DBT in managing data transformations. Participants were encouraged to engage actively in the upcoming lab sessions to solidify their understanding of the concepts discussed. ### Chat with Zach URL: https://www.dataexpert.io/lesson/week-1-zach-qa-v4 Enrolled students Join Zach, our founder/CEO and instructor of our data engineering bootcamp, for an insightful Q&A session covering crucial topics in the field. Zach dives into data management challenges, recommending tools like Trino and Presto, and explores infrastructure optimization with Kafka. He emphasizes individual submissions for collaborative assignments and discusses practical considerations for capstone projects, stressing originality and effective presentation. Transitioning to Apache Iceberg, Zach provides insights into SQL query processing and shares his perspective on the use of classmates and TAs for troubleshooting. Note: Correction at 13:00 - Zach mentions that students will receive up to two chances to correct their assignments before the grade is finalized, with TAs providing two reviews. However, in version 4 of our bootcamp, automatic tests and immediate LLM-generated feedback have been introduced, allowing unlimited revisions before the deadline. As a result, revisions after the deadline are no longer permitted, and the final version submitted at the deadline will be graded. Please refer to the 'assignments' page for further details. [Recorded May 9, 2024] ### TA Office Hour 6 (2025-01-29) URL: https://www.dataexpert.io/lesson/ta-office-hour-6-jan2025-p736-l791 Enrolled students Office hours provide an interactive platform for students to ask questions and clarify concepts related to their coursework. In this session, participants discussed topics such as cloning tables in Snowflake, managing micropartitions, and the use of DBT for data transformations. The session also covered how to handle incremental models and the implications of different configurations in DBT, making it beneficial for students seeking deeper understanding of data engineering practices. ### Snowflake Basics Day 1 Lab URL: https://www.dataexpert.io/lesson/snowflake-basics-day-1-lab-jan2025 Enrolled students Welcome to day one of Snowflake Basics Lab! In this lab, Bruno demonstrates how to create tables and views in Snowflake, highlighting the ease and flexibility compared to traditional methods. He showcases features like dark theme, tables, views, UDFs, AI, ML, and project organization. ### TA Office Hour 5 URL: https://www.dataexpert.io/lesson/ta-office-hour5-july2025 Enrolled students Office Hour 5 focuses on troubleshooting and refining Python environments and dependencies, particularly for projects using Langchain and related libraries. Participants discuss resolving installation conflicts, managing Python versions, and setting up virtual environments. The session also covers assignment expectations, including building a dependency scanner and vulnerability report, and utilizing Neo4j for data relationships. This lesson is beneficial for learners seeking practical guidance on handling common issues in data engineering projects. ### Tech Talk with Carly URL: https://www.dataexpert.io/lesson/tech-talk4-oct2025 Enrolled students ### Chat with Zach [July 27, 2023] URL: https://www.dataexpert.io/lesson/week-2-zach-qa-v2 Enrolled students Delve into the world of data engineering with our weekly Q&A / Fireside Chat series, featuring the founder of EcZachly Inc and lead instructor of the Advanced Data Engineering Bootcamp. Each session covers a different topic, providing invaluable insights and real-world experiences to empower data enthusiasts in their journey. (Recorded July 27, 2023) ### Apache Iceberg: Data Contracts Lecture URL: https://www.dataexpert.io/lesson/apache-iceberg-lecture-day-3-april2025-p735-l960 Module: Week 5: Data Modeling on Iceberg · Enrolled students In this lecture, Zach dives into the concept of data contracts and their importance in analytics engineering. He discusses how data contracts help define the relationship between data producers and consumers, ensuring clarity on data usage and expectations. He also highlights the significance of quality checks and the role of tools like Iceberg in managing data effectively. ### Azure - Week 5 (Copy) URL: https://www.dataexpert.io/lesson/azure-week-5-3166 Enrolled students Azure Data Factory (ADF) is a powerful tool for creating data pipelines, and this session covers its essential components and functionalities. Participants will learn how to set up ADF accounts, create linked services, and build pipelines for data movement and transformation, specifically from an Azure SQL database to Azure Blob Storage in CSV format. This lesson is ideal for data engineers and professionals looking to enhance their skills in cloud data integration and ETL processes using Azure. ### Loom URL: https://www.dataexpert.io/lesson/loom-4046 Enrolled students ### Fact Data Modeling: Core Concepts, Deduplication Day 1 Lecture URL: https://www.dataexpert.io/lesson/fact-data-modeling-day-1-lecture-yt-p736-l662 Enrolled students In this lecture, we delve into the realm of fact data modeling, focusing on the core concept of facts and addressing the challenges associated with handling duplicates. Our exploration underscores the pivotal role of facts in data engineering, shedding light on their significance. ### Q&A with Zach Week2 URL: https://www.dataexpert.io/lesson/qawithzachweek2-feb26 Enrolled students The week two Q&A session covers a range of topics related to data engineering, with a focus on Apache Spark and its significance in the industry. Participants have the opportunity to ask questions about technical concepts, including data modeling in Databricks and the implications of AI on job markets. The discussion also touches on the importance of storytelling and human skills in an evolving tech landscape, providing insights for those pursuing careers in data and AI engineering. ### Book Club 2 URL: https://www.dataexpert.io/lesson/bookclub2-mar26 Enrolled students This book club session focuses on Chapter 6, which explores Retrieval-Augmented Generation (RAG) and its implications in AI. Participants engage in discussions about the dynamic context of models, comparing keyword-based and embedding-based retrieval methods. The session includes breakout discussions, allowing attendees to share insights on RAG's applications, challenges, and the balance between using RAG and traditional models in various scenarios. ### Career Development - Resume Review URL: https://www.dataexpert.io/lesson/career-development-resume-review-jan2025 Enrolled students A comprehensive resume review session focuses on optimizing resumes for data engineering roles. Participants learn essential components to include, such as a concise summary, professional experience, and relevant projects, while emphasizing the importance of quantifying achievements and aligning skills with job postings. The session also covers formatting tips and common pitfalls to avoid, ensuring that attendees can effectively present their qualifications to recruiters and hiring managers. ### Write a comprehensive pipeline spec document Lab URL: https://www.dataexpert.io/lesson/pipeline-spec-building-lab Module: Pipeline Spec Building + Airflow Fundamentals · Enrolled students In this lab, Zach discusses the importance of data metrics and effective communication strategies. He highlights various data sources, such as banks, rental properties, and Excel sheets, and explain how they can be used to track metrics related to health, career, and recreation. He also emphasizes the significance of considering context and building relationships when communicating with stakeholders. The information shared can help them gain valuable insights and improve their communication skills. ### TA Office Hour 2 (2025-10-09) URL: https://www.dataexpert.io/lesson/ta-office-hour2-oct2025-p737-l1537 Enrolled students ### Day 1 Lab: Vectorizing Inputs into Milvus URL: https://www.dataexpert.io/lesson/understanding-rag-day1-lab-july2025 Enrolled students The Day 1 Lab focuses on vectorizing inputs into Milvus, utilizing FastAPI and Python for creating a new collection of YouTube videos. Participants will learn how to set up a Milvus account, create collections, and integrate video transcripts using OpenAI for transcription. This hands-on session is ideal for data engineers and AI practitioners looking to enhance their skills in managing unstructured data and building retrieval-augmented generation (RAG) systems. ### Introduction to Spark Day 2 Lecture URL: https://www.dataexpert.io/lesson/introduction-to-spark-day-2-lecture-jan2025 Enrolled students In this lecture, Yared dives into the fundamentals of Apache Spark, discussing its architecture and how it processes big data. He highlights the importance of Spark's DataFrames and the various programming languages it supports, such as SQL, Python, and Scala. He also touches on the execution process and the concept of lazy evaluation. ### TA Office Hour 6 URL: https://www.dataexpert.io/lesson/ta-office-hour-6-jan2025 Enrolled students Office Hour 6 provides an interactive session focused on Snowflake's cloning functionality and DBT (Data Build Tool) operations. Participants can ask questions about cloning tables, managing micro-partitions, and the implications of incremental models in DBT. The discussion also covers the semantic layer in DBT, including how to define metrics and dimensions, making it beneficial for learners seeking to deepen their understanding of data modeling and transformation techniques in a cloud environment. ### TA Office Hour 5 (2025-01-29) URL: https://www.dataexpert.io/lesson/ta-office-hour-5-jan2025-p736-l790 Enrolled students Office hours provide an interactive platform for students to ask questions and discuss their capstone projects. Key topics covered include guidance on using Azure for cloud projects, integrating tools like Airflow and Spark, and strategies for effective project design. Students also share their project ideas, emphasizing the importance of demonstrating business value and robust documentation. This session is beneficial for anyone seeking clarification on project requirements or looking for peer support in their data engineering journey. ### Introduction to IDEs URL: https://www.dataexpert.io/lesson/introduction-to-ides-af3aa Module: Web Dev Fundamentals · Enrolled students Participants explore popular IDEs, programming languages, and version control systems like Git. They learn how to navigate within IDEs, understand language nuances, and grasp basic Git operations. Additionally, they gain insights into the differences between compiled and interpreted languages. ### Q&A with Zach Week 1 (2025-07-10) URL: https://www.dataexpert.io/lesson/qa-with-zach-week1-july2025-p737-l1286 Module: Q&A with Zach · Enrolled students The week one Q&A session covers various topics related to the AI Engineering boot camp, including auto prompt optimization and vibe coding. Participants can ask questions about capstone project requirements, unstructured data processing, and the use of tools like vector databases. The session also addresses concerns about course logistics, certification criteria, and the importance of self-guided learning in project development. This interactive format is beneficial for students seeking clarity and guidance as they embark on their boot camp journey. ### Q&A with Zach Week4 URL: https://www.dataexpert.io/lesson/qawithzachweek4-feb26 Enrolled students The Week 4 Q&A session focuses on streaming pipelines, Spark Streaming, and Delta Live Tables, providing insights for data engineers and analysts. Participants discuss the cost-effectiveness of Snowflake versus Databricks, considerations for setting up data ingestion pipelines, and the importance of team skill sets. The conversation also touches on adaptive execution, scalability, and the implications of using various technologies like Trino and Iceberg, making it a valuable resource for those navigating data architecture decisions. ### OpenClaw Safeguards & Measuring business value for AI Agents Day2 Lab URL: https://www.dataexpert.io/lesson/openclawsafeguardsmeasuringbusinessvalueforaiagentsday2lab-mar26 Enrolled students Please see the instructions here - https://road-suede-29f.notion.site/Participant-guide-34bfce66e714800b996dd75dcf92b9a1 Instructions to set up Mission Control Run on your vm with openclaw: git clone https://github.com/robsannaa/openclaw-mission-control.git cd openclaw-mission-control ./setup.sh Run locally: ssh -N -L 3333:127.0.0.1:3333 root@yourserver ### TA Office Hour 4 (2025-06-17) URL: https://www.dataexpert.io/lesson/ta-office-hour-4-p736-l1153 Enrolled students Office hours provide a platform for students to seek assistance on their assignments and technical issues. Participants discuss challenges related to running code, creating folders, and querying tables in their schemas. The session also covers general questions about common mistakes in homework, with guidance on using specific tools and techniques effectively. This interactive format is beneficial for students needing support in their data engineering tasks. ### Growth Accounting Lab URL: https://www.dataexpert.io/lesson/growthaccountinglab-c17bb Enrolled students ### Databricks and Advanced Spark Day1 Lecture URL: https://www.dataexpert.io/lesson/databricksandadvancedsparkday1lecture-feb26 Enrolled students In this lecture, Zach discusses the intricacies of Shuffle Joins and Optimization in Apache Spark. He highlights the importance of understanding business needs when processing data. Zach also shares insights on the app-to-app communication analysis for security, where processing the entire dataset was crucial. ### Analytics Data Quality - Mastering DBT Projects with Bruno: Practical Overview and Hands-On Demonstrations (Day 2 Lecture) URL: https://www.dataexpert.io/lesson/analytics-data-quality-day-2-lecture-v3 Module: Week 5: DBT and Pipeline Spec Building · Enrolled students In this lecture, the speaker provides an insightful overview of a DBT (Data Build Tool) project and offers practical demonstrations. Explaining the process of running dbt debug and troubleshooting potential issues, offering a hands-on learning experience for mastering dbt basics. Emphasis is placed on the significance of profiles and dbt project files, along with discussions on snapshots, slowly changing dimensions, and tests. Viewers gain a comprehensive understanding of DBT and its functionalities through this informative video.[Recorded on Dec 13th, 2023] ### advanced dbt Day 1 Lecture URL: https://www.dataexpert.io/lesson/advanceddbtday1lecture-55678 Enrolled students ### LinkedIn like a pro URL: https://www.dataexpert.io/lesson/career-dev-session2-july2025-p737-l1301 Module: Career Development · Enrolled students Mastering LinkedIn is essential for career development, and this lesson provides actionable strategies to enhance your profile and visibility. Learn how to create an attractive LinkedIn profile, optimize your headline and summary, and connect with industry professionals and recruiters effectively. Discover the importance of endorsements, recommendations, and skill assessments, as well as tips for engaging with content to boost your online presence and attract job opportunities. ### Tech Talk - Joe Reis URL: https://www.dataexpert.io/lesson/v4-tech-talk-joe-reis-906 Enrolled students In this Tech Talk with Joe Reis, he discusses the importance of data modeling and its impact on data management. He also explains the different levels of data modeling (conceptual, logical, and physical) and emphasize the need for a well-designed data model to ensure data accuracy, clarity, and ease of use. He touches on the drawbacks of not implementing proper data modeling and provide real-life examples to illustrate the concepts. [Recorded on May29th, 2024] ### Expanding Scope & Influence Without Authority URL: https://www.dataexpert.io/lesson/expandingscopeinfluencewithoutauthority-1f6e2 Enrolled students Creating impact beyond your immediate team Technical vision and strategy: what it looks like at Staff+ Building credibility across organizations Navigating ambiguity and driving consensus Hands-on: Scope Expansion Mapping—identify opportunities in your current role ### Scala Dataset vs Dataframe API [Dec 15, 2023] URL: https://www.dataexpert.io/lesson/spark-bonus-v3 Module: Interview Skills · Free In this bonus session on Spark, Zach provides a comprehensive overview of the Gradle plugin portal, dependencies, and the data frame API. Participants will gain valuable insights into specifying Maven Central, building a jar, and leveraging the data set API for seamless data processing. Zach also discusses the benefits of using Scala and navigates through the challenges of integrating with Java. This session offers a wealth of knowledge for enthusiasts and professionals alike.[Recorded on Dec 15, 2023] ### Fact Data Modeling Lecture URL: https://www.dataexpert.io/lesson/fact-data-modeling-lecture Module: Bonus · Enrolled students In this lecture, we delve into the realm of fact data modeling, focusing on the core concept of facts and addressing the challenges associated with handling duplicates. Our exploration underscores the pivotal role of facts in data engineering, shedding light on their significance. Practical insights into deduplication techniques are provided, and we discuss the ramifications of retention on fact data. Join us in this video to gain a comprehensive understanding of fact data modeling and its crucial implications in the field of data engineering. ### TA Office Hour 1 URL: https://www.dataexpert.io/lesson/ta-office-hour1-jul2025 Enrolled students The session covers the first office hours for a boot camp, focusing on two key assignments: prompt engineering and creating a FastAPI app. Participants learn how to manually craft prompts to identify countries with repeated characters in their names and explore various OpenAI models for optimal results. The second assignment involves building a simple FastAPI application with a single endpoint for integration with ChatGPT, providing practical experience with API development and prompt optimization techniques. ### YZ URL: https://www.dataexpert.io/lesson/yz-jan2025 Enrolled students A live Q&A session featuring YZ, an experienced data scientist with a background at Amazon, Tencent, and Statsig. The discussion covers various topics including the importance of A/B testing, the role of data scientists, and the evolving landscape of data science in the era of AI. YZ shares insights on the significance of domain knowledge, the necessity of rigorous thinking in data analysis, and the impact of generative AI on the field. This session is beneficial for aspiring data scientists and those interested in understanding the practical applications of data science in business. ### Applying Analytical Patterns Day 1 Lab URL: https://www.dataexpert.io/lesson/applying-analytical-patterns-day-1-lab-jan2025 Enrolled students The lab focuses on applying analytical patterns using a bootcamp web events table in Starburst. Participants will learn to create a daily dataset of user metrics, including sign-up, landing, and contact page visits, and implement state transition tracking for user behavior analysis. By the end, learners will understand how to manage user states such as new, retained, churned, and resurrected, enabling them to derive insights on user engagement and retention over time. ### Special Guests, Stephanie Nuesi and Joe Reis [July 26, 2023] URL: https://www.dataexpert.io/lesson/v2-special-guests-stephanie-nuesi-and-joe-reis-july-26-2023-3258 Enrolled students Join us each week as we sit down with visionary speakers from diverse fields, engaging in enlightening conversations that explore cutting-edge insights, trends, and their wealth of expertise. (Recorded July 26, 2023) ### TA Office hour 8 URL: https://www.dataexpert.io/lesson/taofficehour8-feb26 Enrolled students This office hours session addresses common issues faced by students, particularly around permissions and data management in their projects. Participants discuss challenges with creating tables and inserting data in specific directories, as well as clarifying submission processes for capstone proposals. The session also touches on accessing past boot camp materials and scholarship application timelines, providing valuable support for students navigating their coursework and project requirements. ### Advanced dbt Lab Day 1 URL: https://www.dataexpert.io/lesson/advanced-dbt-lab-day-1-april2025 Enrolled students The first day of DBT Advanced Week successfully introduced participants to advanced concepts of Jinja and incremental models, providing practical demonstrations and encouraging hands-on practice. Participants left with a clearer understanding of how to implement these techniques in their data transformation processes. ### Apache Spark Memory Turning, Partitioning Day 2 Lecture URL: https://www.dataexpert.io/lesson/apache-spark-memory-turning-partitioning-day-2-lecture-jan2025 Enrolled students In this lecture, Zach dives into how to troubleshoot slow Spark jobs, particularly focusing on common bottlenecks that can occur in big tech environments. He discusses the impact of upstream data formats like JSON and CSV, and how they can hinder performance. He also shares insights on optimizing data processing and managing skew data effectively. ### Snowflake Basics Day 1 Lab URL: https://www.dataexpert.io/lesson/snowflake-basics-day-1-lab-jan2025-p736-l770 Enrolled students Welcome to day one of Snowflake Basics Lab! In this lab, Bruno demonstrates how to create tables and views in Snowflake, highlighting the ease and flexibility compared to traditional methods. He showcases features like dark theme, tables, views, UDFs, AI, ML, and project organization. ### TA Office hours 3 URL: https://www.dataexpert.io/lesson/taofficehours3-mar26-p737-l2131 Enrolled students Office hours provide a platform for students to discuss assignments and clarify doubts. In this session, participants engaged in a conversation about assignment expectations, the use of cloud tools, and the differences between Snowflake and Databricks. Key topics included coding practices, the importance of understanding tools rather than just using them, and strategies for managing coursework and deadlines effectively. This session benefits students seeking guidance on practical applications in data engineering and AI. ### TA Office hour 4 URL: https://www.dataexpert.io/lesson/taofficehour4-3892c-p736-l2050 Enrolled students ### Special Guests, Stephanie Nuesi and Joe Reis [July 26, 2023] URL: https://www.dataexpert.io/lesson/week-2-speaker-v2 Enrolled students Join us each week as we sit down with visionary speakers from diverse fields, engaging in enlightening conversations that explore cutting-edge insights, trends, and their wealth of expertise. (Recorded July 26, 2023) ### Dimensional Data Modeling: Graph Data Modeling Day 3 Lecture URL: https://www.dataexpert.io/lesson/dimensional-data-modeling-day-3-lecture-yt-p736-l666 Free In this lecture we talk about dimensional additivity and how to build a flexible data model ready for graph database consumption! ### RAG optimisation + Reranking Day 1 Lecture URL: https://www.dataexpert.io/lesson/rag-optimisation-reranking-day1-lecture-oct-2025 Enrolled students In this video, Zach discusses the concepts of vectors and RAG. He emphasizes the importance of identifying business problems suitable for AI, noting that AI is not a self-starter and requires specific trigger points to deliver value. Zach explores various interfaces beyond chatbots, such as CICD and cron jobs, and explains how these can enhance performance and relevance in RAG systems. Additionally, he outlines strategies for improving vector search performance and making RAG outputs more relevant. ### Career Development - Data Modeling Interview URL: https://www.dataexpert.io/lesson/career-development-data-modeling-interview-jan2025-p736-l801 Module: Career Development · Enrolled students Gain insights into the data modeling interview process, focusing on how to effectively communicate with interviewers. Learn the importance of asking clarifying questions, designing schemas, and understanding data flow, particularly in the context of a ride-sharing app. This lesson is ideal for aspiring data engineers and analysts looking to enhance their interview skills and grasp the nuances of data architecture discussions. ### Youtube URL: https://www.dataexpert.io/lesson/56ab37f4-896d-43a1-afbf-e0b7aebf8c18cours Enrolled students testing updating ### Understanding Read Streams and SCD Implementation Lab URL: https://www.dataexpert.io/lesson/understandingreadstreamsandscdimplementationlab-jan2026 Enrolled students Read streams and slowly changing dimensions (SCD) are crucial for tracking data changes over time. This lab focuses on creating a table to monitor active NBA players and implementing SCD Type 2 using SQL commands like MERGE and INSERT. Participants will learn how to effectively manage data streams, materialize changes, and handle player status updates, making it ideal for data engineers and analysts looking to enhance their skills in change data capture and data modeling. ### How to do Growth Accounting like Meta URL: https://www.dataexpert.io/lesson/howtodogrowthaccountinglikemeta-jan2026-p735-l1903 Module: Week 7: Analytical Patterns & CDC · Enrolled students In this video, Zach shares a crash course on applying analytical patterns based on the two years at Meta in growth analytics. He explores three main types of analyses: aggregation-based, state change tracking, and survivorship analysis, emphasizing the importance of repeatable analyses to reduce cognitive load. He also highlights the significance of understanding dimensions and their additive properties to avoid common pitfalls in data interpretation. ### Managing Unstructured Data - Day 1 Lab URL: https://www.dataexpert.io/lesson/managing-unstructured-data-day-1-lab-may2025 Enrolled students In this video, Zach walks viewers through the process of setting up a RAG LLM example lab, with a focus on working with unstructured data. He demonstrates how to create an index using the text embedding model and addresses the transcription of video files to extract searchable vectors. Additionally, the importance of adjusting parameters, such as top K, for effectively querying the vector database is emphasized. ### Apache Spark Shuffle Joins Day 1 Lab URL: https://www.dataexpert.io/lesson/apache-spark-shuffle-joins-day-1-lab-may2025 Enrolled students In this lab, Zach walks through the process of joining two datasets related to NBA games and players, focusing on the performance implications of different join strategies. He demonstrates how to disable broadcast joins and analyze the resulting execution plans to better understand the underlying processes. He also highlights the importance of using formatted modes for clarity in job execution. ### Snowflake Basics Day 1 Lecture URL: https://www.dataexpert.io/lesson/snowflake-basics-day-1-lecture-jan2025 Enrolled students In this lecture, Bruno introduces Snowflake and DBT, focusing on basic concepts like time travel, fail safe, and data types. He also discusses the differences between Snowflake and Databricks, emphasizing the ease of scaling in Snowflake. ### Chat with Zach [Nov 15, 2023] URL: https://www.dataexpert.io/lesson/v3-2-chat-with-zach-nov-15-2023-6706 Enrolled students Delve into the world of data engineering with our weekly Q&A / Fireside Chat series, featuring the founder of EcZachly Inc and lead instructor of the Advanced Data Engineering Bootcamp. Each session covers a different topic, providing invaluable insights and real-world experiences to empower data enthusiasts in their journey. (Recorded Nov 15, 2023) ### Apache iceberg: WAP patterns and Querying branched Data Lab URL: https://www.dataexpert.io/lesson/apache-iceberg-lab-day-3-april2025-p735-l961 Module: Week 5: Data Modeling on Iceberg · Enrolled students In this lab, Zach walks you through the Apache Iceberg Day 3 lab, focusing on iceberg branching and working with stock data from an API. He demonstrates how to create a new version of a table and run queries using Spark and Glue. He also discusses the importance of data quality checks and how to manage branches effectively. ### Bootcamp Kickoff URL: https://www.dataexpert.io/lesson/bootcamp-kickoff-yt-p736-l703 Module: Kickoff · Free In this video, Zach talks about the free youtube data engineering bootcamp kickoff. ### Q&A with Zach Week1 URL: https://www.dataexpert.io/lesson/qawithzachweek1-jan2026-p735-l1929 Module: Q&A with Zach · Enrolled students A live Q&A session with Zach covers various topics related to the analytics engineering boot camp, including onboarding processes, course structure, and available resources. Participants ask questions about group assignments, tech stacks, and capstone projects, while Zach emphasizes the importance of engagement and adapting to the fast-paced learning environment. The discussion also touches on personal growth, job application strategies, and the significance of mentorship in navigating the boot camp experience. ### Tech Talk - Li URL: https://www.dataexpert.io/lesson/tech-talk-li-july2025 Enrolled students Li Yan, founder of SILFAI and former AI researcher at Meta, shares her insights on AI engineering and the evolving landscape of the field. This session covers essential skills for aspiring AI engineers, including the importance of project experience, understanding user needs, and navigating ambiguity in product development. Li emphasizes the shift in demand towards experienced engineers and the value of building unique projects to stand out in job applications, making this lesson valuable for anyone looking to enter the AI space. ### Tech Talk - 2 URL: https://www.dataexpert.io/lesson/tech-talk2-oct2025 Enrolled students Unfortunately, Andrew couldn't join the Tech Talk, so this is Q&A session with Zach. ### Apache Spark: Managing Spark Jobs and Notebooks Day 2 Lecture URL: https://www.dataexpert.io/lesson/apache-spark-day-2-lecture-yt-p736-l680 Enrolled students In this lecture, Zach discusses the intricacies of managing Spark jobs and notebooks, highlighting the challenges of achieving code modularity and the nuances of jar submissions for Scala Spark and PySpark. ### Advanced SQL and Applying Analytical patterns (Day 1 Lecture) URL: https://www.dataexpert.io/lesson/analytical-patterns-day-1-lecture-v4 Module: Week 5: Analytical Patterns & Analysis in Iceberg · Enrolled students This is a pre-recorded class. In this video lecture, we explore fundamental concepts in data engineering, with a primary focus on the pivotal role of SQL as the foundational element in this field. The discussion extends to the patterns and structures essential for scaling data engineering projects, providing a comprehensive understanding of their implementation. Delving into aggregation-based analysis, we address considerations surrounding dimensions and granularity. The lecture also touches upon the significance of retention analysis and survivorship curves, offering insights that are valuable for career growth. Join us in this session to gain a profound understanding of the key elements shaping data engineering and the strategic insights necessary for success in this dynamic field. [Recorded on Nov 27, 2023] ### Azure - Week 4 URL: https://www.dataexpert.io/lesson/azure-week-4-may2025 Enrolled students Understanding Azure Data Factory (ADF) is crucial for anyone involved in data engineering on the Azure platform. This session covers the fundamentals of ADF, including its components like link services, datasets, and activities, which are essential for creating data pipelines. Participants will learn how to use ADF for ETL processes, including moving data from various sources to destinations, all while leveraging a low-code environment that simplifies the pipeline creation process. ### Spark Data Quality (Day 3 Lab) URL: https://www.dataexpert.io/lesson/spark-batch-day-3-lab-v4 Module: Week 4: Batch Pipelines with Apache Spark · Enrolled students This is a pre-recorded class. In this lab, we embark on an exploration of Spark, offering a comprehensive overview of its main components and the intricacies of data transformation within a Spark job. The focus then shifts to the testing paradigm in PySpark code, where I guide you through the creation of input and output data frames and the utilization of named tuples for comparison. Delving into the significance of testing, we unravel its role in enhancing development velocity. The discussion is enriched with insights into common challenges and practical tips for conducting effective tests. Join us in this video to deepen your comprehension of Spark and acquire essential skills in testing PySpark code.[Recorded on Dec 12, 2023] ### How to load data into Iceberg with Airflow URL: https://www.dataexpert.io/lesson/howtoloaddataintoicebergwithairflow-16dc3 Enrolled students ### Wednesday Quiz URL: https://www.dataexpert.io/lesson/wednesdayquiz-0d421 Enrolled students ### Q&A with Zach 4 URL: https://www.dataexpert.io/lesson/qawithzach4-mar26-p737-l2122 Module: Q&A with Zach · Enrolled students Week 4 of the Q&A session with Zach focuses on student engagement and feedback, as well as insights into the skills needed for data engineering and AI roles. Participants discuss their experiences, including job placements and effective strategies for standing out in the job market, such as leveraging LinkedIn. Key topics include the importance of storytelling in interviews, the evolving landscape of data and AI engineering roles, and the significance of continuous learning and community involvement in achieving career success. ### Applying Advanced SQL for Analytical Insights - Aggregations and Cardinality Reduction in Bootcamp Web Events (Day 2 Lab) URL: https://www.dataexpert.io/lesson/analytical-patterns-day-2-lab-v3 Module: Week 3: Analytical Patterns & Analysis in Iceberg · Enrolled students In this lab session, Zach demonstrates various queries and aggregations that can be applied to the bootcamp web events table. The talk emphasizes the significance of reducing cardinality and showcases the use of functions like coalesce and rollup. Zach also touches on the concept of self-joins and how they can aid in analyzing a users journey on the website. [Recorded on Nov 29, 2023] ### Azure - Week 3 URL: https://www.dataexpert.io/lesson/azure-week-3-may2025-p736-l1149 Enrolled students Understanding cloud databases is crucial for data engineers, and this session focuses on Azure's database offerings. It covers the distinctions between structured, semi-structured, and unstructured data, and how to select appropriate storage solutions like Azure SQL Database and NoSQL options. Participants will learn about cloud migration, database provisioning, and the importance of firewall settings and access management, providing a solid foundation for building scalable data solutions in Azure. ### Advanced Spark (Day 2 Lab) URL: https://www.dataexpert.io/lesson/spark-batch-day-2-lab-v4 Module: Week 4: Batch Pipelines with Apache Spark · Enrolled students In this lab, Zach shows the students how to use Glue Job Runner and Iceberg to optimize the data processing. He goes over setting up the job, running Python functions, and using UDFs. He also demonstrates how to monitor the job and view the output table. Plus, he explains the benefits of using Iceberg for data compression and partitioning. [Recorded on May30th, 2024] ### MCP and Agentic AI Day 1 Lecture URL: https://www.dataexpert.io/lesson/mcpandagenticaiday1lecture-oct2025 Enrolled students In this video, Zach dives into the Model Context Protocol (MCP) and its tools, emphasizing how they can transform AI from merely providing advice to taking actionable steps. He highlights that only about 20% of internet users engage with ChatGPT, and a mere 1-2% have used Python to prompt it, showcasing how early we are in this AI trend. Zach will explore how MCP servers can streamline tool management and enhance AI capabilities, guiding viewers through the process of setting up their own MCP server by the end of the week. ### Carly Taylor - Field CTO at Databricks (2.0) URL: https://www.dataexpert.io/lesson/tech-talk-with-carly-taylor-p735-l592 Module: Tech Talks · Enrolled students Carly Taylor, a Field CTO at Databricks, shares her journey from chemist to data and machine learning expert, offering insights for career changers. She discusses the importance of boot camps, the value of community in learning, and the significance of understanding data pipelines and analytics engineering. Carly emphasizes the need for effective communication and the balance between technical skills and storytelling in data roles, making this session valuable for aspiring data professionals looking to navigate their careers. ### Special Guests, Carly Taylor and Ben Rogojan [July 19, 2023] URL: https://www.dataexpert.io/lesson/week-1-speaker-v2 Enrolled students Join us each week as we sit down with visionary speakers from diverse fields, engaging in enlightening conversations that explore cutting-edge insights, trends, and their wealth of expertise. (Recorded July 19, 2023) ### TA Office hour 2 URL: https://www.dataexpert.io/lesson/taofficehour2-jan2026 Enrolled students This office hours session provides an interactive platform for students to ask questions about the data and AI engineering boot camp. Key topics discussed include onboarding processes, assignment submissions, and project requirements, particularly around the use of tools like Airflow and Astronomer. Students can benefit from personalized guidance on their capstone projects and clarification on technical aspects of their coursework, fostering a collaborative learning environment. ### Building functional AI agents using LangChain Day1 Lecture URL: https://www.dataexpert.io/lesson/buildingfunctionalaiagentsusinglangchainday1lecture-mar26-p737-l2137 Module: Week 1: Prompt Engineering & Basic RAG · Enrolled students In this lesson, Zach dives into the concept of AI agents, exploring what they are and how they differ from simple prompts and tools. He emphasises the importance of understanding the AI pyramid, where most people are still at the bottom, and encouraged everyone to feel confident in their learning journey. Zach also introduces the idea of using tools and middleware to enhance AI capabilities, and the class discussed the need for human-in-the-loop validation for risky actions. ### Book Club 4 URL: https://www.dataexpert.io/lesson/bookclub4-feb26 Enrolled students A discussion on various aspects of data engineering, particularly focusing on streaming and batch processing, unfolds in this session. Participants share their experiences with building streaming pipelines, including challenges faced and the complexity of implementation. The conversation also touches on the evolving landscape of data engineering tools, such as the shift from Scala to more accessible languages, and the importance of balancing performance with maintainability in code. ### Day 1 Lecture: Graph RAG vs Vector RAG URL: https://www.dataexpert.io/lesson/advanced-rag-and-agentic-ai-day1-lecture-july2025-p737-l1178 Module: Week 2: Advanced RAG, Reranking & Dev Workflow · Enrolled students In this lecture, Ashwani provides an overview of vector retrieval augmented generation (RAG) and introduce knowledge graphs, highlighting their evolution and significance in enhancing search capabilities. He discuss the limitations of vector RAG, particularly in handling deep relationship queries and fragmented context, and how knowledge graphs can address these challenges. He also explains the process of constructing knowledge graphs from structured and unstructured data, emphasizing the importance of defining schemas for effective graph creation. Lastly, he outlines the retrieval patterns and strategies for leveraging knowledge graphs. ### Build your first pipeline with dbt URL: https://www.dataexpert.io/lesson/buildyourfirstpipelinewithdbt-jan2026 Enrolled students Building your first pipeline with dbt introduces essential concepts and practices for analytics engineers looking to transform raw data into actionable insights. The lesson covers key topics such as data modeling, sources, and seeds, emphasizing the importance of SQL and software engineering best practices. Participants will learn how to effectively use dbt to create models and manage data transformations within a data warehouse environment, making it suitable for both beginners and those with prior experience in dbt. ### Snowflake and Glue Catalog Integration Lab URL: https://www.dataexpert.io/lesson/snowflakeandgluecatalogintegrationlab-jan2026-p735-l2005 Module: Week 7: Analytical Patterns & CDC · Enrolled students Learn how to integrate Snowflake with AWS Glue Catalog in this lab, focusing on creating a seamless connection for data querying and management. You'll explore the steps to establish trust between Snowflake and AWS, create an Iceberg volume for data storage, and set up Glue for metadata management. This lesson is ideal for data engineers and analysts looking to enhance their skills in cloud data integration and leverage open table formats for improved data accessibility. ### Your Action Plan and wrap-up URL: https://www.dataexpert.io/lesson/youractionplanandwrapup-2a75b Enrolled students Building your personalized promotion roadmap Quick wins vs. long-term positioning Gap analysis: What do you need to demonstrate that you haven't yet? Setting up accountability partnerships Post-workshop resources and community access ### OpenClaw and Automating business processes Day2 Lecture URL: https://www.dataexpert.io/lesson/openclawandautomatingbusinessprocessesday2lecture-mar26 Enrolled students In this lecture, Carly recaps OpenClaw and then go deep on Pi Agent, the lightweight coding agent at the core. Also discusses about why open source extensibility creates both power and a bigger security risk surface. ### Grouping sets and Window functions Lab URL: https://www.dataexpert.io/lesson/grouping-sets-and-window-functions-lab-p735-l581 Module: Week 7: Analytical Patterns & CDC · Enrolled students In this lab session, Zach demonstrates various queries and aggregations that can be applied to the bootcamp web events table. The talk emphasizes the significance of reducing cardinality and showcases the use of functions like coalesce and rollup. Zach also touches on the concept of self-joins and how they can aid in analyzing a users journey on the website. ### Data Visualization and Impact: Mastering Data Engineering Day 1 Lecture URL: https://www.dataexpert.io/lesson/data-visualization-and-impact-day-1-lecture-yt-p736-l690 Enrolled students Join this insightful video as it delves into the world of data engineering. In this session, common challenges encountered in data engineering are explored, accompanied by practical solutions and best practices for constructing efficient data pipelines. ### Deep Dive On Workflows Day 3 Lecture URL: https://www.dataexpert.io/lesson/deep-dive-on-workflows-day-3-lecture-jan2025 Enrolled students Explore the intricacies of orchestration within Databricks in this detailed lecture. Learn how to effectively manage workflows that involve data ingestion, transformation, and machine learning, utilizing tools like Delta Live Tables and Spark. This session is ideal for data engineers and data scientists looking to optimize their data pipelines, reduce costs, and streamline processes by leveraging Databricks' unified orchestration capabilities. ### Deep Dive On Workflows Day 3 Lab URL: https://www.dataexpert.io/lesson/deep-dive-on-workflows-day-3-lab-jan2025 Enrolled students Explore the intricacies of building and managing data workflows in this lab session focused on executing sequential pipelines and handling dependencies. Gain hands-on experience with Delta Live Tables, including data ingestion, monitoring, and quality checks, using tools like Databricks and Auto Loader. Ideal for data engineers and analysts, this lesson provides practical insights into creating robust data pipelines, managing data quality, and building dashboards for effective data visualization. ### Tuesday Quiz URL: https://www.dataexpert.io/lesson/tuesdayquiz-3abf8 Enrolled students ### Prasad Rao - Principal Solutions Architect URL: https://www.dataexpert.io/lesson/tech-talk-8-prasad-rao-april2025-p735-l1062 Module: Tech Talks · Enrolled students Prasad Rao, a Principal Solutions Architect at AWS, shares insights from his extensive career in tech, focusing on the journey to achieving principal-level roles. He discusses the importance of articulating experience during interviews, the significance of influence over execution, and strategies for effective promotion documentation. This session is beneficial for engineers aspiring to advance their careers, as it highlights the skills and mindset needed to scale impact within organizations. ### Intro to Javascript URL: https://www.dataexpert.io/lesson/intro-to-javascript-f6146 Module: Web Dev Fundamentals · Enrolled students JavaScript is a versatile programming language essential for creating interactive web experiences. This lesson covers fundamental concepts such as variable declaration using let and const, data types, control flow, and functions. Learners will also explore DOM manipulation, event handling, and error management, providing a solid foundation for web development and enhancing their coding skills in JavaScript. ### Advanced dbt: Understanding incremental models Lab URL: https://www.dataexpert.io/lesson/advanced-dbt-understanding-incremental-models-lab-p735-l569 Module: Week 2: Advanced dbt · Enrolled students In this lab, Bruno demonstrates the implementation of incremental models using DBT. He showcases the process of setting up a new staging model to display incremental models, emphasizing the importance of defining conditions for merging logic. Throughout the lab, he explains the concept of incremental materialization and the use of full refresh flags. ### Tech Talk 5 - Carly Taylor URL: https://www.dataexpert.io/lesson/tech-talk-5-carly-taylor-april2025 Enrolled students Carly Taylor shares her unique journey from chemist to data engineer, discussing the importance of analytics and data engineering in the evolving tech landscape. She emphasizes the need for professionals to adapt their skill sets in response to automation and AI advancements, particularly in tools like DataBricks and Cursor. This session is ideal for aspiring data engineers and analysts seeking insights on career growth, skill development, and the future of data roles in business. ### Tech talk with Tim Chan URL: https://www.dataexpert.io/lesson/tech-talk- with-tim-chan Module: Tech Talk · Enrolled students Tim Chan, head of data science at Statsig, shares insights on experimentation in data science during this engaging tech talk. He discusses the nuances of designing A/B tests, the importance of statistical rigor, and how to interpret results effectively. This lesson is particularly beneficial for analytics engineers and data scientists looking to enhance their understanding of experimentation methodologies, metrics, and the balance between statistical significance and business impact. ### Dimensional Data Modeling - Complex Data Types Arrays and Structs with Iceberg (Day 1 Lab) - Pre-recorded from V3 URL: https://www.dataexpert.io/lesson/dimensional-data-modeling-day-1-lab-v4 Module: Week 1: Dimensional Data Modeling in Iceberg · Enrolled students This is a pre-recorded lesson from V3. In this lab, we explore Slowly Changing Dimensions (SCDs) and design Cumulative Tables using the nba_players dataset. This is our first stress test of our web-based interface linked to a Presto/Trino database running in Starburst. Presto (formerly PrestoSQL) or Trino are both query engines that allow users to query and analyze data in a data lake using SQL, while Starburst offers a distribution of Presto/Trino. During this session, attendees executed 1742 queries in one hour and Starburst handled the workload without any issues! This lab not only demonstrates how to design cumulative tables but also showcases the power of running analytics in the data lake! [Recorded Nov 7, 2023] ### Special Guests, Parth Parekh and Nick Singh [August 16, 2023] URL: https://www.dataexpert.io/lesson/week-5-speaker-v2 Enrolled students Join us each week as we sit down with visionary speakers from diverse fields, engaging in enlightening conversations that explore cutting-edge insights, trends, and their wealth of expertise. (Recorded August 16, 2023) ### Tech Talk - Sai URL: https://www.dataexpert.io/lesson/techtalks3-mar26 Enrolled students Tech Talk - Sai provides valuable insights into leveraging LinkedIn for career advancement in data analytics and engineering. Sai, a lead data analyst, shares his journey and emphasizes the importance of building a personal brand through consistent posting and networking. He discusses the evolving role of AI in data jobs, highlighting the need for strong communication and problem-solving skills, and offers practical tips for using AI tools to enhance productivity and career growth. ### Azure - Week 2 (Copy) URL: https://www.dataexpert.io/lesson/azure-week-2-7072 Enrolled students The session focuses on Azure storage accounts, specifically discussing the differences between Blob Storage and Azure Data Lake Storage (ADLS) for data engineering. Participants will learn about access methods such as access keys and Shared Access Signatures (SAS), which provide fine-grained access control. Additionally, the lesson covers redundancy levels in Azure storage, including Locally Redundant Storage (LRS), Zone-Redundant Storage (ZRS), and Geo-Redundant Storage (GRS), emphasizing their importance for data availability and disaster recovery. ### Advanced Spark Optimization Techniques Day 1 Lab URL: https://www.dataexpert.io/lesson/advanced-spark-optimization-techniques-day-1-lab-jan2025-p736-l813 Module: Week 4: Structured Streaming with Spark & Kafka · Enrolled students Advanced Spark Optimization Techniques Day 1 Lab focuses on enhancing your skills in optimizing Spark applications, particularly in structured streaming with Spark and Kafka. Participants will learn to read and filter CSV data, analyze physical execution plans, and troubleshoot performance issues using Spark UI metrics. This lab is ideal for data engineers and data scientists looking to deepen their understanding of Spark's optimization capabilities and improve data processing efficiency in real-time applications. ### TA Office Hour 6 URL: https://www.dataexpert.io/lesson/taofficehour6-3254f Enrolled students ### Chat with Zach [July 20, 2023] URL: https://www.dataexpert.io/lesson/week-1-zach-qa-v2 Enrolled students Delve into the world of data engineering with our weekly Q&A / Fireside Chat series, featuring the founder of EcZachly Inc and lead instructor of the Advanced Data Engineering Bootcamp. Each session covers a different topic, providing invaluable insights and real-world experiences to empower data enthusiasts in their journey. (Recorded July 20, 2023) ### advanced dbt Day 1 Lab URL: https://www.dataexpert.io/lesson/advanceddbtday1lab-jan2026 Enrolled students Advanced DBT Day 1 Lab focuses on practical applications of DBT, specifically using Jinja for macros and implementing incremental models. Participants will learn how to create and utilize macros for SQL generation, manage incremental data loads in Snowflake, and handle data updates efficiently. This lab is ideal for data engineers and analysts looking to enhance their DBT skills and optimize their data transformation workflows. ### The Data Modeling Interview URL: https://www.dataexpert.io/lesson/the-data-modeling-interview Module: Interview Skills · Free This video covers the main things you need to remember to pass the Data Modeling interview ### Tech Talks - Carly URL: https://www.dataexpert.io/lesson/techtalksweek3-jan2026 Enrolled students Carly Taylor shares her diverse career journey from chemist to data scientist, emphasizing the value of varied experiences in problem-solving across industries like gaming, real estate, and marketing. She discusses the evolving expectations for junior engineers in the age of AI, highlighting the importance of soft skills and storytelling in interviews. This session is ideal for aspiring data professionals looking to navigate their careers and understand how to leverage their unique backgrounds effectively. ### RAG optimisation + Reranking Day 1 Lab URL: https://www.dataexpert.io/lesson/rag-optimisation-reranking-day-oct Enrolled students ### Q&A with Zach Week 3 URL: https://www.dataexpert.io/lesson/qa-with-zach-week-3-april2025-p735-l1000 Module: Q&A with Zach · Enrolled students A live Q&A session with Zach covers various topics related to the boot camp experience, including mentorship, capstone projects, and data engineering techniques. Participants ask questions about project requirements, data sourcing, and transitioning into data engineering roles. Zach emphasizes the importance of completing assignments on time and discusses strategies for effective project execution, including the use of tools like Snowflake and DBT. This session is beneficial for students seeking guidance on their projects and career paths in data engineering. ### Dimensional Data Modeling - Graph Data Modeling with Iceberg (Day 3 Lecture) URL: https://www.dataexpert.io/lesson/dimensional-data-modeling-day-3-lecture-v3 Module: Week 1: Dimensional Data Modeling in Iceberg · Enrolled students In this presentation, we cover crucial concepts: the difference between additive and non-additive dimensions, the significance of Enums, and the decision-making process behind using flexible data types. Gain insights into how these elements play a role in effective data modeling and learn when to leverage each approach in your projects. [Recorded Nov 9, 2023] ### Thursday Quiz URL: https://www.dataexpert.io/lesson/thursdayquiz-3ef15 Enrolled students ### Advanced Data Pipeline Maintenance - Understanding Complex Pipeline Topology (Day 2 Lab) URL: https://www.dataexpert.io/lesson/pipeline-maintenance-day-2-lab-v3 Module: Week 6: Data Pipeline Maintenance · Enrolled students Unlocking the intricacies of complex pipelines, this video underscores the significance of comprehending their topology. Zach sheds light on the primary owner and the various teams integral to the troubleshooting process. Emphasizing key elements such as documentation, SLAs, and data contracts, the video offers insights into managing and optimizing intricate pipelines. This session provides valuable guidance for navigating the challenges of complex data engineering tasks.[Recorded on Dec 21st 2023] ### Day 2 Lecture: How to link your AI agent to value URL: https://www.dataexpert.io/lesson/building-end-to-end-ai-application-day2-lecture-july25 Enrolled students In this lecture Zach dives into end-to-end AI applications, discusses what AI excels at, the roadblocks that hinder its value, and strategies to overcome those challenges. He shares that around 60–70% of AI projects fail, slightly better than the 80% failure rate often cited for data science projects. Zach explores key AI use cases—prediction, detection, and optimization—while emphasizing that not every problem requires large language models. ### Flink Lab Setup URL: https://www.dataexpert.io/lesson/flink-lab-setup-yt-p736-l708 Enrolled students ### TA Office hour 4 URL: https://www.dataexpert.io/lesson/taofficehour4-jan2026 Enrolled students A TA office hour session focused on addressing student questions related to their capstone projects and assignments. Key topics included project planning, incorporating analytics, data engineering, and AI elements, as well as troubleshooting technical issues with tools like PostgreSQL and Snowflake. Participants received guidance on project timelines, mentor communications, and accessing course materials, ensuring they are well-prepared for their upcoming assignments and capstone submissions. ### Apache Iceberg Day 2 Lab URL: https://www.dataexpert.io/lesson/apache-iceberg-day-2-lab-jan2025 Enrolled students In this lab, Zach walks through data compression techniques, demonstrating how to reduce data size and enhance efficiency. He explores concepts like run-length encoding and bin-pack compression, focusing on optimizing data storage. ### Apache Iceberg Lab Day1 URL: https://www.dataexpert.io/lesson/apacheiceberglabday1-cbfc8 Enrolled students Apache Iceberg's Day 1 lab focuses on essential features like partitioning, snapshotting, and time travel, using Trino as the execution engine. Participants will learn to create and manage Iceberg tables, explore snapshot metadata, and understand the importance of Sargable queries for efficient data retrieval. This hands-on session is ideal for data engineers looking to deepen their understanding of Iceberg's capabilities and improve their SQL skills in a data lake environment. ### Tech Talk - Carly Taylor URL: https://www.dataexpert.io/lesson/guest-speaker-10-v4 Enrolled students In this tech talk series with Carly Taylor, she shares insights on career progression in tech, emphasizing the importance of skill set rarity and effective communication. Acknowledging the need for direction-setting and team management, she reflects on personal growth challenges and the value of feedback. [Recorded on June5th, 2024] ### Introduction to Spark Day 2 Lab URL: https://www.dataexpert.io/lesson/introduction-to-spark-day-2-lab-jan2025-p736-l776 Module: Week 1: Databricks Basics · Enrolled students In this lab, Yared walks you through the process of running Spark jobs in Databricks, focusing on the differences between shared and personal clusters. He highlights the importance of understanding job IDs, stages, and tasks when executing jobs. He also requests everyone to switch to their personal clusters for better visibility of job details. Please make sure to destroy your cluster after use to avoid unnecessary costs. ### Sunday Quiz URL: https://www.dataexpert.io/lesson/sundayquiz-1b699 Enrolled students ### QnA with Zach 2 URL: https://www.dataexpert.io/lesson/qnawithzach2-aug26 Enrolled students A live Q&A session addresses various topics related to data engineering, particularly focusing on watermarking and deduplication in streaming data pipelines. Participants discuss the importance of watermarking in Spark Streaming, the implications of using micro-batch architectures, and strategies for handling duplicates effectively. The conversation also touches on capstone project requirements, including the need for real-time data processing and the integration of various data sources. ### Apache Iceberg Data Contracts Lecture URL: https://www.dataexpert.io/lesson/apache-iceberg-data-contracts-day-3-lecture-may2025 Enrolled students In this lecture, Zach dives into the concept of data contracts, which are crucial for data engineers to establish clear agreements between data producers and consumers. He discusses the importance of these contracts in minimizing burnout and ensuring that stakeholders understand their responsibilities. ### Databricks Platform Overview Day 1 Lab URL: https://www.dataexpert.io/lesson/databricks-platform-overview-day-1-lab-jan2025-p736-l772 Module: Week 1: Databricks Basics · Enrolled students In this lab, we go through the databricks platform. ### CDC Pipelines and Triggers URL: https://www.dataexpert.io/lesson/cdc-and-push-pipelines Enrolled students In this lab we go over how to propagate data from production to Lakehouse and build a triggered pipeline. Visit this URL to play around with the production data from github https://lakebase-cdc-app-1352785079224954.aws.databricksapps.com/ ### Yuzheng Sun - Staff Data Scientist at Statsig URL: https://www.dataexpert.io/lesson/yz-jan2025-p736-l802 Module: Guest Speakers · Enrolled students Yuzheng Sun, a seasoned data scientist with experience at Amazon, Facebook, and Tencent, shares insights on A/B testing, data science roles, and the evolving landscape of AI. He emphasizes the importance of experimentation in data-driven decision-making, the necessity of domain knowledge, and the value of understanding business contexts. This session is ideal for aspiring data scientists and analytics engineers looking to deepen their understanding of practical data science applications and the cultural shifts in the industry. ### Dimensional Data Modeling: Building Slowly Changing Dimensions Day 2 Lab URL: https://www.dataexpert.io/lesson/dimensional-data-modeling-day-2-lab-yt-p736-l661 Free In this video we build an slowly changing dimension type 2 model on top of the players table we built in the first lab! ### End-to-End AI Applications Day 2 Lab URL: https://www.dataexpert.io/lesson/building-end-to-end-ai-application-day2-lab-oct25 Enrolled students In this lab, Zach walks through the integration of various components, concentrating on n8n and webhooks to build a LinkedIn shout‑out workflow. Zach demonstrates how to set up triggers, parse and handle JSON data, and implement a human‑in‑the‑loop validation process using Discord for approvals. He guides through troubleshooting steps, ensuring the application is deployment‑ready. ### Building functional AI agents using LangChain Day2 Lab URL: https://www.dataexpert.io/lesson/buildingfunctionalaiagentsusinglangchainday2lab-mar26 Enrolled students In this lab, Zach dives into deep research agents and MCP tools. He explores the GitHub agent and its functionality, including how to manage pull requests and create issues on Linear. He demonstrates how to refine the agent's prompts to ensure it uses the correct tools and avoids confusion. He also introduces the deep research agent that utilizes web searches and saves findings to Supabase, emphasizing the importance of semantic matching to avoid duplicate research. ### Unit Testing URL: https://www.dataexpert.io/lesson/unit-testing-cd643 Module: Office Hours · Enrolled students ### Structured Streaming Kafka to Delta Live Table Day1 Lecture URL: https://www.dataexpert.io/lesson/structuredstreamingkafkatodeltalivetableday1lecture-feb26-p736-l2029 Module: Week 4: Structured Streaming with Spark & Kafka · Enrolled students In this lecture, Zach dives into the world of real-time data processing, focusing on the velocity problem and how technologies like Kafka, Iceberg, and Delta Lake fit into the picture. He explores the Kafka producer and consumer APIs, and shares insights on when to choose streaming over batch processing. He also highlights the advantages of Delta Live Table for making streaming pipelines more accessible. ### The 2026 Context Engineering QnA URL: https://www.dataexpert.io/lesson/the2026contextengineering-qna Enrolled students The 2026 Context Engineering Q&A covers a range of topics relevant to data engineering and AI, including insights on the DataBricks platform, LTAP modeling, and master data management (MDM). Participants gain valuable perspectives on the evolving landscape of data engineering, interview preparation strategies, and the integration of MDM with data engineering roles. This session is ideal for data professionals looking to enhance their understanding of current trends and best practices in the field. ### Q&A with Zach Week5 URL: https://www.dataexpert.io/lesson/qawithzachweek5-jan2026-p735-l1933 Module: Q&A with Zach · Enrolled students The session features a live Q&A with Zach, focusing on the final week of the analytics engineering boot camp. Key topics include project deadlines, capstone project requirements, and the distinction between excellence and superb certificates. Participants can expect insights on using tools like DBT, Airflow, and Snowflake, as well as guidance on project presentations and mentorship opportunities. This session is ideal for boot camp participants seeking clarity on their projects and certification paths. ### OpenClaw Safeguards & Measuring business value for AI Agents Day1 Lab URL: https://www.dataexpert.io/lesson/openclawsafeguardsmeasuringbusinessvalueforaiagentsday1lab-mar26-p737-l2150 Module: Week 4: MLOps, Guardrails & Business Value · Enrolled students Explore the intricacies of OpenClaw's safeguards and the measurement of business value for AI agents in this hands-on lab session. Participants will learn how to implement prompt injection tests, tool scoping, and least privilege principles while interacting with OpenClaw. This lesson is ideal for AI engineers and data professionals looking to enhance their understanding of AI safety protocols and improve their AI deployment strategies. ### TA Office Hour 1 (2025-01-14) URL: https://www.dataexpert.io/lesson/ta-office-hour-1-jan2025-p736-l721 Enrolled students Office hours provide a platform for students to ask questions and seek clarification on course materials and assignments. Key topics discussed include data modeling, handling .env files, and expectations for the capstone project. Participants share their struggles with homework, seek advice on tools and techniques, and discuss the importance of collaboration and peer support. This session is particularly beneficial for students looking for guidance on practical applications and troubleshooting in data engineering. ### Azure - Week 1 URL: https://www.dataexpert.io/lesson/azure-week-1-may2025-p736-l1080 Enrolled students Understanding cloud computing is essential for data engineers, and this session introduces Azure's foundational services, particularly the storage account. Participants will learn how to create and utilize Azure storage accounts, focusing on Blob storage and Azure Data Lake Storage (ADLS), which are crucial for data management and analytics. The lesson also covers key concepts like scalability, cost considerations, and authentication methods, providing a comprehensive overview for those new to Azure or transitioning from other cloud platforms. ### Tech Talk 4 - John Kutay URL: https://www.dataexpert.io/lesson/tech-talk-4-john-kutay-april2025 Enrolled students Change Data Capture (CDC) is a crucial technique for data engineers, allowing them to efficiently track and replicate changes in databases. This session features John Kutay from Stream, who shares insights on implementing CDC using log-based methods and discusses its benefits over traditional batch processing. Learn about real-world applications, including how companies like Stripe and American Airlines leverage CDC for operational efficiency and real-time analytics, making it essential for modern data engineering practices. ### Build your first pipeline with dbt URL: https://www.dataexpert.io/lesson/buildyourfirstpipelinewithdbt-93144 Enrolled students ### TA Office Hour 6 (2025-06-19) URL: https://www.dataexpert.io/lesson/ta-office-hour-6-may2025-p736-l1158 Enrolled students This office hours session focuses on addressing student inquiries related to Delta Live Tables and Databricks workflows. Participants discuss querying APIs, loading JSON data into Delta Live Tables, and implementing medallion architecture. The conversation also touches on capstone project proposals, sharing resources, and navigating pipeline access within Databricks, making it beneficial for students seeking guidance on these topics. ### Navigating Data Engineering -Tips, Tools, and Career Insights URL: https://www.dataexpert.io/lesson/navigating-data-engineering-tips-tools-and-career-insights-may2025-p736-l1196 Module: Q&A with Zach · Enrolled students This Q&A session focuses on practical tips and insights for data professionals navigating their careers in data engineering. Key topics include transitioning from notebooks to IDEs, effective use of tools like AWS Glue and DataBricks, and strategies for enhancing technical skills in data engineering fundamentals. Participants will benefit from career advice, discussions on industry trends, and insights into the tools and technologies that are shaping the data engineering landscape. ### dbt Basics Day 3 Lecture URL: https://www.dataexpert.io/lesson/dbt-basics-day-3-lecture-jan2025 Enrolled students In this lecture, Bruno dives into the functionalities of DBT, particularly focusing on its role in data transformation and testing. He discusses the importance of unit tests and data tests, explaining how they can help ensure data integrity and business logic. He also touches on the challenges DBT aims to solve, such as documentation and testing chaos. ### TA Office Hour 2 URL: https://www.dataexpert.io/lesson/ta-office-hour-2-may2025 Enrolled students Office hours provide an opportunity for students to clarify their understanding of data engineering concepts and homework assignments. This session covered topics such as creating main and audit branches in data tables, using Trino for table creation, and performing data quality checks. Participants also discussed the structure of cumulative tables and the importance of specific fields in production tables, addressing various questions related to homework and project proposals. ### Jason Reid - Co-founder at Tabular URL: https://www.dataexpert.io/lesson/jason-reid-cofounder-of-tabular-p736-l719 Module: Guest Speakers · Enrolled students Jason Reid, co-founder of Tabular, shares insights on the evolution of data engineering and the Iceberg project in this engaging discussion. He reflects on his journey from Netflix to launching Tabular, emphasizing the importance of table abstractions and the challenges faced with traditional data formats. This lesson is valuable for data engineers and tech enthusiasts looking to understand the future of data lakes, the significance of open-source contributions, and the interplay between cloud technologies and data management. ### Snowflake Lab URL: https://www.dataexpert.io/lesson/snowflake-lab-jan2025 Enrolled students In this lab, Bruno walks you through using Snowflake for data management without dbt, focusing on creating and merging tables. He demonstrates how to handle new data, including updates and inserts, and explain the importance of clustering for efficient querying. He also covers change data capture and how to create a slow change dimension. ### Q&A with Zach Week1 URL: https://www.dataexpert.io/lesson/qawithzachweek1-jan2026 Enrolled students A live Q&A session with Zach covers various topics related to the analytics engineering boot camp. Participants discuss onboarding resources, the structure of the program, and the importance of live instruction versus pre-recorded content. Zach addresses concerns about course workload, group assignments, and capstone project ideas, emphasizing the need for effective time management and the value of building a strong portfolio. The session also touches on the significance of mentorship and networking in the tech industry. ### Building AI Agents with Databricks Day2 Lecture URL: https://www.dataexpert.io/lesson/buildingaiagentswithdatabricksday2lecture-feb26 Enrolled students In this lecture, Eumar discusses the critical transition from demo to production for AI agents, emphasizing that 80% of GenAI projects fail to reach production due to issues like data quality and user trust. He explores essential technologies such as Databricks Model Serving, Guardrails, and MLflow Evaluation, which can help ensure successful deployment. He highlights the importance of observability and monitoring in maintaining agent performance post-launch. ### Q&A with Zach Week 5 URL: https://www.dataexpert.io/lesson/qa-with-zach-week5-oct2025 Enrolled students ### Joe Reis - Author of Fundamentals of Data Engineering URL: https://www.dataexpert.io/lesson/techtalksweek1-jan2026-p735-l1934 Module: Tech Talks · Enrolled students Joe Reis, author of "Fundamentals of Data Engineering," shares insights on the evolving landscape of data engineering in this engaging tech talk. He discusses the importance of foundational concepts, the interplay between orchestration and AI, and the necessity of understanding data modeling amidst emerging technologies. This session is valuable for data engineers and aspiring professionals seeking to navigate the complexities of modern data systems and the future of AI integration in data workflows. ### Week 6 lab part 1 URL: https://www.dataexpert.io/lesson/week-6-lab-part-1 Module: Advanced Frontend Development · Enrolled students The lab session focuses on building a Next.js application, emphasizing the use of NPX for creating new apps without installing global modules. Participants will learn about server-side rendering, middleware, and the structure of Next.js projects, including the app and layout folders. The session also covers how to manage authentication with Supabase, utilize environment variables, and implement dynamic and static rendering. This lesson is ideal for developers looking to enhance their frontend skills with Next.js and modern web application practices. ### Book Club 1 URL: https://www.dataexpert.io/lesson/bookclub1-c98da Enrolled students ### Career Dev Session 4 URL: https://www.dataexpert.io/lesson/careerdevsession4-94f20 Enrolled students During this session, Mitali provided guidance on students' resumes and LinkedIn profiles. So, the session was not recorded. ### Prompt Engineering Day1 Lab URL: https://www.dataexpert.io/lesson/prompt-engineering-day1-lab-oct2025 Enrolled students In this video, Zach dives into prompt engineering exercises, focusing on the system prompt and how to enhance chat and resume parsing applications. He emphasizes the importance of setting up environment variables, particularly the OpenAI key, and demonstrates how to interact with the FastAPI backend. He also highlights the significance of the system prompt in shaping AI responses and shares insights on managing response times for API calls. ### Book Club 4 URL: https://www.dataexpert.io/lesson/bookclub4-mar26 Enrolled students The Book Club 4 session focuses on engineering architecture and user feedback, providing insights into designing effective applications. Participants discuss key concepts such as starting simple, adding context with vector databases, implementing guardrails for user interactions, and the importance of user feedback in refining models. The conversation also covers trade-offs between speed, safety, and accuracy, as well as the role of orchestration tools like LangChain in managing complex pipelines. This session benefits data and AI engineers looking to enhance their application design skills. ### Exploring UDFs and SQL Benchmarks in Spark Streaming URL: https://www.dataexpert.io/lesson/exploringudfsandsqlbenchmarksinsparkstreaming-feb26-p736-l2113 Module: Week 4: Structured Streaming with Spark & Kafka · Enrolled students In this video, Zach dives into the differences between UDFs and built-in SQL functions in Spark Streaming, using a benchmark that processes 5 million random numbers. He explores how UDFs can be slower, with results showing about a 10% speed-up when using built-in functions, but this can vary based on caching and the complexity of operations. Zach also experiments with increasing the row count to 100 million and even 1 billion to see how performance changes. He encourages everyone to run similar benchmarks and observe the results for themselves, as there are nuances that can affect performance. ### TA Office hours 7 URL: https://www.dataexpert.io/lesson/taofficehours7-mar26-p737-l2135 Enrolled students This session is a live office hours discussion focused on navigating the Spring AI boot camp program. Participants receive guidance on accessing the overview tab and finding relevant sections within the program. Ideal for learners seeking support with the course structure and tools, this session provides an interactive platform for addressing specific questions and challenges. ### Onboarding Guide for the Data and AI Engineering Bootcamp URL: https://www.dataexpert.io/lesson/onboardingguideforthedataandaiengineeringbootcamp-jan2026 Enrolled students ### Tech Talk with Xinran Waibel URL: https://www.dataexpert.io/lesson/tech-talk1-oct2025 Enrolled students Tech Talk with Xinran Waibel explores the evolving landscape of data engineering and AI. Xinran, a data engineer at Netflix and founder of the Data Engineer Things community, shares insights on the intersection of AI and data engineering, discussing the blurred lines between roles, the impact of AI on workflows, and the importance of foundational skills over specific tools. This session is beneficial for aspiring data engineers and AI professionals looking to understand current industry trends and enhance their skill sets. ### Advanced Snowflake: Data Modeling and UDFs Lecture URL: https://www.dataexpert.io/lesson/advanced-snowflake-data-modeling-and-udfs-lecture Module: Advanced Snowflake · Enrolled students Advanced Snowflake Data Modeling and UDFs covers essential concepts for data engineers looking to deepen their understanding of Snowflake's capabilities. The lesson explores complex data types such as arrays, maps, and the variant type, highlighting their benefits and use cases. Additionally, it delves into User Defined Functions (UDFs), including their types and when to use them, alongside practical examples of integrating external APIs, making it ideal for professionals aiming to optimize their data modeling and processing strategies. ### Injecting quality into your Airflow DAG Lab URL: https://www.dataexpert.io/lesson/injectingqualityintoyourairflowdaglab-jan2026 Enrolled students In this video, Zach walk you through the setup and execution of a simple data pipeline using Airflow, focusing on reading data from Kafka and storing it in a production table. He covers key components like ExecutionTimeout, MaxActiveRuns, and the importance of data quality checks. He demonstrates how to handle missing data and ensure our pipeline is idempotent, meaning it won't create duplicates when rerun. He also highlights the significance of staging tables and the write-audit-publish pattern for maintaining data integrity. ### Write a comprehensive pipeline spec document Lab URL: https://www.dataexpert.io/lesson/pipeline-spec-building-lab-p735-l556 Module: Week 6: Airflow & Pipeline Specs with Trino · Enrolled students In this lab, Zach discusses the importance of data metrics and effective communication strategies. He highlights various data sources, such as banks, rental properties, and Excel sheets, and explain how they can be used to track metrics related to health, career, and recreation. He also emphasizes the significance of considering context and building relationships when communicating with stakeholders. The information shared can help them gain valuable insights and improve their communication skills. ### Delta Table Day 1 Lab URL: https://www.dataexpert.io/lesson/delta-table-day-1-lab-may2025-p736-l1087 Module: Week 3: Data Lakes with Delta Table · Enrolled students In this lab, Yared walks you through the structure of Databricks, focusing on the Unity Catalog and Delta Lake. He explains how to create and manage tables, schemas, and permissions, and he also touches on the differences between managed and external tables. He encourages you to access the lab materials and complete the tasks outlined, including creating Delta tables and performing data transformations. ### Slowly Changing Dimensions Lecture URL: https://www.dataexpert.io/lesson/slowly-changing-dimensions-lecture-p735-l576 Module: Bonus · Enrolled students In this presentation, we delve into the concept of idempotent pipelines, exploring their significance, advantages, and potential drawbacks. We provide insights into Slowly Changing Dimensions (SCDs) and discuss the various types. Discover whether SCDs are inherently idempotent or not and grasp the implications of their usage in data pipelines. Join us to gain a clear understanding of these foundational concepts in data processing. ### Applying Analytical Patterns Day 1 Lecture URL: https://www.dataexpert.io/lesson/applying-analytical-patterns-day-1-lecture-jan2025 Enrolled students Understanding analytical patterns is crucial for data engineers and analysts aiming to optimize data pipelines. This lecture covers three key analytical patterns: aggregation-based analysis, cumulation-based patterns, and window-based analysis. Participants will learn how to apply these patterns to streamline SQL queries, track user behavior over time, and analyze trends effectively, enhancing their ability to design efficient data pipelines and derive actionable insights from data. ### Data Quality Patterns: Spec-Building Document Day 1 Lab URL: https://www.dataexpert.io/lesson/data-quality-patterns-day-1-lab-yt-p736-l699 Enrolled students In this presentation, learn how to make a spec-building document. ### Airflow Data Quality (Day 1 Lecture) URL: https://www.dataexpert.io/lesson/airflow-data-quality-day-1-lecture-v4 Module: Week 3: Airflow and Airflow Data Quality · Enrolled students In this lecture, Zach explains the concept of cumulative DAGs in Airflow and their unique architecture. He also discusses the importance of using partition sensors and setting concurrency limits on DAGs. Additionally, he shares best practices for backfilling data and highlight the potential challenges and resource-intensive nature of backfilling. This lecture explains how to handle both small and large datasets in your pipeline and airflow. [Recorded on May22nd, 2024] ### Local Video URL: https://www.dataexpert.io/lesson/localvideo-963b1 Enrolled students ### Core Data Modeling Elements Saturday Quiz URL: https://www.dataexpert.io/lesson/core-data-modeling-quiz Enrolled students ### Chat with Zach URL: https://www.dataexpert.io/lesson/v4-5-chat-with-zach-686 Enrolled students Exclusive Q&A for homework doers. [Recorded on June 6th, 2024] ### Intro to HTML & CSS URL: https://www.dataexpert.io/lesson/intro-to-html-and-css-94d04 Module: Web Dev Fundamentals · Free Overview of HTML, the various components, markup language, how to run it in the browser (include the inspect/console tool in chrome). Also understand CSS basics, including styling HTML elements using classes, ids, attributes, and other selectors. ### Advanced SQL Patterns Day 2 Lecture URL: https://www.dataexpert.io/lesson/advanced-sql-patterns-day-2-lecture-jan2025-p736-l818 Enrolled students Advanced SQL techniques are essential for data engineers looking to excel in their roles and interviews. This lecture covers critical topics such as window functions, recursive CTEs, and advanced aggregation methods like grouping sets, roll-up, and cube. Participants will learn how to differentiate between SQL interview expectations and real-world applications, ensuring they can write efficient queries and improve data modeling practices for better performance in analytics. ### Youtube URL: https://www.dataexpert.io/lesson/youtube-2165 Enrolled students testing updating ### V4 Boot Camp Kick-Off URL: https://www.dataexpert.io/lesson/boot-camp-kick-off-v4 Module: Boot Camp Orientation · Enrolled students In this video, Zach will be sharing some exciting updates and important information about the V4 Bootcamp. He will be discussing topics like the capstone project, policies, refund policies, and the LinkedIn masterclass. He will also introduce some of the amazing mentors and TA's who will be guiding you throughout the bootcamp. So get ready for an immersive learning experience and let's dive in together! [Recorded on May6th, 2024] ### Advanced dbt: Production Pipelines and dbt with Airflow Lab URL: https://www.dataexpert.io/lesson/advanceddbtday2lab-jan2026-p735-l1924 Module: Week 2: Advanced dbt · Enrolled students The lesson covers advanced techniques in dbt, focusing on implementing the right audit-publish pattern and integrating dbt with Airflow. Participants will learn how to create an audit table for critical data, ensuring data quality through testing before publishing to production tables. The session also demonstrates how to configure Airflow to manage dbt tasks, including setting environment variables and using Docker for deployment, making it ideal for data engineers looking to enhance their workflow automation and data integrity practices. ### Managing Unstructured Data - Day 2 Lecture URL: https://www.dataexpert.io/lesson/managing-unstructured-data-day-2-lecture-may2025-p736-l1198 Module: Week 5: Managing Unstructured Data · Enrolled students In this lecture, Zach focuses on retrieval augmented generation (RAG) and large language models (LLMs). He emphasises the importance of identifying business problems that are well-suited for AI solutions. Zach outlined key topics for the day, including ensuring that LLMs return valid JSON and tuning RAG to achieve more relevant results. ### Q&A with Zach Week 3 URL: https://www.dataexpert.io/lesson/qa-with-zach-week3-oct2025 Enrolled students ### Loom URL: https://www.dataexpert.io/lesson/loom-8070 Enrolled students ### Q&A with Zach Week1 URL: https://www.dataexpert.io/lesson/qawithzachweek1-feb26-p736-l2037 Module: Q&A with Zach · Enrolled students The session features an interactive Q&A with Zach, focusing on setup instructions for the lab environment, including GitHub integration and DataBricks workspace navigation. Participants can gain insights into best practices for managing data engineering workflows and the importance of Git in collaborative projects. The discussion also touches on various topics, such as Delta tables, local execution of code, and the evolving landscape of data engineering roles in light of AI advancements. ### Apache Spark Unit Testing Day 3 Lab URL: https://www.dataexpert.io/lesson/apache-spark-unit-testing-day-3-lab-jan2025 Enrolled students In this lab, Zach walks you through the process of creating unit tests in PySpark, including how to handle bad data and ensure your tests pass. He demonstrates the importance of testing logic without impacting datasets and share tips on using named tuples and data frames effectively. He also discusses the significance of CI/CD practices and how to properly deploy changes. ### MLOPs and Guardrails Day 1 Lab URL: https://www.dataexpert.io/lesson/ml-ops-day1-lab-oct25 Enrolled students In this video, Zach walks through the setup of guardrails for his MLOPS project, focusing on the database agent. He explores how to manage toxic inputs and ensure that the AI remains respectful during user interactions. Zach demonstrates the middleware functionalities, including toxicity detection and on-topic checks, while emphasizing the importance of keeping the agent aligned with relevant data tables. He also highlights the need for the middleware to adapt dynamically as new tables are created in the database. ### Apache Spark Shuffle Joins Day 1 Lab URL: https://www.dataexpert.io/lesson/apache-spark-shuffle-joins-day-1-lab-jan2025-p736-l789 Module: Week 2: Databricks & Advanced Spark · Enrolled students In this lab, Zach walks through the process of joining two datasets related to NBA games and players, focusing on the performance implications of different join strategies. He demonstrates how to disable broadcast joins and analyze the resulting execution plans to better understand the underlying processes. He also highlights the importance of using formatted modes for clarity in job execution. ### Lesson 4 URL: https://www.dataexpert.io/lesson/lesson-4-8341 Enrolled students ### TA Office Hour 4 (2025-10-09) URL: https://www.dataexpert.io/lesson/ta-office-hour4-oct2025-p737-l1539 Enrolled students ### LLMs - Using LLMs to provide business value Day 1 Lab URL: https://www.dataexpert.io/lesson/machine-learning-day-1-lab-v4 Module: Week 6: Machine Learning · Free In this lab, Zach walks you through a Python script he created for automated code review. He demonstrates how the script generates comments on pull requests, the setup process involving GitHub tokens, and the prompt engineering for feedback. Recorded on June 20th, 2024. ### Tech Talk 8 - Prasad Rao URL: https://www.dataexpert.io/lesson/tech-talk-8-prasad-rao-april2025 Enrolled students Prasad Rao, a principal solutions architect at AWS, shares insights from his journey in tech, emphasizing the importance of communication and influence in career advancement. He discusses strategies for transitioning from senior to principal roles, including the significance of articulating experiences, understanding organizational dynamics, and creating scalable solutions. This session is beneficial for engineers aiming to navigate their career paths effectively and enhance their impact within their organizations. ### Infrastructure Data Quality - Implementing Data Quality Measures with Astronomer and Airflow: Hands-On Lab with Marc Lamberti (Day 2 Lab) URL: https://www.dataexpert.io/lesson/infra-data-quality-day-2-lab-v3 Module: Week 5: Unit Testing Pipelines & Write-audit-publish · Enrolled students Embark on a hands-on learning journey with Marc Lamberti in this lab session focused on Astronomer and Airflow. Marc guides you through practical exercises, demonstrating the implementation of data quality measures within infrastructure using Astronomer and Airflow. From setting up workflows to optimizing processes, this lab equips you with the skills needed to streamline data pipelines and ensure data quality in an infrastructure context.[Recorded on Dec 14th, 2023] ### Chat with Zach URL: https://www.dataexpert.io/lesson/v4-4-chat-with-zach-5051 Enrolled students In this exclusive Q&A with Zach, he discusses various topics related to data engineering, including partitioning, data dependencies, and the use of Airflow and serverless functions. He shares his experiences working on a project before starting a bootcamp and provide insights into forming an LLC or an S Corp. The Q&A aims to provide valuable information and prompt viewers to consider different approaches to data engineering. [Recorded on May30th, 2024] ### Local Video URL: https://www.dataexpert.io/lesson/local-video-1727 Enrolled students ### Analytics Data Quality - Exploring Data Modeling and Quality Checks (Day 1 Lab) URL: https://www.dataexpert.io/lesson/analytics-data-quality-day-1-lab-v3 Module: Week 5: DBT and Pipeline Spec Building · Enrolled students Zach discusses the intricacies of data modeling and the crucial role of data quality checks in this informational lab session. He expounds on the significance of video intervals in relation to log tables, offering insightful examples of various log types and their relevance. Zach also addresses the dimensions and metrics inherent in data modeling, emphasizing the importance of implementing thorough data quality checks. [Recorded on Dec 11, 2023] ### Analytics Engineering Bootcamp Kickoff URL: https://www.dataexpert.io/lesson/analytics-engineering-bootcamp-kickoff Module: Bootcamp Orientation · Enrolled students The Analytics Engineering Bootcamp Kickoff introduces participants to the exciting world of analytics engineering, focusing on key tools like Snowflake, DBT, and Airflow. Attendees will learn about the bootcamp structure, including live lectures, Q&A sessions, and a capstone project that emphasizes real-world applications. This session sets the stage for a comprehensive six-week program designed for both beginners and those looking to enhance their skills in analytics engineering. ### MCP and Agentic AI Day 1 Lecture URL: https://www.dataexpert.io/lesson/mcpandagenticaiday1lecture-oct2025-p737-l1606 Module: Week 3: Agentic AI, MCP & Automation · Enrolled students In this video, Zach dives into the Model Context Protocol (MCP) and its tools, emphasizing how they can transform AI from merely providing advice to taking actionable steps. He highlights that only about 20% of internet users engage with ChatGPT, and a mere 1-2% have used Python to prompt it, showcasing how early we are in this AI trend. Zach will explore how MCP servers can streamline tool management and enhance AI capabilities, guiding viewers through the process of setting up their own MCP server by the end of the week. ### Book Club 4 URL: https://www.dataexpert.io/lesson/book-club-4-oct2025 Enrolled students ### TA Office Hour 7 URL: https://www.dataexpert.io/lesson/ta-office-hour-7-may2025 Enrolled students This office hours session focuses on deploying ETL pipelines using Databricks and the Medallion architecture. Participants discuss creating separate notebooks for each layer—bronze, silver, and gold—and the importance of properly structuring their scripts. The session also covers troubleshooting deployment issues, creating dashboards, and best practices for submitting assignments, making it beneficial for learners looking to enhance their practical skills in data engineering. ### Orchestration and Airflow Fundamentals Lecture URL: https://www.dataexpert.io/lesson/orchestration-and-airflow-fundamentals-lecture Module: Pipeline Spec Building + Airflow Fundamentals · Enrolled students Welcome to the very first lecture of the Analytics Engineering Bootcamp where we delve into the concept of orchestration, automations that drive business success through data. Throughout the lecture, we differentiate between orchestration and tools like Airflow, emphasizing the importance of data quality and efficient data processing techniques. ### dbt basics: creating a full dbt project lecture URL: https://www.dataexpert.io/lesson/dbt-basics-creating-a-full-dbt-project-lecture Module: dbt basics · Enrolled students In this lecture, Bruno delves into the world of DBT testing and snapshots, focusing on how DBT handles transformation tasks. He explains the significance of DBT as a tool for SQL-savvy individuals, particularly in the context of Snowflake warehouses. Throughout the lecture, he discusses the creation of materialized views, testing methodologies, and the importance of unit tests versus data tests. By the end, he touches upon the concept of snapshots and their role in maintaining historical data integrity within a database. ### Q&A with Zach 3 URL: https://www.dataexpert.io/lesson/qawithzach3-mar26 Enrolled students Week 3 features a Q&A session focused on troubleshooting and optimizing AI agent performance, particularly regarding token usage and proxy server limitations. Participants discuss the implications of token burning as a metric for AI impact, the challenges of using proxy servers like Heroku, and strategies for improving agent responses through logging and tracing. This session is beneficial for students looking to deepen their understanding of AI agent architecture and performance optimization techniques. ### Chat with Zach [Aug 10, 2023] URL: https://www.dataexpert.io/lesson/week-4-zach-qa-v2 Enrolled students Delve into the world of data engineering with our weekly Q&A / Fireside Chat series, featuring the founder of EcZachly Inc and lead instructor of the Advanced Data Engineering Bootcamp. Each session covers a different topic, providing invaluable insights and real-world experiences to empower data enthusiasts in their journey. (Recorded Aug 10, 2023) ### TA Office Hour 3 URL: https://www.dataexpert.io/lesson/ta-office-hour-3-may2025 Enrolled students Office hours provide an interactive platform for students to discuss their assignments and clarify doubts. Key topics covered include building a simple data warehouse using a star schema, generating synthetic data with Python's Faker library, and utilizing Databricks for data loading and optimization techniques. Students can also seek guidance on best practices for structuring their projects and addressing specific challenges related to their assignments. ### OpenClaw Safeguards & Measuring business value for AI Agents Day1 Lab URL: https://www.dataexpert.io/lesson/openclawsafeguardsmeasuringbusinessvalueforaiagentsday1lab-mar26 Enrolled students Explore the OpenClaw safeguards and learn how to measure the business value of AI agents in this hands-on lab session. Participants will engage in activities like prompt injection testing, tool scoping, and implementing least privilege principles to enhance security. By utilizing OpenClaw's capabilities, attendees will gain practical insights into configuring AI systems for safety and efficiency, making this lesson ideal for AI engineers and data professionals looking to strengthen their AI implementations. ### advanced dbt Day 2 Lab URL: https://www.dataexpert.io/lesson/advanceddbtday2lab-jan2026 Enrolled students The advanced dbt Day 2 Lab focuses on implementing the right audit publish pattern in dbt, particularly for critical tables like fact orders. Participants will learn how to create an audit table for testing data integrity before publishing to production, utilizing dbt with Airflow for orchestration. The session covers essential techniques, including incremental models and environment variable management, ensuring that data quality is maintained throughout the ETL process. ### Spark Structured Streaming Day 2 Lecture URL: https://www.dataexpert.io/lesson/spark-structured-streaming-day-2-lecture-jan2025 Enrolled students Explore the fundamentals of Spark Structured Streaming, focusing on its advantages, use cases, and core concepts such as sources, sinks, and streaming aggregates. This lecture is ideal for data engineers and professionals interested in real-time data processing, covering practical examples and scenarios like fraud detection, cybersecurity, and real-time decision-making. Gain insights into configuring micro-batch processing, handling duplicates, and utilizing various streaming sources like Kafka and Azure Event Hub. ### advanced dbt Day 1 Lab URL: https://www.dataexpert.io/lesson/advanceddbtday1lab-e01c7 Enrolled students ### Q&A with Zach 1 URL: https://www.dataexpert.io/lesson/qawithzach1-mar26-p737-l2119 Module: Q&A with Zach · Enrolled students The Q&A session with Zach covers a range of topics relevant to participants in the AI engineering boot camp. Key discussions include troubleshooting issues with proxies and API setups, particularly with Anthropic and OpenAI, as well as guidance on setting up the environment for building AI agents. Participants can expect insights on creating agents using LangChain, addressing common concerns about homework and project ideas, and exploring best practices for AI implementation and security. ### MCP and Agentic AI Day 2 Lab URL: https://www.dataexpert.io/lesson/mcp-and-agentic-ai-day2lab-oct2025-p737-l1609 Module: Week 3: Agentic AI, MCP & Automation · Enrolled students In this lab, Zach dives into middleware and MCP servers, focusing on logging and adding new features. He introduces new middleware files for tracing, PII detection, and cost analytics, emphasizing the importance of creating an audit trail for their runs. He explores how to generate unique run IDs and improves their tracing middleware to better log tool calls and errors. He demonstrates how to implement a new function for querying RAG content, which successfully retrieves resumes from their system. ### Special Guests, Parth Parekh and Nick Singh [August 16, 2023] URL: https://www.dataexpert.io/lesson/v2-special-guests-parth-parekh-and-nick-singh-august-16-2023-4496 Enrolled students Join us each week as we sit down with visionary speakers from diverse fields, engaging in enlightening conversations that explore cutting-edge insights, trends, and their wealth of expertise. (Recorded August 16, 2023) ### Friday Quiz URL: https://www.dataexpert.io/lesson/fridayquiz-efa43 Enrolled students ### dbt Basics Day 3 Lab URL: https://www.dataexpert.io/lesson/dbt-basics-day-3-lab-jan2025 Enrolled students In this lab, Bruno walks you through the second day of our DBT Basics series, focusing on adding tests and snapshots to our DBT project. He covers various types of tests, including data tests and unit tests, and demonstrate how to implement them effectively. Additionally, he introduces the concept of snapshots and how to set them up in your project. ### Connecting the database to the service URL: https://www.dataexpert.io/lesson/9108e798-515c-438e-9344-1610f25ff638connectingthedatabasetotheservice Module: Intermediate Backend Development · Enrolled students Connecting a database to a service is crucial for backend development, and this lesson focuses on implementing input validation and database integration using tools like Express and Zod. Participants will learn how to validate incoming data, manage middleware for input checks, and utilize libraries for database connections and migrations. This lesson is ideal for intermediate developers looking to enhance their backend skills and ensure data integrity in their applications. ### QnA with Zach 4 URL: https://www.dataexpert.io/lesson/qnawithzach4-aug26 Enrolled students Mastering the behavioral interview is crucial for advancing your career in data engineering. This session covers effective strategies to excel in this interview round, emphasizing the importance of storytelling using the STAR method, showcasing past projects, and understanding company culture. Participants will learn how to present their experiences positively and authentically, which can significantly impact their chances of landing senior or staff positions in tech companies. ### Building AI Agents with Databricks Day1 Lab URL: https://www.dataexpert.io/lesson/buildingaiagentswithdatabricksday1lab-feb26-p736-l2034 Module: Week 6: Building AI Agents with Databricks · Enrolled students In this video, Eumar walks you through the new repository he created for week 2, which includes five labs focusing on deploying a custom MCP app using Databricks asset bundles. He explains the importance of role-level security and how to ensure users can only access their own data. He also dives into the process of bringing GitHub data into the vector database and discusses chunking strategies for optimal data retrieval. ### TA Office Hour 1 (2025-06-04) URL: https://www.dataexpert.io/lesson/ta-office-hour-1-may2025-p736-l1085 Enrolled students A TA office hour session focused on clarifying the first assignment for a boot camp involving the creation of a data pipeline using Airflow and Iceberg. Participants discussed the steps required to build the pipeline, including setting up DAGs, managing environment variables, and implementing data quality checks. Key topics included handling API data fetching, creating audit branches, and ensuring proper data accumulation over specified periods, with participants encouraged to refer back to lab examples for guidance. ### Communicating Ideas and Getting Alignment URL: https://www.dataexpert.io/lesson/communicating-ideas-and-getting-alignment Module: Soft Skills · Enrolled students ### Q&A with Zach Week 5 (2025-10-09) URL: https://www.dataexpert.io/lesson/qa-with-zach-week5-oct2025-p737-l1548 Module: Q&A with Zach · Enrolled students ### Day 1 Lab: Auto Prompt Optimization with dspy URL: https://www.dataexpert.io/lesson/prompt-engineering-day1-lab-jul2025 Enrolled students In this lab, Zach dives into analyzing code in a GitHub repository and discussed auto prompt optimization techniques, specifically focusing on two examples: the auto optimize candidates pool and counting.py. He guides through setting up their Python environment, generating a GitHub personal access token, and running the code to fetch repository contents. Zach emphasized the importance of structured data in prompt engineering and how to create effective prompts to ensure consistent JSON output. ### Apache Flink Streaming - Exploring Data Collection and Processing with Kafka in Flink (Day 2 Lecture) URL: https://www.dataexpert.io/lesson/infra-flink-streaming-day-2-lecture-v3 Module: Week 3: Streaming Pipelines with Apache Flink · Enrolled students In this presentation, the focus is on elucidating how data is collected and processed utilizing Kafka. The step-by-step walkthrough covers the process of making a request and the seamless passage of data to a Kafka producer. Attention is given to a common challenge: data flowing into Kafka during development, including insights into a encountered bug and its resolution. Furthermore, we touch upon the advantages of leveraging the Kappa architecture and explore the functionality of emerging metastores such as Iceberg.[Recorded Nov 30, 2023] ### TA Office Hour 5 URL: https://www.dataexpert.io/lesson/ta-office-hour5-oct2025 Enrolled students ### Building functional AI agents using LangChain Day1 Lab URL: https://www.dataexpert.io/lesson/buildingfunctionalaiagentsusinglangchainday1lab-mar26-p737-l2138 Module: Week 1: Prompt Engineering & Basic RAG · Enrolled students In this lab, Zach walks you through the LangChain starter repo, highlighting key components such as the requirements for libraries, setting up the environment, and the different types of agents you'll be working with, including memory agents and tool agents. Zach emphasises the importance of specificity in prompts to reduce token usage. He also discusses how memory functions in agents and how to store and retrieve user information effectively. ### Special Guests, Stephanie Nuesi and Joe Reis [July 26, 2023] URL: https://www.dataexpert.io/lesson/v2-special-guests-stephanie-nuesi-and-joe-reis-july-26-2023-6878 Enrolled students Join us each week as we sit down with visionary speakers from diverse fields, engaging in enlightening conversations that explore cutting-edge insights, trends, and their wealth of expertise. (Recorded July 26, 2023) ### Career Dev Session 5 URL: https://www.dataexpert.io/lesson/careerdevsession5-f1228 Enrolled students During this session, Mitali provided guidance on students' resumes and LinkedIn profiles. So, the session was not recorded. ### Tuesday Quiz URL: https://www.dataexpert.io/lesson/tuesdayquiz-a58fc Enrolled students ### TA Office hour 1 URL: https://www.dataexpert.io/lesson/taofficehour1-jan2026-p735-l1939 Enrolled students This session serves as an office hours Q&A for students in the boot camp, addressing various topics related to assignments, project guidelines, and technical issues. Participants discuss challenges with schema permissions in Trino, deadlines for capstone project proposals, and homework submission processes. The session emphasizes the importance of timely homework submissions for participation in live Q&A sessions and provides guidance on using tools like Python and Airflow effectively. ### Special Guest, Alex Freberg [August 2, 2023] URL: https://www.dataexpert.io/lesson/week-3-speaker-v2 Enrolled students Join us each week as we sit down with visionary speakers from diverse fields, engaging in enlightening conversations that explore cutting-edge insights, trends, and their wealth of expertise. (Recorded August 2, 2023) ### Advanced dbt: Understanding incremental models Lecture URL: https://www.dataexpert.io/lesson/advanced-dbt-understanding-incremental-models-lecture-p735-l568 Module: Week 2: Advanced dbt · Enrolled students In this lecture, Bruno delves into the intricacies of incremental models in dbt. He explains how to define and implement incremental strategies, showcasing the use of unique keys and filters for efficient data processing. Additionally, he touches upon the significance of materialization types and the impact on data modeling. The explanation of the is_incremental() function got a bit messed up. Go to the hands-on lab to watch a better explanation of it. ### Test Lesson URL: https://www.dataexpert.io/lesson/a4859c01-6e2f-4d69-b0ed-b86960deaf7ftestlesson Enrolled students ### The Data Architecture Interview URL: https://www.dataexpert.io/lesson/the-data-architecture-interview Module: Interview Skills · Free This video covers the main things you need to remember to pass the Data Architecture interview ### LLMs - Vector database for RAG applications Day 2 Lab URL: https://www.dataexpert.io/lesson/machine-learning-day-2-lab-v4 Module: Week 6: Machine Learning · Free This is new content and will be recorded on June11th 2024. ### Q&A with Zach 5 URL: https://www.dataexpert.io/lesson/qawithzach5-mar26-p737-l2123 Module: Q&A with Zach · Enrolled students A live Q&A session with Zach wraps up the AI engineering boot camp, featuring discussions on personal success stories, course feedback, and insights into the learning journey. Participants share their experiences, including job placements and project implementations, while Zach addresses questions about AI applications, project management, and the future of the boot camp. This session is ideal for students seeking guidance and community support as they transition into AI engineering roles. ### TA Office Hour 1 URL: https://www.dataexpert.io/lesson/ta-office-hour-1-april2025 Enrolled students The TA Office Hour session provides an interactive platform for students to clarify doubts regarding their first assignment in the data engineering boot camp. Key topics include fetching data from the Polygon API, using PyIceberg and PySpark for data processing, and creating partitioned tables. Students can also ask questions about environment setup, handling API responses, and best practices for data insertion and management. This session is beneficial for students seeking guidance on practical implementation and troubleshooting in their assignments. ### OpenClaw Safeguards & Measuring business value for AI Agents Day1 Lecture URL: https://www.dataexpert.io/lesson/openclawsafeguardsmeasuringbusinessvalueforaiagentsday1lecture-mar26 Enrolled students Understanding the complexities of AI agents is crucial for anyone working with OpenClaw. This lecture covers essential topics such as the importance of reliability, types of failures, and effective guardrails to mitigate risks associated with agentic AI. Participants will learn about failure modes like hallucinations, prompt injections, and scope creep, along with practical strategies for measuring business value and ensuring safe operations when integrating AI agents into workflows. ### Brian Pulliam - Career Developement Coach (2.0) URL: https://www.dataexpert.io/lesson/brian-pulliam-jan2025-p736-l783 Module: Guest Speakers · Enrolled students Brian Pulliam shares insights on career development, focusing on job searching, interviewing, and negotiation strategies for tech professionals. His experience spans over 26 years in various tech roles, and he emphasizes the importance of storytelling in interviews, understanding industry dynamics, and preparing for job transitions. This session is particularly beneficial for mid-level data engineers, analysts transitioning to data engineering, and software engineers seeking new opportunities. ### advanced dbt Day 2 Lecture URL: https://www.dataexpert.io/lesson/advanceddbtday2lecture-jan2026 Enrolled students Advanced DBT techniques are explored, focusing on transforming raw data into production-ready datasets using orchestration tools like Airflow. Participants will learn about production pipelines, including the right audit publish pattern, efficient task management, and the integration of CI/CD practices. This lesson is ideal for data engineers and analysts looking to enhance their DBT skills and streamline their data transformation workflows. ### Conceptual Data Modeling (Day 3 Lecture) URL: https://www.dataexpert.io/lesson/dimensional-data-modeling-day-3-lecture-v4 Module: Week 1: Dimensional Data Modeling in Iceberg · Enrolled students Zach discusses the importance of data modeling and metrics in building a successful pipeline. He emphasizes that conceptual data modeling is accessible to everyone and highlight the need to consider the right people for feedback and collaboration. He also touches on the challenges of understanding stakeholders' true requirements and share his experience with choosing the right database solution. Throughout the video, he emphasizes the importance of metrics-driven approaches and the potential impact of well-designed dashboards. Overall, this lecture provides valuable insights and guidance for data engineers looking to optimize their pipeline. [Recorded May 9, 2024] ### test 5 URL: https://www.dataexpert.io/lesson/test5-ac697 Enrolled students ### Career Dev Session 3 URL: https://www.dataexpert.io/lesson/career-dev-session3-oct2025-p737-l1530 Module: Career Development · Enrolled students ### Q&A with Zach Week 2 URL: https://www.dataexpert.io/lesson/qa-with-zach-week2-july2025 Enrolled students A live Q&A session with Zach covers various topics related to the boot camp, including capstone project requirements, technical challenges, and project ideas. Participants discuss integrating LLMs and RAG into their projects, troubleshooting technical issues, and exploring the evolving roles in data and AI engineering. Zach emphasizes the importance of business value in projects and encourages students to leverage their unique skills and experiences to stand out in the job market. ### Real Time data with Delta Live Table Day1 Lab URL: https://www.dataexpert.io/lesson/realtimedatawithdeltalivetableday1lab-aug26 Enrolled students In this lab, Zach walks through a Spark Streaming lab for processing Data Expert events into Delta tables, emphasizing how streaming data is logged and how streaming jobs are configured and run. He first shows the Event logging pipeline, where a WebEvents REST endpoint creates an event schema and publishes data to both Kafka and Postgres, and then demonstrates a Kafka producer and Spark Streaming job consuming from a Kafka topic into a “bronze” Delta table with checkpointing and Kafka SSL settings.He also covers how continuous processing works using await termination plus trigger intervals, why watermarking is required for streaming aggregations, and how windowed aggregations with watermark and window duration produce sliding windows with overlap. ### OpenClaw Safeguards & Measuring business value for AI Agents Day2 Lecture URL: https://www.dataexpert.io/lesson/openclawsafeguardsmeasuringbusinessvalueforaiagentsday2lecture-mar26 Enrolled students Understanding what tasks are worth automating is crucial for maximizing efficiency in any workflow. This lecture covers frameworks to evaluate automation opportunities, including the intern test and identifying hidden costs. Participants will engage in a lab session to audit their tasks and collaboratively identify and implement useful automations tailored to their specific needs, ensuring they focus on high-frequency, low-load tasks while avoiding common pitfalls. ### You must learn Databricks in 2026 - The DataExpert.io Roadmap Lab URL: https://www.dataexpert.io/lesson/youmustlearndatabricksin2026thedataexpertioroadmap-lab Enrolled students The Delta Table Lab focuses on using Databricks to manage and manipulate Delta tables effectively. Participants will learn how to create a Git folder, link API keys, and handle stock market data ingestion, including data cleansing and optimization techniques. This lesson is ideal for data engineers and analysts looking to enhance their skills in data processing and management within the Databricks environment. ### Analytical Patterns Recognizing Business Value Day 3 Lecture URL: https://www.dataexpert.io/lesson/analytical-patterns-recognizing-business-value-day-3-lecture-jan2025-p736-l820 Enrolled students Recognizing business value in data engineering is crucial for driving impactful decision-making. This lecture covers analytical patterns, machine learning, and the importance of dashboards and master data. Participants will learn how to assess the business value of data pipelines, prioritize projects effectively, and understand the role of analytics engineers in experimentation and machine learning, ultimately enhancing their ability to contribute to high-value data initiatives. ### How to use Snowpark and Snowflake Optimizations Lab Day 2 URL: https://www.dataexpert.io/lesson/how-to use-snowpark-and-snowflake-optimizations-lab-day-2-april2025 Enrolled students In this lab, Zach walks through optimizing the compute of our Snowflake warehouse, focusing on larger ETL loads and the integration with Iceberg data. He also discusses the importance of clustering and how to manage tables effectively. ### Airflow Data Quality (Day 2 Lecture) URL: https://www.dataexpert.io/lesson/airflow-data-quality-day-2-lecture-v4 Module: Week 3: Airflow and Airflow Data Quality · Enrolled students In this lecture, Zach discusses the importance of data quality checks and usability in data pipelines. He explains how data practitioners can ensure that their data is reliable, usable, and of high quality. He also shares insights from his experience working at Facebook and highlight the need for effective data marketing.[Recorded on May 23rd, 2024] ### Dimensional Data Modeling - Graph Data Modeling with Iceberg (Day 3 Lab) URL: https://www.dataexpert.io/lesson/dimensional-data-modeling-day-3-lab-v3 Module: Week 1: Dimensional Data Modeling in Iceberg · Enrolled students In this lab, we delve into the NBA game dataset, specifically the nba_player_seasons table. Learn the nuances of using team ID and team abbreviation in the graph, and follow along as we demonstrate the insertion of data into the vertices. Uncover the concept of bi-directional edges, supported by practical examples of data filtering and joining. Throughout, we share optimization tips, highlighting the advantages of leveraging graph traversal for complex data analysis. These insights prove valuable for data analysts and developers engaging with graph databases.[Recorded Nov 9, 2023] ### Live SQL Training January 25th, Window Functions and Common Table Expressions URL: https://www.dataexpert.io/lesson/live-sql-training-jan-25-window-functions-common-table-expressions Module: Interview Skills · Enrolled students In this video we cover window functions LAG and LEAD. We also cover common table expressions in depth comparing them to temp tables and subqueries! ### How to deploy AI Agents using Claude managed Agents Lab URL: https://www.dataexpert.io/lesson/endtoendaiapplicationsday2lab-mar26-p737-l2156 Module: Week 5: End-to-End AI Applications · Enrolled students Deploying AI agents using Claude managed Agents Lab equips learners with practical skills in creating and managing AI agents in a cloud environment. The lesson covers essential topics such as setting up the environment, utilizing the Anthropics API key, and implementing tool calls for various tasks. Participants will benefit from hands-on demonstrations that illustrate how to create agents, manage sessions, and integrate custom tools, making it ideal for those looking to enhance their AI application deployment capabilities. ### Backfilling with Airflow URL: https://www.dataexpert.io/lesson/backfillingwithairflow-3483a-p735-l2003 Module: Week 6: Airflow & Pipeline Specs with Trino · Enrolled students Backfilling with Airflow focuses on optimizing data pipeline performance while using Airflow and AWS Glue. The lesson covers best practices for managing memory usage during backfills, highlighting the inefficiencies of Python in handling large datasets. Participants will learn how to effectively orchestrate ETL processes, ensuring that heavy lifting is delegated to Glue, and gain insights into managing data loads and table swaps for efficient data handling. ### MLOPs and Guardrails Day 2 Lecture URL: https://www.dataexpert.io/lesson/ml-ops-day2-lecture-oct25 Enrolled students In this lecture, Zach dives into the critical topics of evaluations and reliability in AI, emphasizing that AI agents are not magical black boxes but statistical models that need careful management. He highlights how shifts in user behavior, code updates, and data changes can break AI systems and shares strategies for monitoring these issues effectively. Zach walks through various evaluation methods such as ROUGE and cosine distance and shows how they can be used to optimize prompts with DSPY. ### Cumulative DAGs in Production Day 3 Lab URL: https://www.dataexpert.io/lesson/cumulative-dags-in-production-day-3-lab-jan2025 Enrolled students Cumulative table design and airflow concurrent writes are explored in this lab session, focusing on the creation and management of production and staging tables within Airflow. Participants will learn how to handle data from Kafka, implement data quality checks, and address race conditions in concurrent writes using Trino and Iceberg. This lesson is ideal for data engineers looking to enhance their skills in building robust data pipelines and managing data integrity in production environments. ### Chip Huyen - Author of AI Engineering URL: https://www.dataexpert.io/lesson/tech-talk-chip-july25-p737-l1379 Module: Tech Talks · Enrolled students Chip Huyen, a best-selling author and AI expert, shares insights from her extensive experience in AI engineering during this engaging tech talk. She discusses the evolution of AI, the importance of user experience in AI applications, and the challenges of integrating generative AI into existing systems. This session is valuable for engineers and data scientists looking to understand the practical implications of AI technologies and how to navigate the rapidly changing landscape of AI development. ### Apache Spark Unit Testing Day 3 Lecture URL: https://www.dataexpert.io/lesson/apache-spark-unit-testing-day-3-lecture-may2025-p736-l1147 Module: Week 2: Databricks & Advanced Spark · Enrolled students In this lecture, Zach discusses the critical aspects of data engineering, focusing on the importance of maintaining data quality and minimizing customer pain. He highlights the challenges we face in balancing speed and quality, especially when dealing with machine learning models. He also emphasizes the need for best practices and frameworks to ensure efficient data pipelines. ### Apache Flink Streaming - Mastering Streaming and Real-Time Data Processing with Apache Flink (Day 1 Lecture) URL: https://www.dataexpert.io/lesson/infra-flink-streaming-day-1-lecture-v3 Module: Week 3: Streaming Pipelines with Apache Flink · Enrolled students This comprehensive video presentation explores the concepts of streaming and real-time data processing, distinguishing between low latency and high latency and elucidating their impact on data processing. The discussion encompasses the differences between real-time and near real-time processing, emphasizing the necessity of clear communication with stakeholders. Challenges and considerations in working with streaming data, such as out-of-order events and late-arriving data, are addressed. Additionally, the importance of fault tolerance and its relation to data recovery is touched upon. The video serves as a valuable resource for data engineers and anyone interested in gaining insights into the intricacies of streaming and real-time data processing. [Recorded on Nov 28th] ### TA Office hours 2 URL: https://www.dataexpert.io/lesson/taofficehours2-mar26 Enrolled students A live Q&A session was conducted to address various technical issues related to cloud services, API connections, and homework assignments. Participants discussed troubleshooting strategies for connecting to cloud environments, managing API keys, and resolving errors in Python scripts. The session also covered best practices for storing session data and handling assignments, providing valuable insights for students working on their projects and exercises. ### Orchestration and Airflow Fundamentals Lab URL: https://www.dataexpert.io/lesson/orchestration-and-airflow-fundamentals-lab-p735-l554 Module: Week 6: Airflow & Pipeline Specs with Trino · Enrolled students In this lab, Zach walks you through using Astro for deployment, focusing on creating and running data pipelines with airflow. He demonstrate steps like creating new DAGs, running Trino data quality checks, and shifting to the next pipeline. ### Book Club 5 URL: https://www.dataexpert.io/lesson/bookclub5-12234 Enrolled students ### How to deploy AI Agents using Claude managed Agents Lecture URL: https://www.dataexpert.io/lesson/endtoendaiapplicationsday2lecture-mar26-p737-l2155 Module: Week 5: End-to-End AI Applications · Enrolled students Learn how to deploy AI agents using Claude managed agents in this comprehensive lecture on end-to-end AI applications. This session covers the importance of human-in-the-loop strategies, the balance between human and AI decision-making, and the technical aspects of cloud-managed agents, including their architecture and functionality. Ideal for data engineers and AI practitioners, this lesson emphasizes best practices for building robust AI applications while avoiding common pitfalls in AI project development. ### Databricks and Advanced Spark Day2 Lab URL: https://www.dataexpert.io/lesson/databricksandadvancedsparkday2lab-feb26-p736-l2028 Module: Week 2: Databricks & Advanced Spark · Enrolled students In this Lab, Zach dives into performance tuning and incremental data processing using a substantial dataset of around 50 gigabytes of tweets. He demonstrates the importance of partitioning. He explores the impact of repartitioning strategies on performance. He encourages everyone to experiment with their own partitioning strategies and to utilize the Spark UI for performance analysis. ### Snowflake and Glue Catalog Integration Lab URL: https://www.dataexpert.io/lesson/snowflakeandgluecatalogintegrationlab-jan2026 Enrolled students Learn how to integrate Snowflake with AWS Glue Catalog in this lab, focusing on creating a seamless connection for data management. You'll explore the steps to establish trust between Snowflake and AWS, create an Iceberg volume, and set up Glue integration for metadata. This lesson is ideal for data engineers and analysts looking to enhance their skills in cloud data architecture and improve data accessibility across platforms. ### Boot Camp Database Setup URL: https://www.dataexpert.io/lesson/boot-camp-database-setup-yt-p736-l704 Module: Kickoff · Free ### Book Club 1 URL: https://www.dataexpert.io/lesson/bookclub1-51a9d Enrolled students ### Tim Chan - Data at OpenAI URL: https://www.dataexpert.io/lesson/tech-talk- with-tim-chan-p735-l596 Module: Tech Talks · Enrolled students Tim Chan, head of data science at Statsig, shares insights on experimentation in data science during this engaging talk. He discusses the importance of A/B testing, statistical rigor, and experiment design, emphasizing how analytics engineers can compute results and interpret metrics effectively. This lesson is ideal for data professionals looking to deepen their understanding of experimentation, its application in real-world scenarios, and the balance between statistical analysis and business decision-making. ### Day 1 Lab: Setting up CI/CD for an AI application URL: https://www.dataexpert.io/lesson/ml-ops-day1-lab-july25-p737-l1184 Module: Week 4: MLOps, Guardrails & Business Value · Enrolled students Setting up CI/CD for an AI application involves integrating various guardrails to ensure safety and compliance. This lab teaches participants how to implement content filtering, PII injection detection, and toxicity detection using tools like OpenAI's moderation API and regex. Ideal for data engineers and AI practitioners, the lesson emphasizes practical examples and the importance of balancing performance and cost in deploying machine learning models effectively. ### Managing Unstructured Data Day1 Lecture URL: https://www.dataexpert.io/lesson/managingunstructureddataday1lecture-feb26-p736-l2084 Module: Week 5: Managing Unstructured Data · Enrolled students In this video, Eumar talks about building an enterprise AI agent named Zachy, designed to assist students and employees with various queries related to the bootcamp and expense reports. He focus on setting up their environment and understanding the foundational aspects of unstructured data, including the importance of Unity Catalog and AI functions for parsing documents. He emphasises the need for context engineering and governance on unstructured data, introducing Unity Catalog Volumes as a solution. ### Structure of a Program URL: https://www.dataexpert.io/lesson/structure-of-a-program-596e3 Module: Web Dev Fundamentals · Enrolled students Understanding the structure of a program is crucial for web development. This lesson covers foundational programming concepts such as variables, constants, functions, and control flow, particularly in JavaScript. Learners will explore how to write clean, efficient code, recognize good coding practices, and understand the role of HTTP and DNS in web communication, benefiting both beginners and those looking to solidify their programming skills. ### Special Guests, Jitender Aswani and Bill Inmon [August 23, 2023] URL: https://www.dataexpert.io/lesson/v2-special-guests-jitender-aswani-and-bill-inmon-august-23-2023-4321 Enrolled students Join us each week as we sit down with visionary speakers from diverse fields, engaging in enlightening conversations that explore cutting-edge insights, trends, and their wealth of expertise. (Recorded August 23, 2023) ### Apache Spark programming with Databricks Day 1 Lecture URL: https://www.dataexpert.io/lesson/apache-spark-programming-with-databricks-day-1-lecture-may2025-1-p736-l1157 Module: Week 4: Structured Streaming with Spark & Kafka · Enrolled students Gain a comprehensive understanding of Apache Spark programming with this lecture focused on Spark architecture and its integration with Databricks. Learn about key concepts such as the roles of the driver and workers, the differences between Spark and Hadoop, and the significance of in-memory processing for big data. This lesson is ideal for data engineers and data scientists looking to enhance their skills in building efficient data pipelines using Spark SQL, Python, and streaming techniques. ### Setting up your Databricks App URL: https://www.dataexpert.io/lesson/setting-up-your-databricks-app Enrolled students In this video, you'll watch how to setup your first Databricks App to finish homework 1! ### TA Office hour 4 URL: https://www.dataexpert.io/lesson/taofficehour4-jan2026-p735-l1942 Enrolled students Office hours provide a platform for students to ask questions and seek guidance on their projects and assignments. Key topics discussed include capstone project planning, incorporating analytics, data engineering, and AI, as well as troubleshooting access issues with tools like Snowflake and PostgreSQL. Students also receive advice on managing timelines and expectations for their assignments, ensuring they understand how to effectively use the resources available to them. ### Day 2 Lab: Tracing LLM reasoning with LangChain + LangSmith URL: https://www.dataexpert.io/lesson/ml-ops-day2-lab-july25 Enrolled students Learn how to trace the reasoning of large language models (LLMs) using LangChain and LangSmith in a practical lab setting. This session covers the integration of the Tali web search API and OpenAI's API to create a simple application that generates explainable answers to queries, such as "What is photosynthesis?" Participants will gain hands-on experience in setting up tracing for function calls, analyzing performance metrics, and optimizing application latency. Ideal for data engineers and AI practitioners looking to enhance their understanding of LLMs and application performance. ### Understanding Capstone Requirements URL: https://www.dataexpert.io/lesson/understanding-capstone-requirements-p735-l655 Module: Kickoff · Enrolled students The session focuses on clarifying the requirements for the capstone project, specifically the need for multiple data sources. Participants engage in an open Q&A format, discussing various topics including project ideas, API usage, and the importance of business value in project selection. Key tools mentioned include Snowflake, Airflow, and various APIs like F1 and LinkedIn, providing insights into how to effectively structure and execute capstone projects in data engineering. ### TA Office Hour 5 URL: https://www.dataexpert.io/lesson/taofficehour5-fea37 Enrolled students ### Analytical Patterns (Day 2 Lecture) URL: https://www.dataexpert.io/lesson/analytical-patterns-day-2-lecture-v4 Module: Week 5: Analytical Patterns & Analysis in Iceberg · Enrolled students This is a pre-recorded class. In this presentation, Zach explores a range of topics related to Advanced SQL Analytical Patterns. The discussion covers the application of recursive CTEs, count case when, grouping sets, cross join unnest, cube, rollup, and window functions. Drawing from Zachs experience in conducting interviews and working with these concepts, the presentation offers insights and practical advice. [Recorded on Nov 29, 2023] ### Advanced dbt Day 2 Lab URL: https://www.dataexpert.io/lesson/advanced-dbt-day-2-lab-jan2025 Enrolled students In this lab, Bruno walks you through the essentials of defining packages in DBT and how to utilize macros effectively. He also covers the importance of logs for debugging and demonstrate how to set up incremental models. Please pay special attention to the examples he provides, as they will help clarify these concepts. ### Tech Talk - 2 URL: https://www.dataexpert.io/lesson/tech-talk2-oct2025-p737-l1550 Module: Tech Talks · Enrolled students Unfortunately, Andrew couldn't join the Tech Talk, so this is Q&A session with Zach. ### Data Quality Patterns: MIDAS Process from Airbnb Day 1 Lecture URL: https://www.dataexpert.io/lesson/data-quality-patterns-day-1-lecture-yt-p736-l698 Enrolled students In this presentation, we delve into the pivotal role of data quality in the realm of analytics engineering, unraveling its profound impact on the success of data pipelines. The discussion underscores the necessity for automated data quality checks, shedding light on the challenges encountered by data engineers in their quest for ensuring data accuracy. ### Advanced pipelines: Blue-green and WAP with Airflow Lab URL: https://www.dataexpert.io/lesson/advanced-pipelines-blue-green-and-wap-with-airflow-lab Module: Advanced dbt · Enrolled students In this lab, Bruno dives into the Airflow part of the project, showcasing how to work within the DBT project folder and run commands efficiently. He demonstrate setting up Airflow, running DBT tasks, and managing connections. ### Data Pipeline Maintenance: Strategies for Maintenance and Dock Building Day 2 Lecture URL: https://www.dataexpert.io/lesson/data-pipeline-maintenance-day-2-lecture-yt-p736-l688 Enrolled students In this video, Zach delves into the crucial realm of maintenance and dock building within data engineering. Unveiling the impact these often-overlooked tasks can have on our work, the discussion provides valuable insights. ### Fact Data Modeling Lecture URL: https://www.dataexpert.io/lesson/fact-data-modeling-lecture-p735-l578 Module: Bonus · Enrolled students In this lecture, we delve into the realm of fact data modeling, focusing on the core concept of facts and addressing the challenges associated with handling duplicates. Our exploration underscores the pivotal role of facts in data engineering, shedding light on their significance. Practical insights into deduplication techniques are provided, and we discuss the ramifications of retention on fact data. Join us in this video to gain a comprehensive understanding of fact data modeling and its crucial implications in the field of data engineering. ### Data Impact Communication & Visualization - Insights and Best Practices (Day 2 Lecture) URL: https://www.dataexpert.io/lesson/visual-impact-day-2-lecture-v3 Module: Week 6: Data Impact Communication & Visualization with Tableau · Enrolled students In this insightful lecture, we delve into the crucial realm of dashboard metrics. Exploring the importance of various metric types, including total aggregates, time and entity-based aggregates, dimensional mix, and retention and survivorship numbers, the video provides a comprehensive understanding. The discussion extends to common pitfalls to steer clear of when crafting dashboards. This video serves as a valuable guide to assist you in selecting the most suitable metrics for your dashboard, ultimately enhancing its overall performance. [Recorded on Dec 20, 2023] ### Apache Iceberg Lecture Day3 URL: https://www.dataexpert.io/lesson/apacheiceberglectureday3-6fb59 Enrolled students Data contracts are essential for establishing clear agreements between data producers and consumers, ensuring data quality and efficient processing. This lecture explores the concept of data contracts, their implementation in big tech, and the WAP (Write, Audit, Publish) pattern used for managing data quality. Participants will learn about the Iceberg branching model, time travel features, and the importance of understanding downstream impacts, making it beneficial for data engineers and analytics professionals looking to enhance their data management practices. ### Full Stack Bootcamp Orientation URL: https://www.dataexpert.io/lesson/fullstack-bootcamp-orientation Module: Fullstack Bootcamp Orientation · Enrolled students The Full Stack Bootcamp Orientation introduces participants to the structure and expectations of the upcoming six-week program. Attendees will learn foundational skills in JavaScript, TypeScript, and frameworks like Next.js and React, while also exploring backend development and database integration. This session is ideal for beginners and those looking to enhance their full stack capabilities, with an emphasis on community support, interactive learning, and practical project experience. ### Local Video 2 URL: https://www.dataexpert.io/lesson/lesson4-7ec44 Enrolled students The session features Madison, an experienced analytics engineer and author of a prominent newsletter on analytics engineering. Attendees gain insights into the evolving role of analytics engineers, the importance of effective communication with stakeholders, and the balance between data quality and business needs. Madison discusses practical experiences with tools like DBT, the challenges of data modeling, and the impact of AI on the analytics landscape, providing valuable perspectives for aspiring analytics professionals. ### Applying KPIs and Experimentation - Setting Up and Analyzing Experiments in Statsig (Day 1 Lab) URL: https://www.dataexpert.io/lesson/kpi-metrics-day-1-lab-v3 Module: Week 4: KPIs and Experimentation with Statsig · Enrolled students Learn how to practically apply the concepts from the lecture by setting up a web experiment in Statsig. Observe how data is collected in this process, gaining hands-on experience in defining KPIs and conducting experiments for measurement and improvement. ### End-to-End AI Applications Day2 Lab URL: https://www.dataexpert.io/lesson/endtoendaiapplicationsday2lab-mar26 Enrolled students Explore the practical aspects of building cloud-managed AI agents in this lab session. Participants will learn how to set up their development environment, create agents using the Anthropics API, and implement custom tool calls for enhanced functionality. This lesson is ideal for developers and data engineers looking to deepen their understanding of AI applications and cloud integration, as it covers essential techniques in asynchronous programming and API interactions. ### Hard Orchestration Lessons Day 2 Lab URL: https://www.dataexpert.io/lesson/hard-orchestration-lessons-day-2-lab-jan2025 Enrolled students In this lab, Zach takes you through the optimization of a data pipeline, with a focus on making it idempotent and predictable. He demonstrates steps such as setting up dependencies, handling data waits, and ensuring task success. Zach also emphasizes the importance of scheduling and avoiding duplicates. ### Databricks Boot Camp Kickoff URL: https://www.dataexpert.io/lesson/databricksbootcampkickoff-feb2026 Enrolled students ### Apache Spark Memory Turning, Partitioning Day 2 Lab URL: https://www.dataexpert.io/lesson/apache-spark-memory-turning-partitioning-day-2-lab-may2025 Enrolled students In this lab, Zach dives into the intricacies of Spark data processing, focusing on how to manage CSV files and optimize shuffle partitions. He highlights the importance of partitioning and how it affects job performance, especially when dealing with large datasets. He also discusses some common pitfalls and provide insights on how to improve efficiency. ### Q&A with Zach Week 1 URL: https://www.dataexpert.io/lesson/qa-with-zach-week1-oct2025 Enrolled students A live Q&A session addresses common setup issues and clarifies key concepts for participants in the boot camp. Topics include configuring the local environment, managing Git and SSH keys, and understanding the implications of scraping data from platforms like LinkedIn. The session also touches on the importance of foundational Python skills for data engineering roles and discusses strategies for navigating the job market. This session is beneficial for anyone looking to solidify their technical setup and gain insights into the course structure. ### Wednesday SQL Quiz URL: https://www.dataexpert.io/lesson/window-functions-wednesday-quiz Enrolled students ### Jason Reid (cofounder of Tabular) URL: https://www.dataexpert.io/lesson/jason-reid-cofounder-of-tabular Enrolled students Jason Reid, co-founder of Tabular, shares insights into the evolution of data engineering and the Iceberg project in this engaging discussion. He reflects on his journey from Netflix to Tabular, emphasizing the importance of correctness and productivity in data management. Reid also explores the future of Iceberg, its role in the open-source community, and the impact of cloud storage solutions like S3 on data architecture. This lesson is ideal for data engineers and tech enthusiasts looking to deepen their understanding of modern data frameworks and industry trends. ### Apache Spark programming with Databricks Day 2 Lab URL: https://www.dataexpert.io/lesson/apache-spark-programming-with-databricks-day-2-lab-may2025-p736-l1160 Module: Week 4: Structured Streaming with Spark & Kafka · Enrolled students In this video, Yared walks through the key features of Delta Live Tables (DLT) and explains how it simplifies structured streaming using both SQL and Python. He demonstrates the creation of streaming tables and materialized views, emphasizing the importance of using Auto Loader for efficient file ingestion, which can process thousands of files simultaneously. A real-life scenario involving airport operations is shared to illustrate data engineering concepts. ### Li Yin - Founder at Adal (2026-03-13) URL: https://www.dataexpert.io/lesson/techtalks1-mar26-p737-l2114 Module: Tech Talks · Enrolled students Li Yin, founder of SILFAI, shares her journey from working at MetaAI to launching her own AI startup focused on coding agents. This session explores the evolution of AI roles, the importance of entrepreneurship in the tech landscape, and insights into building effective AI agents. Attendees will gain valuable perspectives on navigating the AI engineering field, the significance of multi-agent architectures, and the future of AI-driven tools in various applications. ### Analytical Patterns (Day 2 Lab) URL: https://www.dataexpert.io/lesson/analytical-patterns-day-2-lab-v4 Module: Week 5: Analytical Patterns & Analysis in Iceberg · Enrolled students This is a pre-recorded class. In this lab session, Zach demonstrates various queries and aggregations that can be applied to the bootcamp web events table. The talk emphasizes the significance of reducing cardinality and showcases the use of functions like coalesce and rollup. Zach also touches on the concept of self-joins and how they can aid in analyzing a users journey on the website. [Recorded on Nov 29, 2023] ### Friday Quiz URL: https://www.dataexpert.io/lesson/fridayquiz-53b9c Enrolled students ### TA Office Hour 3 (2025-06-11) URL: https://www.dataexpert.io/lesson/ta-office-hour-3-may2025-p736-l1144 Enrolled students A TA office hour session focused on addressing student questions regarding assignments and project requirements. Key topics included building a simple data warehouse using a star schema, generating synthetic data with Python libraries like Faker, and best practices for organizing data in staging and final layers. Students also discussed the use of Databricks features such as clustering and optimization, as well as submission guidelines for their assignments and capstone projects. ### dbt basics Day2 Lab URL: https://www.dataexpert.io/lesson/dbtbasicsday2lab-c7486 Enrolled students ### AI Engineering Bootcamp Kick off URL: https://www.dataexpert.io/lesson/aiengineeringbootcampkickoff-mar26 Enrolled students ### Day 1 Lecture: RAG Fundamentals and types of RAG URL: https://www.dataexpert.io/lesson/understanding-rag-lecture-day1-july2025-p737-l1174 Module: Week 1: Prompt Engineering & Basic RAG · Enrolled students In this lecture, Zach shares some staggering statistics, like the fact that there are currently 160 zettabytes of unstructured data in the world, which is projected to double every two to three years. He discusses the importance of understanding unstructured data and how AI can help us make sense of it. He emphasizes the need to convert this data into more structured formats for better analysis and retrieval. ### Chat with Zach URL: https://www.dataexpert.io/lesson/week-3-zach-qa-v4 Enrolled students Exclusive Q&A for homework doers. [Recorded on May23rd, 2024] ### dbt day 1 lecture URL: https://www.dataexpert.io/lesson/dbt-day-1-lecture Module: DBT · Enrolled students In this lecture, Bruno provides an introduction to DBT (Data Build Tool) and discuss its key features. He share my personal experience as a DBT certified professional and a former student of the V2 Bootcamp. He explains how DBT differs from cloud data warehouses like BigQuery and Snowflake, and highlight its SQL-first approach and support for transformation pipelines. He also touches upon topics such as models, tests (including data tests and unit tests), sources, seeds, and snapshots. ### How to use snowpark and Snowflake optimization Lecture URL: https://www.dataexpert.io/lesson/how-to-use-snowpark-and-snowflake-optimization-lecture-p735-l651 Module: Week 7: Analytical Patterns & CDC · Enrolled students Welcome to Snowflake Day 2! In this lecture, Zach delves into various optimization techniques for Snowflake, including cost optimizations and utilizing Snowpark for data processing. He explains the intricacies of Snowpark, its similarities to Spark, and its role in ETL processes within Snowflake. Additionally, He discusses micropartitioning, lazy execution in Snowpark, and the importance of understanding query patterns for cost management. ### Tech talk with Jess Ramos URL: https://www.dataexpert.io/lesson/tech-talk-with-jess-ramos Module: Tech Talk · Enrolled students This is the tech talk with Jess Ramos. ### RAG optimisation + Reranking Day 2 Lab URL: https://www.dataexpert.io/lesson/rag-optimisation-reranking-day2-lab-oct2025-p737-l1574 Module: Week 2: Advanced RAG, Reranking & Dev Workflow · Enrolled students In this lab, Zach focuses on re-ranking and working with an external worker process, which are crucial for optimizing the retrieval system. He implements metadata filtering and re-ranking to improve query results, ensuring that he accurately classifies document types and determines the optimal number of documents to retrieve. He demonstrates how to set up a PubNub channel for asynchronous processing, allowing him to handle longer tasks without timing out. ### Optimize your dev workflow with Claude Code and Codex Day2 Lecture URL: https://www.dataexpert.io/lesson/optimizeyourdevworkflowwithclaudecodeandcodexday2lecture-mar26 Enrolled students In this lecture, Kevin walk throughs the compound engineering loop with Claude Code, focusing on agentic context management. He explains Claude md versus skills, how skills are reusable workflows invoked by slash commands, and how skills compound over time. He also covers sub agents and agent teams for parallel work while protecting context, plus agentic validation with multiple fail safes like lint, tests, and UI checks. He briefly introduces MCP and cattle not pets development setup with a VPS, tmux, and SSH. ### Managing Unstructured Data Day2 Lab URL: https://www.dataexpert.io/lesson/managingunstructureddataday2lab-feb26 Enrolled students In this video, Eumar explores document parsing and chunking strategies using AI tools. Hw successfully upload a PDF syllabus, parse it, and save the content into a database, while also discussing advanced web crawling techniques. He encourages everyone to review the 10 chunking strategies he outlined in Lab 2, as understanding these will be crucial for their upcoming projects. ### Local Video URL: https://www.dataexpert.io/lesson/local-video-769 Enrolled students ### RAG optimisation + Reranking Day 2 Lab URL: https://www.dataexpert.io/lesson/rag-optimisation-reranking-day2-lab-oct2025 Enrolled students In this lab, Zach focuses on re-ranking and working with an external worker process, which are crucial for optimizing the retrieval system. He implements metadata filtering and re-ranking to improve query results, ensuring that he accurately classifies document types and determines the optimal number of documents to retrieve. He demonstrates how to set up a PubNub channel for asynchronous processing, allowing him to handle longer tasks without timing out. ### OpenClaw and Automating business processes Day2 Lab URL: https://www.dataexpert.io/lesson/openclawandautomatingbusinessprocessesday2lab-mar26-p737-l2148 Module: Week 3: Agentic AI, MCP & Automation · Enrolled students In this lab, Drew connects the OpenClaw to Zapier via an MCP server, using a fresh Gmail for the agent, least privilege tool access, and locking Gmail tools to only send or read from your agent email. He demonstrates the test email, calendar and tasks, set up a heartbeat, and do prompt injection and tool audit exercises. ### Injecting quality into your Airflow DAG Lab URL: https://www.dataexpert.io/lesson/injectingqualityintoyourairflowdaglab-4c365 Enrolled students ### Q&A with Zach Week5 URL: https://www.dataexpert.io/lesson/qawithzachweek5-feb26-p736-l2041 Module: Q&A with Zach · Enrolled students Week 5 features a live Q&A session where Zach discusses the upcoming AI boot camp and introduces new instructors, including Carly Taylor and Kevin Notten. Participants will learn about setting up Open Claw agents and explore foundational AI concepts such as prompt engineering and RAG. This session is ideal for those looking to deepen their understanding of AI tools and techniques while receiving guidance on their projects and career paths in data engineering. ### dbt basics: creating a full dbt project lab URL: https://www.dataexpert.io/lesson/dbt-basics-creating-a-full-dbt-project-lab Module: dbt basics · Enrolled students In this lab, Bruno walks you through our DBT project, emphasizing the importance of running DBT dependencies and configuring freshness settings. He demonstrates how to handle errors, set configurations, and write data tests effectively. Pay attention to the different ways to define configurations and the significance of testing models for accuracy. ### Thursday Quiz URL: https://www.dataexpert.io/lesson/thursdayquiz-5bc94 Enrolled students ### Managing Unstructured Data Day1 Lab URL: https://www.dataexpert.io/lesson/managingunstructureddataday1lab-feb26-p736-l2085 Module: Week 5: Managing Unstructured Data · Enrolled students In this video, Eumar walks you through the four notebooks focused on Unity Catalog, synthetic data generation, and using Databricks AI functions to parse receipts. He discusses the importance of handling PII data and how to effectively process data in parallel using Spark. He demonstrates how to create schemas, tables, and volumes, and encourages everyone to follow along and try the exercises at home. Additionally, he highlights the use of the Databricks AI DevKit for generating code and managing data. ### Building AI Agents with Databricks Day2 Lab URL: https://www.dataexpert.io/lesson/buildingaiagentswithdatabricksday2lab-feb26-p736-l2036 Module: Week 6: Building AI Agents with Databricks · Enrolled students In this video, Eumer walks you through the implementation of the app during the lab session, highlighting key aspects of the Supervisor agent and the importance of on-behalf user authorization for security. He also demonstrates how to create connections for external APIs and manage permissions effectively. He encourages everyone to complete Lab 4 before moving on to the next sessions, as it lays the groundwork for the upcoming tasks. Additionally, he shares insights on how to register models in Unity Catalog and the significance of monitoring performance through tracing and evaluation. ### Q&A with Zach Week 2 (2025-10-09) URL: https://www.dataexpert.io/lesson/qa-with-zach-week2-oct2025-p737-l1545 Module: Q&A with Zach · Enrolled students Week 2's Q&A session covers various topics crucial for students in the AI engineering boot camp. Participants engage in discussions about the capstone project, strategies to avoid tutorial hell, and the importance of building practical projects. Zach also addresses questions about keeping up with industry trends, the role of AI in job markets, and the significance of foundational knowledge over specific tools. This session is ideal for learners seeking guidance on project development and career progression in AI engineering. ### Saturday Quiz URL: https://www.dataexpert.io/lesson/saturdayquiz-01a8a Enrolled students ### Unit Testing Spark Jobs: Mastering Spark and PySpark Testing Lab URL: https://www.dataexpert.io/lesson/apache-spark-day-3-lab-yt-p736-l707 Enrolled students In this lab, we embark on an exploration of Spark, offering a comprehensive overview of its main components and the intricacies of data transformation within a Spark job. ### MCP and Agentic AI Day 2 Lecture URL: https://www.dataexpert.io/lesson/mcp-and-agentic-ai-day2-lecture-oct2025-p737-l1608 Module: Week 3: Agentic AI, MCP & Automation · Enrolled students In this lecture, Zach dives deeper into the Model Context-X protocol, focusing on the importance of middleware in keeping AI agents safe and effective. He discusses how to optimize for a happy path with MCP agents, when to use middleware, and how to avoid reinventing the wheel with REST and MCP servers. ### Apache Iceberg Lecture Day1 URL: https://www.dataexpert.io/lesson/apacheiceberglectureday1-4c361-p735-l1991 Module: Week 5: Data Modeling on Iceberg · Enrolled students Apache Iceberg is a powerful data management framework designed for modern data lake architectures. This lecture introduces the evolution of big data technologies, focusing on Iceberg's advantages over predecessors like Apache Hive, particularly in metadata management and data governance. Data engineers and analysts will learn about Iceberg's features, including time travel, snapshot management, and efficient partitioning strategies, which enhance data processing and accessibility in large-scale environments. ### How to connect Iceberg and Snowflake with Catalogs URL: https://www.dataexpert.io/lesson/howtoconnecticebergandsnowflakewithcatalogs-jan2026 Enrolled students In this lab, Zach dives into processing unstructured data, focusing on earnings call transcripts and SEC filings. He explores how to calculate file checksums to avoid reprocessing unchanged files and successfully load earnings transcripts into Snowflake using Snowpark. Zach demonstrates the importance of managing file compression and encryption settings when creating stages in Snowflake. ### Day 1 Lab: Building Graph RAG in production URL: https://www.dataexpert.io/lesson/advanced-rag-and-agentic-ai-day1-lab-july2025 Enrolled students In this lab, Ashwani walks you through the process of setting up a knowledge graph using Neo4j and OpenAI. He demonstrates how to clone the repository, configure environment variables, and create a graph with entities like person, organization, and location. He also explores how to perform hybrid searches and generate responses using a language model. ### TA Office hour 3 URL: https://www.dataexpert.io/lesson/taofficehour3-feb26 Enrolled students This office hours session addresses common issues encountered in homework assignments related to API usage and function execution in coding labs. Participants can learn how to troubleshoot a 401 client error, modify base URLs, and adjust code to ensure proper function execution. This session is beneficial for students seeking clarification on practical coding challenges and improving their understanding of API integration and catalog management. ### OpenClaw and Automating business processes Day2 Lecture URL: https://www.dataexpert.io/lesson/openclawandautomatingbusinessprocessesday2lecture-mar26-p737-l2147 Module: Week 3: Agentic AI, MCP & Automation · Enrolled students In this lecture, Carly recaps OpenClaw and then go deep on Pi Agent, the lightweight coding agent at the core. Also discusses about why open source extensibility creates both power and a bigger security risk surface. ### Data modeling with Date lists Lab URL: https://www.dataexpert.io/lesson/data-modeling-with-date-lists-lab-p735-l586 Module: Bonus · Enrolled students This lab guides you through the process of analyzing user activity using compact tables. The lab demonstrates how to create a table that efficiently captures user IDs and their corresponding activity status. We address the intricacies of handling date calculations and utilizing bitwise operations to represent user activity. The lab captures the presenter encountering and troubleshooting various challenges, providing valuable insights and explanations. By the end of the lab, you will have gained a clear understanding of how to design and leverage compact tables for the purpose of analyzing user activity. ### Apache Spark Memory Turning, Partitioning Day 2 Lecture URL: https://www.dataexpert.io/lesson/apache-spark-memory-turning-partitioning-day-2-lecture-may2025 Enrolled students In this lecture, Zach dives into how to troubleshoot slow Spark jobs, particularly focusing on common bottlenecks that can occur in big tech environments. He discusses the impact of upstream data formats like JSON and CSV, and how they can hinder performance. He also shares insights on optimizing data processing and managing skew data effectively. ### Data Accessibility and Privacy (Day 4 Lab) URL: https://www.dataexpert.io/lesson/fact-data-modeling-day-4-lab-v4 Module: Week 2: Fact Data Modeling in Iceberg · Enrolled students In this lab, Zach discusses the concepts of anonymization and retention strategies in data management. He explains how to create tables and schemas, and how to handle anonymized data. He also demonstrates how to join tables and perform queries to retrieve specific data. Throughout the lab, he provides examples and explanations to help viewers understand the concepts better. [Recorded on May16th, 2024] ### Infrastructure Data Quality - Elevating Infrastructure Data Quality: Insights and Best Practices with Marc Lamberti (Day 2 Lecture) URL: https://www.dataexpert.io/lesson/infra-data-quality-day-2-lecture-v3 Module: Week 5: Unit Testing Pipelines & Write-audit-publish · Enrolled students In this illuminating lecture, Marc Lamberti takes the stage to unravel the complexities of ensuring data quality within the infrastructure landscape. Delving into the intricacies of data validation and integrity, Marc shares invaluable insights and best practices to elevate your understanding of infrastructure data quality. With a wealth of experience in the field, he navigates through challenges and opportunities, providing a comprehensive overview that empowers you to enhance data quality in infrastructure settings. Join us for a deep dive into this critical aspect of data management[Recorded on Dec 14th, 2023] ### TA Office Hour 7 (2025-10-09) URL: https://www.dataexpert.io/lesson/ta-office-hour7-oct2025-p737-l1542 Enrolled students ### Lesson 4 URL: https://www.dataexpert.io/lesson/lesson-4-758 Enrolled students ### Q&A with Zach Week 4 URL: https://www.dataexpert.io/lesson/qa-with-zach-week-4-april2025-p735-l1013 Module: Q&A with Zach · Enrolled students A live Q&A session with Zach covers various topics related to the ongoing boot camp, including project proposals, API usage, and insights into analytics engineering roles. Participants discuss the challenges of accessing paid APIs, the importance of project feedback, and strategies for optimizing data engineering practices. Zach shares personal experiences and advice on navigating the complexities of data engineering, making this session valuable for students seeking guidance on their projects and career paths. ### Advanced snowflake: Deploying the Capstone dag Lecture URL: https://www.dataexpert.io/lesson/advanced-snowflake-deploying-the-capstone-dag-lecture-p735-l561 Module: Week 4: Advanced Snowflake · Enrolled students In this lecture, Zach delves into the key aspects of creating impactful capstone projects to maximize your bootcamp experience. He discusses the importance of longevity, impact, quality, and proof in project development. I share insights on project complexity, data pipeline management, and showcasing your work effectively. ### Data Lakes with Delta Table Day1 Lab URL: https://www.dataexpert.io/lesson/datalakeswithdeltatableday1lab-feb26-p736-l2022 Module: Week 3: Data Lakes with Delta Table · Enrolled students In this lab, Zach walks you through setting up your Git repository in Databricks and creating a Delta Table with synthetic data. He explores schema generation, data manipulation, and how to query the table's history, including operations like deletes and updates. Zach also highlights the importance of time travel in Delta Tables for data recovery. ### TA Office Hour 3 (2025-01-21) URL: https://www.dataexpert.io/lesson/ta-office-hour-3-jan2025-p736-l768 Enrolled students Office hours provide an interactive platform to discuss homework assignments and clarify concepts in data engineering. This session focused on Homework 2, covering the integration of the Polygon API with Apache Airflow for data processing. Key topics included the setup of data pipelines, the use of SQL and Python for data quality checks, and the implementation of cumulative tables. Participants also explored troubleshooting techniques and best practices for using Airflow effectively. ### Complex Data Types Lecture URL: https://www.dataexpert.io/lesson/complex-data-types-lecture-p735-l583 Module: Bonus · Enrolled students In this presentation, we explore the fundamental concepts essential for effective data modeling and database management. We dive into understanding your data consumer, the distinctions between OLTP and OLAP data modeling, the principles of Cumulative Table design, and the tradeoff between compactness and usability. We also address the challenges of temporal cardinality explosion and the potential pitfalls of run-length encoding compression. By the end of this lecture, you will have a strong foundation in the key concepts of data modeling and be prepared to apply these principles in practice. ### Change Data Capture Lab URL: https://www.dataexpert.io/lesson/changedatacapturelab-jan2026-p735-l1904 Module: Week 7: Analytical Patterns & CDC · Enrolled students Change Data Capture (CDC) is crucial for tracking data changes in real-time, and this lab focuses on implementing it using database triggers. Participants will learn how to create an audit tracking table for time zone changes, utilize PostgreSQL with DataGrip, and set up triggers to capture updates efficiently. This lesson benefits data engineers and analysts looking to enhance their skills in CDC and understand the implications of real-time data tracking in production environments. ### RAG and LLMs Day1 Lecture URL: https://www.dataexpert.io/lesson/rag-and-llms-day-1-lecture-jan2025 Enrolled students In this lecture, Zach dives into the powerful implications of LLMs and RAG techniques for engineers, emphasizing the importance of adapting to these changes. He discusses the differences between fine-tuning and RAG, and how they can be leveraged together for better outcomes. He also highlights the significance of creating effective prompts for generating SQL with LLMs. ### Intro to CI/CD URL: https://www.dataexpert.io/lesson/intro-to-ci-cd-0a1f8 Module: Q&A with Dan · Enrolled students Learn about Continuous Integration and Deployment (CI/CD) using Jenkins, GitLab, GitHub Actions, CircleCI, and ArgoCD. ### Apache Spark Memory Turning, Partitioning Day 2 Lecture URL: https://www.dataexpert.io/lesson/apache-spark-memory-turning-partitioning-day-2-lecture-may2025-p736-l1145 Module: Week 2: Databricks & Advanced Spark · Enrolled students In this lecture, Zach dives into how to troubleshoot slow Spark jobs, particularly focusing on common bottlenecks that can occur in big tech environments. He discusses the impact of upstream data formats like JSON and CSV, and how they can hinder performance. He also shares insights on optimizing data processing and managing skew data effectively. ### Apache Spark: Hands-On for Broadcast and Hash Joins Day 1 Lab URL: https://www.dataexpert.io/lesson/apache-spark-day-1-lab-yt-p736-l679 Enrolled students Dive into practical Spark join optimization techniques in this hands-on video. We kick off by setting up the Spark directory and executing Docker compose for a seamless start. ### Day 2 Lecture: RAG Tradeoffs URL: https://www.dataexpert.io/lesson/understanding-rag-lecture-day2-july2025-p737-l1176 Module: Week 1: Prompt Engineering & Basic RAG · Enrolled students In this lecture Zach dives deeper into identifying trigger points for implementing AI in our businesses, discussing how to make our RAG chunks more relevant, and exploring cloud deployment and unification of applications. He covers four key trigger points: chatbots, CICD, cron jobs, and event queues, highlighting their risks and benefits. He will also address how to improve the relevance of our RAG results and the importance of proper data chunking. ### Thursday Slowly Changing Dimensions Quiz URL: https://www.dataexpert.io/lesson/scd-quiz Enrolled students ### Tech Talk - Li URL: https://www.dataexpert.io/lesson/techtalks1-mar26 Enrolled students Tech Talk features Li Yin, founder of Self AI and former researcher at MetaAI, discussing her journey from corporate environments to entrepreneurship in AI. The session covers insights on building AI agents, including methodologies for tool selection and the importance of documentation in coding. Li also shares her experiences with vibe coding and the development of innovative tools like Ada Video, emphasizing the collaborative nature of working with AI agents. This talk is ideal for aspiring AI engineers and entrepreneurs looking to navigate the evolving landscape of AI technology. ### Q&A with Zach Week1 URL: https://www.dataexpert.io/lesson/qawithzachweek1-feb26 Enrolled students The session features a Q&A with Zach, focusing on setup instructions for the lab environment, including integrating GitHub with DataBricks and creating Git folders. Participants will benefit from insights on managing data workflows, using Delta tables, and understanding the importance of Git in data engineering. The discussion also touches on broader topics like data architecture, the future of data engineering roles, and the implications of AI on the industry, making it valuable for both beginners and experienced professionals in data engineering. ### Backfilling with Airflow URL: https://www.dataexpert.io/lesson/backfillingwithairflow-18fa4 Enrolled students ### Delta Table Day 2 Lecture URL: https://www.dataexpert.io/lesson/delta-table-day-2-lecture-may2025-p736-l1130 Module: Week 3: Data Lakes with Delta Table · Enrolled students In this video, Yared dives into advanced features of Delta Lake, focusing on liquid clustering and optimization techniques. He explains how liquid clustering improves query performance by dynamically organizing data and how the optimize command compacts smaller files for better read efficiency. He also touches on versioning and the importance of maintaining historical data. ### Day 8: Reverse a singly linked list URL: https://www.dataexpert.io/lesson/reverse-a-singly-linked-list-p913-d8 Enrolled students Reverse a singly linked list iteratively with three pointers in one pass, proving O(n) time and O(1) space and handling empty/one-node inputs safely. ### dbt basics: creating a full dbt project lab URL: https://www.dataexpert.io/lesson/dbt-basics-creating-a-full-dbt-project-lab-p735-l657 Module: Week 1: dbt Basics · Enrolled students In this lab, Bruno walks you through our DBT project, emphasizing the importance of running DBT dependencies and configuring freshness settings. He demonstrates how to handle errors, set configurations, and write data tests effectively. Pay attention to the different ways to define configurations and the significance of testing models for accuracy. ### TA Office hour 7 URL: https://www.dataexpert.io/lesson/taofficehour7-9b936 Enrolled students ### Q&A with Zach Week3 URL: https://www.dataexpert.io/lesson/qawithzachweek3-feb26-p736-l2039 Module: Q&A with Zach · Enrolled students The Week 3 Q&A session features Zach and Umar discussing various topics related to data engineering and unstructured data. Participants can gain insights into the practical applications of DataBricks, including its advantages for handling large-scale data and unstructured formats. The session also covers the importance of governance in data management, the evolving landscape of data technologies, and the integration of foundational models in data workflows. Attendees benefit from real-world examples and expert advice on leveraging these tools effectively. ### Onboarding Guide for the Data and AI Engineering Bootcamp URL: https://www.dataexpert.io/lesson/onboardingguideforthedataandaiengineeringbootcamp-jan2026-p735-l1959 Module: Kickoff · Enrolled students ### Prasad Rao URL: https://www.dataexpert.io/lesson/prasad-rao-jan2025 Enrolled students Prasad Rao, a principal solutions architect at AWS, shares insights on succeeding in behavioral interviews, particularly for roles in big tech companies. This session covers key strategies for articulating past experiences, emphasizing the importance of storytelling, and preparing for common behavioral questions. Ideal for aspiring solutions architects and professionals looking to enhance their interview skills, the lesson highlights the significance of both technical and interpersonal competencies in the hiring process. ### Wiring up Agent Bricks to MCP URL: https://www.dataexpert.io/lesson/wiringupagentbrickstomcp-aug26 Enrolled students In this lecture, Eumar focuses on how to productionize an LLM agent with Databricks, emphasizing LLM Ops through model serving, Unity AI Gateway, guardrails, inference tables, and evaluation. He explains that only about 20 percent of agents typically reach production due to non-determinism, inconsistent results, missing success criteria, and the need for continuous feedback and observability. He distinguish model serving as the compute layer and Unity AI Gateway as the proxy with governance features such as rate limits, PII handling, topic restrictions, and jailbreak prevention, noting Unity AI Gateway became GA about a month ago. He also covers inference tables for logging input and output and using MLflow 3 with LLMs as judges plus human-in-the-loop review apps to validate against a golden dataset, with a workflow for iteration and redeployment. ### Book Club 3 URL: https://www.dataexpert.io/lesson/bookclub3-mar26 Enrolled students The discussion focuses on Chapter 8 of a dataset engineering book, emphasizing the importance of clean, high-quality data for successful AI model training. Participants share their experiences with data engineering techniques, including data cleaning, augmentation, and validation, as well as the relevance of these practices in analytics and AI. The session also explores the balance between model quality and data quality, highlighting the critical role of diverse and well-curated datasets in developing effective AI systems. ### QnA with Zach 5 URL: https://www.dataexpert.io/lesson/qnawithzach5-aug26 Enrolled students The final Q&A session of the Data Bricks AI bootcamp offers participants an opportunity to engage directly with Zach, discussing topics such as resume reviews, job placement strategies, and insights into the capstone projects. Attendees can gain valuable advice on navigating technical interviews and enhancing their career prospects in data engineering, particularly in relation to AI and machine learning roles. This relaxed session encourages open dialogue, making it ideal for those seeking guidance and community support as they transition into their next career steps. ### Sundas Khalid URL: https://www.dataexpert.io/lesson/sundas-khalid-jan2025 Enrolled students Sundas Khalid shares her extensive journey through various roles in data engineering, data science, and analytics, drawing from her experiences at Amazon and Google. She discusses the importance of financial literacy and salary negotiation strategies, providing insights on how to advocate for oneself during the hiring process. This session is beneficial for aspiring data professionals looking to understand career progression, negotiation techniques, and the realities of working in big tech. ### Career Dev Session 5 URL: https://www.dataexpert.io/lesson/career-dev-session5-oct2025 Enrolled students ### The History of Data Lakes Lecture URL: https://www.dataexpert.io/lesson/apache-iceberg-day-1-lecture-jan2025 Enrolled students In this lecture, we explore the evolution of big data processing technologies, starting with Java MapReduce, the impact of Hive, and the rise of Spark. Zach shares experiences from adopting Scala and Spark at Airbnb, discuss the power of Iceberg in data engineering, and how object storage technologies like S3 have changed the game. We also look at the benefits of Iceberg, its role in data restoration, and why partitioning strategies are so important for data engineers. ### Apache Spark Core Day 3 Lab URL: https://www.dataexpert.io/lesson/apache-spark-core-day-3-lab-jan2025 Enrolled students In this lab, Yared walks you through the basics of Spark SQL and the DataFrame API, highlighting how to create and interact with DataFrames. He explains the difference between transformations and actions, and how they affect job creation in Spark. He also emphasizes the importance of understanding narrow and wide transformations, as well as schema inference. ### Apache Spark programming with Databricks Day 2 Lecture URL: https://www.dataexpert.io/lesson/apache-spark-programming-with-databricks-day-2-lecture-may2025-p736-l1159 Module: Week 4: Structured Streaming with Spark & Kafka · Enrolled students In this video, Yared discusses the concept of stream processing, drawing parallels between data streams and water flow to illustrate the importance of continuous data movement. He highlights various applications, such as fraud detection, predictive maintenance in aircraft, and real-time marketing, emphasizing the need for low-latency decision-making in these areas. He also introduces popular stream processing tools like Kafka and Flink, and explores how to implement these concepts using Databricks. ### TA Office Hour 3 URL: https://www.dataexpert.io/lesson/ta-office-hour-3-jan2025 Enrolled students Office hours provide a platform for students to clarify doubts and discuss their homework assignments. This session focused on Homework 2, covering the integration of the Polygon API with Iceberg for stock price data processing. Key topics included using Airflow for workflow orchestration, implementing data quality checks, and managing cumulative tables. Students also explored the use of Python and SQL for various tasks, with discussions on best practices and troubleshooting common issues. ### Monday Quiz URL: https://www.dataexpert.io/lesson/mondayquiz-e949d Enrolled students ### Building AI Agents with Databricks Day1 Lab URL: https://www.dataexpert.io/lesson/buildingaiagentswithdatabricksday1lab-feb26 Enrolled students In this video, Eumar walks you through the new repository he created for week 2, which includes five labs focusing on deploying a custom MCP app using Databricks asset bundles. He explains the importance of role-level security and how to ensure users can only access their own data. He also dives into the process of bringing GitHub data into the vector database and discusses chunking strategies for optimal data retrieval. ### Apache Spark Memory Turning, Partitioning Day 2 Lab URL: https://www.dataexpert.io/lesson/apache-spark-memory-turning-partitioning-day-2-lab-may2025-p736-l1146 Module: Week 2: Databricks & Advanced Spark · Enrolled students In this lab, Zach dives into the intricacies of Spark data processing, focusing on how to manage CSV files and optimize shuffle partitions. He highlights the importance of partitioning and how it affects job performance, especially when dealing with large datasets. He also discusses some common pitfalls and provide insights on how to improve efficiency. ### Q&A with Zach Week 4 (2025-08-01) URL: https://www.dataexpert.io/lesson/qa-with-zach-week4-july2025-p737-l1348 Module: Q&A with Zach · Enrolled students A live Q&A session with Zach covers various topics related to AI engineering and project development. Participants discuss the challenges of using AI tools for homework, the importance of understanding system design in AI applications, and strategies for effective prompt engineering. Zach emphasizes the value of hands-on experience, the significance of clear requirements, and the balance between leveraging AI and maintaining foundational coding skills. This session is beneficial for learners seeking guidance on applying AI in practical scenarios and enhancing their engineering skills. ### Career Dev Session 2 URL: https://www.dataexpert.io/lesson/career-dev-session2-july2025 Enrolled students Career development on LinkedIn is essential for job seekers looking to enhance their visibility and opportunities. This session covers how to create an impactful LinkedIn profile, including tips on optimizing your headline, summary, and endorsements. Participants will learn strategies for connecting with industry professionals and recruiters, as well as the importance of maintaining an active presence through posts and engagement, ultimately increasing their chances of landing job interviews. ### Slowly Changing Dimensions Lab URL: https://www.dataexpert.io/lesson/slowly-changing-dimensions-lab Module: Bonus · Enrolled students In this lab, we reinforce lecture concepts with a hands-on approach. We dive into implementing a Slowly Changing Dimension (SCD) table. Learn to track user activity changes using lag functions and identify active and inactive player seasons. We also tackle common errors in query optimization, providing practical solutions for a real-world understanding. Watch the video for step-by-step guidance on implementing SCDs in your data pipelines. ### Data Modeling and intro to backend URL: https://www.dataexpert.io/lesson/data-modeling-intro-to-backend Module: Fundamentals of Backend Development · Enrolled students Data modeling is essential for designing effective databases, focusing on relational data modeling and SQL. This lesson covers the principles of creating a well-structured database schema, emphasizing the importance of avoiding data redundancy and ensuring data integrity through normalization techniques, particularly third normal form. Ideal for aspiring data engineers and backend developers, this session also introduces REST services and prepares learners for building backend applications using Rust in future lessons. ### Monetizing Part 2: Sponsorship by Carly Taylor URL: https://www.dataexpert.io/lesson/linkedin-masterclass-5 Module: Making Content on LinkedIn · Enrolled students Monetizing Part 2: Sponsorship by Carly Taylor. [Recorded on June 7th, 2024] ### Airflow Day2 Lab URL: https://www.dataexpert.io/lesson/airflowday2lab-jan2026 Enrolled students The Airflow Day2 Lab focuses on enhancing data quality in ETL processes using the Pie Iceberg API. Participants will learn to identify and resolve common data quality issues, such as duplicates and incorrect currency formats, while working with stock ticker data. This hands-on session is ideal for data engineers and analysts looking to improve their skills in data validation and pipeline management using Apache Airflow and Python. ### Applying Advanced SQL for Analytical Insights - Exploring SQL, Scaling Projects, and Aggregation Analysis (Day 1 Lecture) URL: https://www.dataexpert.io/lesson/analytical-patterns-day-1-lecture-v3 Module: Week 3: Analytical Patterns & Analysis in Iceberg · Enrolled students In this video lecture, we explore fundamental concepts in data engineering, with a primary focus on the pivotal role of SQL as the foundational element in this field. The discussion extends to the patterns and structures essential for scaling data engineering projects, providing a comprehensive understanding of their implementation. Delving into aggregation-based analysis, we address considerations surrounding dimensions and granularity. The lecture also touches upon the significance of retention analysis and survivorship curves, offering insights that are valuable for career growth. Join us in this session to gain a profound understanding of the key elements shaping data engineering and the strategic insights necessary for success in this dynamic field. [Recorded on Nov 27, 2023] ### Data Visualization with tableau (Day 2 Lab) URL: https://www.dataexpert.io/lesson/visual-impact-day-2-lab-v4 Module: Week 6: Data Impact Communication & Visualization with Tableau · Enrolled students This is a pre-recorded class. Welcome to a hands-on lab where we explore diverse data aggregation and visualization techniques using a sample dataset. The video takes you through the intricacies of aggregating data at different levels, such as host and source, and demonstrates the creation of meaningful visualizations. Emphasizing the advantages of pre-aggregating data for accelerated query performance, this lab provides valuable insights into data analysis and visualization. [Recorded on Dec 20, 2023] ### Delta Lake Bonus URL: https://www.dataexpert.io/lesson/delta-lake-bonus-may2025 Enrolled students Explore the advanced features of Delta Lake in this bonus session designed for data engineers using Databricks. Learn how to implement Delta Lake for data management, including batch and streaming operations, time travel, and data quality enforcement. This lesson covers practical techniques such as creating Delta tables, performing DML operations, and leveraging metadata for efficient data processing, making it ideal for those looking to enhance their data engineering skills. ### LLMs Day3 Part2 URL: https://www.dataexpert.io/lesson/llms-day-3-part-2-jan2025-p736-l831 Module: Bonus - LLMs · Enrolled students Explore the end-to-end process of building and training a language model application in this lesson. You'll learn how to set up the Aflow library, utilize OpenAI datasets, and implement a task that involves counting objects using language models. This lesson is ideal for data scientists and AI engineers looking to enhance their skills in model training, prompt engineering, and performance evaluation using various models like GPT-3.5 and DeepSeek. ### New Test Lesson URL: https://www.dataexpert.io/lesson/new-test-lesson-7028 Module: test module · Enrolled students Learn how to leverage AI for auto grading in data assignments using the Data Expert platform. This lesson demonstrates the process of submitting homework, receiving instant feedback, and understanding grading criteria, showcasing the efficiency and speed of AI in evaluating student work. Ideal for educators and students looking to enhance their assessment methods with innovative technology. ### Q&A Week 1 URL: https://www.dataexpert.io/lesson/qa-week-1-jan2025 Enrolled students The Week 1 Q&A session covers a range of topics crucial for data engineering, focusing on tools like Apache Iceberg, Hive, Trino, and Spark. Participants gain insights into the practical applications of these technologies, including when to use Trino versus Spark for data processing and how to integrate Iceberg with AWS Glue. The discussion also addresses common challenges in data engineering, such as managing large datasets and the importance of understanding data lakes versus data warehouses. ### End-to-End AI Applications Day2 Lecture URL: https://www.dataexpert.io/lesson/endtoendaiapplicationsday2lecture-mar26 Enrolled students End-to-end AI applications require a careful balance of human and AI involvement, especially in high-stakes scenarios. This lecture covers essential concepts such as the importance of human oversight, the role of data quality in AI performance, and the distinctions between cloud-managed agents and custom solutions. Participants will gain insights into building effective AI workflows, managing risks, and leveraging AI's capabilities while maintaining accountability and quality control in their projects. ### Q&A with Zach Week4 URL: https://www.dataexpert.io/lesson/qawithzachweek4-jan2026 Enrolled students A live Q&A session with Zach focused on various topics related to data engineering, including job opportunities, interview strategies, and technical discussions about tools like DBT, Airflow, and Delta tables. Participants shared their experiences with job interviews and capstone projects, while Zach provided insights on navigating the job market and improving data quality. This session is beneficial for aspiring data engineers looking to enhance their skills and understand industry expectations. ### Capstone Project Day 1 Lab URL: https://www.dataexpert.io/lesson/capstone-project-day-1-lab-april2025-p735-l1052 Module: Week 8: End-to-End Project Hacking · Enrolled students In this lab, Zach dives into our current work with Airflow and how we're building queries for our data projects. He also discusses the importance of creating unique IDs for our data and maintaining them effectively. By next class, we should have a solid project that addresses many questions, and he encourages everyone to engage with the upcoming Kafka and Spark sessions. ### Optimize your dev workflow with Claude Code and Codex Day1 Lecture URL: https://www.dataexpert.io/lesson/optimizeyourdevworkflowwithclaudecodeandcodexday1lecture-mar26-p737-l2141 Module: Week 2: Advanced RAG, Reranking & Dev Workflow · Enrolled students In this video, Kevin introduces to AI Coding and Context Engineering course, exploring the evolution of programming and the importance of embracing AI as a natural progression in the field. He highlights the need for context engineering to maximize AI's effectiveness in coding. Kevin discusses the significance of planning, executing, and validating work with AI, emphasizing that coding tools are here to stay and are rapidly improving. ### How to build job orchestration that matters URL: https://www.dataexpert.io/lesson/howtobuildjoborchestrationthatmatters-jan2026-p735-l1906 Module: Week 6: Airflow & Pipeline Specs with Trino · Enrolled students In this lecture, Zach dives into the concept of orchestration and its critical role in data pipelines, emphasizing that orchestration is not just about tools like Airflow, but about the entire process from data collection to ROI. He shares insights on common pitfalls in orchestration, such as bad logging and schema changes, and discussed best practices like the right audit publish pattern. He also highlights the importance of idempotency in pipelines to avoid inconsistencies. ### dbt day 2 lab URL: https://www.dataexpert.io/lesson/dbt-day-2-lab Module: DBT · Enrolled students In this video, Bruno demonstrates how to create incremental models and use the merge function in dbt. He explains the concept of incremental models and show how to configure them in dbt. He also demonstrates how to use the merge function to update existing data in a table. Throughout the video, he provides examples and explanations to help the students understand the process. ### Claude Code Onboarding URL: https://www.dataexpert.io/lesson/claude-code-onboarding Enrolled students Learn how to set up Cloud Code for the AI engineering boot camp, including installation via terminal and configuring API keys for use with Anthropic. This lesson is ideal for participants looking to effectively manage their coding environment and understand the limitations of different models like Opus, Sonnet, and Haiku. Gain insights into logging sessions and receiving feedback on prompts, enhancing your overall learning experience. ### Tech Talk - Parth URL: https://www.dataexpert.io/lesson/techtalksweek5-jan2026 Enrolled students Parth discusses the evolving landscape of data engineering in the context of AI advancements, emphasizing the shift towards AI-native systems. He highlights the importance of context engineering over traditional prompt engineering, advocating for a focus on creating rich semantic layers that enhance model performance. This lesson is beneficial for data engineers and analytics professionals looking to adapt to the rapid changes in technology and improve their skills in delivering actionable insights and automation in their work. ### Applying Analytical Patterns: Aggregations and Cardinality Reduction Day 2 Lab URL: https://www.dataexpert.io/lesson/applying-analytical-patterns-day-2-lab-yt-p736-l673 Enrolled students In this lab session, Zach demonstrates various queries and aggregations that can be applied to the bootcamp web events table. The talk emphasizes the significance of reducing cardinality and showcases the use of functions like coalesce and rollup. ### Q&A with Zach Week 4 URL: https://www.dataexpert.io/lesson/q-a-with-zach-week-4-may2025 Enrolled students A live Q&A session with Zach covers various topics related to capstone projects and data engineering concepts. Participants ask questions about using real-time data, project feasibility, and integrating tools like DBT and Grafana. Discussions also touch on the differences between data engineering and backend engineering, as well as best practices for data modeling in both batch and streaming contexts. This session is beneficial for students seeking guidance on their projects and understanding key data engineering principles. ### Managing Unstructured Data - Day 1 Lab URL: https://www.dataexpert.io/lesson/managing-unstructured-data-day-1-lab-may2025-p736-l1194 Module: Week 5: Managing Unstructured Data · Enrolled students In this video, Zach walks viewers through the process of setting up a RAG LLM example lab, with a focus on working with unstructured data. He demonstrates how to create an index using the text embedding model and addresses the transcription of video files to extract searchable vectors. Additionally, the importance of adjusting parameters, such as top K, for effectively querying the vector database is emphasized. ### Test Course URL: https://www.dataexpert.io/lesson/2fe0167a-a85b-4342-884b-82f43cb3166ftestcourse Module: test · Enrolled students ### John Kutay - Director of Engineering at Rippling URL: https://www.dataexpert.io/lesson/tech-talk-4-john-kutay-april2025-p735-l1006 Module: Tech Talks · Enrolled students Change Data Capture (CDC) is a crucial topic for data engineers, and this session features John Kutay, Director of Engineering at Rippling, who shares his extensive experience with CDC. Learn how CDC works by mining database logs to efficiently replicate data changes in real-time, which is essential for maintaining data integrity and performance in transactional systems. This lesson is ideal for data engineers and analytics professionals looking to enhance their understanding of CDC implementations and best practices. ### Advance Snowflake Day 2 Lecture URL: https://www.dataexpert.io/lesson/advancesnowflake day2lecture-jan2025 Enrolled students In this lecture, Zach dives into the exciting realm of context engineering and its evolution from traditional data engineering. He explores the nine different types of context that data engineers must understand, emphasizing the importance of business context, organizational context, and semantic context. Zach shares insights on how to create rich contexts from unstructured data and the role of vectors in this process. ### TA Office hour 8 URL: https://www.dataexpert.io/lesson/taofficehour8-jan2026-p735-l1946 Enrolled students Office hours provide an opportunity for students to ask questions and seek clarification on course material and assignments. Key topics discussed include submission deadlines for homework and capstone projects, scheduling additional support sessions, and specific queries related to analytics engineering projects. Students also inquire about using APIs, data modeling, and Snowflake connections, ensuring they understand the requirements and tools needed for their assignments. ### Chat with Zach [Nov 15, 2023] URL: https://www.dataexpert.io/lesson/zach-qa-2023-11-15-v3 Enrolled students Delve into the world of data engineering with our weekly Q&A / Fireside Chat series, featuring the founder of EcZachly Inc and lead instructor of the Advanced Data Engineering Bootcamp. Each session covers a different topic, providing invaluable insights and real-world experiences to empower data enthusiasts in their journey. (Recorded Nov 15, 2023) ### Saturday Quiz URL: https://www.dataexpert.io/lesson/saturdayquiz-d2fd4 Enrolled students ### Complex Data Types Lab URL: https://www.dataexpert.io/lesson/complex-data-types-lab-p735-l584 Module: Bonus · Enrolled students In this lab, we explore Slowly Changing Dimensions (SCDs) and design Cumulative Tables using the nba_players dataset. This is our first stress test of our web-based interface linked to a Presto/Trino database running in Starburst. Presto (formerly PrestoSQL) or Trino are both query engines that allow users to query and analyze data in a data lake using SQL, while Starburst offers a distribution of Presto/Trino. ### Sunday Quiz URL: https://www.dataexpert.io/lesson/sundayquiz-29862 Enrolled students ### TA Office hour 6 URL: https://www.dataexpert.io/lesson/taofficehour6-7abba Enrolled students This office hours session focuses on addressing student queries related to multi-factor authentication (MFA) issues with DBT commands and Snowflake connectivity. Participants discuss solutions for bypassing MFA prompts, including the use of RSA keys and alternative authentication methods. The session also covers capstone project expectations, including the use of live APIs and project proposals, providing students with guidance on their assignments and technical challenges. ### TA Office hours 1 URL: https://www.dataexpert.io/lesson/taofficehours1-mar26 Enrolled students Office hours provide a platform for students to seek guidance on their coursework and technical issues. This session covers topics such as API key management, troubleshooting GitHub token errors, and capstone project requirements. Students can benefit from personalized support in resolving technical challenges and understanding submission criteria for assignments and certifications. ### Applying Analytical Patterns: Recursive CTEs and Window Functions Day 2 Lecture URL: https://www.dataexpert.io/lesson/applying-analytical-patterns-day-2-lecture-yt-p736-l672 Free In this presentation, Zach explores a range of topics related to Advanced SQL Analytical Patterns. The discussion covers the application of recursive CTEs, count case when, grouping sets, cross join unnest, cube, rollup, and window functions. ### Thursday Quiz URL: https://www.dataexpert.io/lesson/thursdayquiz-4d179 Enrolled students ### Structured Streaming Kafka to Delta Live Table Day2 Lab URL: https://www.dataexpert.io/lesson/structuredstreamingkafkatodeltalivetableday2lab-feb26 Enrolled students In this video, Zach walks you through the Delta Live Table (DLT) pipeline, focusing on the Medallion architecture with Bronze, Silver, and Gold layers. He sets up data quality checks and explores how to quarantine problematic data, specifically addressing null country values that arose during processing. Zach demonstrates how to create alerts for data quality issues and discusses the importance of scheduling pipeline runs efficiently to manage costs. Zach reminds everyone to review the data quality checks and alerts to enhance their data processing workflows. ### TA Office Hour 5 (2025-10-09) URL: https://www.dataexpert.io/lesson/ta-office-hour5-oct2025-p737-l1540 Enrolled students ### Q&A Week 4 URL: https://www.dataexpert.io/lesson/qa-week-4-jan2025 Enrolled students A live Q&A session covers various topics related to Spark, data engineering, and the boot camp experience. Participants ask questions about Spark job optimization, data processing techniques, and integration with tools like Snowflake and Airflow. The instructor shares insights on real-world applications, including experiences from alumni who secured roles at major tech companies. This session is beneficial for students seeking clarity on complex concepts and practical advice for their data engineering careers. ### End-to-End AI Applications Day1 Lecture URL: https://www.dataexpert.io/lesson/endtoendaiapplicationsday1lecture-mar26 Enrolled students In this lecture, Zach talks about end to end AI apps, focusing on MLOps, evaluations, and reliability. He explains where agents break, like user behavior drift, code and tool changes via MCP, model upgrades, RAG data changes, and even tool or web downtime, plus latency and context window issues. He breaks down unit tests, integration tests, and end to end evaluation, and four evaluation styles, cosine distance, ROUGE, exact match, custom scoring, and LLM as a judge. ### dbt basics: How dbt works: sources, models, packages Lecture URL: https://www.dataexpert.io/lesson/dbt-basics-how-dbt-works-sources-models-packages-lecture-952 Module: Week 1: dbt Basics · Enrolled students In this lecture, Bruno delves into the intricacies of dbt, focusing on its role in data engineering and best practices. He explains how dbt simplifies data transformation processes, emphasizing the importance of software engineering practices in data projects. Students are encouraged to grasp the significance of dbt in ETL pipelines and its integration with various platforms. He highlights key features like models, seeds, and sources, offering insights into how dbt streamlines data workflows. ### The Data Structures and Algorithms Interview URL: https://www.dataexpert.io/lesson/the-data-structures-and-algorithms-interview Module: Interview Skills · Free This video covers the main things you need to remember to pass the Data structures and Algorithms interview ### Tech Talk - Shubham URL: https://www.dataexpert.io/lesson/tech-talk-shubham-may2025 Enrolled students The Tech Talk session features Shubham, a senior data engineer at Amazon, who shares insights from his extensive career in data engineering and analytics. He discusses his journey from Deloitte to PayTM and then to Amazon, emphasizing the importance of adaptability and strategic planning in a fast-paced environment. Shubham also highlights the evolving role of data engineers, the significance of mentorship, and the impact of AI on workflow efficiency, making this session valuable for aspiring data professionals looking to navigate their careers effectively. ### Q&A with Zach Week 1 URL: https://www.dataexpert.io/lesson/qa-with-zach-week-1-may2025 Enrolled students The first Q&A session of the boot camp covers a range of topics relevant to data engineering, including techniques for validating pipeline performance and data quality, the relevance of functional programming, and strategies for handling late arriving data. Participants will benefit from insights on creating item potent pipelines, effective cross-functional leadership, and the differences between data engineering and analytics roles. This session is ideal for those looking to deepen their understanding of practical data engineering challenges and solutions. ### Advanced dbt Day 3 Lecture URL: https://www.dataexpert.io/lesson/advanced-dbt-day-3-lecture-jan2025 Enrolled students In this lecture, Bruno dives into advanced techniques for running DBT projects, particularly focusing on integrating with Airflow. He discusses the importance of selecting specific models and the implications of running stale data. He also covers the benefits of using CI/CD for testing changes in isolated schemas to avoid merging bad code. ### OpenClaw and Automating business processes Day1 Lecture URL: https://www.dataexpert.io/lesson/openclawandautomatingbusinessprocessesday1lecture-mar26 Enrolled students In this lecture, Carly dives into what OpenCLAW is. She discuss its rapid growth to 250,000 GitHub stars in just two months and set it up in a safe Docker environment ### MCP and Agentic AI Day 2 Lecture URL: https://www.dataexpert.io/lesson/mcp-and-agentic-ai-day2-lecture-oct2025 Enrolled students In this lecture, Zach dives deeper into the Model Context-X protocol, focusing on the importance of middleware in keeping AI agents safe and effective. He discusses how to optimize for a happy path with MCP agents, when to use middleware, and how to avoid reinventing the wheel with REST and MCP servers. ### Alex Merced - Head of DevRel at Dremio URL: https://www.dataexpert.io/lesson/tech-talk-alex-may2025-p736-l1081 Module: Guest Speakers · Enrolled students Alex Merced, Head of DevRel at Dremio, shares insights on the evolving landscape of data engineering, focusing on the significance of Apache Iceberg and Lake House architecture. He discusses the interoperability of various data tools and the role of catalogs in managing metadata effectively. This session is beneficial for data engineers and professionals looking to enhance their understanding of modern data architectures, including the integration of streaming data and the advantages of using open-source technologies. ### Day 2 Lecture: Types of Agents (reflection, plan-and-execute, rewoo) URL: https://www.dataexpert.io/lesson/advanced-rag-and-agentic-ai-day2-lecture-july2025-p737-l1181 Module: Week 2: Advanced RAG, Reranking & Dev Workflow · Enrolled students Understanding AI agents is crucial for those looking to leverage advanced technologies in data and AI engineering. This lecture covers the definition and architecture of AI agents, including types such as reflection, plan-and-execute, and rewoo. Participants will learn about the components of AI agents, including planning, reasoning, and memory, as well as the importance of tool access and feedback loops in enhancing agent performance. The session also introduces industry trends and frameworks relevant to AI agents, providing a comprehensive overview for aspiring engineers. ### Special Guest, Joe Reis [Recorded Dec 8, 2023] URL: https://www.dataexpert.io/lesson/v3-3-special-guest-joe-reis-recorded-dec-8-2023-2760 Enrolled students In this captivating speaker series, Joe Reis delivers an insightful talk on the pivotal role of data engineering in facilitating informed decision-making. With a profound understanding of the challenges data engineering addresses, Joe shares valuable suggestions drawn from his extensive experience. Delving into the historical evolution of the data industry, he underscores the critical importance of navigating the vast landscape of data. Whether you are a seasoned data professional or someone keen on understanding the impact of data engineering on decision-making, this talk offers a wealth of knowledge and expertise. Dive into this special session for an enriching exploration of the dynamic intersection between data, engineering, and strategic decision-making.[Recorded on Dec 8th, 2023] ### Dimensional Data Modeling - Slowly-changing Dimensions and Idempotent Queries in Iceberg (Day 2 Lecture) - Pre-recorded from V3 URL: https://www.dataexpert.io/lesson/dimensional-data-modeling-day-2-lecture-v4 Module: Week 1: Dimensional Data Modeling in Iceberg · Enrolled students This is a pre-recorded lesson from V3. In this presentation, we delve into the concept of idempotent pipelines, exploring their significance, advantages, and potential drawbacks. We provide insights into Slowly Changing Dimensions (SCDs) and discuss the various types. Discover whether SCDs are inherently idempotent or not and grasp the implications of their usage in data pipelines. Join us to gain a clear understanding of these foundational concepts in data processing. [Recorded Nov 8, 2023] ### Career Dev Session 2 - LinkedIn like a Pro! URL: https://www.dataexpert.io/lesson/career-dev-session2-oct2025-p737-l1529 Module: Career Development · Enrolled students ### Nextjs Introduction URL: https://www.dataexpert.io/lesson/nextjs-introduction Module: Advanced Frontend Development · Enrolled students Next.js is a powerful full-stack framework built on React that simplifies the development of modern web applications. This lesson covers essential concepts such as server-side rendering, file-based routing, and code optimization techniques that enhance performance and SEO. Ideal for frontend developers looking to deepen their understanding of Next.js, it also addresses the integration of client and server components, providing insights into best practices for building efficient web applications. ### Day 2 Lecture: Understanding LLM Reasoning traceability URL: https://www.dataexpert.io/lesson/ml-ops-day2-lecture-july25-p737-l1186 Module: Week 4: MLOps, Guardrails & Business Value · Enrolled students Understanding LLM reasoning traceability is essential for evaluating and optimizing AI applications in production. This lesson covers key concepts such as tracing, evaluation techniques, and the importance of guardrails when deploying models. Participants will learn how to assess model performance, address production challenges, and utilize tools like Lang Smith for effective monitoring and evaluation, benefiting data engineers and AI practitioners looking to enhance their applications. ### Tech Talk - Joe Reis URL: https://www.dataexpert.io/lesson/v4-tech-talk-joe-reis-3537 Enrolled students In this Tech Talk with Joe Reis, he discusses the importance of data modeling and its impact on data management. He also explains the different levels of data modeling (conceptual, logical, and physical) and emphasize the need for a well-designed data model to ensure data accuracy, clarity, and ease of use. He touches on the drawbacks of not implementing proper data modeling and provide real-life examples to illustrate the concepts. [Recorded on May29th, 2024] ### Airflow Data Quality (Day 2 Lab) URL: https://www.dataexpert.io/lesson/airflow-data-quality-day-2-lab-v4 Module: Week 3: Airflow and Airflow Data Quality · Enrolled students In this lab, Zach explains how automated data quality checks work. He discusses the process of running checks on an output table, ensuring that certain columns are not null, and executing Trino queries to validate the results. He also covers the steps involved in inserting and cleaning up data, as well as the challenges we faced with permissions. [Recorded on May23rd, 2024] ### Thursday Quiz URL: https://www.dataexpert.io/lesson/thursdayquiz-4b3ee Enrolled students ### Monday SQL Quiz URL: https://www.dataexpert.io/lesson/sql-growth-accounting-quiz Enrolled students ### Spark Fundamentals Spark Lab on REST API consumption (Day 1 Lab) URL: https://www.dataexpert.io/lesson/spark-batch-day-1-lab-v4 Module: Week 4: Batch Pipelines with Apache Spark · Enrolled students This is new content and will be recorded on May27th 2024. ### Building functional AI agents using LangChain Day2 Lecture URL: https://www.dataexpert.io/lesson/buildingfunctionalaiagentsusinglangchainday2lecture-mar26-p737-l2139 Module: Week 1: Prompt Engineering & Basic RAG · Enrolled students In this lecture, Zach builds on prior discussions, focuses on why agents fail, how to provide the right context, and when to use multi-agent architectures. He introduces new endpoints to the Data Expert API that help participants create a functioning version of Zach GPT. He encourages attendees to consider the difference between false positives and false negatives and to share their thoughts in the chat. ### Incorporating Evaluation Criteria into CI/CD URL: https://www.dataexpert.io/lesson/incorporating-evaluation-criteria-into-cicd-oct2025 Enrolled students ### RAG and LLMs Day2 Lab URL: https://www.dataexpert.io/lesson/rag-and-llms-day-2-lab-jan2025 Enrolled students In this lab, Zach walks you through some key concepts in natural language processing, particularly focusing on lemmatization and its importance in our projects. He also discusses how to manage environment variables and the integration of OpenAI keys into our repository. ### Deploying AI agents with Agent Bricks Day1 Lab URL: https://www.dataexpert.io/lesson/deployingaiagentswithagentbricksday1lab-aug26 Enrolled students In this lab, Eumar covers the key workflow for packaging an AI agent (Zachy) and deploying it with Databricks services. The session focuses on showing how to wrap Zachy in an MLflow ResponsesAgent-based Python implementation, smoke test it, log and register the model in Unity Catalog, and then query the deployed endpoint through AI Gateway. He also demonstrates deploying the chat UI app that connects to the deployed endpoint and stores chat history in LakeBase. Finally, he briefly reviews MLflow tracing and LLM-as-a-judge evaluation and MLflow labeling sessions for user feedback. ### Gourav Khanijoe - Engineering Lead at HubSpot URL: https://www.dataexpert.io/lesson/tech-talk-with-gourav-khanijoe-p735-l591 Module: Tech Talks · Enrolled students Gourav Khanijoe, a staff engineer at HubSpot, shares insights on the journey to becoming a staff engineer, emphasizing the importance of scaling oneself and tackling critical organizational problems. He discusses the significance of building relationships with leadership to identify impactful projects and highlights the shift from complexity to simplicity in problem-solving. This session is beneficial for aspiring engineers looking to advance their careers and understand the nuances of engineering leadership. ### AI Engineering Bootcamp Kick off URL: https://www.dataexpert.io/lesson/aiengineeringbootcampkickoff-mar26-p737-l2224 Module: Kickoff & Setup · Enrolled students ### Day 2 Lab: Enhancing RAG with Agentic AI URL: https://www.dataexpert.io/lesson/advanced-rag-and-agentic-ai-day2-lab-july2025-p737-l1182 Module: Week 2: Advanced RAG, Reranking & Dev Workflow · Enrolled students Enhancing Retrieval-Augmented Generation (RAG) with Agentic AI focuses on building a multi-modal research agent using Langchain and OpenAI tools. Participants will learn to deploy a research agent that performs web searches, analyzes YouTube transcripts, and generates reports and podcasts. This lesson is ideal for data engineers and AI practitioners looking to enhance their workflows with advanced RAG techniques and agent-based architectures. ### Injecting quality into your Airflow DAG Lab URL: https://www.dataexpert.io/lesson/injectingqualityintoyourairflowdaglab-jan2026-p735-l1907 Module: Week 6: Airflow & Pipeline Specs with Trino · Enrolled students In this video, Zach walk you through the setup and execution of a simple data pipeline using Airflow, focusing on reading data from Kafka and storing it in a production table. He covers key components like ExecutionTimeout, MaxActiveRuns, and the importance of data quality checks. He demonstrates how to handle missing data and ensure our pipeline is idempotent, meaning it won't create duplicates when rerun. He also highlights the significance of staging tables and the write-audit-publish pattern for maintaining data integrity. ### Azure - Week 3 URL: https://www.dataexpert.io/lesson/azure-week-3-may2025 Enrolled students Understanding cloud databases is essential for data engineers working with Azure. This session covers the differences between structured, semi-structured, and unstructured data, and how to choose appropriate storage solutions like Azure SQL Database, NoSQL databases, and cloud storage options. Participants will learn about database creation, firewall settings, and the importance of Azure Active Directory for authentication, along with recent developments like the integration of OLTP and OLAP in Lake House architectures. ### TA Office hour 2 URL: https://www.dataexpert.io/lesson/taofficehour2-jan2026-p735-l1940 Enrolled students This office hour session provides an interactive platform for students to ask questions about the data and AI engineering boot camp. Participants can seek clarification on topics like onboarding processes, assignment submissions, and project requirements, including the use of tools such as Airflow and Astronomer. The session also addresses specific project concerns, allowing students to discuss their capstone projects and receive guidance on data handling and task management within their assignments. ### Monday Data Engineer Interview Quiz URL: https://www.dataexpert.io/lesson/mondaydataengineerinterviewquiz-39afc Enrolled students ### Apache Flink Streaming - Setting Up Streaming Pipelines with Kafka Integration (Day 1 Lab) URL: https://www.dataexpert.io/lesson/infra-flink-streaming-day-1-lab-v3 Module: Week 3: Streaming Pipelines with Apache Flink · Enrolled students In this lab session, you will learn the ropes of setting up and configuring streaming pipelines with Apache Flink. Write a job that connects to Kafka, filter out unwanted events, and write the relevant data to Postgres. ### Apache Iceberg Lecture Day2 URL: https://www.dataexpert.io/lesson/apacheiceberglectureday2-bc90f Enrolled students Understanding data lake architectures is crucial for data engineers aiming to transition from building simple pipelines to designing comprehensive data solutions. This lesson covers key concepts such as agnostic storage, Lambda and Kappa architectures, and the implications of file compaction and row-level updates in Apache Iceberg. Participants will learn about the trade-offs between hot, warm, and cold storage, and how to optimize data retrieval and processing efficiency, making it ideal for aspiring data engineers and analytics professionals. ### Fact Data Modeling - Core Concepts, Deduplication Techniques, and Retention Considerations with Iceberg (Day 1 Lab) - Pre-recorded from V3 URL: https://www.dataexpert.io/lesson/fact-data-modeling-day-1-lab-v4 Module: Week 2: Fact Data Modeling in Iceberg · Enrolled students This is pre-recorded from V3. In this hands-on lab session, we analyze NBA game data, unraveling the intricacies of data modeling and analysis. Breaking down the data into distinct columns, we elucidate the importance of each element. Through practical demonstrations, we showcase how to execute queries and perform joins, extracting meaningful insights from the dataset. Throughout the video, valuable tips are shared on data modeling, query optimization, and the significance of empathy when collaborating with stakeholders. Tune in to this video to acquire practical skills in NBA game data analysis and learn essential techniques for effective data modeling and analysis.[Recorded Nov 14, 2023] ### Mastering Apache Spark Day1 Lab URL: https://www.dataexpert.io/lesson/masteringapachesparkday1lab-aug26 Enrolled students In this lab, Zach explains how Spark join types and shuffle behavior affect performance, using Spark UI, query plans, and Delta table experiments. It covers ingesting approximately 200,000 rows from Polygon/Massive into a Delta table, then examines an intraday self-join for best and worst trading days for Apple, including how broadcast joins can avoid shuffle but window functions with order by introduce additional sort and shuffle. He also compares repartition versus coalesce and different partition strategies on a 38 million row tweets dataset, showing how partition counts and defaults like auto change runtime, and discusses skew versus shuffle with adaptive query execution. He concludes with guidance on when repartitioning is likely unnecessary for well-modeled Delta inputs and when it can help for less structured sources. ### Managing Unstructured Data - Day 2 Lab URL: https://www.dataexpert.io/lesson/managing-unstructured-data-day-2-lab-may2025 Enrolled students In this session, Zach explores how to process unstructured data effectively and generate structured outputs using AI. He demonstrates creating a schema for code review feedback, a crucial aspect of enhancing collaboration and quality in software development. He also highlighted the integration of this data into Iceberg and the use of GitHub for analysis. ### Book Club 4 URL: https://www.dataexpert.io/lesson/book-club4-july2025 Enrolled students The session focuses on Chapter 10, which covers AI engineering, architecture, and user feedback. Participants engage in discussions about creating AI systems that enhance user experience, emphasizing the importance of understanding user needs over wants. Key topics include the balance between automation and human oversight in AI systems, the significance of user joy in product design, and the challenges of ensuring user privacy while monitoring AI models in sensitive fields like healthcare and finance. ### Q&A with Zach Week 5 (2025-08-08) URL: https://www.dataexpert.io/lesson/qa-with-zach-week5-july25-p737-l1386 Module: Q&A with Zach · Enrolled students A live Q&A session with Zach covers a range of topics related to AI engineering, including insights on using tools like NAN and Ollama for building AI applications. Participants discuss the importance of understanding AI fundamentals, the role of data engineering, and the significance of networking and personal branding on platforms like LinkedIn. Zach emphasizes the value of effort in learning and the potential for AI to enhance personalized education and business processes. ### Mastering Change Data Capture (CDC) URL: https://www.dataexpert.io/lesson/masteringchangedatacapturecdc-jan2026-p735-l1902 Module: Week 7: Analytical Patterns & CDC · Enrolled students In this session, Zach dives into Change Data Capture (CDC) and explored why Snowflake is so popular among SQL users for its ease of use and ability to handle big data efficiently. He discusses the differences between Snowflake and Databricks, emphasizing that each serves different user needs—analysts versus engineers. He also highlights the importance of Merge operations and when they can be beneficial or problematic. He also covers various methods for implementing CDC, including read streams and database triggers. ### Orchestration and Airflow Fundamentals Day 1 Lecture URL: https://www.dataexpert.io/lesson/orchestration-and-airflow-fundamentals-day-1-lecture-jan2025 Enrolled students In this lecture, Zach breaks down the key aspects of orchestration, focusing on why it's crucial to understand the orchestration process for effective data engineering. He covers what orchestration isn't, the role it plays in data pipelines, and highlights common mistakes to avoid for smooth orchestration. ### Capstone May 2025 URL: https://www.dataexpert.io/lesson/capstone-may-2025-p736-l1139 Module: Capstone Project · Enrolled students ### Big Picture AI Applications Lecture URL: https://www.dataexpert.io/lesson/building-end-to-end-ai-application-day2-lecture-oct2025-p737-l1521 Module: Week 5: End-to-End AI Applications · Enrolled students In this lecture Zach discusses about AI applications and stresses the critical importance of understanding business requirements before diving into any project. He recounts a cautionary tale from his time at Netflix, where a $25 million initiative collapses because the team over‑promised and under‑delivered, underscoring that 80 % of AI projects fail. Zach urges the students to invest time in the foundational layers of development, warning that rushing leads to costly mistakes. He introduces the concept of “human in the loop,” explaining when it becomes essential—particularly in high‑risk scenarios. ### Jess Ramos - Data Analyst URL: https://www.dataexpert.io/lesson/tech-talk-1-jess-ramos-april2025-p735-l989 Module: Tech Talks · Enrolled students Jess Ramos shares her insights as a seasoned data analyst in a candid Q&A session, discussing the challenges of job transitions in today's market. She emphasizes the importance of building a strong professional network and diversifying income streams for job security. The conversation also covers effective strategies for collaborating with data engineers and analytics engineers, highlighting the nuances of project management and the significance of clear communication in avoiding scope creep. ### Delta Table Day 1 Lecture URL: https://www.dataexpert.io/lesson/delta-table-day-1-lecture-may2025 Enrolled students In this lecture, Yared dives into the key concepts of Delta Lake and its significance in data engineering. He discusses the differences between Delta Lake and traditional data formats like CSV and JSON, emphasizing the importance of performance and data management. ### Q&A with Zach Week 5 URL: https://www.dataexpert.io/lesson/q-a-with-zach-week-5-may2025-p736-l1197 Module: Q&A with Zach · Enrolled students A live Q&A session with Zach covers a range of topics related to data engineering and AI, focusing on techniques for chunking video transcriptions, the use of overlapping chunks, and the importance of context in LLMs. Participants ask questions about streaming data with Kafka, integrating batch and streaming processes, and strategies for handling PDF documents in AI applications. The session also touches on advanced concepts like change data capture and the challenges of maintaining data integrity in real-time systems. ### TA Office Hour 2 URL: https://www.dataexpert.io/lesson/ta-office-hour-2-jan2025 Enrolled students The office hour begins from Minute 13. ### Apache Iceberg Lab Day1 URL: https://www.dataexpert.io/lesson/apacheiceberglabday1-cbfc8-p735-l1992 Module: Week 5: Data Modeling on Iceberg · Enrolled students Explore the foundational concepts of Apache Iceberg in this lab session focused on partitioning, snapshotting, and time travel. Participants will learn how to create and manage tables using Trino, understand the significance of metadata in Iceberg, and implement efficient query practices, including Sargable queries for optimal performance. This lesson is ideal for data engineers and analysts looking to enhance their skills in data lake management and efficient querying techniques. ### Q&A with Zach Week 1 URL: https://www.dataexpert.io/lesson/qa-with-zach-week1-july2025 Enrolled students The Week 1 Q&A session covers various topics related to the AI Engineering boot camp, including auto prompt optimization and vibe coding. Participants can ask questions about capstone projects, focusing on implementing RAG models and utilizing unstructured data. The discussion also touches on the importance of self-guided learning and the expectations for assignments and certifications, providing valuable insights for both new and returning students in the program. ### Career Dev Session 2 - LinkedIn like a Pro! URL: https://www.dataexpert.io/lesson/career-dev-session2-oct2025 Enrolled students ### TA Office Hour 4 URL: https://www.dataexpert.io/lesson/ta-office-hour-4 Enrolled students Office hours provide a platform for students to ask questions and seek guidance on their assignments. Participants discuss various challenges they face, such as running Spark applications, creating folders for data uploads, and issues with querying tables. The session emphasizes hands-on troubleshooting and offers insights into common mistakes in homework assignments, particularly regarding data generation and usage. This interactive format benefits students needing clarification and support in their coursework. ### Advanced dbt: Jinja, Macros and variables Lab URL: https://www.dataexpert.io/lesson/advanceddbtday1lab-jan2026-p735-l1922 Module: Week 2: Advanced dbt · Enrolled students Explore advanced dbt techniques in this lab focused on Jinja, macros, and incremental models. Learn how to define and utilize macros for dynamic SQL generation, manage variables, and implement incremental models to optimize data processing in Snowflake. This lesson is ideal for data engineers looking to enhance their dbt skills and streamline their data workflows. Gain hands-on experience with practical examples and debugging strategies to effectively manage your data transformations. ### Vin Vashishtha - Author of From Data to Profit URL: https://www.dataexpert.io/lesson/tech-talk-6-vin-vashishtha-april2025-p735-l1053 Module: Tech Talks · Enrolled students Vin Vashishtha, a seasoned expert in data science and AI, shares insights on transforming data from a cost center into a profit-generating asset. He discusses the importance of early involvement in projects, effective communication with C-suite executives, and the necessity of understanding business needs to enhance the impact of analytics engineering. This session is beneficial for professionals looking to elevate their roles in data and analytics by aligning technical work with strategic business goals. ### TA Office Hour 3 (2025-07-23) URL: https://www.dataexpert.io/lesson/ta-office-hour-3-july2025-p737-l1298 Enrolled students This office hours session covers various topics related to data engineering assignments, focusing on creating sparse and dense indexes in vector databases. Students discuss challenges in indexing collections, chunking files for processing, and using repositories for data retrieval. The session also addresses specific coding issues and strategies for effectively managing assignments, making it beneficial for learners seeking clarification on practical implementation and troubleshooting in their projects. ### test 3 URL: https://www.dataexpert.io/lesson/test3-78cf9 Enrolled students ### Apache Spark Shuffle Joins Day 1 Lecture URL: https://www.dataexpert.io/lesson/apache-spark-shuffle-joins-day-1-lecture-may2025 Enrolled students In this lecture, Zach discusses how we handle two petabytes of data daily at Netflix, focusing on different sampling techniques to optimize processing. He shares insights on the importance of precision in data analysis and how we managed to reduce processing time and costs significantly by using a 0.1% sample. He also touches on the challenges of dynamic IP addresses in our cloud environment and the need for collaboration with application owners to implement effective logging. ### Building AI Agents with Databricks Day2 Lab URL: https://www.dataexpert.io/lesson/buildingaiagentswithdatabricksday2lab-feb26 Enrolled students In this video, Eumer walks you through the implementation of the app during the lab session, highlighting key aspects of the Supervisor agent and the importance of on-behalf user authorization for security. He also demonstrates how to create connections for external APIs and manage permissions effectively. He encourages everyone to complete Lab 4 before moving on to the next sessions, as it lays the groundwork for the upcoming tasks. Additionally, he shares insights on how to register models in Unity Catalog and the significance of monitoring performance through tracing and evaluation. ### Advanced pipelines: Blue-green and WAP with Airflow Lecture URL: https://www.dataexpert.io/lesson/advanced-pipelines-blue-green-and-wap-with-airflow-lecture-p735-l570 Module: Week 2: Advanced dbt · Enrolled students In this lecture,Bruno delves into the intricacies of DBT pipelines, focusing on structuring pipelines, selecting project components, and understanding downstream dependencies. He explains the importance of organizing models in folders for efficient maintenance and debugging. Additionally, he highlights the significance of manifest files in tracking model changes and ensuring data integrity. ### Day 1 Lecture: Graph RAG vs Vector RAG URL: https://www.dataexpert.io/lesson/advanced-rag-and-agentic-ai-day1-lecture-july2025 Enrolled students In this lecture, Ashwani provides an overview of vector retrieval augmented generation (RAG) and introduce knowledge graphs, highlighting their evolution and significance in enhancing search capabilities. He discuss the limitations of vector RAG, particularly in handling deep relationship queries and fragmented context, and how knowledge graphs can address these challenges. He also explains the process of constructing knowledge graphs from structured and unstructured data, emphasizing the importance of defining schemas for effective graph creation. Lastly, he outlines the retrieval patterns and strategies for leveraging knowledge graphs. ### Apache Spark: Architecture, Optimization, and Best Practices Day 1 Lecture URL: https://www.dataexpert.io/lesson/apache-spark-day-1-lecture-yt-p736-l678 Enrolled students Explore the essentials of Spark in this video, uncovering key insights and strategies to enhance your skills. We delve into Spark's architecture, distinguish it from technologies like Hive, and offer best practices for job optimization. ### Dimensional Data Modeling Complex Data Type and Cumulation Day 1 Lab URL: https://www.dataexpert.io/lesson/dimensional-data-modeling-lab-day-1-yt-p736-l659 Free In this video will be building a cumulative table design (https://github.com/EcZachly/cumulative-table-design) with complex data types STRUCT and ARRAY. We will take the table player_seasons and create a new table called players that holds onto an historical record of all NBA players and the seasons that they played! ### Xinran Waibel - Data Engineer at OpenAI URL: https://www.dataexpert.io/lesson/techtalks4-feb26-p736-l2045 Module: Guest Speakers · Enrolled students Gain insights from Shenron Waibel, a data engineer at OpenAI, as he discusses the evolving landscape of data engineering in the age of AI. This session covers the shift from traditional coding practices to leveraging AI tools like Codex for enhanced productivity. Shenron emphasizes the importance of adaptability, continuous learning, and the human element in decision-making, making this lesson invaluable for aspiring data engineers looking to navigate the future of their field. ### dbt Basics Day 2 Lab URL: https://www.dataexpert.io/lesson/dbt-basics-day-2-lab-jan2025-p736-l774 Enrolled students In this lab, Bruno walks you through the setup of our Airflow DBT project, focusing on the installation of dbt core, dbt snowflake, and sqlfluff. He also demonstrates how to define models and organize your project structure effectively. Please make sure to follow along and set up your environment as discussed. ### Databricks and Advanced Spark Day2 Lecture URL: https://www.dataexpert.io/lesson/databricksandadvancedsparkday2lecture-feb26 Enrolled students In this lesson, Zach focuses on the five key reasons why Spark jobs can be slow, including data model bottlenecks and job misconfigurations. He emphasizes the importance of processing only the necessary data, advocating for incremental refresh strategies over full dataset refreshes, which can lead to significant performance improvements. Additionally, Zach discusses the impact of source file formats and the need for proper configurations to avoid congestion and misconfigurations. ### January 2025 Bootcamp Kickoff URL: https://www.dataexpert.io/lesson/bootcamp-kickoff-jan2025-p736-l709 Module: Kickoff · Enrolled students In this video, Zach will be sharing some exciting updates and important information about the January 2025 Bootcamp. He will be discussing topics like the capstone project, policies, refund policies, and the mentorship program. He will also introduce some of the amazing mentors and TA's who will be guiding you throughout the bootcamp. So get ready for an immersive learning experience and let's learn together. ### Career Dev Session 3 URL: https://www.dataexpert.io/lesson/careerdevsession3-jan2026 Enrolled students Career development focuses on mastering interview techniques essential for success in any industry. Participants will learn how to effectively respond to common interview questions, such as "Tell me about yourself" and "Why do you want to work here," while emphasizing the importance of soft skills and preparation. This session is ideal for job seekers looking to enhance their interview performance and make a strong impression on potential employers. ### TA Office Hour 2 URL: https://www.dataexpert.io/lesson/ta-office-hour-2-april2025-p735-l985 Enrolled students Office hours provide a platform for students to discuss homework and clarify concepts related to data engineering and cloud technologies. Key topics covered include the use of AWS Glue and Iceberg for data processing, as well as the integration of Airflow and DBT for project management. Participants can benefit from shared insights on setting up their environments, understanding API data handling, and troubleshooting common issues in their assignments. ### Advanced ReactJS and Tailwind URL: https://www.dataexpert.io/lesson/0b787442-0857-4ab7-8e32-ea32373842f8advancedreactjsandtailwind Module: Intermediate Frontend Development · Enrolled students Advanced ReactJS and Tailwind teaches developers how to effectively utilize React hooks, including useState and useEffect, to manage component state and lifecycle. The lesson covers the use of the Context API for global state management, along with practical examples of creating reusable components and custom hooks. Additionally, it introduces Tailwind CSS for rapid styling, emphasizing utility-first design principles that enhance productivity and maintain consistency across applications. This lesson is ideal for intermediate frontend developers looking to deepen their React and CSS skills. ### Q&A with Zach Week 3 URL: https://www.dataexpert.io/lesson/q-a-with-zach-week-3-may2025 Enrolled students A live Q&A session with Zach covers various topics related to data engineering, including the use of Databricks, Spark, and Iceberg. Participants ask questions about capstone projects, technology choices, and best practices for SQL and Spark performance. Zach emphasizes the importance of selecting technologies that align with career goals and shares insights on managing workloads and optimizing data pipelines. This session is beneficial for data engineering students seeking practical advice and deeper understanding of industry tools and techniques. ### Wednesday Quiz URL: https://www.dataexpert.io/lesson/wednesdayquiz-9e3a3 Enrolled students ### Fact Data Modeling - Core Concepts, Deduplication Techniques, and Retention Considerations with Iceberg (Day 1 Lecture) URL: https://www.dataexpert.io/lesson/fact-data-modeling-day-1-lecture-v3 Module: Week 2: Fact Data Modeling in Iceberg · Enrolled students In this lecture, we delve into the realm of fact data modeling, focusing on the core concept of facts and addressing the challenges associated with handling duplicates. Our exploration underscores the pivotal role of facts in data engineering, shedding light on their significance. Practical insights into deduplication techniques are provided, and we discuss the ramifications of retention on fact data. Join us in this video to gain a comprehensive understanding of fact data modeling and its crucial implications in the field of data engineering.[Recorded Nov 14, 2023] ### Thursday Quiz URL: https://www.dataexpert.io/lesson/thursdayquiz-c78e7 Enrolled students ### Product Sense Interview URL: https://www.dataexpert.io/lesson/product-sense-interview Module: Interview Skills · Enrolled students In this lecture, Zach delves into the crucial aspects of defining product metrics for success, emphasizing the importance of leading metrics in product sense interviews. He shares insights on the key elements of a successful product, highlighting the significance of user engagement and long-term success. Additionally, he discusses the balance between profitability and user experience, offering valuable tips on identifying success metrics and running effective experiments to optimize product performance. ### Conceptual Data Modeling Lab Day 1 URL: https://www.dataexpert.io/lesson/conceptualdatamodelinglabday1-3ae1d Enrolled students Creating a conceptual data model involves identifying key metrics that define quality of life, such as happiness, health, and productivity. This interactive lab session guides participants through brainstorming and categorizing various metrics, like sleep, exercise, and social interactions, into leading and lagging indicators. Ideal for data engineers and analysts, this lesson emphasizes the importance of structuring data for meaningful insights and decision-making. ### Delta Lake Bonus URL: https://www.dataexpert.io/lesson/delta-lake-bonus-may2025-p736-l1057 Module: Week 3: Data Lakes with Delta Table · Enrolled students Explore the advanced features of Delta Lake in this bonus session, designed for data engineers looking to enhance their skills with Databricks. Learn about key functionalities such as time travel, DML support, and the differences between Delta Lake and other formats like Iceberg and Hoodie. Gain insights into creating and managing Delta tables, handling batch and streaming data, and implementing data quality constraints, all while leveraging the power of metadata for improved performance and reliability. ### TA Office Hour 3 URL: https://www.dataexpert.io/lesson/ta-office-hour-3-april2025 Enrolled students Office hours provide an opportunity for students to ask questions and clarify doubts regarding their assignments. This session focused on the current week's SQL assignment in Snowflake, specifically using common table expressions (CTEs) and user-defined functions (UDFs) to analyze an actor films dataset. Key topics included generating sequences of years to identify inactive periods for actors and leveraging Python for UDFs, along with discussions on project proposals and homework assistance. ### Chat with Zach [Nov 30, 2023] URL: https://www.dataexpert.io/lesson/v3-3-chat-with-zach-nov-30-2023-5466 Enrolled students Delve into the world of data engineering with our weekly Q&A / Fireside Chat series, featuring the founder of EcZachly Inc and lead instructor of the Advanced Data Engineering Bootcamp. Each session covers a different topic, providing invaluable insights and real-world experiences to empower data enthusiasts in their journey. (Recorded Nov 30, 2023) ### Advanced Snowflake Data Modeling and UDFs Lab Day 3 URL: https://www.dataexpert.io/lesson/advanced-snowflake-data-modeling-and-udfs-lab-day-3-april2025 Enrolled students In this lab, Zach dives into how to analyze NBA player performance using data, specifically focusing on players who have had five or more consecutive seasons averaging at least 20 points. He walks through the process of using SQL and Python to extract and manipulate this data, showcasing some interesting streaks, including those of notable players like Steph Curry and LeBron James. ### Databricks and Advanced Spark Day1 Lab URL: https://www.dataexpert.io/lesson/databricksandadvancedsparkday1lab-feb26-p736-l2026 Module: Week 2: Databricks & Advanced Spark · Enrolled students In this lab, Zach walks you through how to work with the Massive API, to extract stock prices for various companies like Apple, Netflix, and MSFT. Zach also highlights the importance of understanding Spark's explain plans to optimize their queries and avoid performance issues, particularly regarding shuffle operations. ### Conceptual Data Modeling Lecture Day 1 URL: https://www.dataexpert.io/lesson/conceptual-data-modeling-lecture-day-1-april2025-p735-l979 Module: Week 3: Data Modeling on Snowflake · Enrolled students In this lecture, Zach discusses the essential steps for building effective data models, especially for those looking to advance in their careers. He shares his experiences and lessons learned from past projects, emphasizing the importance of understanding stakeholder requirements and avoiding false requirements that can complicate data pipelines. He also touches on the significance of balancing technical and business insights in data engineering. ### Lesson 2 URL: https://www.dataexpert.io/lesson/lesson2-c112d Enrolled students Creating an integration between Snowflake and AWS Glue is essential for effective data management in cloud environments. This lesson covers the steps to establish trust between Snowflake and AWS, including creating an Iceberg volume and a Glue integration for metadata management. Data engineers and analysts will benefit from understanding how to set up this integration, allowing seamless querying and data access across platforms like Snowflake, Glue, and Iceberg. ### Analytical Patterns Recognizing Business Value Day 3 Lab URL: https://www.dataexpert.io/lesson/analytical-patterns-recognizing-business-value-day-3-lab-jan2025-p736-l821 Enrolled students Creating a feature store is essential for data engineers looking to enhance machine learning models. This lab focuses on building a feature store using Trino, covering key concepts such as one-hot encoding and the creation of user features from web event data. Participants will learn to generate and manipulate features, including categorical and numerical data, to prepare datasets for predictive modeling, ultimately improving their data engineering skills and understanding of analytical patterns. ### How is LTAP different from OLAP and OLTP URL: https://www.dataexpert.io/lesson/how-is-ltap-different-from-olap-and-oltp-aug26 Enrolled students In this lesson, Zach covers the shift in data engineering from traditional ETL pipelines to leveraging AI agents for real-time data-driven outcomes, introducing LTAP (Lake House Transaction Analytical Processing) as a model that connects operational (OLTP) and analytical (OLAP) databases. He highlights the unique roles of these systems, discusses challenges with newer models like HTAP, and emphasizes the importance of aligning data strategies with business needs for successful implementation. ### Conceptual Data Modeling (Day 3 Lab) URL: https://www.dataexpert.io/lesson/dimensional-data-modeling-day-3-lab-v4 Module: Week 1: Dimensional Data Modeling in Iceberg · Enrolled students In this lab, Zach discusses the importance of data metrics and effective communication strategies. He highlights various data sources, such as banks, rental properties, and Excel sheets, and explain how they can be used to track metrics related to health, career, and recreation. He also emphasizes the significance of considering context and building relationships when communicating with stakeholders. The information shared can help them gain valuable insights and improve their communication skills. [Recorded May 9, 2024] ### dbt basics Day2 Lab URL: https://www.dataexpert.io/lesson/dbtbasicsday2lab-jan2026 Enrolled students The lab focuses on completing a DBT project by exploring key features such as snapshots, tests, and packages. Participants will learn how to implement slowly changing dimensions using snapshots to track historical changes in customer data, as well as how to write and run data tests for quality assurance. The session also covers the use of packages to enhance DBT functionality, making it valuable for analytics engineers looking to deepen their understanding of data transformation processes. ### SQL URL: https://www.dataexpert.io/lesson/sql-intereview-p735-l571 Module: Interview Skills · Enrolled students Zach discusses key insights from SQL interviews, highlighting common mistakes to avoid and emphasizing the importance of problem-solving skills over just getting the right answer. He also touches on the evolving nature of interview processes at different companies, offering tips on how to stand out and optimize your performance. ### Creating a full dbt Project Lecture URL: https://www.dataexpert.io/lesson/dbtbasicsday2lecture-jan2026-p735-l1926 Module: Week 1: dbt Basics · Enrolled students Creating a full dbt project provides a comprehensive overview of essential dbt concepts, focusing on tests, snapshots, and packages. Participants will learn how to implement data quality tests, including data and unit tests, to ensure the integrity of their data models. The lesson also covers the creation of snapshots for tracking slowly changing dimensions and introduces packages that enhance dbt functionality, making it ideal for analytics engineers looking to streamline their data transformation processes. ### Azure - Week 5 URL: https://www.dataexpert.io/lesson/azure-week-5-may2025 Enrolled students Azure Data Factory (ADF) is a powerful tool for creating data pipelines, and this session focuses on its core components and functionalities. Participants will learn how to set up ADF accounts, create linked services for Azure SQL databases and Azure Data Lake Storage, and build pipelines for data transfer and transformation. This lesson is ideal for data engineers and professionals looking to enhance their skills in cloud data integration and management using Azure. ### Spark Batch Processing - Managing Spark Jobs and Notebooks: Challenges, Caching, and Performance Optimization (Day 2 Lecture) URL: https://www.dataexpert.io/lesson/spark-batch-day-2-lecture-v3 Module: Week 4: Batch Pipelines with Apache Spark · Enrolled students In this lecture, Zach discusses the intricacies of managing Spark jobs and notebooks, highlighting the challenges of achieving code modularity and the nuances of jar submissions for Scala Spark and PySpark. Exploring the advantages and distinctions between notebooks and spark-submit for production jobs, Zach also sheds light on caching, the auto broadcast join threshold, and the effective use of UDFs in PySpark. Closing with expert insights, he offers valuable tips on performance optimization and the optimal language selection for Spark jobs. [Recorded on Dec 7 2023] ### Advanced dbt Lecture Day 2 URL: https://www.dataexpert.io/lesson/advanced-dbt-lecture-day-2-april2025 Enrolled students Advanced pipelines in dbt are crucial for optimizing data workflows in production environments. This lesson covers techniques for running dbt with Airflow, including the use of selection methods to manage model execution frequency and the implementation of write-audit-publish patterns to ensure data quality. Participants will learn how to improve efficiency and reliability in their dbt projects, making it beneficial for data engineers and analysts looking to enhance their data transformation processes. ### Tech Talk - Raaid URL: https://www.dataexpert.io/lesson/techtalksweek4-jan2026 Enrolled students Raaid shares insights from his extensive experience in data science and analytics, including his time at Facebook and as a venture capitalist. He discusses the transition from senior to staff engineer roles, emphasizing the importance of systems thinking, ownership, and leveraging team dynamics. Attendees will benefit from practical advice on navigating career growth, managing ambiguity, and developing leadership skills, making this session valuable for aspiring data professionals and engineers. ### Tech Talk with Xinran Waibel URL: https://www.dataexpert.io/lesson/tech-talk1-oct2025-p737-l1549 Module: Tech Talks · Enrolled students Tech Talk with Xinran Waibel explores the evolving landscape of data engineering and AI. Xinran, a seasoned data engineer at Netflix, shares insights on the integration of AI tools in data workflows, the blurred lines between software and data engineering roles, and the importance of understanding foundational concepts over specific technologies. This session is beneficial for aspiring data engineers and AI professionals looking to navigate the complexities of modern data practices and enhance their skills in a rapidly changing environment. ### Day 5: Doubly linked lists URL: https://www.dataexpert.io/lesson/doubly-linked-lists-p913-d5 Enrolled students Use doubly linked lists to insert/delete in O(1) with a node reference and traverse in both directions, while understanding space–time trade-offs vs singly lists. ### Apache Spark Memory Turning, Partitioning Day 2 Lab URL: https://www.dataexpert.io/lesson/apache-spark-memory-turning-partitioning-day-2-lab-jan2025 Enrolled students In this lab, Zach dives into the intricacies of Spark data processing, focusing on how to manage CSV files and optimize shuffle partitions. He highlights the importance of partitioning and how it affects job performance, especially when dealing with large datasets. He also discusses some common pitfalls and provide insights on how to improve efficiency. ### Azure - Week 5 URL: https://www.dataexpert.io/lesson/azure-week-5-may2025-p736-l1246 Enrolled students Azure Data Factory (ADF) is a powerful tool for creating ETL pipelines, and this session focuses on building a pipeline to copy data from an Azure SQL database to Azure Data Lake Storage in CSV format. Participants will learn how to create linked services, datasets, and pipelines using ADF's user-friendly interface, as well as how to manage and scale their data integration processes effectively. Ideal for data engineers and analysts looking to enhance their skills in Azure data solutions. ### Data modeling with Date lists Lecture URL: https://www.dataexpert.io/lesson/data-modeling-with-date-lists-lecture Module: Bonus · Enrolled students In this lecture, we cover the fundamental concepts of fact and dimension data structures. The lecture delves into the significance of fact data, highlighting its role as the core element in data modeling. Additionally, we explore dimension data and its crucial function in grouping and categorizing information. The discussion emphasizes the importance of capturing changes within dimensions and introduces the benefits of employing slowly changing dimensions. The lecture provides examples and insights, offering viewers a comprehensive understanding of fact and dimension data structures in the context of data engineering. ### Structured Streaming Kafka to Delta Live Table Day2 Lab URL: https://www.dataexpert.io/lesson/structuredstreamingkafkatodeltalivetableday2lab-feb26-p736-l2032 Module: Week 4: Structured Streaming with Spark & Kafka · Enrolled students In this video, Zach walks you through the Delta Live Table (DLT) pipeline, focusing on the Medallion architecture with Bronze, Silver, and Gold layers. He sets up data quality checks and explores how to quarantine problematic data, specifically addressing null country values that arose during processing. Zach demonstrates how to create alerts for data quality issues and discusses the importance of scheduling pipeline runs efficiently to manage costs. Zach reminds everyone to review the data quality checks and alerts to enhance their data processing workflows. ### Vaishali Macwan - Senior Data Scientist URL: https://www.dataexpert.io/lesson/techtalks4-mar26-p737-l2117 Module: Tech Talks · Enrolled students Vaishali Macwan, a Senior Data Scientist with extensive experience in FinTech and data science, shares insights on the evolving landscape of data roles and the importance of adaptability in job searching. This session covers the shift towards AI engineering roles, the significance of soft skills in hiring, and practical strategies for building a personal brand and client relationships. Ideal for aspiring data professionals and those looking to navigate the current job market effectively. ### Brian Pulliam - Career Development Coach URL: https://www.dataexpert.io/lesson/techtalks2-feb26-p736-l2043 Module: Guest Speakers · Enrolled students Brian Pulliam, a seasoned career development coach with extensive experience in tech, shares insights on navigating the job market and mastering the interview process. He discusses the importance of storytelling, understanding hiring dynamics, and adapting resumes for different company sizes. This session is invaluable for job seekers looking to enhance their interview skills and stand out in a competitive landscape, especially in the evolving tech environment influenced by AI. ### Data Modeling Sunday Quiz URL: https://www.dataexpert.io/lesson/data-modeling-quiz Enrolled students ### TA Office hour 7 URL: https://www.dataexpert.io/lesson/taofficehour7-jan2026-p735-l1945 Enrolled students This office hours session provides insights into the capstone project for students in the analytics engineering boot camp. Participants discuss submission deadlines, local development challenges, and best practices for handling API keys and environment variables. The session is beneficial for students seeking clarification on project requirements and troubleshooting deployment issues, particularly in using tools like Airflow and managing local testing environments. ### Welcome and Course Intro URL: https://www.dataexpert.io/lesson/welcomeandcourseintro-00cd8 Enrolled students ### Mastering Data Lake Architectures Lecture URL: https://www.dataexpert.io/lesson/apache-iceberg-day-2-lecture-jan2025 Enrolled students In today’s lecture, Zach dives into the world of data lake architectures, covering Iceberg, Delta Lake, and Hive. He highlights their key features, including the flexibility of file formats, transactional capabilities, and seamless integration with various platforms. ### Day 3: Linked list tails and O(1) append URL: https://www.dataexpert.io/lesson/linked-list-tails-and-o-1-append-p913-d3 Enrolled students Use a tail pointer to append to a singly linked list in O(1) while correctly handling empty and singleton lists and keeping tail valid on deletions. ### Apache Spark programming with Databricks Day 1 Lab URL: https://www.dataexpert.io/lesson/apache-spark-programming-with-databricks-day-1-lab-may2025 Enrolled students In this video, Yared discusses about Adaptive Query Execution (AQE) in Spark, highlighting how it optimizes partitioning and joins under the hood, particularly in Spark 3.x. He explaines the differences between on-heap and off-heap memory management, and how Spark handles memory allocation across various components. He also explores Spark UI to track job execution and performance metrics, emphasizing the importance of understanding narrow versus wide transformations. ### Tech Talk - Madison URL: https://www.dataexpert.io/lesson/techtalksweek2-jan2026 Enrolled students Madison shares her extensive experience as an analytics engineer, discussing the evolution of the role and the importance of stakeholder communication. She addresses the nuances of ELT versus ETL, the significance of data modeling, and the impact of AI on analytics engineering. This session is beneficial for aspiring analytics engineers looking to enhance their understanding of the field, improve their technical skills, and learn how to effectively bridge the gap between data and business needs. ### Parth Parekh - Director of Data Engineering at Meta (2.0) URL: https://www.dataexpert.io/lesson/techtalksweek5-jan2026-p735-l1938 Module: Tech Talks · Enrolled students Parth Parekh, Director of Data Engineering at Meta, shares insights on the evolving landscape of data engineering in the context of AI. He discusses the shift from traditional data roles to a more dynamic, AI-native approach, emphasizing the importance of context engineering and the need for data professionals to adapt to new tools and methodologies. This session benefits data engineers, analysts, and product managers looking to enhance their understanding of AI integration and the future of data-driven decision-making. ### Joe Reis - Author of Fundamentals of Data Engineering (2.0) URL: https://www.dataexpert.io/lesson/joe-reis-jan2025-p736-l871 Module: Guest Speakers · Enrolled students Joe Reis, author of "Fundamentals of Data Engineering," shares insights on data modeling and its evolving role in the data engineering landscape. He discusses the importance of moving beyond traditional table-centric approaches to embrace diverse data forms, including unstructured data and AI integration. This session benefits data engineers and aspiring professionals by highlighting the significance of robust data modeling practices and the impact of AI on data workflows, encouraging a forward-thinking mindset in an ever-changing field. ### TA Office hour 1 URL: https://www.dataexpert.io/lesson/taofficehour1-feb26-p736-l2047 Enrolled students Office hours provide an opportunity for students to discuss challenges and clarify concepts related to their coursework. This session focused on addressing a student's concerns about the disconnect between lecture material and homework assignments, emphasizing the importance of hands-on practice with labs and the necessity of prior knowledge in Python and SQL. Participants are encouraged to reach out for support and engage with peers for collaborative learning. ### TA Office hour 3 URL: https://www.dataexpert.io/lesson/taofficehour3-jan2026-p735-l1941 Enrolled students A live Q&A session focused on addressing student questions about assignments and technical issues related to data engineering tools. Key topics discussed include creating Snow Park jobs, using Airflow for orchestration, and troubleshooting Docker and Portman environments. Students also sought clarification on assignment requirements and accessing necessary credentials for databases. This session is beneficial for learners needing support with practical applications in their coursework. ### Capstone Project URL: https://www.dataexpert.io/lesson/capstone-project-v3 Module: Capstone Project · Free Zach Wilson presents a concise overview of the Capstone project in this informative video. He guides viewers on accessing materials on GitHub and the project flexible approach, including options for team collaboration. Zach emphasizes the requirement of using multiple data sources with significant data volume and the importance of early project scoping and spec creation. He also outlines the key criteria for a successful project, such as creating detailed diagrams and ensuring the project addresses a complex use case. This video is a crucial guide for students embarking on their Capstone project journey. Recorded on Dec 4th, 2023 ### TA Office Hour 5 (2025-07-30) URL: https://www.dataexpert.io/lesson/ta-office-hour5-july2025-p737-l1307 Enrolled students A live Q&A session focused on troubleshooting dependency issues in Python environments, particularly with Langchain and related libraries. Participants discuss error messages, version conflicts, and installation problems, with guidance on creating and managing virtual environments. Additionally, the session covers an assignment involving building a dependency scanner and vulnerability report, utilizing tools like Neo4j and GitHub Security advisories, and emphasizes the importance of testing and managing dependencies effectively. ### How to load data into Iceberg with Airflow URL: https://www.dataexpert.io/lesson/howtoloaddataintoicebergwithairflow-jan2026-p735-l1908 Module: Week 6: Airflow & Pipeline Specs with Trino · Enrolled students Learn how to effectively load data into Iceberg using Airflow in this comprehensive lesson. Gain insights into orchestration techniques through real-world examples from companies like Airbnb and Netflix, focusing on pipeline design, data availability definitions, and performance optimization. This lesson is ideal for data engineers and data scientists looking to enhance their skills in building efficient data pipelines with Airflow and Iceberg. ### Snowflake Lab URL: https://www.dataexpert.io/lesson/snowflake-lab-jan2025-p736-l787 Enrolled students In this lab, Bruno walks you through using Snowflake for data management without dbt, focusing on creating and merging tables. He demonstrates how to handle new data, including updates and inserts, and explain the importance of clustering for efficient querying. He also covers change data capture and how to create a slow change dimension. ### Chat with Zach URL: https://www.dataexpert.io/lesson/week-5-zach-qa-v4 Enrolled students Exclusive Q&A for homework doers. [Recorded on June 6th, 2024] ### Spark Basics URL: https://www.dataexpert.io/lesson/spark-basics-may2025-p736-l1061 Enrolled students ### dbt Basics Day 2 Lecture URL: https://www.dataexpert.io/lesson/dbt-basics-day-2-lecture-jan2025-p736-l773 Enrolled students In this lecture, Bruno dives into the importance of DBT for data transformation within cloud data warehouses like Snowflake and BigQuery. He discusses how the shift from ETL to ELT has changed the landscape, making it more cost-effective to store and process data. He also highlights best practices for using DBT and the challenges that come with it. ### Shwetha - Data Engineer at OpenAI URL: https://www.dataexpert.io/lesson/techtalks5-mar26-p737-l2118 Module: Tech Talks · Enrolled students Sweta, a data engineer at OpenAI, shares insights from her experience in the safety systems team, focusing on handling unstructured data and building detection pipelines for safety classifiers. She discusses the differences between her roles at OpenAI and Google, emphasizing the shift towards unstructured data and the importance of data quality and logging. This session is beneficial for aspiring data engineers looking to understand the evolving landscape of data engineering, particularly in AI contexts. ### Prompt Engineering + Basic RAG Day 2 Lecture URL: https://www.dataexpert.io/lesson/prompt-engineering-day2-lecture-oct2025-p737-l1526 Module: Week 1: Prompt Engineering & Basic RAG · Enrolled students In this video, Zach explores the world of vectors, retrieval-augmented generation (RAG), and fine-tuning, underscoring the significance of understanding vectors in AI engineering. He examines the two types of vectors—dense and sparse—elaborating on their differences and advising when to use each. Additionally, he discusses how to select the appropriate embedding model, with a particular emphasis on OpenAI's options, along with the trade-offs between RAG and fine-tuning in relation to data dynamics and access patterns. ### Capstone Project Day 3 Class URL: https://www.dataexpert.io/lesson/capstone-project-day-3-class-april2025 Enrolled students In this lab, zach walks through the data pipeline process, emphasizing the importance of not writing directly to production. He discusses various data quality checks and how we aggregate data to ensure everything is functioning correctly. He also highlights the significance of monitoring declines in data quality. ### Book Club 2 URL: https://www.dataexpert.io/lesson/bookclub2-1f234 Enrolled students ### Advance Snowflake Day 1 Lecture URL: https://www.dataexpert.io/lesson/advancesnowflakeday1lecture-cedbb Enrolled students ### TA Office Hour 5 URL: https://www.dataexpert.io/lesson/ta-office-hour-5-april2025-p735-l1005 Enrolled students A TA office hour session focused on the GPT basics assignment provides an overview of the assignment's requirements and troubleshooting tips. Participants engage in discussions about setting up their environments, using DBT for data modeling, and addressing common issues with Snowflake. The session is beneficial for students seeking clarification on assignment tasks, including building models, creating YAML files, and implementing tests in DBT. ### Advanced Spark Optimization Techniques Day 1 Lecture URL: https://www.dataexpert.io/lesson/advanced-spark-optimization-techniques-day-1-lecture-jan2025-p736-l812 Module: Week 4: Structured Streaming with Spark & Kafka · Enrolled students Advanced Spark optimization techniques are explored, focusing on performance and query optimization strategies within Spark. Participants will learn about the Catalyst Optimizer, Adaptive Query Execution (AQE), and memory management, which are crucial for efficient data processing. The session includes a real-life use case related to airport data, demonstrating how to apply these techniques to optimize queries and manage resources effectively in data engineering projects. ### Capstone Project Day 1 Lab URL: https://www.dataexpert.io/lesson/capstone-project-day-1-lab-april2025 Enrolled students In this lab, Zach dives into our current work with Airflow and how we're building queries for our data projects. He also discusses the importance of creating unique IDs for our data and maintaining them effectively. By next class, we should have a solid project that addresses many questions, and he encourages everyone to engage with the upcoming Kafka and Spark sessions. ### Advanced snowflake: Deploying the Capstone dag Lab URL: https://www.dataexpert.io/lesson/advanced-snowflake-deploying-the-capstone-dag-lab-p735-l562 Module: Week 4: Advanced Snowflake · Enrolled students In this lab, Zach covers key aspects of our Data Engineering Bootcamp. He discusses setting up Airflow DAGs, managing secrets with AWS Secrets Manager, and working with Glue Spark. He also highlights the importance of not storing secrets in code and provide guidance on project deadlines. ### Tech Talk - Vikram URL: https://www.dataexpert.io/lesson/tech-talk-vikram-july2025 Enrolled students Vikram discusses the innovative approach of building the world's first AI data engineer at Argent, focusing on automating complex data engineering tasks. He shares insights on leveraging machine learning and language models to enhance data pipeline management, including metadata ingestion and sandbox testing for safe implementation. This session is valuable for data engineers and AI enthusiasts interested in the intersection of AI and data engineering, particularly in improving efficiency and reducing manual workload. ### Complex Data Types Lecture URL: https://www.dataexpert.io/lesson/complex-data-types-lecture Module: Bonus · Enrolled students In this presentation, we explore the fundamental concepts essential for effective data modeling and database management. We dive into understanding your data consumer, the distinctions between OLTP and OLAP data modeling, the principles of Cumulative Table design, and the tradeoff between compactness and usability. We also address the challenges of temporal cardinality explosion and the potential pitfalls of run-length encoding compression. By the end of this lecture, you will have a strong foundation in the key concepts of data modeling and be prepared to apply these principles in practice. ### QnA with Zach 3 URL: https://www.dataexpert.io/lesson/qnawithzach3-aug26 Enrolled students The Q&A session focuses on data modeling interviews, discussing methodologies and techniques essential for success. Participants learn about the importance of asking questions, understanding data usage, and the trade-offs between different table designs, such as cumulative and daily tables. This session benefits aspiring data engineers preparing for interviews by providing insights into effective data modeling practices, tools like LucidChart, and the nuances of dimensional data modeling. ### LLM-Driven Data Engineering Day 1 Lecture URL: https://www.dataexpert.io/lesson/large-language-models-day-1-lecture Free In this bonus lecture, we dive into one of the hottest topics in tech right now: "Will data engineering be replaced by AI and large language models?" We explore where Large Language Models (LLMs) like GPT-4 are poised to have the most significant impact, identify areas ripe for disruption, and discern the domains where LLMs have a long way to go before they can replace human ingenuity. But most importantly, discover how you can use these insights to harness the power of LLMs and elevate your career as a data engineer. ### Win with Soft Skills URL: https://www.dataexpert.io/lesson/careerdevsession3-jan2026-p735-l1949 Module: Career Development · Enrolled students Mastering soft skills is essential for succeeding in job interviews across all industries. This session covers key strategies for answering common interview questions, such as how to effectively respond during screening calls and behavioral interviews. Participants will learn how to articulate their experiences clearly, prepare thoughtful questions, and demonstrate their value to potential employers, making it ideal for job seekers looking to enhance their interview performance. ### Q&A with Zach Week 2 (2025-07-18) URL: https://www.dataexpert.io/lesson/qa-with-zach-week2-july2025-p737-l1295 Module: Q&A with Zach · Enrolled students A live Q&A session with Zach covers various topics related to the boot camp, including capstone project ideas, AI engineering, and deployment challenges. Participants discuss their projects, such as a crypto design and a movie recommendation app, while seeking advice on integrating LLMs and RAG models. Zach emphasizes the importance of business value in projects and addresses technical issues, providing insights into the evolving roles of data engineers and AI engineers. This session is valuable for anyone looking to enhance their understanding of AI applications and project development. ### Special Guest, Ben Rogajan [Dec 1, 2023] URL: https://www.dataexpert.io/lesson/guest-speaker-2023-12-01-v3 Enrolled students Join us for this installment of the Spotlight Series with Ben Rogajan, as we explore the landscape of data engineering and consulting. He presents insights and experiences in data engineering and consulting. The discussion covers industry challenges and opportunities, emphasizing the significance of building a strong personal brand. Practical tips on effective outreach and recommended resources for continuous learning are provided. Watch the video to gain insights and elevate your career in these dynamic fields.[Recorded Dec 1st, 2023] ### Grouping sets and Window functions Lecture URL: https://www.dataexpert.io/lesson/grouping-sets-and-window-functions-lecture-p735-l580 Module: Week 7: Analytical Patterns & CDC · Enrolled students In this presentation, Zach explores a range of topics related to Advanced SQL Analytical Patterns. The discussion covers the application of recursive CTEs, count case when, grouping sets, cross join unnest, cube, rollup, and window functions. Drawing from Zachs experience in conducting interviews and working with these concepts, the presentation offers insights and practical advice. ### Jess Ramos - Data Analyst (2.0) URL: https://www.dataexpert.io/lesson/tech-talk-with-jess-ramos-p735-l593 Module: Tech Talks · Enrolled students This is the tech talk with Jess Ramos. ### Building functional AI agents using LangChain Day1 Lab URL: https://www.dataexpert.io/lesson/buildingfunctionalaiagentsusinglangchainday1lab-mar26 Enrolled students In this lab, Zach walks you through the LangChain starter repo, highlighting key components such as the requirements for libraries, setting up the environment, and the different types of agents you'll be working with, including memory agents and tool agents. Zach emphasises the importance of specificity in prompts to reduce token usage. He also discusses how memory functions in agents and how to store and retrieve user information effectively. ### Databricks and Advanced Spark Day2 Lecture URL: https://www.dataexpert.io/lesson/databricksandadvancedsparkday2lecture-feb26-p736-l2027 Module: Week 2: Databricks & Advanced Spark · Enrolled students In this lesson, Zach focuses on the five key reasons why Spark jobs can be slow, including data model bottlenecks and job misconfigurations. He emphasizes the importance of processing only the necessary data, advocating for incremental refresh strategies over full dataset refreshes, which can lead to significant performance improvements. Additionally, Zach discusses the impact of source file formats and the need for proper configurations to avoid congestion and misconfigurations. ### Hard Orchestration Lessons Day 2 Lecture URL: https://www.dataexpert.io/lesson/hard-orchestration-lessons-day-2-lecture-jan2025-p736-l725 Module: Week 6: Airflow & Pipeline Specs with Trino · Enrolled students In this lecture, Zach explores the complexities of Airbnb's PNA system, focusing on key issues and solutions. He discusses challenges related to availability definitions, backfilling historical data, and optimizing pipeline designs. Zach suggests actions around communication frequency, stakeholder engagement, and iterative design improvements. ### Apache Iceberg Day 3 Lecture URL: https://www.dataexpert.io/lesson/apache-iceberg-day-3-lecture-jan2025 Enrolled students In today’s lecture, Zach dives into the essential concept of data contracts, explaining how they build trust between data producers and consumers by establishing clear guidelines for data generation and usage. Understanding data contracts is key to making informed decisions and preventing data misuse. It is a must know for anyone in data engineering! ### Chat with Zach URL: https://www.dataexpert.io/lesson/v4-1-chat-with-zach-5269 Enrolled students Join Zach, our founder/CEO and instructor of our data engineering bootcamp, for an insightful Q&A session covering crucial topics in the field. Zach dives into data management challenges, recommending tools like Trino and Presto, and explores infrastructure optimization with Kafka. He emphasizes individual submissions for collaborative assignments and discusses practical considerations for capstone projects, stressing originality and effective presentation. Transitioning to Apache Iceberg, Zach provides insights into SQL query processing and shares his perspective on the use of classmates and TAs for troubleshooting. Note: Correction at 13:00 - Zach mentions that students will receive up to two chances to correct their assignments before the grade is finalized, with TAs providing two reviews. However, in version 4 of our bootcamp, automatic tests and immediate LLM-generated feedback have been introduced, allowing unlimited revisions before the deadline. As a result, revisions after the deadline are no longer permitted, and the final version submitted at the deadline will be graded. Please refer to the 'assignments' page for further details. [Recorded May 9, 2024] ### TA Office hours 6 URL: https://www.dataexpert.io/lesson/taofficehours6-mar26-p737-l2134 Enrolled students Office hours provide an opportunity for students to clarify their capstone project proposals and discuss technical questions related to their projects. Participants engage in discussions about project ideas, data storage options, and the use of APIs for data analysis. Key topics include project criteria, database choices like AWS and Superbase, and the timeline for submissions, making this session beneficial for students seeking guidance on their capstone projects. ### Q&A with Zach Week 2 URL: https://www.dataexpert.io/lesson/qa-with-zach-week-2-may2025 Enrolled students A live Q&A session with Zach covers various topics related to data engineering and AI, including discussions on homework, course content, and industry trends. Key points include the potential integration of Iceberg V3 into the curriculum, strategies for enhancing job prospects through networking and personal branding, and insights on managing data pipelines and retention policies. Participants engage with questions about capstone projects, data processing challenges, and best practices for using tools like DataBricks and Airflow. ### TA Office hours 1 URL: https://www.dataexpert.io/lesson/taofficehours1-mar26-p737-l2129 Enrolled students Office hours provide an interactive platform for students to seek guidance on their coursework and technical challenges. Key topics discussed include troubleshooting API key issues with OpenAI and Anthropic, navigating assignment submissions, and understanding certification requirements. Students looking for support with their projects or facing technical difficulties will benefit from this session, as it addresses common concerns and offers practical solutions. ### Special Guest, Joseph Machado [Dec 15,2023] URL: https://www.dataexpert.io/lesson/guest-speaker-2023-12-15-v3 Enrolled students In this video, Joseph Matado, a data engineer at LinkedIn, shares insights on data engineering, discusses the Iceberg data format, and provides career tips. He emphasizes the importance of integrating multiple systems and being cautious with running multiple inserts. Joseph also touches on the popularity of Python and the benefits of preparing for interviews. The video offers valuable information for data engineers and those interested in the field.[Recorded on Dec 15th, 2023] ### Spark Data Quality (Day 3 Lecture) URL: https://www.dataexpert.io/lesson/spark-batch-day-3-lecture-v4 Module: Week 4: Batch Pipelines with Apache Spark · Enrolled students This is a pre-recorded class. In this presentation, we delve into the pivotal role of data quality in the realm of analytics engineering, unraveling its profound impact on the success of data pipelines. The discussion underscores the necessity for automated data quality checks, shedding light on the challenges encountered by data engineers in their quest for ensuring data accuracy. Exploring the leadership dimension, we examine the crucial role of management in prioritizing data quality and the far-reaching consequences of subpar data quality on business outcomes. Join us in this insightful video to glean a deeper understanding of the critical role that data quality plays in analytics engineering, unraveling its implications for informed decision-making. [Recorded on Dec 12, 2023] ### Youtube URL: https://www.dataexpert.io/lesson/youtube-3 Enrolled students testing updating ### How to connect Iceberg and Snowflake with Catalogs URL: https://www.dataexpert.io/lesson/howtoconnecticebergandsnowflakewithcatalogs-jan2026-p735-l1910 Module: Week 4: Advanced Snowflake · Enrolled students In this lab, Zach dives into processing unstructured data, focusing on earnings call transcripts and SEC filings. He explores how to calculate file checksums to avoid reprocessing unchanged files and successfully load earnings transcripts into Snowflake using Snowpark. Zach demonstrates the importance of managing file compression and encryption settings when creating stages in Snowflake. ### Day 2 Lecture: Optimizing Dev Workflows with Cursor and Windsurf URL: https://www.dataexpert.io/lesson/prompt-engineering-day2-lecture-jul2025-p737-l1171 Module: Week 1: Prompt Engineering & Basic RAG · Enrolled students In this lecture, Zach dives into how AI is transforming workflow productivity and the importance of adapting skills to stay relevant. He discusses the different categories of work and how AI will impact tasks, particularly highlighting that routine tasks done solo are at risk, while collaborative and creative tasks will thrive. Zach shares insights on the best practices for using AI in coding and the various tools available, emphasizing the need for engineers to become more technical and socially adept. ### Tech Talk - Parth Parekh URL: https://www.dataexpert.io/lesson/guest-speaker-4-v4 Enrolled students In this tech talk, Zach discusses his experience working with Parth Parekh and how his knowledge has influenced his understanding of data engineering. They cover topics such as date lists, cumulative tables, full outer joins, and processing facts. He also explores the challenges of scaling data processing and the new frontier of data engineering in the context of text models, image models, and multi-models. Throughout the talk, he emphasizes the importance of structuring unstructured data and the role of data engineering in creating labeled and structured data for machine learning models. [Recorded on 17th May, 2024] ### Graph Data Modeling Lecture URL: https://www.dataexpert.io/lesson/graph-data-modeling-lecture-p735-l587 Module: Bonus · Enrolled students Zach discusses the importance of data modeling and metrics in building a successful pipeline. He emphasizes that conceptual data modeling is accessible to everyone and highlight the need to consider the right people for feedback and collaboration. He also touches on the challenges of understanding stakeholders' true requirements and share his experience with choosing the right database solution. Throughout the video, he emphasizes the importance of metrics-driven approaches and the potential impact of well-designed dashboards. Overall, this lecture provides valuable insights and guidance for data engineers looking to optimize their pipeline. ### Setting Up Airflow for Week1 for Mac URL: https://www.dataexpert.io/lesson/setting-up-airflow-for-week1-for-mac-may2025 Enrolled students Learn how to set up Apache Airflow on a Mac for your data engineering projects. This lesson covers essential steps such as editing your data expert profile, cloning the Airflow DBT project repository, and installing necessary dependencies like Python and Astronomer CLI. Ideal for data engineers and aspiring professionals, this session ensures you have the foundational setup to effectively manage and run your data workflows using Airflow. ### Advanced Snowflake Data Modeling and UDFs Lab Day 3 URL: https://www.dataexpert.io/lesson/advanced-snowflake-data-modeling-and-udfs-lab-day-3-april2025-p735-l987 Module: Week 3: Data Modeling on Snowflake · Enrolled students In this lab, Zach dives into how to analyze NBA player performance using data, specifically focusing on players who have had five or more consecutive seasons averaging at least 20 points. He walks through the process of using SQL and Python to extract and manipulate this data, showcasing some interesting streaks, including those of notable players like Steph Curry and LeBron James. ### Data Lakes with Delta Table Day1 Lecture URL: https://www.dataexpert.io/lesson/datalakeswithdeltatableday1lecture-feb26-p736-l2021 Module: Week 3: Data Lakes with Delta Table · Enrolled students In this lecture, Zach dives into what a lake house is, how it differs from data lakes and warehouses, and the nuances between Delta and Iceberg. He emphasizes the importance of understanding the architecture, anti-patterns, and the unique features of Delta tables, including time travel and data immutability. Zach also highlights the need for proper data governance, especially concerning PII, and the significance of using vacuuming to manage data effectively. ### TA Office hour 3 URL: https://www.dataexpert.io/lesson/taofficehour3-jan2026 Enrolled students A live Q&A session focused on addressing student questions regarding assignments and technical issues. Key topics discussed include the Snow Park job creation, connecting to PostgreSQL databases, and using Airflow for orchestrating data workflows. Students also sought clarification on the capstone project requirements and troubleshooting Docker and Portman setups. This session is beneficial for learners needing support with practical applications of data engineering concepts and tools. ### Claude Code Onboarding URL: https://www.dataexpert.io/lesson/claude-code-onboarding-p737-l2225 Module: Kickoff & Setup · Enrolled students Learn how to set up Cloud Code for the AI engineering boot camp, including installation via terminal and configuring a proxy for session management. This lesson covers essential steps such as exporting API keys and navigating the dashboard, making it ideal for participants looking to effectively utilize Cloud Code and manage their sessions on the Data Expert server. Gain practical insights into using the platform and troubleshooting common issues. ### Azure - Week 1 (Copy) URL: https://www.dataexpert.io/lesson/azure-week-1-1976 Enrolled students Understanding cloud computing is essential for data engineers, and this session introduces Azure's foundational concepts. Participants will learn about Azure's storage account service, including its scalability and various types such as Blob storage and Azure Data Lake Storage (ADLS). The lesson emphasizes the importance of these services for data management and analytics, providing insights into practical use cases, authentication methods, and best practices for leveraging Azure in data engineering roles. ### Career Dev Session 1 - Win with soft skills! URL: https://www.dataexpert.io/lesson/career-dev-session1-oct2025-p737-l1528 Module: Career Development · Enrolled students ### Tech Talk 1 - Jess Ramos URL: https://www.dataexpert.io/lesson/tech-talk-1-jess-ramos-april2025 Enrolled students A live Q&A session with Jess Ramos, a seasoned data analyst and content creator, explores navigating job transitions and building resilience in a challenging job market. Jess shares her personal experiences with job loss, the importance of networking, and strategies for diversifying income. The discussion also delves into the roles of data analysts and engineers, emphasizing collaboration, project management, and the significance of maintaining clear communication to avoid scope creep. This session is beneficial for anyone interested in career development in data and analytics. ### Advanced Snowflake: Data Modeling and UDFs Lab URL: https://www.dataexpert.io/lesson/advanced-snowflake-data-modeling-and-udfs-lab Module: Advanced Snowflake · Enrolled students Explore the advanced capabilities of Snowflake in this lab focused on data modeling and User Defined Functions (UDFs). Participants will learn how to effectively implement UDFs for complex SQL queries, particularly in scenarios involving consecutive data streaks, such as tracking NBA player performance. The session covers practical coding techniques, including window functions and array handling, while also demonstrating the integration of external APIs for enhanced data insights. Ideal for data engineers and analysts looking to deepen their Snowflake expertise. ### Dimensional Data Modeling - Slowly-changing Dimensions and Idempotent Queries in Iceberg (Day 2 Lecture) URL: https://www.dataexpert.io/lesson/dimensional-data-modeling-day-2-lecture-v3 Module: Week 1: Dimensional Data Modeling in Iceberg · Enrolled students In this presentation, we delve into the concept of idempotent pipelines, exploring their significance, advantages, and potential drawbacks. We provide insights into Slowly Changing Dimensions (SCDs) and discuss the various types. Discover whether SCDs are inherently idempotent or not and grasp the implications of their usage in data pipelines. Join us to gain a clear understanding of these foundational concepts in data processing. [Recorded Nov 8, 2023] ### Apache Spark programming with Databricks Day 2 Lecture URL: https://www.dataexpert.io/lesson/apache-spark-programming-with-databricks-day-2-lecture-may2025 Enrolled students In this video, Yared discusses the concept of stream processing, drawing parallels between data streams and water flow to illustrate the importance of continuous data movement. He highlights various applications, such as fraud detection, predictive maintenance in aircraft, and real-time marketing, emphasizing the need for low-latency decision-making in these areas. He also introduces popular stream processing tools like Kafka and Flink, and explores how to implement these concepts using Databricks. ### Tech Talk - Vaishali URL: https://www.dataexpert.io/lesson/techtalks4-mar26 Enrolled students A guest speaker session featuring Vaishali, an experienced data professional from the FinTech sector, discusses the evolving landscape of data roles and interviews. She shares insights on the importance of adaptability in job titles, the impact of AI on the industry, and strategies for effective outreach to potential clients. This session benefits anyone interested in data science, AI engineering, or those navigating the job market, providing practical advice on building a career in these fields. ### Building AI Agents with Databricks Day1 Lecture URL: https://www.dataexpert.io/lesson/buildingaiagentswithdatabricksday1lecture-feb26-p736-l2033 Module: Week 6: Building AI Agents with Databricks · Enrolled students In this session, Eumar walks you through the process of transforming Zachy into a real agent capable of answering questions about code, course syllabus, and employee expenses using tools like Unity catalog functions, Langgraph, and Databricks. He discusses the differences between RAGs and agents, emphasizing how agents can perform multi-step actions and interact with various tools. He also highlights the importance of using the Supervisor Agent for orchestrating multiple tools and shares his excitement about the capabilities of Langgraph for simplifying this process. ### Prompt Engineering + Basic RAG Day 2 Lab URL: https://www.dataexpert.io/lesson/prompt-engineering-day2-lab-oct2025 Enrolled students In this lab, Zach sets up a retrieval-augmented generation (RAG) system, which included creating a vector database in Supabase. During the session, he demonstrates how to load synthetic job and profile data into this table, emphasizing the importance of avoiding duplicate vectors to preserve system performance. He also creates a Python script designed to generate embeddings and retrieve relevant job listings based on user queries. ### Fact Data Modeling - Practical Insights into Data Modeling and Analysis with Iceberg (Day 1 Lab) URL: https://www.dataexpert.io/lesson/fact-data-modeling-day-1-lab-v3 Module: Week 2: Fact Data Modeling in Iceberg · Enrolled students In this hands-on lab session, we analyze NBA game data, unraveling the intricacies of data modeling and analysis. Breaking down the data into distinct columns, we elucidate the importance of each element. Through practical demonstrations, we showcase how to execute queries and perform joins, extracting meaningful insights from the dataset. Throughout the video, valuable tips are shared on data modeling, query optimization, and the significance of empathy when collaborating with stakeholders. Tune in to this video to acquire practical skills in NBA game data analysis and learn essential techniques for effective data modeling and analysis.[Recorded Nov 14, 2023] ### Introduction to Delta Live Table URL: https://www.dataexpert.io/lesson/introductiontodeltalivetable-aug26 Enrolled students In this lab, Zach explains streaming data pipelines, focusing on when to use streaming versus batch and how Delta Live Tables fits in. He describes how DataExpert logs user events to Kafka in real time, then streaming compute engines read from Kafka and write to long-term sinks like Delta, Iceberg, or Postgres. He emphasizes that “real time” is misleading and the key decision is acceptable latency, warning that streaming has higher complexity and often higher cost, with data quality and maintenance burdens that can require strong on-call and skills coverage. He also contrasts Flink versus Spark Structured Streaming, including Spark watermarking trade-offs and the checkpoint and offset options for failure recovery, and concludes by noting Delta Live Tables removes much of the operational overhead. ### Wednesday Quiz URL: https://www.dataexpert.io/lesson/wednesdayquiz-8c099 Enrolled students ### Day 1 Lecture: Leveraging MCP and tools to integrate effectively URL: https://www.dataexpert.io/lesson/building-end-to-end-ai-application-day1-lecture-july25-p737-l1187 Module: Week 5: End-to-End AI Applications · Enrolled students In this session , Zach discusses the Model Context Protocol (MCP) and how it enhances AI capabilities beyond simple prompts. He explores the five levels of AI engagement, from basic prompting with ChatGPT to advanced AI agent work, highlighting the importance of tools in enabling AI to take meaningful actions. Zach emphasizes that without tools, AI is limited to providing information rather than executing tasks. ### Hard Orchestration Lessons Day 2 Lecture URL: https://www.dataexpert.io/lesson/hard-orchestration-lessons-day-2-lecture-jan2025 Enrolled students In this lecture, Zach explores the complexities of Airbnb's PNA system, focusing on key issues and solutions. He discusses challenges related to availability definitions, backfilling historical data, and optimizing pipeline designs. Zach suggests actions around communication frequency, stakeholder engagement, and iterative design improvements. ### Special Guest, Madison Schott [Recorded Dec 22, 2023] URL: https://www.dataexpert.io/lesson/guest-speaker-2023-12-22-v3 Enrolled students In this session, Madison discusses data engineering, analytics, and tech industry trends. Madison shares her unique journey as an analytics engineer at Convert Kit. They explore differences between data and analytics engineering, emphasize software engineering role in analytics, and touch on industry shifts. The conversation also touches on data quality, AI, and effective metric management. They conclude by sharing experiences with role transitions and career growth. This conversation offers valuable insights into the evolving world of data and analytics.[Recorded on Dec 22,2023] ### Kafka Fundamentals, consumer APIs with Confluent (Day 1 Lecture) URL: https://www.dataexpert.io/lesson/flink-streaming-day-1-lecture-v4 Module: Week 5: Kafka-Flink Streaming Pipelines · Enrolled students In this lecture, Zach delves into the intricacies of Kafka, highlighting its versatility beyond data engineering. He explains how Kafka functions, its categorization of data lines, and its applications in various industries, including notifications and sockets. He touches on the differences between Kafka, Flink, and Spark Streaming, emphasizing the importance of real-time processing and the considerations when choosing between streaming and batch pipelines. [Recorded on June 5th, 2024] ### Monday Data Pipelines Quiz URL: https://www.dataexpert.io/lesson/mondaydatapipelinesquiz-8f0e2 Enrolled students ### Complex Data Types Lab URL: https://www.dataexpert.io/lesson/complex-data-types-lab Module: Bonus · Enrolled students In this lab, we explore Slowly Changing Dimensions (SCDs) and design Cumulative Tables using the nba_players dataset. This is our first stress test of our web-based interface linked to a Presto/Trino database running in Starburst. Presto (formerly PrestoSQL) or Trino are both query engines that allow users to query and analyze data in a data lake using SQL, while Starburst offers a distribution of Presto/Trino. ### Orchestration and Airflow Fundamentals Day 1 Lab URL: https://www.dataexpert.io/lesson/orchestration-and-airflow-fundamentals-day-1-lab-jan2025-p736-l1826 Module: Week 6: Airflow & Pipeline Specs with Trino · Enrolled students In this lab, Zach will provide an overview of airflow backfilling practices and discuss the pipeline code using pyspark. He will explain the importance of including the username in the DAG name and demonstrate how to run glue jobs and backfill data. He will also show a table that was backfilled and discuss the comparison of tables using union all. [Recorded on May 22nd, 2024] ### Loom URL: https://www.dataexpert.io/lesson/loom-7747 Enrolled students ### Lakehouse Syncing to Lakebase and Closed Loop AI Engineering URL: https://www.dataexpert.io/lesson/lakehouse-sync-for-closed-loop-system Enrolled students In this lab, Zach creates a "closed loop" system that takes Lakebase data CDC'd into Lakehouse and then does dependency analysis with ai_query ### MCP and Agentic AI Day 2 Lab URL: https://www.dataexpert.io/lesson/mcp-and-agentic-ai-day2lab-oct2025 Enrolled students In this lab, Zach dives into middleware and MCP servers, focusing on logging and adding new features. He introduces new middleware files for tracing, PII detection, and cost analytics, emphasizing the importance of creating an audit trail for their runs. He explores how to generate unique run IDs and improves their tracing middleware to better log tool calls and errors. He demonstrates how to implement a new function for querying RAG content, which successfully retrieves resumes from their system. ### Data Pipeline Maintenance: Navigating the Complexities of Data Engineering Day 1 Lecture URL: https://www.dataexpert.io/lesson/data-pipeline-maintenance-day-1-lecture-yt-p736-l686 Free Explore the dynamic world of data engineering in this insightful video. Zach shares a wealth of experiences and insights, addressing key challenges faced in the field. ### LLMs Day3 Part2 URL: https://www.dataexpert.io/lesson/llms-day-3-part-2-jan2025 Enrolled students Explore the end-to-end process of building and training a language model application in this lesson. Participants will learn how to set up the Aflow library, utilize OpenAI datasets, and implement a task pipeline for counting objects using various models like GPT-3.5 and DeepSeek R1. The lesson covers essential concepts such as data preparation, model evaluation, and optimization techniques, making it ideal for data engineers and AI practitioners looking to enhance their skills in model training and application development. ### Lesson 4 URL: https://www.dataexpert.io/lesson/lesson-4-5623 Enrolled students ### Data Lakes with Delta Table Day2 Lab URL: https://www.dataexpert.io/lesson/datalakeswithdeltatableday2lab-feb26 Enrolled students In this video, Zach walks you through Delta tables, partitioning, and the Unity Catalog. He explore how to create schemas and tables, and demonstrates the importance of using Unity Catalog to manage table names more effectively. He also discusses liquid clustering and its advantages over traditional partitioning, emphasizing best practices for clustering columns. ### Advanced Spark Optimization Techniques Day 1 Lab URL: https://www.dataexpert.io/lesson/advanced-spark-optimization-techniques-day-1-lab-jan2025 Enrolled students Advanced Spark Optimization Techniques Day 1 Lab focuses on enhancing your skills in Spark by exploring data frame operations, including reading CSV files, filtering, and aggregating data. Participants will learn to analyze physical plans, optimize memory utilization, and troubleshoot performance issues using the Spark UI. This lab is ideal for data engineers and analysts looking to improve their data processing efficiency and gain insights into operational metrics, particularly in contexts like airport management. ### TA Office Hour 2 (2025-01-14) URL: https://www.dataexpert.io/lesson/ta-office-hour-2-jan2025-p736-l722 Enrolled students The office hour begins from Minute 13. ### Madison Schott - Analytics Engineer URL: https://www.dataexpert.io/lesson/tech-talk-2-madison-april2025-p735-l991 Module: Tech Talks · Enrolled students A live Q&A session with Madison Schott, a senior analytics engineer at KIT, explores the nuances of analytics engineering. Madison discusses the differences between analytics engineering and data engineering, emphasizing the importance of stakeholder interaction and data modeling. The conversation also covers challenges in collaboration with data engineers, the significance of setting boundaries with stakeholders, and insights into the analytics engineering interview process, including case studies and the importance of clear communication. ### TA Office Hour 3 URL: https://www.dataexpert.io/lesson/ta-office-hour-3-july2025 Enrolled students Office Hour 3 focuses on troubleshooting and clarifying concepts related to creating sparse and dense indexes in a vector database. Participants discuss the process of chunking files, using Python for data ingestion, and the importance of context windows for model performance. The session also covers assignment details, including using GitHub repositories for data and the integration of OpenAI for enhanced query results. This lesson is beneficial for students looking to deepen their understanding of vector databases and data processing techniques. ### TA Office hours 4 URL: https://www.dataexpert.io/lesson/taofficehours4-mar26 Enrolled students Office hours provide an interactive platform for students to seek help with specific technical issues. Participants discuss troubleshooting Docker installation problems, resolving inactive API key errors, and integrating cloud services. The session also covers project collaboration options and access to resources after the boot camp, ensuring students receive the support they need for their data and AI engineering projects. ### Special Guests, Sundar Velayutham and Aishwarya Srinivasan [August 9, 2023] URL: https://www.dataexpert.io/lesson/v2-special-guests-sundar-velayutham-and-aishwarya-srinivasan-august-9-2023-7416 Enrolled students Join us each week as we sit down with visionary speakers from diverse fields, engaging in enlightening conversations that explore cutting-edge insights, trends, and their wealth of expertise. (Recorded August 9, 2023) ### Fact Data Modeling: Practical Guide to Formatting and Aggregating Data Day 3 Lab URL: https://www.dataexpert.io/lesson/fact-data-modeling-day-3-lab-yt-p736-l669 Enrolled students This lab demonstration provides a practical guide on working with daily metrics. The lab walks through the process of formatting data, highlighting key steps such as partitioning and grouping by date and metric name. ### TA Office hour 5 URL: https://www.dataexpert.io/lesson/taofficehour5-jan2026 Enrolled students This office hour session focuses on addressing student questions regarding the capstone project and assignment submissions within the analytics engineering boot camp. Participants discuss issues related to email communications, mentor availability, and project expectations, including the integration of data engineering concepts and tools like Airflow and Python. This session is beneficial for students seeking clarification on their course requirements and project guidelines. ### AI Application Setup URL: https://www.dataexpert.io/lesson/ai-application-setup-22551 Enrolled students In this video we go over how to setup Supabase, OpenAI, Render, and PubNub. These are the free services we will be using to create amazing applications together! ### Orchestration and Airflow Fundamentals Lecture URL: https://www.dataexpert.io/lesson/orchestration-and-airflow-fundamentals-lecture-p735-l555 Module: Week 6: Airflow & Pipeline Specs with Trino · Enrolled students Welcome to the very first lecture of the Analytics Engineering Bootcamp where we delve into the concept of orchestration, automations that drive business success through data. Throughout the lecture, we differentiate between orchestration and tools like Airflow, emphasizing the importance of data quality and efficient data processing techniques. ### Data Pipelines Maintenance (Day 1 Lecture) URL: https://www.dataexpert.io/lesson/pipeline-maintenance-day-1-lecture-v4 Module: Week 6: Data Pipeline Maintenance (V4) · Enrolled students This is a pre-recorded class. Explore the dynamic world of data engineering in this insightful video. Zach shares a wealth of experiences and insights, addressing key challenges faced in the field. Dive into discussions on the delicate balance between depth and breadth, emphasizing the significance of domain knowledge and effective communication. Uncover the impact of ownership and accountability on data engineering projects. Gain practical tips on pipeline optimization, navigating tech debt, and handling migrations.[Recorded on Dec 19th 2023] ### Staff Engineer Career Progression URL: https://www.dataexpert.io/lesson/staff-engineer-career-progression-4308 Free This is a panel discussion on how to scale as an engineer, featuring insights from Rahul Pandey, Zach Wilson, Ryan Peterman, Carly Taylor, and Lee McKeeman. Throughout this conversation, they delve into the challenges and opportunities they've encountered in their journeys at leading tech companies like Microsoft, Netflix, Amazon, Meta, Instagram and Airbnb. The panelists share their experiences with managing teams during the COVID pandemic, the importance of undertaking ambitious projects, and the role of strategic job hopping in career advancement. Additionally, they highlight the crucial aspects of visibility and communication in securing promotions. Tune in to this discussion to explore valuable strategies for progressing in your engineering career. ### Staff Engineer Career Progression URL: https://www.dataexpert.io/lesson/staff-engineer-career-progression Free This is a panel discussion on how to scale as an engineer, featuring insights from Rahul Pandey, Zach Wilson, Ryan Peterman, Carly Taylor, and Lee McKeeman. Throughout this conversation, they delve into the challenges and opportunities they've encountered in their journeys at leading tech companies like Microsoft, Netflix, Amazon, Meta, Instagram and Airbnb. The panelists share their experiences with managing teams during the COVID pandemic, the importance of undertaking ambitious projects, and the role of strategic job hopping in career advancement. Additionally, they highlight the crucial aspects of visibility and communication in securing promotions. Tune in to this discussion to explore valuable strategies for progressing in your engineering career. ### Q&A with Zach Week 2 URL: https://www.dataexpert.io/lesson/qa-with-zach-week-2-april2025-p735-l988 Module: Q&A with Zach · Enrolled students The Week 2 Q&A session with Zach provides an interactive platform for students in the analytics engineering boot camp to discuss their capstone project ideas and seek guidance. Key topics include project scope, budget considerations for cloud tools, and the importance of selecting impactful data sources. Students can gain insights on how to effectively structure their projects, manage costs, and leverage real-time data for enhanced learning experiences. This session is ideal for those looking to refine their project concepts and receive personalized feedback. ### Special Guests, Stephanie Nuesi and Joe Reis [July 26, 2023] URL: https://www.dataexpert.io/lesson/v2-special-guests-stephanie-nuesi-and-joe-reis-july-26-2023-4972 Enrolled students Join us each week as we sit down with visionary speakers from diverse fields, engaging in enlightening conversations that explore cutting-edge insights, trends, and their wealth of expertise. (Recorded July 26, 2023) ### Supplemental Session on Auto-prompt Optimization URL: https://www.dataexpert.io/lesson/supplemental-session-on-auto-prompt-optimization-jul2025 Enrolled students In this video, Zach provides an overview of the Autoprompt optimization session, where he discusses the key components of Adallflow pipelines, including the system prompt, examples, and input strings. He emphasizes the importance of having a ground truth for effective auto-optimization and explains how we can iteratively improve our system prompts through a structured training process. ### Q&A with Zach Week4 URL: https://www.dataexpert.io/lesson/qawithzachweek4-jan2026-p735-l1932 Module: Q&A with Zach · Enrolled students The Week 4 Q&A session with Zach focuses on various topics relevant to data engineering, including insights on job opportunities in the field, particularly with a specific company looking to hire staff and senior data engineers. Participants discuss their experiences with job interviews, data modeling, and the use of tools like DBT, Snowflake, and Airflow. The session also emphasizes the importance of aligning skills with industry needs and preparing effectively for interviews, making it beneficial for aspiring data engineers seeking guidance and practical advice. ### Tech Talk 4 URL: https://www.dataexpert.io/lesson/techtalk4-ab513 Enrolled students ### KPIs and Experimentation: Leading and Lagging Metrics Day 2 Lecture URL: https://www.dataexpert.io/lesson/kpis-and-experimentation-day-2-lecture-yt-p736-l696 Enrolled students Leading and Lagging Metrics ### RAG and LLMs Day2 Lecture URL: https://www.dataexpert.io/lesson/rag-and-llms-day-2-lecture-jan2025-p736-l828 Module: Bonus - LLMs · Enrolled students In this lecture, Zach dives into the intricacies of full stack engineering, particularly focusing on retrieval-augmented generation (RAG) systems. He discusses the importance of chunking data effectively and the role of AI in processing assignments, emphasizing the need for fine-tuning as our platform scales. ### Growth Accounting Models Lecture URL: https://www.dataexpert.io/lesson/growth-accounting-models-lecture-p735-l559 Module: Week 7: Analytical Patterns & CDC · Enrolled students In this video lecture, we explore fundamental concepts in data engineering, with a primary focus on the pivotal role of SQL as the foundational element in this field. The discussion extends to the patterns and structures essential for scaling data engineering projects, providing a comprehensive understanding of their implementation. Delving into aggregation-based analysis, we address considerations surrounding dimensions and granularity. The lecture also touches upon the significance of retention analysis and survivorship curves, offering insights that are valuable for career growth. Join us in this session to gain a profound understanding of the key elements shaping data engineering and the strategic insights necessary for success in this dynamic field. ### Build your first pipeline with dbt Lab URL: https://www.dataexpert.io/lesson/dbtbasicsday1lab-jan2026-p735-l1925 Module: Week 1: dbt Basics · Enrolled students Learn how to build your first data pipeline using dbt in this hands-on lab session. Participants will explore key concepts such as models, sources, and seeds while working with Snowflake and Airflow. The lesson covers essential dbt commands, environment variable setup, and best practices for organizing and referencing data, enabling data engineers to effectively manage and transform data in their projects. ### Optimizing Spark in production and balancing costs URL: https://www.dataexpert.io/lesson/optimizingsparkinproductionandbalancingcosts-aug26 Enrolled students In this lecture, Zach explains why Spark jobs run slowly, emphasizing that tuning memory and partitions is often a red herring. He outlines main bottlenecks in order of commonness, starting with data model issues where pipelines process too much data that is not changing, and then source file bottlenecks such as the small file or non-splittable file problem. He shares examples including breaking a 11 hour daily deduplication group by into hourly runs to finish in about 30 minutes, and an Airbnb case where reading a single 250 GB CSV took four hours until it was changed to 50 parquet files, reducing runtime to five minutes. He also covers platform congestion and misconfiguration, and notes that only eight categories of fixes typically help, mainly data modeling, upstream formats, staging, shuffle partitions, driver behavior, and adaptive query execution. ### TA Office Hour 2 URL: https://www.dataexpert.io/lesson/ta-office-hour-2-july2025 Enrolled students Office hours provide an interactive platform for students to clarify doubts and discuss assignments related to AI and data engineering. Key topics include homework expectations, prompt engineering techniques, and the use of FastAPI and Milvis for building applications. Participants benefit from real-time Q&A, sharing experiences, and receiving guidance on completing assignments effectively, enhancing their understanding of the course material. ### TA Office hour 4 URL: https://www.dataexpert.io/lesson/taofficehour4-3892c Enrolled students ### Lesson 4 URL: https://www.dataexpert.io/lesson/lesson-4-829 Enrolled students ### Advance Snowflake Day 1 Lab URL: https://www.dataexpert.io/lesson/advancesnowflakeday1lab-c6310 Enrolled students ### Book Club 2 URL: https://www.dataexpert.io/lesson/book-club2-july2025 Enrolled students The Book Club session focuses on Retrieval-Augmented Generation (RAG) and its applications in AI engineering. Participants discuss insights from the assigned reading, including the importance of understanding the differences between vector and graph RAG, and the role of data engineering skills in optimizing AI solutions. This session is beneficial for students looking to deepen their comprehension of RAG concepts and their practical implications in real-world scenarios. ### Carly Taylor - Field CTO at Databricks (3.0) URL: https://www.dataexpert.io/lesson/techtalksweek3-jan2026-p735-l1936 Module: Tech Talks · Enrolled students Carly Taylor, a seasoned data professional with experience across various industries, shares insights on navigating a career in data science and engineering. This session covers the importance of diverse experiences, the evolving expectations for junior candidates in the age of AI, and the significance of soft skills in interviews. Carly emphasizes the need for candidates to articulate their unique value and understand business problems, making this lesson valuable for aspiring data professionals looking to enhance their career prospects. ### TA Office hours 5 URL: https://www.dataexpert.io/lesson/taofficehours5-mar26 Enrolled students This office hours session provides insights into troubleshooting issues with Telegram bot integration and Docker container management. Participants discuss creating and connecting bots, resolving authentication errors, and ensuring Docker services are running smoothly. Ideal for students and developers working with Telegram bots and Open Claw, this session offers practical advice and support for common technical challenges. ### E2E Testing URL: https://www.dataexpert.io/lesson/e2e-testing-03c32 Module: Office Hours · Enrolled students ### Spark Batch Processing - Architecture, Optimization, and Best Practices (Day 1 Lecture) URL: https://www.dataexpert.io/lesson/spark-batch-day-1-lecture-v3 Module: Week 4: Batch Pipelines with Apache Spark · Enrolled students Explore the essentials of Spark in this video, uncovering key insights and strategies to enhance your skills. We delve into Spark's architecture, distinguish it from technologies like Hive, and offer best practices for job optimization. This session equips you with practical tips to overcome common challenges, ensuring a deeper understanding of Spark and refining your development skills. Join us on this journey to make Spark work seamlessly for you. [Recorded on Dec 5th, 2023 ### Book Club 5 URL: https://www.dataexpert.io/lesson/bookclub5-feb26 Enrolled students The Book Club 5 session focuses on reflections from participants at the end of a rigorous boot camp in data engineering. Attendees share their experiences, insights on data engineering concepts, and personal growth throughout the program. The discussion also includes networking opportunities and strategies for continued learning and professional development in the field, making it beneficial for both aspiring and experienced data engineers looking to enhance their skills and connections. ### TA Office Hour 3 URL: https://www.dataexpert.io/lesson/taofficehour3-aug26 Enrolled students Office Hour 3 features a live Q&A session where students discuss various topics related to their coursework and homework assignments. Key discussions include clarifications on homework requirements, schema changes, and project guidelines. Students also inquire about capstone project expectations and the use of tools like Amazon S3. This session is beneficial for those seeking direct support and guidance on their assignments and project development in data engineering. ### TA Office hour 5 URL: https://www.dataexpert.io/lesson/taofficehour5-b1b83 Enrolled students ### Conceptual Data Modeling Lecture Day 1 URL: https://www.dataexpert.io/lesson/conceptual-data-modeling-lecture-day-1-april2025 Enrolled students In this lecture, Zach discusses the essential steps for building effective data models, especially for those looking to advance in their careers. He shares his experiences and lessons learned from past projects, emphasizing the importance of understanding stakeholder requirements and avoiding false requirements that can complicate data pipelines. He also touches on the significance of balancing technical and business insights in data engineering. ### Advanced Data Pipeline Maintenance - Practical Exercises and Runbook Creation (Day 1 Lab) (V2) URL: https://www.dataexpert.io/lesson/pipeline-maintenance-day-1-lab-v3 Module: Week 6: Data Pipeline Maintenance · Enrolled students This lab session was recorded in V2. Please refer to the lecture for V3 content. In this lab session, we simulate data pipeline maintenance scenarios and begin creating a runbook for on-call procedures, preparing you to handle real-world data pipeline maintenance tasks effectively. ### TA Office hours 7 URL: https://www.dataexpert.io/lesson/taofficehours7-mar26 Enrolled students This session features a live Q&A format where participants discuss navigating the program interface, specifically focusing on the overview and lessons tabs within the Spring AI boot camp. Attendees benefit from real-time assistance and clarification on accessing course materials, enhancing their understanding of the platform's features. ### Joe Reis URL: https://www.dataexpert.io/lesson/joe-reis-jan2025 Enrolled students Joe Reis shares insights on data modeling and its evolving role in data engineering during this engaging session. He discusses the importance of thinking beyond traditional table-centric approaches and explores the integration of AI in data workflows. Attendees will benefit from his expertise on data modeling techniques, the significance of robust data structures for AI, and the future of data engineering in a rapidly changing landscape. This session is ideal for data professionals looking to enhance their understanding of modern data practices. ### Q&A with Zach 5 URL: https://www.dataexpert.io/lesson/qawithzach5-mar26 Enrolled students A live Q&A session with Zach wraps up the AI engineering boot camp, featuring discussions on participants' experiences and successes throughout the program. Attendees share their personal journeys, including job placements and project implementations, highlighting the practical applications of AI and data engineering skills learned. Key topics include the importance of foundational knowledge, the integration of AI in business, and strategies for effective project management in AI engineering. ### Mastering Apache Spark Day2 Lab URL: https://www.dataexpert.io/lesson/masteringapachesparkday2lab-aug26 Enrolled students In this lab, Zach explains cumulative table design in Spark, focusing on how to build faster downstream historical metrics from minute and daily aggregates. He walks through creating a daily stock metrics table and then building a cumulative table using a full outer join with coalesce and case logic to track day over day and all time highs. He compares cumulative incremental updates to full recomputation, noting trade offs: cumulative can process far less data but is sequential and adds operational overhead for backfills. He then demonstrates large non splittable file issues using a single gzip, CSV and contrasts it with splittable bzip and Delta, and closes with how Spark skew is handled by adaptive query execution and how to diagnose and mitigate it when needed. ### week-5-Lab 2 URL: https://www.dataexpert.io/lesson/week-5-lab2 Module: Advanced Backend Development · Enrolled students This lab focuses on building an event-driven service using Red Panda and feature toggling with Unleash. Participants will learn to set up a Kafka cluster, create producers and consumers, and implement message handling in a Node.js environment. The lesson also covers integrating feature flags to control the deployment of new features, making it ideal for developers looking to enhance their backend development skills with modern tools and techniques. ### Apache Iceberg Lab Day3 URL: https://www.dataexpert.io/lesson/apacheiceberglabday3-9e57d Enrolled students The Apache Iceberg Lab Day 3 focuses on implementing the write audit published pattern using the Polygon API and Apache Spark. Participants will learn how to create and manage Iceberg tables, handle paginated API responses, and perform data quality checks on stock ticker data. This session is ideal for data engineers looking to enhance their skills in data ingestion, API consumption, and data auditing practices within the Iceberg framework. ### Special Guest, Alex Freberg [August 2, 2023] URL: https://www.dataexpert.io/lesson/v2-special-guest-alex-freberg-august-2-2023-6626 Enrolled students Join us each week as we sit down with visionary speakers from diverse fields, engaging in enlightening conversations that explore cutting-edge insights, trends, and their wealth of expertise. (Recorded August 2, 2023) ### Tuesday SQL Quiz URL: https://www.dataexpert.io/lesson/sql-aggregation-tuesdayquiz Enrolled students ### Q&A with Zach 2 URL: https://www.dataexpert.io/lesson/qawithzach2-mar26 Enrolled students This session features a live Q&A with Zach, addressing various topics related to the bootcamp and AI engineering. Key discussions include group dynamics, API limits, and troubleshooting homework reviews. Participants also explore agentic architecture, the use of tools like LangChain and Snowflake, and strategies for effective project management in AI. This interactive format allows students to ask questions and clarify concepts, making it beneficial for anyone looking to deepen their understanding of AI tools and project workflows. ### Enhancements and Implementations in MLflow URL: https://www.dataexpert.io/lesson/enhancementsandimplementationsinmlflow-feb26 Enrolled students In this video, Eumar provides an update on the GitHub repository enhancements for the Day 2 labs in the agentic module, focusing on the latest MLflow 3 implementation. He demonstrates how to leverage this new version for improved model tracing, observability, and deployment, including A-B testing capabilities. He encourages everyone to run the updated labs, as they now include features like human and LLM evaluations, synthetic dataset generation, and a streamlined user experience for feedback. He reminds everyone to check the changes in the catalogs and engage with the new evaluation sessions, emphasizing that their feedback is crucial as the team refines the models and processes. ### Scaling Delta Live Table and Kafka URL: https://www.dataexpert.io/lesson/scalingdeltalivetableandkafka-aug26 Enrolled students In this lecture, Zach explains streaming pipeline design, focusing on why Kappa architecture and Delta Live Table have become more practical than older Lambda approaches. He contrasts Lambda streaming plus batch true up for correctness versus Kappa’s single streaming pipeline for low latency, noting Lambda doubles code and was historically needed due to limitations like Hive metastore lacking append. He explains how Python UDFs hurt performance because Python. He also describes Delta Live Table as an orchestration layer like Airflow that defines DAG dependencies, discusses medallion bronze silver gold concepts, and warns about DLT drawbacks such as cost for low-volume data and limited streaming join support. ### Spark Structured Streaming Day 2 Lecture URL: https://www.dataexpert.io/lesson/spark-structured-streaming-day-2-lecture-jan2025-p736-l816 Module: Week 4: Structured Streaming with Spark & Kafka · Enrolled students Gain a comprehensive understanding of Spark Structured Streaming in this lecture, focusing on its advantages, use cases, and key concepts like sources, sinks, and streaming aggregates. Ideal for data engineers and professionals interested in real-time data processing, the session covers practical examples, low-latency requirements, and integration with tools like Kafka. Learn how to handle duplicates, manage watermarks, and configure processing intervals to optimize your streaming applications. ### Capstone Showcase Jan 2025 URL: https://www.dataexpert.io/lesson/capstone-showcase-jan2025-p736-l910 Module: Capstone Project · Enrolled students The Capstone Showcase features a diverse range of projects from aspiring data engineers, highlighting their final presentations. Participants demonstrate their skills in real-time data integration, analytics, and visualization using various tools and technologies. Key topics include train delay monitoring applications, eSports voice analytics, stock market indicators, and air quality monitoring systems. Each project showcases unique problem-solving approaches and the application of data engineering principles, providing valuable insights for potential employers and peers in the field. ### Tech Talk - Shubham URL: https://www.dataexpert.io/lesson/techtalks1-feb26 Enrolled students Shubham, a principal data engineer at Amazon, shares insights from his extensive career in data engineering during this engaging tech talk. He discusses the evolution of data engineering practices at Amazon, including the transition from legacy systems to modern data architectures, and the importance of ownership and critical thinking in engineering roles. This session is beneficial for aspiring data engineers looking to understand career progression, the impact of AI on the industry, and how to align technical work with business outcomes. ### Ben Rogojan - Fractional Head of Data URL: https://www.dataexpert.io/lesson/tech-talk-3-ben-rogojan-april2025-p735-l995 Module: Tech Talks · Enrolled students The analytics engineering Q&A session features Ben Rogojan, a fractional head of data and consultant with experience at Meta. He shares insights on transitioning from big tech to consulting, emphasizing the importance of relationship-building and trust with clients. Ben discusses strategies for effective client engagement, pricing models, and the challenges of data infrastructure, making this session valuable for aspiring consultants and data professionals looking to enhance their skills and understanding of the consulting landscape. ### TA Office hours 6 URL: https://www.dataexpert.io/lesson/taofficehours6-mar26 Enrolled students Office hours provide an opportunity for students to ask questions and seek clarification about their capstone projects. Discussions include project proposals, data storage options using AWS and Superbase, and the criteria for successful project submissions. Participants also inquire about homework grading and the timeline for project submissions, ensuring they stay on track as they navigate their data engineering and AI boot camp experience. ### Sunday Quiz URL: https://www.dataexpert.io/lesson/sundayquiz-83952 Enrolled students ### Master Advanced Snowflake Data Modeling with the VARIANT data type URL: https://www.dataexpert.io/lesson/masteradvancedsnowflakedatamodelingwiththevariantdatatype-jan2026 Enrolled students In this lesson, Zach dives deep into unstructured data, data modeling, and some exciting features of Snowflake. He explores the exponential growth of unstructured data, driven by the fact that nearly five billion people now have smartphones, creating massive amounts of content daily. Zach also introduces Cortex, their AI tool, and how it can help them process and model unstructured data effectively. ### Career Development - Interview Help URL: https://www.dataexpert.io/lesson/career-development-interview-help-jan2025-p736-l782 Module: Career Development · Enrolled students Career development in data engineering interviews involves understanding various stages, including recruiter screening, technical assessments, and behavioral evaluations. This session provides insights into the typical interview process, covering key topics such as data modeling, debugging, and end-to-end data architecture. Ideal for job seekers in data engineering, it offers strategies for effectively communicating your skills and experiences to potential employers, ensuring you're well-prepared for each interview round. ### Processing data with Apache Flink (Day 2 Lecture) URL: https://www.dataexpert.io/lesson/flink-streaming-day-2-lecture-v4 Module: Week 5: Kafka-Flink Streaming Pipelines · Enrolled students In this lecture, Zach discusses real-time data processing, focusing on streaming and the importance of tools like Flink and Kafka. He demonstrates how these tools handle data flow and the significance of maintaining schema contracts. [Recorded on June 6th, 2024] ### Tech Talk - Jason URL: https://www.dataexpert.io/lesson/techtalks5-feb26 Enrolled students Jason Reid shares insights from his extensive experience in data engineering, including his time at Netflix and as co-founder of Tabular. He discusses the evolution of data engineering roles, emphasizing the shift towards action-oriented architectures and the impact of AI on data modeling. This session is beneficial for data engineers and aspiring founders, providing valuable perspectives on navigating the changing landscape of data technologies and the importance of focusing on outcomes over processes. ### Live SQL Training January 15th URL: https://www.dataexpert.io/lesson/live-sql-training-jan-15 Module: Interview Skills · Enrolled students In this video I go over some SQL concepts like JOINs, GROUP BY, etc ### Azure - Week 1 URL: https://www.dataexpert.io/lesson/azure-week-1-may2025 Enrolled students Understanding cloud computing is essential for data engineers, and this session introduces Azure's foundational concepts. Participants will learn about Azure's storage account service, its role in data management, and how to leverage it for scalable data storage. Key topics include the differences between blob storage and Azure Data Lake Storage (ADLS), the importance of region selection, and best practices for authentication using access keys and SAS tokens. This lesson is ideal for beginners looking to build a solid foundation in Azure cloud services. ### Advanced dbt Day 2 Lab URL: https://www.dataexpert.io/lesson/advanced-dbt-day-2-lab-jan2025-p736-l793 Enrolled students In this lab, Bruno walks you through the essentials of defining packages in DBT and how to utilize macros effectively. He also covers the importance of logs for debugging and demonstrate how to set up incremental models. Please pay special attention to the examples he provides, as they will help clarify these concepts. ### Kick off URL: https://www.dataexpert.io/lesson/kickoff-8982f Enrolled students ### Structured Streaming Kafka to Delta Live Table Day2 Lecture URL: https://www.dataexpert.io/lesson/structuredstreamingkafkatodeltalivetableday2lecture-feb26 Enrolled students In this lecture, Zach dives deeper into Delta Live Table and its evolution from traditional streaming architectures like Lambda and Kappa. He discusses the challenges of Python UDFs, emphasizing their scalability issues, and highlights the advantages of Delta Tables, which simplify the transition from streaming to batch processing. He explores the Medallion Architecture and its implications for data quality, and shares insights on managing bad data in their pipelines. ### Li Yin - Founder of Adal (2025-07-10) URL: https://www.dataexpert.io/lesson/tech-talk-li-july2025-p737-l1287 Module: Tech Talks · Enrolled students Li Yin, founder of SILFAI, shares insights from her decade-long experience in AI engineering, including her time at Meta. This session covers the importance of understanding user needs, the evolving landscape of AI engineering, and practical advice for aspiring AI engineers, such as focusing on project work and leveraging tools like Ataflow for prompt optimization. Attendees will benefit from Li's perspective on navigating the AI field and the skills needed to stand out in job applications. ### Dimensional Data Modeling - Slowly-changing Dimensions and idempotent queries with Trino (Day 2 Lab) - Pre-recorded from V3 URL: https://www.dataexpert.io/lesson/dimensional-data-modeling-day-2-lab-v4 Module: Week 1: Dimensional Data Modeling in Iceberg · Enrolled students This is a pre-recorded lesson from V3. In this lab, we reinforce lecture concepts with a hands-on approach. We dive into implementing a Slowly Changing Dimension (SCD) table. Learn to track user activity changes using lag functions and identify active and inactive player seasons. We also tackle common errors in query optimization, providing practical solutions for a real-world understanding. Watch the video for step-by-step guidance on implementing SCDs in your data pipelines. [Recorded Nov 8, 2023] ### Evaluations and Reliability MLOps Lecture URL: https://www.dataexpert.io/lesson/endtoendaiapplicationsday1lecture-mar26-p737-l2153 Module: Week 5: End-to-End AI Applications · Enrolled students In this lecture, Zach talks about end to end AI apps, focusing on MLOps, evaluations, and reliability. He explains where agents break, like user behavior drift, code and tool changes via MCP, model upgrades, RAG data changes, and even tool or web downtime, plus latency and context window issues. He breaks down unit tests, integration tests, and end to end evaluation, and four evaluation styles, cosine distance, ROUGE, exact match, custom scoring, and LLM as a judge. ### Friday Quiz URL: https://www.dataexpert.io/lesson/fridayquiz-5af2b Enrolled students ### Data Modeling Interview URL: https://www.dataexpert.io/lesson/data-modeling-interview Module: Interview Skills · Enrolled students In this video, Zach talks about how to clear a Data Modeling interview. ### OpenClaw Safeguards & Measuring business value for AI Agents Day1 Lecture URL: https://www.dataexpert.io/lesson/openclawsafeguardsmeasuringbusinessvalueforaiagentsday1lecture-mar26-p737-l2149 Module: Week 4: MLOps, Guardrails & Business Value · Enrolled students Understanding the complexities of AI agents is essential for anyone involved in AI engineering. This lecture covers the importance of establishing guardrails for AI agents like OpenClaw, focusing on reliability, failure modes, and effective logging practices. Participants will learn about various types of failures, including hallucinations and scope creep, and how to implement safeguards to prevent unintended actions. This session is particularly beneficial for AI developers and engineers looking to enhance the safety and reliability of their AI systems. ### Andrew Madson - Head of DevRel at Fivetran URL: https://www.dataexpert.io/lesson/tech-talk-with-andrew-madson-p735-l590 Module: Tech Talks · Enrolled students Andrew Madsen, Head of DevRel at Fivetran, shares insights on the evolving role of analytics engineers in today's data landscape. He discusses the importance of aligning technical outcomes with business results and the skills that set analytics engineers apart, such as effective communication and the ability to prioritize data requests. This session is ideal for professionals seeking to enhance their understanding of analytics engineering and its impact on business strategy. ### MCP and Agentic AI Day 1 Lab URL: https://www.dataexpert.io/lesson/mcpandagenticaiday1lab-0ct2025 Enrolled students In this video, Zach walks through the MCP Agents Lab, focusing on the tools being used, particularly Langchain and its MCP adapters. He dives into how to create and utilize various tools for GitHub, including generating issues and pull requests, while emphasizing the importance of type hinting in Python for better context understanding. Zach also demonstrates the process of creating an agent that interacts with multiple repositories and discusses the challenges faced with rate limits and tool management. ### Cumulative DAGs in Production Day 3 Lecture URL: https://www.dataexpert.io/lesson/cumulative-dags-in-production-day-3-lecture-jan2025 Enrolled students Cumulative DAGs are a complex yet powerful concept in data engineering, particularly within Apache Airflow. This lecture explores the debate between functional and efficient data engineering, focusing on cumulative table design and its implications for data duplication and maintenance. Participants will learn about the trade-offs of cumulative designs, including their benefits for analytics and the challenges they pose for backfilling and managing historical data. ### Apache Flink Streaming - Kafka, Postgres, Spark Integration, and Parallelism (Day 2 Lab) URL: https://www.dataexpert.io/lesson/infra-flink-streaming-day-2-lab-v3 Module: Week 3: Streaming Pipelines with Apache Flink · Enrolled students In this lab, we delve into data processing capabilities of Flink. We will explore components like Kafka, Postgres, and Spark, witnessing how they collaborate in efficient data pipelines. The concept of parallelism and its impact on Flink jobs will be demystified. Throughout the session, we will highlight key features and considerations for managing sizable Flink pipelines. To top it off, we will run a live popularity contest, showcasing Flink in action. Gain a swift understanding of Flink capabilities and optimization strategies in this hands-on lab.[Recorded Nov 30, 2023] ### Capstone Project Day 2 Class URL: https://www.dataexpert.io/lesson/capstone-project-day-2-class-april2025 Enrolled students In this class, Zach walks through our recent lab work on user master DAG and how we can optimize our data management processes. He highlights the importance of parameterization and the need for a clear naming convention in our tables. He also discusses the certification process and how we can track user engagement effectively. ### TA Office Hour 4 URL: https://www.dataexpert.io/lesson/ta-office-hour4-oct2025 Enrolled students ### Day 1 Lab: Using MCP to build more powerful agents URL: https://www.dataexpert.io/lesson/building-end-to-end-ai-application-day1-lab-july25 Enrolled students In this lab, Zach walks through the MCP lab where he explores building AI agents using LangChain and FastAPI. He covers the necessary Python versions, dependencies like LangChain and Pydantic, and how to set up the environment. Zach demonstrates how to interact with the MCP server, including creating and managing endpoints, as well as using the GitHub MCP server for various tasks. ### QnA with Zach - Jun7th 2026 - Lecture URL: https://www.dataexpert.io/lesson/qnawithzachjun7th2026lecture Enrolled students ### Grouping sets and Window functions Lecture URL: https://www.dataexpert.io/lesson/grouping-sets-and-window-functions-lecture Module: Applying Analytical Patterns · Enrolled students In this presentation, Zach explores a range of topics related to Advanced SQL Analytical Patterns. The discussion covers the application of recursive CTEs, count case when, grouping sets, cross join unnest, cube, rollup, and window functions. Drawing from Zachs experience in conducting interviews and working with these concepts, the presentation offers insights and practical advice. ### OpenClaw and Automating business processes Day2 Lab URL: https://www.dataexpert.io/lesson/openclawandautomatingbusinessprocessesday2lab-mar26 Enrolled students In this lab, Drew connects the OpenClaw to Zapier via an MCP server, using a fresh Gmail for the agent, least privilege tool access, and locking Gmail tools to only send or read from your agent email. He demonstrates the test email, calendar and tasks, set up a heartbeat, and do prompt injection and tool audit exercises. ### Delta Table Day 2 Lab URL: https://www.dataexpert.io/lesson/delta-table-day-2-lab-may2025 Enrolled students In this video, Yared walks viewers through various data transformation techniques using SQL, focusing on data profiling, cleaning, and complex transformations. He highlights the importance of understanding data distribution and how to handle JSON data effectively. He also discusses the creation of silver tables and the significance of defining schemas for efficient data processing. ### Monday ML Quiz URL: https://www.dataexpert.io/lesson/mondaymlandaiquiz-e4a32 Enrolled students ### Data Visualization and Impact: Insights and Best Practices Day 2 Lecture URL: https://www.dataexpert.io/lesson/data-visualization-and-impact-day-2-lecture-yt-p736-l692 Enrolled students In this insightful lecture, we delve into the crucial realm of dashboard metrics. Exploring the importance of various metric types, including total aggregates, time and entity-based aggregates, dimensional mix, and retention and survivorship numbers, the video provides a comprehensive understanding. ### Book Club 4 URL: https://www.dataexpert.io/lesson/bookclub4-jan2026 Enrolled students This session focuses on data modeling and the resources available for students in the analytics engineering boot camp. It highlights the importance of cumulative tables, pipeline specification documents, and analytical patterns, while also discussing access to previous boot camp content. Ideal for learners looking to deepen their understanding of data modeling techniques and enhance their skills in analytics engineering. ### Databricks Platform Overview Day 1 Lecture URL: https://www.dataexpert.io/lesson/databricks-platform-overview-day-1-lecture-jan2025-p736-l771 Module: Week 1: Databricks Basics · Enrolled students In this lecture, Yared delves into the intricacies of data architecture, focusing on the concept of lake house architecture, Delta Lake, and Databricks Data Intelligence. He breaks down the evolution from data warehousing to the innovative lake house architecture, highlighting the benefits and applications in real-world scenarios. We dive into the world of structured and unstructured data, the role of data lakes, and the integration of advanced analytics for a comprehensive understanding of modern data management. ### Data Lakes with Delta Table Day1 Lecture URL: https://www.dataexpert.io/lesson/datalakeswithdeltatableday1lecture-feb26 Enrolled students In this lecture, Zach dives into what a lake house is, how it differs from data lakes and warehouses, and the nuances between Delta and Iceberg. He emphasizes the importance of understanding the architecture, anti-patterns, and the unique features of Delta tables, including time travel and data immutability. Zach also highlights the need for proper data governance, especially concerning PII, and the significance of using vacuuming to manage data effectively. ### Capstone Project Day 3 Class URL: https://www.dataexpert.io/lesson/capstone-project-day-3-class-april2025-p735-l1056 Module: Week 8: End-to-End Project Hacking · Enrolled students In this lab, zach walks through the data pipeline process, emphasizing the importance of not writing directly to production. He discusses various data quality checks and how we aggregate data to ensure everything is functioning correctly. He also highlights the significance of monitoring declines in data quality. ### TA Office Hour 6 URL: https://www.dataexpert.io/lesson/ta-office-hour-6-may2025 Enrolled students This office hours session focuses on addressing student inquiries related to Delta Live Tables and Databricks workflows. Participants discuss querying APIs, loading JSON data into Delta Live Tables, and implementing medallion architecture. The session also covers capstone project proposals, providing feedback and sharing resources among students to enhance their understanding and project development. ### Sunday Quiz URL: https://www.dataexpert.io/lesson/sundayquiz-ff389 Enrolled students ### Applying Advanced SQL for Analytical Insights - Mastering Growth Accounting: Hands-On Journey through User Growth and Retention Analysis (Day 1 Lab) URL: https://www.dataexpert.io/lesson/analytical-patterns-day-1-lab-v3 Module: Week 3: Analytical Patterns & Analysis in Iceberg · Enrolled students In this hands-on lab session, we embark on a practical journey of growth accounting. The video guides you through the step-by-step process, illustrating how to create a partition column, calculate active dates, and adeptly track state transitions. Common challenges encountered in this process are addressed, accompanied by effective solutions. By the conclusion of the video, you will have gained a lucid understanding of analyzing user growth and retention. This session equips you with practical skills to navigate and excel in the intricacies of growth accounting, providing valuable insights applicable to real-world scenarios. [Recorded on Nov 27, 2023] ### Career Development - Strategic Networking URL: https://www.dataexpert.io/lesson/career-development-strategic-networking-jan2025-p736-l825 Module: Career Development · Enrolled students Strategic networking is essential for career growth and visibility in the data and AI engineering fields. This lesson explores various networking strategies, including optimizing your LinkedIn profile, engaging in meetups, and leveraging social media platforms to connect with industry influencers and potential mentors. Participants will learn how to identify key individuals to network with, enhance their credibility, and access hidden job markets through effective communication and relationship-building techniques. ### Joining a new team URL: https://www.dataexpert.io/lesson/joining-a-new-team Module: Soft Skills · Enrolled students ### Q&A with Zach Week2 URL: https://www.dataexpert.io/lesson/qawithzachweek2-feb26-p736-l2038 Module: Q&A with Zach · Enrolled students Week two's Q&A session covers a range of topics centered on Apache Spark and the evolving landscape of AI and data engineering. Participants engage in discussions about the implications of AI on job markets, the importance of storytelling as a uniquely human skill, and practical insights into data modeling in Databricks. Attendees can ask questions about their projects and gain valuable perspectives on navigating their careers in this dynamic field. ### Setting Up CI/CD and Unit Testing in Databricks for Reliable Data Pipelines URL: https://www.dataexpert.io/lesson/setting-up-ci-cd-and-unit-testing-in-databricks-for-reliable-data-pipelines-may2025-p736-l1195 Module: Week 2: Databricks & Advanced Spark · Enrolled students In this video, Zach walks through the process of setting up Continuous Integration/Continuous Deployment (CI/CD) and unit testing in Databricks, with a particular emphasis on enhancing the reliability of data pipelines. He begins by creating a Git folder for an Airflow DBT project and explore the YAML configuration for GitHub Actions, which will trigger tests upon pull requests. The demonstration includes running unit tests and addressing bad data scenarios, highlighting the critical importance of ensuring accurate data transformations. ### Q&A Week 3 URL: https://www.dataexpert.io/lesson/qa-week-3-jan2025 Enrolled students In this Q-A session, Zach shares his journey of moving and setting up my new place while also discussing the Capstone project for our data engineering bootcamp. He emphasizes the importance of creating a solid conceptual data model and provide guidelines for tackling the project effectively. He also touches on the significance of collaboration and mentorship during this process. This recording has been cut short, we are working on adding the second part to it. ### Local Video URL: https://www.dataexpert.io/lesson/lession3-664da Enrolled students Understanding memory management and best practices in data engineering is crucial for efficient workflows. This lesson explores the challenges of using Apache Airflow for backfilling stock prices, particularly the memory inefficiencies of Python and the importance of offloading heavy ETL tasks to AWS Glue. Participants will learn how to optimize their data pipelines by implementing best practices, ensuring smoother execution and better resource management in their data engineering projects. ### Madison Schott - Analytics Engineer (3.0) URL: https://www.dataexpert.io/lesson/techtalksweek2-jan2026-p735-l1935 Module: Tech Talks · Enrolled students Madison Schott shares her extensive experience as an analytics engineer, discussing the evolution of the role and the importance of balancing technical skills with stakeholder communication. She emphasizes the significance of data modeling, the nuances of ELT versus ETL, and how AI is reshaping analytics engineering. This session is beneficial for aspiring analytics engineers and data professionals looking to understand the complexities of the field and improve their ability to deliver business value through data. ### Special Guests, Carly Taylor and Ben Rogojan [July 19, 2023] URL: https://www.dataexpert.io/lesson/v2-special-guests-carly-taylor-and-ben-rogojan-july-19-2023-309 Enrolled students Join us each week as we sit down with visionary speakers from diverse fields, engaging in enlightening conversations that explore cutting-edge insights, trends, and their wealth of expertise. (Recorded July 19, 2023) ### Managing Unstructured Data - Day 2 Lab URL: https://www.dataexpert.io/lesson/managing-unstructured-data-day-2-lab-may2025-p736-l1199 Module: Week 5: Managing Unstructured Data · Enrolled students In this session, Zach explores how to process unstructured data effectively and generate structured outputs using AI. He demonstrates creating a schema for code review feedback, a crucial aspect of enhancing collaboration and quality in software development. He also highlighted the integration of this data into Iceberg and the use of GitHub for analysis. ### TA Office Hour 4 URL: https://www.dataexpert.io/lesson/taofficehour4-aug26 Enrolled students Office Hour 4 provides an interactive Q&A session for students to discuss their assignments and projects. Participants address issues related to assignment submissions, feedback on homework, and preparation for interviews in data engineering roles. The session also covers insights into using tools like DataBricks and SQL for data management, alongside strategies for tackling data structures and algorithms in interview settings. This lesson is beneficial for students seeking support and guidance in their data engineering journey. ### Advanced dbt Day 2 Lecture URL: https://www.dataexpert.io/lesson/advanced-dbt-day-2-lecture-jan2025-p736-l792 Enrolled students In this lecture, Bruno dives into advanced DBT concepts, focusing on Jinja, macros, and incremental models. He explains how to use Jinja for Python-like functionality in SQL and the importance of macros in creating reusable SQL code. He also discusses best practices for running DBT models, particularly the significance of selecting models to optimize costs. ### Orchestration and Airflow Fundamentals Lab URL: https://www.dataexpert.io/lesson/orchestration-and-airflow-fundamentals-lab Module: Pipeline Spec Building + Airflow Fundamentals · Enrolled students In this lab, Zach walks you through using Astro for deployment, focusing on creating and running data pipelines with airflow. He demonstrate steps like creating new DAGs, running Trino data quality checks, and shifting to the next pipeline. ### Data Lakes with Delta Table Day2 Lab URL: https://www.dataexpert.io/lesson/datalakeswithdeltatableday2lab-feb26-p736-l2024 Module: Week 3: Data Lakes with Delta Table · Enrolled students In this video, Zach walks you through Delta tables, partitioning, and the Unity Catalog. He explore how to create schemas and tables, and demonstrates the importance of using Unity Catalog to manage table names more effectively. He also discusses liquid clustering and its advantages over traditional partitioning, emphasizing best practices for clustering columns. ### TA Office hour 3 URL: https://www.dataexpert.io/lesson/taofficehour3-feb26-p736-l2049 Enrolled students This office hours session addresses common issues encountered by students during their homework and lab exercises. Participants discuss troubleshooting API errors, specifically a 401 unauthorized error, and how to adjust code to ensure proper execution. The session also covers accessing functions within labs and modifying commented-out code to enable full functionality, making it beneficial for students seeking clarification on practical coding challenges. ### Dimensional Data Modeling Complex Data Type and Cumulation Day 1 Lecture URL: https://www.dataexpert.io/lesson/dimensional-data-modeling-lecture-day-1-yt-p736-l658 Free In this lecture, we talk about one big table vs relational vs dimensional data modeling and their tradeoffs! ### Q&A with Zach Week 4 URL: https://www.dataexpert.io/lesson/q-a-with-zach-week-4-may2025-p736-l1161 Module: Q&A with Zach · Enrolled students A live Q&A session with Zach covers various topics related to capstone projects, including real-time data consumption, project scope, and technical challenges. Participants discuss using APIs, building dashboards, and integrating tools like DBT and Iceberg. Zach provides insights on project feasibility, data modeling, and the importance of user experience in applications. The session also touches on the upcoming AI boot camp and its curriculum, emphasizing practical skills and real-world applications. ### Lession 3 URL: https://www.dataexpert.io/lesson/lession-3-9548 Enrolled students ### Capstone Project Brainstorming URL: https://www.dataexpert.io/lesson/capstoneprojectbrainstorming-feb2026 Enrolled students Capstone Project Brainstorming focuses on generating innovative ideas for capstone projects in data engineering and AI. Participants engage in a collaborative discussion, exploring various project concepts that meet specific requirements, such as integrating multiple data sources and implementing agentic actions. The session also emphasizes the importance of real-time data processing, quality controls, and the use of tools like cloud services and APIs, providing valuable insights for aspiring data engineers and AI practitioners. ### Monitoring Models in Production with Tracing and LangSmith URL: https://www.dataexpert.io/lesson/monitoring-models-in-production-with-tracing-and-langsmith-oct2025 Enrolled students ### RAG and LLMs Day1 Lecture URL: https://www.dataexpert.io/lesson/rag-and-llms-day-1-lecture-jan2025-p736-l826 Module: Bonus - LLMs · Enrolled students In this lecture, Zach dives into the powerful implications of LLMs and RAG techniques for engineers, emphasizing the importance of adapting to these changes. He discusses the differences between fine-tuning and RAG, and how they can be leveraged together for better outcomes. He also highlights the significance of creating effective prompts for generating SQL with LLMs. ### Big Picture AI Applications Lab URL: https://www.dataexpert.io/lesson/building-end-to-end-ai-application-day2-lab-oct25-p737-l1522 Module: Week 5: End-to-End AI Applications · Enrolled students In this lab, Zach walks through the integration of various components, concentrating on n8n and webhooks to build a LinkedIn shout‑out workflow. Zach demonstrates how to set up triggers, parse and handle JSON data, and implement a human‑in‑the‑loop validation process using Discord for approvals. He guides through troubleshooting steps, ensuring the application is deployment‑ready. ### Career Development - Resume Review URL: https://www.dataexpert.io/lesson/career-development-resume-review-jan2025-p736-l764 Module: Career Development · Enrolled students A comprehensive resume is essential for advancing your career, and this session focuses on optimizing your resume for data engineering roles. Learn how to effectively structure your resume, including key sections such as summary, professional experience, and projects, while emphasizing measurable achievements and relevant tools. This lesson is beneficial for job seekers aiming to enhance their resumes to attract recruiters and align with job postings in the data engineering field. ### MLOPs and Guardrails Day 2 Lab URL: https://www.dataexpert.io/lesson/ml-ops-day2-lab-oct25-p737-l1525 Module: Week 4: MLOps, Guardrails & Business Value · Enrolled students In this lab, Zach covers evaluation criteria, tests the RAG system, and explores auto-prompt optimizations. He demonstrates how to test the RAG system using semantic searches and sets evaluation criteria for expected results. Zach emphasizes the importance of integrating these tests into the CI-CD pipeline to ensure robustness. Additionally, he showcases how to optimize prompts using AI-generated suggestions, which results in improved accuracy in the database analytics agent. ### Q&A with Zach Week5 URL: https://www.dataexpert.io/lesson/qawithzachweek5-jan2026 Enrolled students The final Q&A session for the analytics engineering boot camp covers critical deadlines and project requirements for participants. Key topics include the extended submission deadline for capstone projects, certification criteria, and the importance of demonstrating progress in analytics engineering skills using tools like DBT, Airflow, and Snowflake. This session is beneficial for students seeking clarity on project expectations and certification pathways as they prepare for their upcoming submissions. ### Conor Brennan - CEO & Founder at Hypersell URL: https://www.dataexpert.io/lesson/tech-talk-conor-july2025-p737-l1290 Module: Tech Talks · Enrolled students Conor Brennan, CEO of Hyperspell, discusses the importance of context in AI systems, particularly focusing on retrieval-augmented generation (RAG) and how it can enhance AI applications. He shares insights from his experience with unstructured data and explains how Hyperspell serves as a context layer for AI, enabling developers to access relevant information from various sources like Slack, Notion, and Google Drive. This session is beneficial for data engineers and aspiring AI engineers looking to understand the complexities of integrating context into AI systems. ### Shachar Meir - Data Advisor and Public Speaker URL: https://www.dataexpert.io/lesson/shachar-meir-jan2025-p736-l724 Module: Guest Speakers · Enrolled students Shachar Meir shares insights from his extensive career in data engineering and advisory roles, emphasizing the importance of understanding business outcomes over mere technical tasks. He discusses the significance of resilience in navigating career challenges and the necessity of soft skills for professional advancement. This lesson is particularly beneficial for data professionals seeking to enhance their communication, strategic thinking, and career direction while learning how to effectively create and articulate value in their roles. ### Book Club 1 URL: https://www.dataexpert.io/lesson/bookclub1-mar26 Enrolled students The session focuses on prompt engineering, a crucial skill for effectively communicating with AI models. Participants discuss strategies for crafting prompts, such as specificity and context, and how these affect the quality of AI responses. The conversation also explores the evolving nature of prompt engineering, emphasizing its importance in designing AI products and the potential shift towards context engineering as models improve. This lesson benefits anyone looking to enhance their interaction with AI systems. ### Wednesday Cumulative Data Quiz URL: https://www.dataexpert.io/lesson/cumulative-data-quiz Enrolled students ### RAG optimisation + Reranking Day 1 Lab URL: https://www.dataexpert.io/lesson/ragoptimisation-reranking-day1-lab-oct2025 Enrolled students In this lab, Zach focuses on re-ranking and working with an external worker process, both crucial for optimizing their retrieval system. He implements metadata filtering and re-ranking to improve query results, ensuring accurate classification of document types and determining the optimal number of documents to retrieve. Zach demonstrates how to set up a PubNub channel for asynchronous processing, which allows the team to handle longer tasks without timing out. ### Growth Accounting Lab URL: https://www.dataexpert.io/lesson/growthaccountinglab-jan2026 Enrolled students In this lab, Zach walks through building a state transition tracking system to analyze user behavior, including churn and retention rates. He works with several datasets, specifically focusing on platform users and website events, and he demonstrates how to create and manipulate tables using SQL. He also highlights the importance of understanding user activity over time and how to set up a growth accounting DAG with data quality checks. ### TA Office hour 5 URL: https://www.dataexpert.io/lesson/taofficehour5-b1b83-p736-l2051 Enrolled students ### Q&A with Zach Week 4 URL: https://www.dataexpert.io/lesson/qa-with-zach-week4-july2025 Enrolled students A live Q&A session with Zach covers various topics related to AI engineering, homework challenges, and project development. Participants discuss the balance between leveraging AI tools and maintaining their coding skills, as well as strategies for effective system design in AI applications. Zach emphasizes the importance of understanding requirements and the role of AI in enhancing data quality. The session encourages an open dialogue about learning experiences and the evolving landscape of AI technology. ### Orchestration and Airflow Fundamentals Lecture URL: https://www.dataexpert.io/lesson/orchestration-and-airflow-fundamentals-day-1-lecture-may2025 Enrolled students Welcome to the first day of the Data Engineering Bootcamp for Spring 2025! In this session, Zach will dive into the fundamentals of orchestration and Airflow, covering key concepts such as data collection, transformation, and analysis. He will also discuss common pitfalls in orchestration and how to avoid them. ### Hands-on: Crafting Your Promotion Narrative URL: https://www.dataexpert.io/lesson/handsoncraftingyourpromotionnarrative-c2500 Enrolled students Selecting which accomplishments demonstrate Staff+ level work Quantifying impact for different audiences Common pitfalls: vanity metrics, missing context, weak multiplier effects Hands-on: Write you Staff+ accomplishment in STAR format and tranform into Impact Story ### Office Hour 3 URL: https://www.dataexpert.io/lesson/office-hour-3-p735-l603 Enrolled students This TA Office hours was hosted by Sundeep and Weeks 1 and 2 homework was discussed. ### Databricks and Advanced Spark Day1 Lecture URL: https://www.dataexpert.io/lesson/databricksandadvancedsparkday1lecture-feb26-p736-l2025 Module: Week 2: Databricks & Advanced Spark · Enrolled students In this lecture, Zach discusses the intricacies of Shuffle Joins and Optimization in Apache Spark. He highlights the importance of understanding business needs when processing data. Zach also shares insights on the app-to-app communication analysis for security, where processing the entire dataset was crucial. ### MCP and Agentic AI Day 1 Lab URL: https://www.dataexpert.io/lesson/mcpandagenticaiday1lab-0ct2025-p737-l1607 Module: Week 3: Agentic AI, MCP & Automation · Enrolled students In this video, Zach walks through the MCP Agents Lab, focusing on the tools being used, particularly Langchain and its MCP adapters. He dives into how to create and utilize various tools for GitHub, including generating issues and pull requests, while emphasizing the importance of type hinting in Python for better context understanding. Zach also demonstrates the process of creating an agent that interacts with multiple repositories and discusses the challenges faced with rate limits and tool management. ### Introduction to Spark Day 2 Lecture URL: https://www.dataexpert.io/lesson/introduction-to-spark-day-2-lecture-jan2025-p736-l775 Module: Week 1: Databricks Basics · Enrolled students In this lecture, Yared dives into the fundamentals of Apache Spark, discussing its architecture and how it processes big data. He highlights the importance of Spark's DataFrames and the various programming languages it supports, such as SQL, Python, and Scala. He also touches on the execution process and the concept of lazy evaluation. ### Apache iceberg Lab Day 3 URL: https://www.dataexpert.io/lesson/apache-iceberg-lab-day-3-april2025 Enrolled students In this lab, Zach walks you through the Apache Iceberg Day 3 lab, focusing on iceberg branching and working with stock data from an API. He demonstrates how to create a new version of a table and run queries using Spark and Glue. He also discusses the importance of data quality checks and how to manage branches effectively. ### Deploying AI agents with Agent Bricks Day2 Lab URL: https://www.dataexpert.io/lesson/deployingaiagentswithagentbricksday2lab-aug26 Enrolled students In this lesson, Eumar focuses on deploying an AI agent named Zachy using Data Bricks. The key objectives includes packaging the agent for version control, integrating it with the Unity catalog, and deploying a chat application that utilizes the agent. He explains how to monitor agent performance and handle user feedback through ML Flow. He emphasizes the importance of building the agent from scratch without using supervisory agents, promoting a hands-on approach to understanding the underlying processes. ### Special Guest, Joe Reis [Recorded Dec 8, 2023] URL: https://www.dataexpert.io/lesson/v3-3-special-guest-joe-reis-recorded-dec-8-2023-205 Enrolled students In this captivating speaker series, Joe Reis delivers an insightful talk on the pivotal role of data engineering in facilitating informed decision-making. With a profound understanding of the challenges data engineering addresses, Joe shares valuable suggestions drawn from his extensive experience. Delving into the historical evolution of the data industry, he underscores the critical importance of navigating the vast landscape of data. Whether you are a seasoned data professional or someone keen on understanding the impact of data engineering on decision-making, this talk offers a wealth of knowledge and expertise. Dive into this special session for an enriching exploration of the dynamic intersection between data, engineering, and strategic decision-making.[Recorded on Dec 8th, 2023] ### dbt basics: How dbt works: sources, models, packages Lab URL: https://www.dataexpert.io/lesson/dbt-basics-how-dbt-works-lab-p735-l566 Module: Week 1: dbt Basics · Enrolled students In this lab, Bruno delves into DBT best practices, emphasizing the importance of one-to-one staging and the connection of staging stripe payments to payment sources. He discusses the use of prefixes like FACT for clarity and the creation of intermediate models to simplify complex code. ### RAG optimisation + Reranking Day 1 Lab URL: https://www.dataexpert.io/lesson/rag-optimisation-rerankingday1-lab-oct2 Enrolled students ### Decoding Staff+ Promotion Packets URL: https://www.dataexpert.io/lesson/decodingstaffpromotionpackets-e6365 Enrolled students Anatomy of successful Staff+ promotion packets Case studies: what worked and what failed The "impact story" framework that resonates with evaluators Hands-on: Impact Inventory—audit your last 12 months against Staff+ expectations ### Chat with Zach [Dec 14, 2023] URL: https://www.dataexpert.io/lesson/zach-qa-2023-12-14-v3 Enrolled students Delve into the world of data engineering with our weekly Q&A / Fireside Chat series, featuring the founder of EcZachly Inc and lead instructor of the Advanced Data Engineering Bootcamp. Each session covers a different topic, providing invaluable insights and real-world experiences to empower data enthusiasts in their journey.[Recorded on Dec 14, 2023] ### Understanding Read Streams and SCD Implementation Lab URL: https://www.dataexpert.io/lesson/understandingreadstreamsandscdimplementationlab-jan2026-p735-l2006 Module: Week 7: Analytical Patterns & CDC · Enrolled students Learn how to implement Slowly Changing Dimensions (SCD) using read streams in this lab focused on NBA player data. Participants will explore creating and managing tables to track player activity over time, utilizing SQL commands such as CREATE TABLE, INSERT, and MERGE. This lesson is ideal for data engineers and analysts looking to enhance their skills in change data capture and data modeling techniques. ### Streaming Pipelines: Mastering Streaming and Real-time Pipelines Day 1 Lecture URL: https://www.dataexpert.io/lesson/streaming-pipelines-day-1-lecture-yt-p736-l674 Enrolled students This comprehensive video presentation explores the concepts of streaming and real-time data processing, distinguishing between low latency and high latency and elucidating their impact on data processing. The discussion encompasses the differences between real-time and near real-time processing, emphasizing the necessity of clear communication with stakeholders. ### Databricks Boot Camp Kickoff URL: https://www.dataexpert.io/lesson/databricksbootcampkickoff-feb2026-p736-l2093 Module: Kickoff · Enrolled students ### Data Impact Communication & Visualization - Exploring Data Aggregation and Visualization Techniques (Day 2 Lab) URL: https://www.dataexpert.io/lesson/visual-impact-day-2-lab-v3 Module: Week 6: Data Impact Communication & Visualization with Tableau · Enrolled students Welcome to a hands-on lab where we explore diverse data aggregation and visualization techniques using a sample dataset. The video takes you through the intricacies of aggregating data at different levels, such as host and source, and demonstrates the creation of meaningful visualizations. Emphasizing the advantages of pre-aggregating data for accelerated query performance, this lab provides valuable insights into data analysis and visualization. [Recorded on Dec 20, 2023] ### Special Guests, Stephanie Nuesi and Joe Reis [July 26, 2023] URL: https://www.dataexpert.io/lesson/v2-special-guests-stephanie-nuesi-and-joe-reis-july-26-2023-9127 Enrolled students Join us each week as we sit down with visionary speakers from diverse fields, engaging in enlightening conversations that explore cutting-edge insights, trends, and their wealth of expertise. (Recorded July 26, 2023) ### Tech Talk - Ryan URL: https://www.dataexpert.io/lesson/tech-talk-ryan-july25 Enrolled students Ryan Brandt shares his extensive experience as an independent AI consultant, focusing on the challenges faced by companies adopting AI technologies. He discusses the agent debugging loop, emphasizing the importance of measuring performance, identifying errors, and iterating on solutions. This lesson is beneficial for data engineers and AI practitioners looking to enhance their understanding of AI agent performance and improve their debugging processes through practical techniques and tools. ### Applying Analytical Patterns Day 1 Lecture URL: https://www.dataexpert.io/lesson/applying-analytical-patterns-day-1-lecture-jan2025-p736-l814 Enrolled students Understanding analytical patterns is crucial for data engineers and analysts looking to streamline their data pipeline processes. This lecture covers three key analytical patterns: aggregation-based analysis, cumulation-based patterns, and window-based analysis. Participants will learn how to apply these patterns to create efficient SQL queries, track user behavior over time, and analyze trends effectively, ultimately enhancing their data analysis skills and improving project outcomes. ### Tech Talk - Vaishali URL: https://www.dataexpert.io/lesson/tech-talk-vaishali-may2025 Enrolled students Vaishali shares her extensive experience as a data scientist, discussing the evolving landscape of data engineering and AI. She emphasizes the importance of collaboration between data scientists and engineers, highlighting how effective communication can enhance project outcomes. The session also explores the impact of AI tools on data workflows and the necessity for professionals to adapt to these changes to remain competitive in the job market. ### Tuesday Quiz URL: https://www.dataexpert.io/lesson/tuesdayquiz-2a59e Enrolled students ### Graph Data Modeling Lecture URL: https://www.dataexpert.io/lesson/graph-data-modeling-lecture Module: Bonus · Enrolled students Zach discusses the importance of data modeling and metrics in building a successful pipeline. He emphasizes that conceptual data modeling is accessible to everyone and highlight the need to consider the right people for feedback and collaboration. He also touches on the challenges of understanding stakeholders' true requirements and share his experience with choosing the right database solution. Throughout the video, he emphasizes the importance of metrics-driven approaches and the potential impact of well-designed dashboards. Overall, this lecture provides valuable insights and guidance for data engineers looking to optimize their pipeline. ### Brian Pulliam - Career Development Coach URL: https://www.dataexpert.io/lesson/techtalks2-mar26-p737-l2115 Module: Tech Talks · Enrolled students Brian Pulliam, a seasoned career development coach with extensive experience in tech, shares insights on navigating the evolving job market in this engaging session. He discusses the importance of adaptability, the cyclical nature of tech hiring, and strategies for standing out in interviews. Participants will benefit from his practical advice on leveraging personal strengths, understanding market dynamics, and building a compelling resume that highlights business impact, making this a valuable resource for anyone looking to advance their tech career. ### How to load data into Iceberg with Airflow URL: https://www.dataexpert.io/lesson/howtoloaddataintoicebergwithairflow-jan2026 Enrolled students Learn how to effectively load data into Iceberg using Airflow by exploring real-world orchestration challenges faced at major tech companies. This lesson covers key topics such as defining availability in data pipelines, optimizing performance with staging steps, and the importance of item potency in data integrity. Data engineers and AI practitioners will benefit from insights on improving pipeline efficiency and handling complex data transformations. ### advanced dbt Day 2 Lab URL: https://www.dataexpert.io/lesson/advanceddbtday2lab-95c0c Enrolled students ### Tuesday Quiz URL: https://www.dataexpert.io/lesson/tuesdayquiz-4fea1 Enrolled students ### Tech Talk - Parth Parekh URL: https://www.dataexpert.io/lesson/v4-tech-talk-parth-parekh-4534 Enrolled students In this tech talk, Zach discusses his experience working with Parth Parekh and how his knowledge has influenced his understanding of data engineering. They cover topics such as date lists, cumulative tables, full outer joins, and processing facts. He also explores the challenges of scaling data processing and the new frontier of data engineering in the context of text models, image models, and multi-models. Throughout the talk, he emphasizes the importance of structuring unstructured data and the role of data engineering in creating labeled and structured data for machine learning models. [Recorded on 17th May, 2024] ### Tech Talk - Brian URL: https://www.dataexpert.io/lesson/techtalks2-mar26 Enrolled students Brian shares insights from his extensive experience in tech, including roles at Coinbase and as a career coach. He discusses the evolving job market, emphasizing the importance of adaptability and the need for strong interpersonal skills alongside technical expertise. This session is beneficial for anyone navigating their career in tech, especially those preparing for interviews or considering a shift to AI-related roles. Brian also highlights the significance of networking and storytelling in securing job opportunities. ### End-to-End AI Applications Day 1 Lecture URL: https://www.dataexpert.io/lesson/leveraging-mcp-and-tools-to-integrate-effectively-oct2025 Enrolled students In this lecture Zach walks the class through what AI excels at—prediction, detection, and optimization—while pointing out common barriers such as vague business objectives and user resistance to change. Zach argues for stronger frameworks and highlights N8N as a superior choice to the OpenAI Agent Kit for constructing AI workflows. ### Apache Spark Shuffle Joins Day 1 Lecture URL: https://www.dataexpert.io/lesson/apache-spark-shuffle-joins-day-1-lecture-may2025-p736-l1142 Module: Week 2: Databricks & Advanced Spark · Enrolled students In this lecture, Zach discusses how we handle two petabytes of data daily at Netflix, focusing on different sampling techniques to optimize processing. He shares insights on the importance of precision in data analysis and how we managed to reduce processing time and costs significantly by using a 0.1% sample. He also touches on the challenges of dynamic IP addresses in our cloud environment and the need for collaboration with application owners to implement effective logging. ### Advance Snowflake Day 2 Lab URL: https://www.dataexpert.io/lesson/advancesnowflakeday2lab-0be54 Enrolled students ### Book Club 2 URL: https://www.dataexpert.io/lesson/book-club-2-oct2025 Enrolled students ### Optimize your dev workflow with Claude Code and Codex Day1 Lab URL: https://www.dataexpert.io/lesson/optimizeyourdevworkflowwithclaudecodeandcodexday1lab-mar26-p737-l2142 Module: Week 2: Advanced RAG, Reranking & Dev Workflow · Enrolled students In this video, Kevin walks viewers through building an RSS feed reader using Cloud and React with TypeScript. He begins with a vague prompt to demonstrate how context affects the output, then refines the approach by planning additional features like tagging and marking articles as read. Kevin emphasizes the importance of validating code and following best practices in development. ### Intro to NodeJS & Express and event driven programming URL: https://www.dataexpert.io/lesson/intro-to-nodejs-express-and-event-driven-programming Module: Intermediate Backend Development · Enrolled students Understanding NodeJS and Express is essential for building efficient backend applications, especially those that require non-blocking I/O operations. This lesson covers the fundamentals of NodeJS, its event-driven architecture, and how to create REST APIs using Express. Ideal for intermediate developers, you'll learn about asynchronous programming, middleware, and the benefits of using NodeJS for scalable applications, alongside practical examples and best practices for structuring your code. ### Q&A with Zach 3 URL: https://www.dataexpert.io/lesson/qawithzach3-mar26-p737-l2121 Module: Q&A with Zach · Enrolled students Week 3 of the Q&A session with Zach covers a range of topics focused on AI token management, proxy server challenges, and strategies for optimizing AI agent performance. Participants discuss the implications of token usage, the importance of logging and tracing in AI systems, and the nuances of building effective AI agents. This session is particularly beneficial for those looking to deepen their understanding of AI infrastructure and improve their engineering practices in real-world applications. ### Data Communication and Impact (Day 1 Lecture) URL: https://www.dataexpert.io/lesson/visual-impact-day-1-lecture-v4 Module: Week 6: Data Impact Communication & Visualization with Tableau · Enrolled students This is a pre-recorded class. Join this insightful video as it delves into the world of data engineering. In this session, common challenges encountered in data engineering are explored, accompanied by practical solutions and best practices for constructing efficient data pipelines. The discussion extends beyond technical aspects, offering valuable insights into career progression, shedding light on the journey to becoming a staff engineer, and emphasizing the pivotal role of mentorship. Enhance your data engineering skills, unlock career growth strategies, and gain actionable tips to propel your professional journey. [Recorded on Dec 18th 2023] ### Understanding Read Streams and SCD Implementation Lab URL: https://www.dataexpert.io/lesson/understandingreadstreamsandscdimplementationlab-ede60 Enrolled students ### Change Data Capture Lab URL: https://www.dataexpert.io/lesson/changedatacapturelab-ec6de Enrolled students ### Book Club 5 URL: https://www.dataexpert.io/lesson/book-club-5-oct2025 Enrolled students ### OpenClaw Safeguards & Measuring business value for AI Agents Day2 Lab URL: https://www.dataexpert.io/lesson/openclawsafeguardsmeasuringbusinessvalueforaiagentsday2lab-mar26-p737-l2152 Module: Week 4: MLOps, Guardrails & Business Value · Enrolled students Please see the instructions here - https://road-suede-29f.notion.site/Participant-guide-34bfce66e714800b996dd75dcf92b9a1 Instructions to set up Mission Control Run on your vm with openclaw: git clone https://github.com/robsannaa/openclaw-mission-control.git cd openclaw-mission-control ./setup.sh Run locally: ssh -N -L 3333:127.0.0.1:3333 root@yourserver ### Test Course URL: https://www.dataexpert.io/lesson/test-course-6038 Module: test · Enrolled students ### Day 1 Lab: Auto Prompt Optimization with dspy URL: https://www.dataexpert.io/lesson/prompt-engineering-day1-lab-jul2025-p737-l1170 Module: Week 1: Prompt Engineering & Basic RAG · Enrolled students In this lab, Zach dives into analyzing code in a GitHub repository and discussed auto prompt optimization techniques, specifically focusing on two examples: the auto optimize candidates pool and counting.py. He guides through setting up their Python environment, generating a GitHub personal access token, and running the code to fetch repository contents. Zach emphasized the importance of structured data in prompt engineering and how to create effective prompts to ensure consistent JSON output. ### OpenClaw and Automating business processes Day1 Lab URL: https://www.dataexpert.io/lesson/openclawandautomatingbusinessprocessesday1lab-mar26 Enrolled students In this video, Drew walks through setting up OpenClaw, an open-source project, using Docker. He covers the installation process, security considerations, and how to create your first agent, the Chief of Staff. He emphasises the importance of understanding security risks and using best practices. ### Deep Dive On Workflows Day 3 Lab URL: https://www.dataexpert.io/lesson/deep-dive-on-workflows-day-3-lab-jan2025-p736-l823 Module: Week 4: Structured Streaming with Spark & Kafka · Enrolled students Explore the intricacies of building and managing workflows with Delta Live Tables in this lab session. Participants will learn how to create sequential pipelines, manage task dependencies, and implement data quality checks using tools like Databricks and Delta Lake. The lesson covers practical techniques for loading, transforming, and monitoring data, making it ideal for data engineers and analysts looking to enhance their skills in structured streaming with Spark and Kafka. ### Langchain Starter Repo Setup URL: https://www.dataexpert.io/lesson/langchain-starter-repo Enrolled students Learn how to set up the Langchain starter repository effectively, ideal for beginners in AI engineering. This lesson covers cloning the repository, creating a Python virtual environment, installing necessary libraries, and configuring API keys, specifically for OpenAI. By following these steps, you'll be prepared to run both the backend and frontend of a Langchain application, enabling you to interact with AI tools seamlessly. ### TA Office Hour 2 (2025-07-17) URL: https://www.dataexpert.io/lesson/ta-office-hour-2-july2025-p737-l1293 Enrolled students This session serves as an office hours Q&A for students working on their homework assignments, focusing on prompt engineering and FastAPI development. Participants discuss various aspects of their assignments, including the structure of markdown files, the use of different models for prompt optimization, and the setup of vector databases. The session provides valuable insights into troubleshooting common issues and clarifying assignment requirements, making it beneficial for students seeking guidance on their projects. ### Book Club 3 URL: https://www.dataexpert.io/lesson/bookclub3-cc669 Enrolled students ### Capstone Showcase with Billy Switzer URL: https://www.dataexpert.io/lesson/capstone-showcase-with-billy-switzer-8747 Enrolled students Join us as we welcome Billy Switzer, a standout student from our V4 cohort, who’s here to showcase his impressive capstone project! In this session, Billy walks us through his project, sharing the strategies and skills that helped him succeed. He covers the challenges he faced, the solutions he crafted, and how he leveraged core concepts to build a project that truly stands out. The session wraps up with an open Q&A, where Billy answered questions from the audience, providing valuable insights on capstone creation, project execution, and making an impact in the field. ### Dimensional Data Modeling: Graph Data Modeling Day 3 Lab URL: https://www.dataexpert.io/lesson/dimensional-data-modeling-day-3-lab-yt-p736-l667 Free In this lab, we build a network of NBA players and see who they play against/with the most and for which teams! ### Chat with Zach [Aug 24, 2023] URL: https://www.dataexpert.io/lesson/v2-6-chat-with-zach-aug-24-2023-8742 Enrolled students Delve into the world of data engineering with our weekly Q&A / Fireside Chat series, featuring the founder of EcZachly Inc and lead instructor of the Advanced Data Engineering Bootcamp. Each session covers a different topic, providing invaluable insights and real-world experiences to empower data enthusiasts in their journey. (Recorded Aug 24, 2023) ### MLOPs and Guardrails Day 2 Lab URL: https://www.dataexpert.io/lesson/ml-ops-day2-lab-oct25 Enrolled students In this lab, Zach covers evaluation criteria, tests the RAG system, and explores auto-prompt optimizations. He demonstrates how to test the RAG system using semantic searches and sets evaluation criteria for expected results. Zach emphasizes the importance of integrating these tests into the CI-CD pipeline to ensure robustness. Additionally, he showcases how to optimize prompts using AI-generated suggestions, which results in improved accuracy in the database analytics agent. ### LinkedIn Like Pro! URL: https://www.dataexpert.io/lesson/linkedin-like-pro-april2025-p735-l990 Module: Career Development · Enrolled students Mastering LinkedIn is essential for job seekers and professionals looking to enhance their personal brand. This lesson covers key strategies for optimizing your LinkedIn profile, including crafting an engaging summary, selecting the right profile picture, and utilizing endorsements and recommendations. Learn how to connect with industry professionals, engage with relevant content, and effectively reach out to recruiters to increase your visibility and job prospects on the platform. ### TA Office Hour 8 URL: https://www.dataexpert.io/lesson/ta-office-hour8-oct2025 Enrolled students ### Unit Testing Spark Jobs: Importance, Challenges, and Leadership Perspectives Lecture URL: https://www.dataexpert.io/lesson/apache-spark-day-3-lecture-yt-p736-l706 Enrolled students Unit testing and integration testing are crucial for ensuring the quality of Spark data pipelines, helping developers catch bugs during the development phase rather than in production. This lecture covers the importance of implementing testing strategies, the challenges faced in maintaining data quality, and the mindset shift required for data engineers to adopt software engineering principles. Ideal for data engineers looking to enhance their testing skills and improve pipeline reliability, this session provides practical insights into effective testing practices. ### The Behavioral Interview URL: https://www.dataexpert.io/lesson/the-behavioral-interview Module: Interview Skills · Free This video covers the main things you need to remember to pass the Behavioral interview ### Day 1 Lecture: Prompt Engineering Theory URL: https://www.dataexpert.io/lesson/prompt-engineering-day1-lecture-jul2025-p737-l1169 Module: Week 1: Prompt Engineering & Basic RAG · Enrolled students In this lecture, Zach will dive into prompt engineering, a concept that emerged with ChatGPT in November 2022. He will explore how prompt engineering is reshaping the tech landscape, effective prompting techniques, and the various tasks suited for different LLMs like Claude and ChatGPT. Zach will also address whether prompt engineering is a waste of time, emphasizing the importance of understanding algorithms and data structures even in this new AI-driven world. ### Ryan Brandt - CEO at VundaAI URL: https://www.dataexpert.io/lesson/tech-talk-ryan-july25-p737-l1308 Module: Tech Talks · Enrolled students Ryan Brandt, an independent AI consultant, shares insights on building effective AI agents in this engaging tech talk. He discusses the agent debugging loop, emphasizing the importance of measuring performance, identifying failure modes, and iterating on solutions. This lesson is ideal for data engineers and AI enthusiasts looking to enhance their understanding of AI agent development, error analysis, and the practical application of tools like open coding and axial coding in real-world scenarios. ### Q&A Week 2 URL: https://www.dataexpert.io/lesson/qa-week-2-jan2025-p736-l762 Module: Q&A with Zach · Enrolled students A live Q&A session covers various topics relevant to data engineering and technology adoption. Key discussions include the challenges of using technologies like Janus Graph with Cassandra and Spark, strategies for handling duplicates in streaming data using tumbling windows, and insights on transitioning to senior roles in tech. Participants also explore the importance of networking, the role of data pipelines in machine learning, and the practicalities of using views versus tables in data architecture. ### Joe Reis - Author of Fundamentals of Data Engineering (08-19-2026) URL: https://www.dataexpert.io/lesson/joereisauthoroffundamentalsofdataengineering-aug26 Module: Guest Speakers · Enrolled students Joe Reis, author of "Fundamentals of Data Engineering," shares insights on the evolving landscape of data engineering and modeling in this engaging session. He discusses the implications of new technologies like Data Bricks' LTAP and the importance of understanding foundational concepts in data engineering. This lesson is ideal for data professionals looking to deepen their knowledge of data modeling, the impact of AI, and the future of data architectures. ### Chat with Zach [Nov 30, 2023] URL: https://www.dataexpert.io/lesson/zach-qa-2023-11-30-v3 Enrolled students Delve into the world of data engineering with our weekly Q&A / Fireside Chat series, featuring the founder of EcZachly Inc and lead instructor of the Advanced Data Engineering Bootcamp. Each session covers a different topic, providing invaluable insights and real-world experiences to empower data enthusiasts in their journey. (Recorded Nov 30, 2023) ### Processing Unstructured Data with Databricks URL: https://www.dataexpert.io/lesson/processingunstructureddatawithdatabricks-aug26 Enrolled students In this video, Eumar explains the foundation and architecture for building production-ready AI agents on Databricks across three personas: prospective students, enrolled students, and employees. He emphasizes that governance, observability, and guardrails come from Unity Catalog volumes and MLflow, while a supervisor agent routes questions to specialized agents. He covers why unstructured data matters and how retrieval augmented generation works by grounding LLMs on a vector database, with chunking and embeddings handled in pipelines (often using Spark). He also discuss Databricks tools via MCP through Unity Catalog functions, and outline three build approaches: Genie for text to SQL, AgentBricks for low-code reg, and code-first custom agents, with evaluation and LLM ops planned for the next week. ### Apache Spark programming with Databricks Day 1 Lab URL: https://www.dataexpert.io/lesson/apache-spark-programming-with-databricks-day-1-lab-may2025-p736-l1156 Module: Week 4: Structured Streaming with Spark & Kafka · Enrolled students In this video, Yared discusses about Adaptive Query Execution (AQE) in Spark, highlighting how it optimizes partitioning and joins under the hood, particularly in Spark 3.x. He explaines the differences between on-heap and off-heap memory management, and how Spark handles memory allocation across various components. He also explores Spark UI to track job execution and performance metrics, emphasizing the importance of understanding narrow versus wide transformations. ### Apace Iceberg: Data Compression, Run length encoding Lab URL: https://www.dataexpert.io/lesson/apache-iceberg-lab-day-2-april2025-p735-l958 Module: Week 5: Data Modeling on Iceberg · Enrolled students In this lab, Zach walks through data compression techniques, demonstrating how to reduce data size and enhance efficiency. He explores concepts like run-length encoding and bin-pack compression, focusing on optimizing data storage. ### Optimize your dev workflow with Claude Code and Codex Day2 Lecture URL: https://www.dataexpert.io/lesson/optimizeyourdevworkflowwithclaudecodeandcodexday2lecture-mar26-p737-l2143 Module: Week 2: Advanced RAG, Reranking & Dev Workflow · Enrolled students In this lecture, Kevin walk throughs the compound engineering loop with Claude Code, focusing on agentic context management. He explains Claude md versus skills, how skills are reusable workflows invoked by slash commands, and how skills compound over time. He also covers sub agents and agent teams for parallel work while protecting context, plus agentic validation with multiple fail safes like lint, tests, and UI checks. He briefly introduces MCP and cattle not pets development setup with a VPS, tmux, and SSH. ### Conceptual Data Modeling Lab Day 1 URL: https://www.dataexpert.io/lesson/conceptual-data-modeling-lab-day-1-april2025 Enrolled students In this lab, Zach walks through the process of developing a data model that measures various metrics related to health and productivity. We discuss leading and lagging metrics, such as time spent with family and productive days, and how to categorize them effectively. He encourages the students to think critically about the data sources we can use and the ease of obtaining this data. ### TA Office hour URL: https://www.dataexpert.io/lesson/taofficehour0-feb26-p736-l2095 Enrolled students This office hour session provides an opportunity for students to clarify questions regarding course material, homework, and capstone projects. Participants discuss topics such as data frame creation, project formats, and the importance of time zone updates for group assignments. The session also addresses issues related to accessing resources and the structure of upcoming homework assignments, making it beneficial for students seeking support and guidance in their learning journey. ### Databricks and Advanced Spark Day2 Lab URL: https://www.dataexpert.io/lesson/databricksandadvancedsparkday2lab-feb26 Enrolled students In this Lab, Zach dives into performance tuning and incremental data processing using a substantial dataset of around 50 gigabytes of tweets. He demonstrates the importance of partitioning. He explores the impact of repartitioning strategies on performance. He encourages everyone to experiment with their own partitioning strategies and to utilize the Spark UI for performance analysis. ### Advanced Snowflake: Data Modeling and UDFs Lab URL: https://www.dataexpert.io/lesson/advanced-snowflake-data-modeling-and-udfs-lab-p735-l564 Module: Week 4: Advanced Snowflake · Enrolled students Explore the advanced capabilities of Snowflake in this lab focused on data modeling and user-defined functions (UDFs). You'll learn how to effectively use UDFs to simplify complex SQL queries, particularly for analyzing consecutive performance metrics, such as 20-point seasons for NBA players. This lesson is ideal for data engineers and analysts looking to enhance their skills in Snowflake and streamline their data processing workflows. ### Dimensional Data Modeling: Building Slowly Changing Dimensions Day 2 Lecture URL: https://www.dataexpert.io/lesson/dimensional-data-modeling-day-2-lecture-yt-p736-l660 Free We cover slowly changing dimensions and idempotent pipelines in this lecture! Enjoy! ### Special Guest, Joe Reis [Recorded Dec 8, 2023] URL: https://www.dataexpert.io/lesson/guest-speaker-2023-12-08-v3 Enrolled students In this captivating speaker series, Joe Reis delivers an insightful talk on the pivotal role of data engineering in facilitating informed decision-making. With a profound understanding of the challenges data engineering addresses, Joe shares valuable suggestions drawn from his extensive experience. Delving into the historical evolution of the data industry, he underscores the critical importance of navigating the vast landscape of data. Whether you are a seasoned data professional or someone keen on understanding the impact of data engineering on decision-making, this talk offers a wealth of knowledge and expertise. Dive into this special session for an enriching exploration of the dynamic intersection between data, engineering, and strategic decision-making.[Recorded on Dec 8th, 2023] ### Navigating Data Engineering -Tips, Tools, and Career Insights URL: https://www.dataexpert.io/lesson/navigating-data-engineering-tips-tools-and-career-insights-may2025 Enrolled students This session serves as a makeup Q&A for week three, focusing on data engineering tips, tools, and career insights. Key topics include transitioning from notebooks to IDEs, using command line tools effectively, and insights into working with AWS Glue, Kafka, and data engineering roles. Participants will gain practical advice on navigating their careers in data engineering, including strategies for skill development and job applications, particularly for those with analytics engineering backgrounds. ### Tech Talk with Ryan URL: https://www.dataexpert.io/lesson/tech-talk3-oct2025 Enrolled students ### Apache Spark Unit Testing Day 3 Lab URL: https://www.dataexpert.io/lesson/apache-spark-unit-testing-day-3-lab-may2025 Enrolled students In this lab, Zach walks you through the process of creating unit tests in PySpark, including how to handle bad data and ensure your tests pass. He demonstrates the importance of testing logic without impacting datasets and share tips on using named tuples and data frames effectively. He also discusses the significance of CI/CD practices and how to properly deploy changes. ### How to use Snowpark and Snowflake Optimizations Lecture Day 2 URL: https://www.dataexpert.io/lesson/how-to use-snowpark-and-snowflake-optimizations-lecture-day-2-april2025-p735-l983 Module: Week 3: Data Modeling on Snowflake · Enrolled students In this lecture, Zach dives into the key aspects of Snowflake Data Modeling, particularly focusing on Snowpark and Iceberg. He discusses the importance of data governance controls, performance tuning, and the trade-offs between speed and cost when it comes to data clustering. He also highlights the need for proper catalog integration and the significance of understanding your data usage patterns. ### dbt basics Day 1 Lecture URL: https://www.dataexpert.io/lesson/dbt-basics-day-1-lecture-april2025 Enrolled students In this lecture, Bruno dives into the dbt framework, explaining its purpose and how it helps manage data chaos through SQL. He discusses its features, including data lineage and dependency management, and highlight its integration with cloud data warehouses. He also touches on the role of the analytics engineer and how dbt facilitates collaboration within teams. ### Saturday Quiz URL: https://www.dataexpert.io/lesson/saturdayquiz-628a3 Enrolled students ### Growth Accounting Models Lab URL: https://www.dataexpert.io/lesson/growth-accounting-models-lab-p735-l582 Module: Week 7: Analytical Patterns & CDC · Enrolled students In this hands-on lab session, we embark on a practical journey of growth accounting. The video guides you through the step-by-step process, illustrating how to create a partition column, calculate active dates, and adeptly track state transitions. Common challenges encountered in this process are addressed, accompanied by effective solutions. By the conclusion of the video, you will have gained a lucid understanding of analyzing user growth and retention. This session equips you with practical skills to navigate and excel in the intricacies of growth accounting, providing valuable insights applicable to real-world scenarios. ### Fact Modeling Friday Quiz URL: https://www.dataexpert.io/lesson/fact-modeling-quiz Enrolled students ### Special Guests, Jitender Aswani and Bill Inmon [August 23, 2023] URL: https://www.dataexpert.io/lesson/v2-special-guests-jitender-aswani-and-bill-inmon-august-23-2023-9372 Enrolled students Join us each week as we sit down with visionary speakers from diverse fields, engaging in enlightening conversations that explore cutting-edge insights, trends, and their wealth of expertise. (Recorded August 23, 2023) ### Write a comprehensive pipeline spec document URL: https://www.dataexpert.io/lesson/write-a-comprehensive-pipeline-spec-document-lecture-p735-l553 Module: Week 6: Airflow & Pipeline Specs with Trino · Enrolled students Zach discusses the importance of data modeling and metrics in building a successful pipeline. He emphasizes that conceptual data modeling is accessible to everyone and highlight the need to consider the right people for feedback and collaboration. He also touches on the challenges of understanding stakeholders' true requirements and share his experience with choosing the right database solution. Throughout the video, he emphasizes the importance of metrics-driven approaches and the potential impact of well-designed dashboards. Overall, this lecture provides valuable insights and guidance for data engineers looking to optimize their pipeline. ### LLM-Driven Data Engineering Day 2 Lab URL: https://www.dataexpert.io/lesson/large-language-models-day-2-lab Free In this bonus LLM lab, you will learn how to extend your LLMs capabilities beyond text generation. This includes tasks like sending emails, crafting LinkedIn posts, and exploring the role of agents in aiding you in your technical work such as when bugs arise in your pipeline or database. ### Wednesday Quiz URL: https://www.dataexpert.io/lesson/wednesdayquiz-33b10 Enrolled students ### Tech Talk - Stephanie Nuesi URL: https://www.dataexpert.io/lesson/guest-speaker-6-v4 Enrolled students In this tech talk, Stephanie reviews Zach's resume and provide feedback on what's working and what's not. She emphasizes the importance of content over formatting and highlight the need to tailor the resume to the desired expertise. She also discusses the significance of including specific details, such as software used and results achieved.[Recorded on May22nd, 2024] ### Saturday Quiz URL: https://www.dataexpert.io/lesson/saturdayquiz-a156e Enrolled students ### Test Loom URL: https://www.dataexpert.io/lesson/test-loom Module: test module · Free Setting up an AI engineering application can be complex, but this lesson guides you through the essential steps. You'll learn how to clone a GitHub repository, create a private repo, set up a Python environment, and install necessary libraries like OpenAI, PubNub, and Superbase. This lesson is ideal for aspiring AI engineers looking to establish a solid foundation for their projects and ensure their applications run smoothly in a cloud environment. ### Implementing LTAP data modeling with Lakebase and Delta URL: https://www.dataexpert.io/lesson/implementingltapdatamodelingwithlakebaseanddelta-aug26 Enrolled students In this lesson, Zach explains the difference between analytics-oriented and action-oriented architectures for data products. Action-oriented architectures allow for faster decision-making and greater business impact, as they leverage machine learning models for real-time actions. The concept of "shift left" emphasizes simplifying processes and reducing the number of people needed in data operations. ### Saturday Quiz URL: https://www.dataexpert.io/lesson/saturdayquiz-a77da Enrolled students ### Loom URL: https://www.dataexpert.io/lesson/loom-5954 Enrolled students ### Mastering chain of thought prompting URL: https://www.dataexpert.io/lesson/622282ec-36b9-41df-80e0-1379ef2e855amasteringchainofthoughtprompting Enrolled students ### Tech Talk 6 - Vin Vashishtha URL: https://www.dataexpert.io/lesson/tech-talk-6-vin-vashishtha-april2025 Enrolled students Vin Vashishtha shares insights from his extensive experience in data science, analytics, and AI strategy. He emphasizes the importance of engaging early in project lifecycles to ensure analytics engineers can effectively contribute to business outcomes and avoid burnout. The discussion covers strategies for monetizing data, the significance of personal branding in the AI landscape, and how to advocate for oneself in professional settings. This session is valuable for anyone looking to enhance their impact in data and AI roles. ### TA Office Hour URL: https://www.dataexpert.io/lesson/taofficehour-oct2025 Enrolled students This office hour session provides an interactive platform for students to ask questions and seek guidance on technical issues related to their boot camp setup. Participants discuss troubleshooting connectivity problems with PubNub, setting up Python environments, and managing GitHub repositories. The session is beneficial for students needing assistance with coding environments, version control, and understanding course expectations. ### Chat with Zach URL: https://www.dataexpert.io/lesson/v4-2-chat-with-zach-6658 Enrolled students Recorded on May16th, 2024. ### Fact Data Modeling: Practical Insights into Data Modeling Day 1 Lab URL: https://www.dataexpert.io/lesson/fact-data-modeling-day-1-lab-yt-p736-l663 Enrolled students In this hands-on lab session, we analyze NBA game data, unraveling the intricacies of data modeling and analysis. Breaking down the data into distinct columns, we elucidate the importance of each element. ### Azure - Week 2 URL: https://www.dataexpert.io/lesson/azure-week-2-may2025-p736-l1140 Enrolled students Understanding Azure storage accounts is crucial for data engineers, as they serve as the backbone for data management in cloud environments. This session covers the differences between Azure Blob Storage and Azure Data Lake Storage (ADLS), focusing on their use cases, access methods like Shared Access Signatures (SAS), and redundancy options such as Locally Redundant Storage (LRS) and Geo-Redundant Storage (GRS). Ideal for data engineers and developers, this lesson provides foundational knowledge for building scalable and resilient data platforms on Azure. ### Local Video URL: https://www.dataexpert.io/lesson/local-video-835 Enrolled students ### Orchestration and Airflow Fundamentals Day 1 Lecture URL: https://www.dataexpert.io/lesson/orchestration-and-airflow-fundamentals-day-1-lecture-jan2025-p736-l723 Module: Week 6: Airflow & Pipeline Specs with Trino · Enrolled students In this lecture, Zach breaks down the key aspects of orchestration, focusing on why it's crucial to understand the orchestration process for effective data engineering. He covers what orchestration isn't, the role it plays in data pipelines, and highlights common mistakes to avoid for smooth orchestration. ### Sunday Quiz URL: https://www.dataexpert.io/lesson/sundayquiz-c4dda Enrolled students ### Data Lakes with Delta Table Day2 Lecture URL: https://www.dataexpert.io/lesson/datalakeswithdeltatableday2lecture-feb26 Enrolled students In this lecture, Zach dives into Unity Catalog, explaining how it serves as the central control plane for their Lakehouse, preventing data swamps and providing fine-grained access control. He explores the evolution of data architectures, comparing Lambda and Kappa architectures, and discuss how Databricks is evolving to resemble Snowflake with features like liquid clustering and enhanced governance. ### Tech talk with Vaishali Macwan URL: https://www.dataexpert.io/lesson/tech-talk-with-vaishali-macwan Module: Tech Talk · Enrolled students This is the tech talk with Vaishali Macwan. ### Applying Analytical Patterns Day 1 Lab URL: https://www.dataexpert.io/lesson/applying-analytical-patterns-day-1-lab-jan2025-p736-l815 Enrolled students Learn how to apply analytical patterns using SQL and Starburst in this hands-on lab focused on web event data. Participants will create daily user metrics from a web events table, build a state tracking system to monitor user engagement, and analyze user behavior over time. This lesson is ideal for data engineers and analysts looking to enhance their skills in data transformation and user analytics. ### Tech Talk - Xinran URL: https://www.dataexpert.io/lesson/techtalks4-feb26 Enrolled students Explore the evolving landscape of data engineering and AI in this insightful discussion featuring Shenron, a seasoned data engineer at OpenAI. Gain valuable insights into the shift from traditional coding to leveraging AI tools like Codex for enhanced productivity. This lesson is ideal for data engineers looking to adapt to new technologies and understand the importance of human judgment in decision-making, as well as those interested in networking and career growth in the field. ### Raaid A. - AI, Data, & Product Strategy Leader URL: https://www.dataexpert.io/lesson/techtalksweek4-jan2026-p735-l1937 Module: Tech Talks · Enrolled students Raaid A. shares insights from his extensive experience in data science and engineering, highlighting the transition from senior to staff engineer roles. He discusses the importance of systems thinking, ownership, and leveraging team strengths to drive outcomes. This session is ideal for data engineers and analytics professionals looking to advance their careers by understanding the nuances of leadership and the evolving landscape of AI and data engineering. ### Joe Reis - Author of Fundamentals of Data Engineering URL: https://www.dataexpert.io/lesson/tech-talk-joe-may2025-p736-l1138 Module: Guest Speakers · Enrolled students Joe Reis, author of "Fundamentals of Data Engineering," discusses the evolving landscape of data engineering, focusing on the future of data modeling and the integration of AI and machine learning. He emphasizes the importance of understanding business needs and the convergence of software and data engineering roles. This session benefits aspiring data engineers and professionals looking to adapt to the rapid changes in technology and methodologies within the field. ### Azure - Week 2 URL: https://www.dataexpert.io/lesson/azure-week-2-may2025 Enrolled students Understanding Azure storage accounts is crucial for data engineers, as they serve as the foundation for data management in cloud environments. This session delves into the differences between Azure Blob Storage and Azure Data Lake Storage (ADLS), focusing on their use cases, access methods like Shared Access Signatures (SAS), and the importance of redundancy levels such as LRS, ZRS, GRS, and GZRS. Participants will learn how to effectively utilize these storage solutions for building scalable data platforms and ensuring data availability. ### Day 9: Fast–slow pointers: middle & cycles URL: https://www.dataexpert.io/lesson/fast-slow-pointers-middle-cycles-p913-d9 Enrolled students Find the middle of a linked list and detect/locate cycles using fast–slow (Floyd’s) pointers in O(n) time and O(1) space. ### Tech Talk 7 - Parth Parekh URL: https://www.dataexpert.io/lesson/tech-talk-7-parth-parekh-april2025 Enrolled students Parth Parekh shares insights on the evolving role of data engineers in the context of generative AI during this engaging tech talk. He discusses the intersection of data engineering and AI, emphasizing the importance of high-quality data for model performance and the need for engineers to adapt to new technologies. This session is ideal for data professionals looking to understand the future landscape of data engineering and how to leverage generative AI to enhance their workflows. ### Advanced dbt Day 2 Lecture URL: https://www.dataexpert.io/lesson/advanced-dbt-day-2-lecture-jan2025 Enrolled students In this lecture, Bruno dives into advanced DBT concepts, focusing on Jinja, macros, and incremental models. He explains how to use Jinja for Python-like functionality in SQL and the importance of macros in creating reusable SQL code. He also discusses best practices for running DBT models, particularly the significance of selecting models to optimize costs. ### test 4 URL: https://www.dataexpert.io/lesson/test4-b78f2 Enrolled students ### How to use snowpark and Snowflake optimization Lecture URL: https://www.dataexpert.io/lesson/how-to-use-snowpark-and-snowflake-optimization-lecture Module: Applying Analytical Patterns · Enrolled students Welcome to Snowflake Day 2! In this lecture, Zach delves into various optimization techniques for Snowflake, including cost optimizations and utilizing Snowpark for data processing. He explains the intricacies of Snowpark, its similarities to Spark, and its role in ETL processes within Snowflake. Additionally, He discusses micropartitioning, lazy execution in Snowpark, and the importance of understanding query patterns for cost management. ### Tech Talks 3 URL: https://www.dataexpert.io/lesson/techtalks3-7f251 Enrolled students ### TA Office Hour 5 URL: https://www.dataexpert.io/lesson/ta-office-hour-5-april2025 Enrolled students Office hours provide an opportunity for students to discuss the GPT basics assignment and clarify any questions regarding the project setup and execution. Key topics covered include cloning the repository, creating a virtual environment, building models in DBT, and troubleshooting common issues. This session is beneficial for students seeking guidance on using DBT with Snowflake and understanding the assignment requirements, including setting up schemas and creating tests. ### Chat with Zach [Nov 8, 2023] URL: https://www.dataexpert.io/lesson/v3-1-chat-with-zach-nov-8-2023-5980 Enrolled students Delve into the world of data engineering with our weekly Q&A / Fireside Chat series, featuring the founder of EcZachly Inc and lead instructor of the Advanced Data Engineering Bootcamp. Each session covers a different topic, providing invaluable insights and real-world experiences to empower data enthusiasts in their journey. (Recorded Nov 9, 2023) ### Tech Talk - Joe Reis URL: https://www.dataexpert.io/lesson/v4-tech-talk-joe-reis-1446 Enrolled students In this Tech Talk with Joe Reis, he discusses the importance of data modeling and its impact on data management. He also explains the different levels of data modeling (conceptual, logical, and physical) and emphasize the need for a well-designed data model to ensure data accuracy, clarity, and ease of use. He touches on the drawbacks of not implementing proper data modeling and provide real-life examples to illustrate the concepts. [Recorded on May29th, 2024] ### TA Office Hour 6 URL: https://www.dataexpert.io/lesson/ta-office-hour-6-april2025 Enrolled students Office hours provide an opportunity for students to ask questions and receive guidance on their homework and projects. This session covered troubleshooting issues with DBT, including schema errors and setting up environment variables. Participants discussed the creation of models, YAML files, and SQL files, as well as the importance of submitting only the required files for grading. Additionally, there were inquiries about data contracts and using APIs for capstone projects, with suggestions for resources and further assistance. ### Carly Taylor - Field CTO at Databricks URL: https://www.dataexpert.io/lesson/tech-talk4-oct2025-p737-l1552 Module: Tech Talks · Enrolled students ### Conceptual Data Modeling Lab Day 1 URL: https://www.dataexpert.io/lesson/conceptual-data-modeling-lab-day-1-april2025-p735-l980 Module: Week 3: Data Modeling on Snowflake · Enrolled students In this lab, Zach walks through the process of developing a data model that measures various metrics related to health and productivity. We discuss leading and lagging metrics, such as time spent with family and productive days, and how to categorize them effectively. He encourages the students to think critically about the data sources we can use and the ease of obtaining this data. ### Week 6 lab part 2 URL: https://www.dataexpert.io/lesson/week-6-lab-part2 Module: Advanced Frontend Development · Enrolled students The lab focuses on building a form to submit new to-do items using Next.js and Zod for input validation. Participants will learn how to create server actions for data submission, implement client-side validation, and manage form state using React Hook Form. This lesson is ideal for developers looking to enhance their skills in form handling and validation within a Next.js application, particularly for their capstone projects. ### Delta Table Day 1 Lab URL: https://www.dataexpert.io/lesson/delta-table-day-1-lab-may2025 Enrolled students In this lab, Yared walks you through the structure of Databricks, focusing on the Unity Catalog and Delta Lake. He explains how to create and manage tables, schemas, and permissions, and he also touches on the differences between managed and external tables. He encourages you to access the lab materials and complete the tasks outlined, including creating Delta tables and performing data transformations. ### Chat with Zach [July 27, 2023] URL: https://www.dataexpert.io/lesson/v2-2-chat-with-zach-july-27-2023-5930 Enrolled students Delve into the world of data engineering with our weekly Q&A / Fireside Chat series, featuring the founder of EcZachly Inc and lead instructor of the Advanced Data Engineering Bootcamp. Each session covers a different topic, providing invaluable insights and real-world experiences to empower data enthusiasts in their journey. (Recorded July 27, 2023) ### January 2025 Bootcamp Kickoff URL: https://www.dataexpert.io/lesson/bootcamp-kickoff-jan2025 Enrolled students In this video, Zach will be sharing some exciting updates and important information about the January 2025 Bootcamp. He will be discussing topics like the capstone project, policies, refund policies, and the mentorship program. He will also introduce some of the amazing mentors and TA's who will be guiding you throughout the bootcamp. So get ready for an immersive learning experience and let's learn together. ### Sunday SQL Quiz URL: https://www.dataexpert.io/lesson/sundayquiz-4bfbb Enrolled students ### Mastering Change Data Capture (CDC) URL: https://www.dataexpert.io/lesson/masteringchangedatacapturecdc-jan2026 Enrolled students In this session, Zach dives into Change Data Capture (CDC) and explored why Snowflake is so popular among SQL users for its ease of use and ability to handle big data efficiently. He discusses the differences between Snowflake and Databricks, emphasizing that each serves different user needs—analysts versus engineers. He also highlights the importance of Merge operations and when they can be beneficial or problematic. He also covers various methods for implementing CDC, including read streams and database triggers. ### Tech Talk - Jon URL: https://www.dataexpert.io/lesson/techtalk2-aug26 Enrolled students Jon Hershoff shares his extensive journey from economics to data science, focusing on experimentation and causal inference. He discusses the importance of understanding the impact of marketing and product decisions, utilizing techniques like geo-based experiments and observational causal inference. This session is beneficial for aspiring data scientists and professionals looking to enhance their interview skills, particularly in product and marketing roles, by leveraging real-world experiences and insights. ### Q&A with Zach Week 2 URL: https://www.dataexpert.io/lesson/qa-with-zach-week-2-may2025-p736-l1132 Module: Q&A with Zach · Enrolled students A live Q&A session with Zach covers a range of topics relevant to data engineering and AI. Participants discuss the integration of new tools like Iceberg V3, strategies for enhancing job prospects through networking and personal branding, and the importance of understanding data retention policies. Zach also addresses technical challenges, such as handling non-idempotent pipelines and the implications of data versioning. This session is beneficial for students seeking insights into real-world applications and best practices in data engineering. ### TA Office Hour 2 URL: https://www.dataexpert.io/lesson/ta-office-hour-2-april2025 Enrolled students This office hour session covers various topics related to homework assignments, specifically focusing on the Airflow and DBT reports, as well as using AWS Glue and Iceberg for data processing. Participants discuss installation instructions, schema creation, and data transformation techniques, providing insights for those new to cloud technologies and data engineering. The session also addresses common questions and challenges faced by students, fostering an open discussion for better understanding of the course material. ### Advanced Snowflake Data Modeling and UDFs Lecture Day 3 URL: https://www.dataexpert.io/lesson/advancedsnowflakedatamodelingandudfslectureday3-c192d Enrolled students Advanced data modeling in Snowflake is explored through physical data modeling techniques, focusing on complex data types like arrays, structs, and the versatile variant data type. Participants will learn how to effectively use User-Defined Functions (UDFs) to enhance data processing, including making external API calls, such as integrating with OpenAI. This lesson is ideal for data engineers looking to deepen their understanding of Snowflake's capabilities and improve their data modeling strategies. ### MLOPs and Guardrails Day 1 Lecture URL: https://www.dataexpert.io/lesson/ml-ops-day1-lecture-oct25-p737-l1512 Module: Week 4: MLOps, Guardrails & Business Value · Enrolled students In this lesson, Zach dives into the critical topic of guardrails for ensuring LLM quality in production. He explores the various risks associated with LLMs, such as jail-breaking, prompt injection, and handling sensitive data, while emphasizing the importance of both input and output guardrails. Zach discusses effective implementation strategies for these guardrails, including the use of middleware and moderation APIs. ### Apache Spark: User-Defined Functions and Broadcast Join Day 2 Lab URL: https://www.dataexpert.io/lesson/apache-spark-day-2-lab-yt-p736-l681 Enrolled students In this Spark lab, Zach explains the concept of User-Defined Functions (UDFs) and their application within the Spark execution environment. Through a practical code example, he guides viewers on schema updates and emphasizes the significance of caching. ### TA Office Hour 10 URL: https://www.dataexpert.io/lesson/taofficehour10-535bd Enrolled students ### Advanced pipelines: Blue-green and WAP with Airflow Lab URL: https://www.dataexpert.io/lesson/advanced-pipelines-blue-green-and-wap-with-airflow-lab-p735-l702 Module: Week 2: Advanced dbt · Enrolled students In this lab, Bruno dives into the Airflow part of the project, showcasing how to work within the DBT project folder and run commands efficiently. He demonstrate setting up Airflow, running DBT tasks, and managing connections. ### Apache Spark Core Day 3 Lecture URL: https://www.dataexpert.io/lesson/apache-spark-core-day-3-lecture-jan2025 Enrolled students In this lecture, Yared dives into the Query Execution Engine and explore the different types of transformations in Spark SQL, specifically narrow and wide transformations. He provides examples to illustrate these concepts and discuss the importance of schema inference for performance. He also touches on how to configure clusters for optimal data processing. ### Guest Speakers / Panel Discussion (The Optics & Politics of Staff+ Promotion) URL: https://www.dataexpert.io/lesson/guestspeakerspaneldiscussionnavigatingtheopticspoliticsofpromotionacandidconversation-8596e Enrolled students Handling the "you're not ready yet" feedback Navigating organizational politics without becoming political What to do when your manager isn't supportive Dealing with competing priorities between promotion work and delivery The uncomfortable conversations you need to have ### Q&A with Zach Week 3 (2025-10-09) URL: https://www.dataexpert.io/lesson/qa-with-zach-week3-oct2025-p737-l1546 Module: Q&A with Zach · Enrolled students ### Spark Structured Streaming Day 2 Lab URL: https://www.dataexpert.io/lesson/spark-structured-streaming-day-2-lab-jan2025-p736-l817 Module: Week 4: Structured Streaming with Spark & Kafka · Enrolled students The Spark Structured Streaming Day 2 Lab focuses on building a real-time data ingestion pipeline using Spark and Kafka. Participants will learn how to capture web traffic data, process it through bronze, silver, and gold tables, and create meaningful dashboards for business insights. Key topics include using Kafka for data streaming, transforming data with Spark, and leveraging APIs for geolocation analysis, making it ideal for data engineers and analysts interested in real-time analytics. ### Infrastructure Data Quality - Mastering Spark and PySpark Testing: Comprehensive Overview and Practical Guidance (Day 1 Lab) URL: https://www.dataexpert.io/lesson/infra-data-quality-day-1-lab-v3 Module: Week 5: Unit Testing Pipelines & Write-audit-publish · Enrolled students In this lab, we embark on an exploration of Spark, offering a comprehensive overview of its main components and the intricacies of data transformation within a Spark job. The focus then shifts to the testing paradigm in PySpark code, where I guide you through the creation of input and output data frames and the utilization of named tuples for comparison. Delving into the significance of testing, we unravel its role in enhancing development velocity. The discussion is enriched with insights into common challenges and practical tips for conducting effective tests. Join us in this video to deepen your comprehension of Spark and acquire essential skills in testing PySpark code.[Recorded on Dec 12, 2023] ### Change Data Capture (CDC) and Snowflake basics Lecture URL: https://www.dataexpert.io/lesson/change-data-capture-and-snowflake-basics-lecture-p735-l557 Module: Week 7: Analytical Patterns & CDC · Enrolled students In this lecture, Zach delves into the hot debate between Snowflake and Databricks, focusing on change data capture and the powerful concept of merge. He explores the impact of these technologies on big data over the last decade, discussing simplicity, usability, and performance enhancements. ### Change Data Capture (CDC) and Snowflake basics Lecture URL: https://www.dataexpert.io/lesson/change-data-capture-and-snowflake-basics-lecture Module: Applying Analytical Patterns · Enrolled students In this lecture, Zach delves into the hot debate between Snowflake and Databricks, focusing on change data capture and the powerful concept of merge. He explores the impact of these technologies on big data over the last decade, discussing simplicity, usability, and performance enhancements. ### Prompt Engineering + Basic RAG Day 2 Lab URL: https://www.dataexpert.io/lesson/prompt-engineering-day2-lab-oct2025-p737-l1527 Module: Week 1: Prompt Engineering & Basic RAG · Enrolled students In this lab, Zach sets up a retrieval-augmented generation (RAG) system, which included creating a vector database in Supabase. During the session, he demonstrates how to load synthetic job and profile data into this table, emphasizing the importance of avoiding duplicate vectors to preserve system performance. He also creates a Python script designed to generate embeddings and retrieve relevant job listings based on user queries. ### Q&A with Zach Week 1 URL: https://www.dataexpert.io/lesson/qa-with-zach-week-1-may2025-p736-l1077 Module: Q&A with Zach · Enrolled students The first Q&A session of the boot camp covers a range of topics relevant to data engineering, including pipeline performance validation, data quality checks, and the relevance of functional programming. Participants can expect insights into hands-on techniques for managing late-arriving data, effective cross-functional leadership, and the differences between various data engineering tools. This session is particularly beneficial for those looking to deepen their understanding of practical data engineering challenges and solutions. ### Sunday Quiz URL: https://www.dataexpert.io/lesson/sundayquiz-a92f1 Enrolled students ### Q&A with Zach Week3 URL: https://www.dataexpert.io/lesson/qawithzachweek3-jan2026-p735-l1931 Module: Q&A with Zach · Enrolled students The Week 3 Q&A session covers various topics related to data engineering, including homework challenges, the use of Trino for complex data modeling, and the integration of Snow Park. Participants share experiences with their assignments, discuss strategies for optimizing data pipelines, and explore the implications of moving to new technologies like Iceberg. This session is beneficial for those seeking to deepen their understanding of data engineering practices and improve their project outcomes. ### Tech Talk - Carly Taylor URL: https://www.dataexpert.io/lesson/v4-5-tech-talk-carly-taylor-8456 Enrolled students In this tech talk series with Carly Taylor, she shares insights on career progression in tech, emphasizing the importance of skill set rarity and effective communication. Acknowledging the need for direction-setting and team management, she reflects on personal growth challenges and the value of feedback. [Recorded on June5th, 2024] ### Apache Iceberg Lecture Day1 URL: https://www.dataexpert.io/lesson/apacheiceberglectureday1-4c361 Enrolled students Apache Iceberg is a powerful tool for managing data lake architectures, focusing on efficient data processing and governance. This lecture covers the evolution of big data technologies, starting from MapReduce and Hive, leading to Iceberg's innovative features like time travel and snapshotting. Data engineers and analysts will benefit from understanding how Iceberg improves data management through better partitioning, schema evolution, and the ability to handle large datasets efficiently. ### Tech Talk - Alex URL: https://www.dataexpert.io/lesson/tech-talk-alex-may2025 Enrolled students The session features Alex Merced, a prominent figure in the data engineering community, discussing his journey and insights into Apache Iceberg and the evolving landscape of data engineering. Attendees will benefit from his expertise on Lake House architecture, catalog management, and the interoperability of data tools, including Dremio, Trino, and various catalog services. This talk is ideal for data engineers and analytics professionals looking to deepen their understanding of modern data frameworks and best practices. ### Growth Accounting Models Lecture URL: https://www.dataexpert.io/lesson/growth-accounting-models-lecture Module: Applying Analytical Patterns · Enrolled students In this video lecture, we explore fundamental concepts in data engineering, with a primary focus on the pivotal role of SQL as the foundational element in this field. The discussion extends to the patterns and structures essential for scaling data engineering projects, providing a comprehensive understanding of their implementation. Delving into aggregation-based analysis, we address considerations surrounding dimensions and granularity. The lecture also touches upon the significance of retention analysis and survivorship curves, offering insights that are valuable for career growth. Join us in this session to gain a profound understanding of the key elements shaping data engineering and the strategic insights necessary for success in this dynamic field. ### Office Hour 4 URL: https://www.dataexpert.io/lesson/office-hour-4-p735-l604 Enrolled students Office Hour 4 focuses on homework assignments related to querying NBA player data. Participants learn how to track state changes of players using the bootcamp NBA players table and explore alternative approaches with the NBA player seasons table. The session also covers aggregation techniques using grouping sets to analyze player performance metrics, such as total points scored, across different dimensions. This lesson is beneficial for students looking to deepen their understanding of SQL queries and data manipulation in sports analytics. ### Apache Spark Shuffle Joins Day 1 Lecture URL: https://www.dataexpert.io/lesson/apache-spark-shuffle-joins-day-1-lecture-jan2025 Enrolled students In this lecture, Zach discusses how we handle two petabytes of data daily at Netflix, focusing on different sampling techniques to optimize processing. He shares insights on the importance of precision in data analysis and how we managed to reduce processing time and costs significantly by using a 0.1% sample. He also touches on the challenges of dynamic IP addresses in our cloud environment and the need for collaboration with application owners to implement effective logging. ### Advanced Spark (Day 2 Lecture) URL: https://www.dataexpert.io/lesson/spark-batch-day-2-lecture-v4 Module: Week 4: Batch Pipelines with Apache Spark · Enrolled students In this lecture, Zach covers the Spark APIs, specifically focusing on Dataset, Dataframe, and Spark SQL. He explains the differences between them and their use cases. Additionally, he discusses PySpark UDFs and when to use them. The lecture provides important insights and recommendations for working with Spark. [Recorded on May30th, 2024] ### Delta Table Day 1 Lecture URL: https://www.dataexpert.io/lesson/delta-table-day-1-lecture-may2025-p736-l1086 Module: Week 3: Data Lakes with Delta Table · Enrolled students In this lecture, Yared dives into the key concepts of Delta Lake and its significance in data engineering. He discusses the differences between Delta Lake and traditional data formats like CSV and JSON, emphasizing the importance of performance and data management. ### Chat with Zach [July 20, 2023] URL: https://www.dataexpert.io/lesson/v2-1-chat-with-zach-july-20-2023-2652 Enrolled students Delve into the world of data engineering with our weekly Q&A / Fireside Chat series, featuring the founder of EcZachly Inc and lead instructor of the Advanced Data Engineering Bootcamp. Each session covers a different topic, providing invaluable insights and real-world experiences to empower data enthusiasts in their journey. (Recorded July 20, 2023) ### Managing Unstructured Data - Day 2 Lecture URL: https://www.dataexpert.io/lesson/managing-unstructured-data-day-2-lecture-may2025 Enrolled students In this lecture, Zach focuses on retrieval augmented generation (RAG) and large language models (LLMs). He emphasises the importance of identifying business problems that are well-suited for AI solutions. Zach outlined key topics for the day, including ensuring that LLMs return valid JSON and tuning RAG to achieve more relevant results. ### TA Office Hour 8 (2025-08-07) URL: https://www.dataexpert.io/lesson/ta-office-hour8-july2025-p737-l1384 Enrolled students This office hour session focuses on addressing student questions related to homework and capstone projects in a data engineering boot camp. Key topics include the creation of synthetic data with vulnerabilities, evaluation metrics for homework assignments, and the use of cursor for generating reports. Participants also discuss submission guidelines and grading criteria, providing insights into managing coursework effectively while preparing for the capstone project. ### Graph Data Modeling Lab URL: https://www.dataexpert.io/lesson/graph-data-modeling-lab-p735-l588 Module: Bonus · Enrolled students In this lab, Zach discusses the importance of data metrics and effective communication strategies. He highlights various data sources, such as banks, rental properties, and Excel sheets, and explain how they can be used to track metrics related to health, career, and recreation. He also emphasizes the significance of considering context and building relationships when communicating with stakeholders. The information shared can help them gain valuable insights and improve their communication skills. ### Prompt Engineering Day 1 Lecture URL: https://www.dataexpert.io/lesson/prompt-engineering-day1-lecture-oct2025 Enrolled students In this video, Zach discusses about prompt engineering, highlighting its evolution from being overhyped to recognizing its true value alongside retrieval augmented generation. He emphasizes that mastering these two concepts is crucial for success as an AI engineer, even if participants forget everything else. He covers the differences between structured and unstructured tasks, the importance of context, and how to effectively prompt large language models (LLMs). ### Parth Parekh - Director of Data Engineering at Meta URL: https://www.dataexpert.io/lesson/tech-talk-7-parth-parekh-april2025-p735-l1054 Module: Tech Talks · Enrolled students Parth Parekh, Director of Data Engineering at Meta, shares insights on the evolving role of data engineers in the context of generative AI. This session covers the intersection of data engineering and AI, focusing on the importance of high-quality data, the impact of generative models on data workflows, and the future of AI technologies. Ideal for data professionals looking to understand how generative AI can enhance their work and the skills needed to thrive in this rapidly changing landscape. ### Introduction to ReactJS URL: https://www.dataexpert.io/lesson/introduction-to-reactjs-faa87 Module: Intermediate Frontend Development · Enrolled students React and TypeScript are powerful tools for building dynamic web applications. This lesson covers the fundamentals of using TypeScript with React, including static typing, interfaces, and props. You'll learn how to create functional components, manage state, and handle events, which will enhance your development efficiency and reduce bugs. Ideal for intermediate frontend developers looking to strengthen their skills in modern web development practices. ### Setting Up CI/CD and Unit Testing in Databricks for Reliable Data Pipelines URL: https://www.dataexpert.io/lesson/setting-up-ci-cd-and-unit-testing-in-databricks-for-reliable-data-pipelines-may2025 Enrolled students In this video, Zach walks through the process of setting up Continuous Integration/Continuous Deployment (CI/CD) and unit testing in Databricks, with a particular emphasis on enhancing the reliability of data pipelines. He begins by creating a Git folder for an Airflow DBT project and explore the YAML configuration for GitHub Actions, which will trigger tests upon pull requests. The demonstration includes running unit tests and addressing bad data scenarios, highlighting the critical importance of ensuring accurate data transformations. ### Build-a-Resume! URL: https://www.dataexpert.io/lesson/build-a-resume-april2025 Enrolled students ### Q&A Week 2 URL: https://www.dataexpert.io/lesson/qa-week-2-jan2025 Enrolled students A live Q&A session covering various topics related to data engineering and technology. Key discussions include the challenges of adopting new technologies like Janus Graph and Cassandra, strategies for handling duplicate streaming data using tumbling windows, and insights on how to prepare for data engineering roles in big tech. The session also touches on the importance of networking, the use of views versus tables, and the evolving landscape of data quality tools. ### Iceberg Partitioning and Metadata Exploration Lab URL: https://www.dataexpert.io/lesson/apache-iceberg-day-1-lab-jan2025 Enrolled students In this lab, we explore the key concepts of partitioning and metadata in data management. Zach explains the importance of creating a schema, using dollar files for queries, and demonstrate how partitions work. We also discuss the value of low cardinality number columns and the idea of partition pruning. ### Day 1 Lecture: RAG Fundamentals and types of RAG URL: https://www.dataexpert.io/lesson/understanding-rag-lecture-day1-july2025 Enrolled students In this lecture, Zach shares some staggering statistics, like the fact that there are currently 160 zettabytes of unstructured data in the world, which is projected to double every two to three years. He discusses the importance of understanding unstructured data and how AI can help us make sense of it. He emphasizes the need to convert this data into more structured formats for better analysis and retrieval. ### Q&A with Zach Week 5 URL: https://www.dataexpert.io/lesson/q-a-with-zach-week-5-may2025 Enrolled students A live Q&A session with Zach covers various topics related to data engineering and AI, focusing on techniques for chunking transcriptions, using overlapping chunks for coherence, and the implications of context size in LLMs. Participants ask about streaming data with Kafka, integrating batch and streaming data, and the challenges of real-time data processing. The discussion also touches on the importance of capturing changes in data and strategies for effective project development in AI and data engineering. ### Spark Batch Processing - Hands-On Techniques for Broadcast Hash Join (Day 1 Lab) URL: https://www.dataexpert.io/lesson/spark-batch-day-1-lab-v3 Module: Week 4: Batch Pipelines with Apache Spark · Enrolled students Dive into practical Spark join optimization techniques in this hands-on video. We kick off by setting up the Spark directory and executing Docker compose for a seamless start. Learn how to efficiently read CSV files using spark.read options with headers set to true. Explore the concept and benefits of broadcast join, encountering and troubleshooting issues along the way. The highlight of this lab is the demonstration of the effectiveness of broadcast hash join, with a discussion on the delicate trade-off between compute and memory usage. Immerse yourself in a real-world Spark optimization experience, gaining valuable insights for your data journey. [Recorded on Dec 5th, 2023 ### Dimensional Data Modeling - Complex Data Types Arrays and Structs with Iceberg (Day 1 Lab) URL: https://www.dataexpert.io/lesson/dimensional-data-modeling-day-1-lab-v3 Module: Week 1: Dimensional Data Modeling in Iceberg · Enrolled students In this lab, we explore Slowly Changing Dimensions (SCDs) and design Cumulative Tables using the nba_players dataset. This is our first stress test of our web-based interface linked to a Presto/Trino database running in Starburst. Presto (formerly PrestoSQL) or Trino are both query engines that allow users to query and analyze data in a data lake using SQL, while Starburst offers a distribution of Presto/Trino. During this session, attendees executed 1742 queries in one hour and Starburst handled the workload without any issues! This lab not only demonstrates how to design cumulative tables but also showcases the power of running analytics in the data lake! [Recorded Nov 7, 2023] ### Test URL: https://www.dataexpert.io/lesson/test-4f528 Enrolled students ### Advance Snowflake Day 2 Lab URL: https://www.dataexpert.io/lesson/advancesnowflakeday2-jan2026 Enrolled students In this lab, Zach dives into chunking, RAG, and parsing, focusing on processing documents with Cortex. He reviews how to efficiently parse filings and transcripts, emphasizing the importance of filtering out short, unhelpful chunks. He also discusses the need to optimize their queries and explore indexing options for faster results. ### How to use Snowpark and Snowflake Optimizations Lecture Day 2 URL: https://www.dataexpert.io/lesson/how-to use-snowpark-and-snowflake-optimizations-lecture-day-2-april2025 Enrolled students In this lecture, Zach dives into the key aspects of Snowflake Data Modeling, particularly focusing on Snowpark and Iceberg. He discusses the importance of data governance controls, performance tuning, and the trade-offs between speed and cost when it comes to data clustering. He also highlights the need for proper catalog integration and the significance of understanding your data usage patterns. ### TA Office hour 5 URL: https://www.dataexpert.io/lesson/taofficehour5-jan2026-p735-l1943 Enrolled students Office hours provide a platform for students to discuss their progress and seek guidance on assignments and capstone projects. Key topics covered include submission issues, mentorship availability, and project expectations for the analytics engineering boot camp. Participants also share their challenges with specific assignments, such as understanding Airflow and Python scripting, while seeking clarification on project requirements and timelines. This session is beneficial for students needing support and direction in their coursework. ### TA Office hours 3 URL: https://www.dataexpert.io/lesson/taofficehours3-mar26 Enrolled students Office hours provide an opportunity for students to discuss assignments and clarify course content. This session covers topics such as using cloud tools for assignments, the differences between Snowflake and Databricks, and the importance of understanding coding concepts rather than just copying solutions. Participants can gain insights into effective project management and the benefits of utilizing AI tools in their work. ### Career Dev Session 4 (2025-08-07) URL: https://www.dataexpert.io/lesson/career-dev-session4-july25-p737-l1385 Module: Career Development · Enrolled students Soft skills are crucial for success in job interviews, and this session focuses on mastering them. Participants will learn effective strategies for screening calls, including how to articulate their experiences clearly and confidently, ask insightful questions, and align expectations with potential employers. The session also covers common interview questions, such as "Tell me about yourself" and "Why do you want to work here?", providing techniques to convey enthusiasm and fit for the role while showcasing relevant skills and experiences. ### How to use Snowpark and Snowflake Optimizations Lab Day 2 URL: https://www.dataexpert.io/lesson/how-to use-snowpark-and-snowflake-optimizations-lab-day-2-april2025-p735-l984 Module: Week 3: Data Modeling on Snowflake · Enrolled students In this lab, Zach walks through optimizing the compute of our Snowflake warehouse, focusing on larger ETL loads and the integration with Iceberg data. He also discusses the importance of clustering and how to manage tables effectively. ### Career Development - Interview Help URL: https://www.dataexpert.io/lesson/career-development-interview-help-jan2025 Enrolled students Career Q&A sessions provide insights into the data engineering interview process, covering key rounds such as recruiter screening, technical assessments, and behavioral interviews. Participants will learn about the structure of interviews, including technical skills evaluation in Python and SQL, data modeling, and data architecture design. This lesson is ideal for job seekers in data engineering looking to enhance their interview preparation and understand what recruiters and hiring managers expect. ### Apache Spark programming with Databricks Day 2 Lab URL: https://www.dataexpert.io/lesson/apache-spark-programming-with-databricks-day-2-lab-may2025 Enrolled students In this video, Yared walks through the key features of Delta Live Tables (DLT) and explains how it simplifies structured streaming using both SQL and Python. He demonstrates the creation of streaming tables and materialized views, emphasizing the importance of using Auto Loader for efficient file ingestion, which can process thousands of files simultaneously. A real-life scenario involving airport operations is shared to illustrate data engineering concepts. ### Data Engineering Interviews with Swetha at OpenAI URL: https://www.dataexpert.io/lesson/dataexpert-community-event-with-openai-swetha Module: Tech Talks · Enrolled students 🎤 Exclusive Fireside Chat: How to Break Into Data Engineering We’re thrilled to host Swetha Sekhar https://www.linkedin.com/in/swetha-sekhar/ — Member of Technical Staff at OpenAI, ex-Google and Twitter Data Engineer for a fireside chat on interview tips to land your next DE role. 💡 What you’ll learn: How to prep for data engineering interviews at top tech firms The common mistakes candidates make (and how to avoid them) Insider tips from interviews at OpenAI, Google, Twitter ### TA Office Hour 8 URL: https://www.dataexpert.io/lesson/taofficehour8-a9a36 Enrolled students ### TA Office hours 8 URL: https://www.dataexpert.io/lesson/taofficehours8-22bc5-p737-l2136 Enrolled students ### Loom URL: https://www.dataexpert.io/lesson/course-2-updating-test Enrolled students ### Loom URL: https://www.dataexpert.io/lesson/loom-420 Enrolled students ### Apache Iceberg Data Contracts Lab URL: https://www.dataexpert.io/lesson/apache-iceberg-data-contracts-day-3-lab-may2025 Enrolled students In this lab, Zach walks you through the process of building data contracts using the Polygon API, focusing on schema enforcement and data quality checks. We aim to create a daily dataset of valid stock tickers, ensuring that our schema correctly identifies nullable and non-nullable fields. He also demonstrates how to handle API responses and enforce data integrity throughout the pipeline. ### Tech Talk - Steve Huynh URL: https://www.dataexpert.io/lesson/v4-tech-talk-steve-huynh-4999 Enrolled students In this tech talk, Steve discusses different ways to grow your career in the tech industry based on his 18 years of experience at Amazon. He touches on the benefits of switching companies, the importance of asking the right questions, and the value of defining problem spaces. He also shares insights on how to navigate ambiguity and provide clarity in your work. Watch this video to gain valuable career advice and actionable strategies for professional growth. ### TA Office Hour 5 URL: https://www.dataexpert.io/lesson/ta-office-hour-5 Enrolled students Office hours focused on a specific assignment related to Databricks and advanced Spark techniques. The session included a walkthrough of the assignment involving a tweets dataset, data cleaning, creating fact and dimension tables, and performing a broadcast join. Participants learned how to analyze tweet volumes, identify top tweeters, and calculate metrics based on user engagement, making it beneficial for students working on data engineering projects using Spark. ### Behavioral Interview URL: https://www.dataexpert.io/lesson/behavioral-interview Module: Interview Skills · Enrolled students Behavioral interviews focus on assessing your interpersonal skills and cultural fit within a company. This lesson explores key strategies to excel in these interviews, including the importance of positive energy, the ability to learn from feedback, and the effective use of the STAR method to articulate your experiences. By understanding company culture and preparing thoughtful responses to common questions, candidates can present themselves as strong collaborators and valuable team members. ### TA Office Hour 8 URL: https://www.dataexpert.io/lesson/ta-office-hour8-july2025 Enrolled students Office hours provide an opportunity for students to ask questions about homework, capstone projects, and evaluation metrics. Participants discuss generating synthetic data, setting up evaluation criteria, and using tools like Cursor for coding tasks. Key topics include understanding vulnerabilities, the importance of ground truth data in JSON format, and clarifying submission deadlines for assignments. This session is beneficial for those seeking guidance on project requirements and effective use of coding tools. ### Analytical Patterns Recognizing Business Value Day 3 Lab URL: https://www.dataexpert.io/lesson/analytical-patterns-recognizing-business-value-day-3-lab-jan2025 Enrolled students Creating a feature store is essential for effective data engineering, and this lab focuses on building one using Trino. Participants will learn to create user feature tables, implement one-hot encoding, and aggregate user behavior data over different time frames. This hands-on experience is beneficial for data engineers looking to enhance their skills in feature engineering and prepare data for machine learning models. ### Advanced dbt: Jinja, Macros and variables Lecture URL: https://www.dataexpert.io/lesson/advanceddbtday1lecture-jan2026-p735-l1921 Module: Week 2: Advanced dbt · Enrolled students Advanced Jinja, macros, and variables in dbt are essential for writing efficient SQL code. This lesson covers how to leverage Jinja's syntax for control structures like loops and conditionals, as well as defining and using macros for reusable SQL functions. Participants will learn about incremental models, including strategies such as append, merge, and delete plus insert, which optimize data processing in large tables. This lesson is ideal for data engineers looking to enhance their dbt skills and improve data transformation efficiency. ### Orchestration and Airflow Fundamentals Day 1 Lab URL: https://www.dataexpert.io/lesson/orchestration-and-airflow-fundamentals-day-1-lab-jan2025 Enrolled students In this lab, Zach will provide an overview of airflow backfilling practices and discuss the pipeline code using pyspark. He will explain the importance of including the username in the DAG name and demonstrate how to run glue jobs and backfill data. He will also show a table that was backfilled and discuss the comparison of tables using union all. [Recorded on May 22nd, 2024] ### Tech Talk - Brian URL: https://www.dataexpert.io/lesson/techtalks2-feb26 Enrolled students Brian shares his extensive experience in tech and insights on navigating the job market, particularly in big tech. He discusses the evolving interview landscape, emphasizing the importance of storytelling and understanding hiring dynamics. Attendees will benefit from his practical advice on tailoring resumes, preparing for interviews, and leveraging AI in the hiring process, making this session invaluable for anyone looking to enhance their job search strategies in data and AI engineering. ### Advanced Snowflake: Data Modeling and UDFs Lecture URL: https://www.dataexpert.io/lesson/advanced-snowflake-data-modeling-and-udfs-lecture-p735-l563 Module: Week 4: Advanced Snowflake · Enrolled students Advanced Snowflake data modeling and user-defined functions (UDFs) are explored in this technical lecture, ideal for data engineers looking to deepen their understanding of Snowflake's capabilities. Key topics include the comparison of Snowflake with Iceberg, complex data types such as arrays, maps, and variants, and the nuances of UDFs, UDAFs, and UDTFs. Participants will learn about effective data modeling strategies, the compression hierarchy, and how to implement external API calls within Snowflake, enhancing their data engineering skill set. ### Book Club 2 URL: https://www.dataexpert.io/lesson/bookclub2-feb26 Enrolled students The session focuses on batch processing in data engineering, discussing its relevance and guarantees such as exactly-once processing, item potency, and replayability. Participants share experiences and challenges related to data quality, pipeline performance, and the transition from batch to streaming architectures. This interactive discussion benefits data engineers and analysts looking to optimize their batch workflows and understand the architectural considerations necessary for effective data processing. ### Chat with Zach [Dec 14, 2023] URL: https://www.dataexpert.io/lesson/v3-4-chat-with-zach-dec-14-2023-9964 Enrolled students Delve into the world of data engineering with our weekly Q&A / Fireside Chat series, featuring the founder of EcZachly Inc and lead instructor of the Advanced Data Engineering Bootcamp. Each session covers a different topic, providing invaluable insights and real-world experiences to empower data enthusiasts in their journey.[Recorded on Dec 14, 2023] ### New Test Lesson URL: https://www.dataexpert.io/lesson/9d3c7e54-e7d3-4ed3-82a4-5709747bc1b7newtestlesson Module: test module · Enrolled students Learn how to leverage AI for auto grading assignments using Data Expert. This lesson demonstrates the process of submitting homework, receiving instant feedback, and understanding the grading criteria, including aspects like partition tables and branch creation. Ideal for students and educators looking to enhance their assessment methods with efficient AI tools. ### Slowly Changing Dimensions Lab URL: https://www.dataexpert.io/lesson/slowly-changing-dimensions-lab-p735-l577 Module: Bonus · Enrolled students In this lab, we reinforce lecture concepts with a hands-on approach. We dive into implementing a Slowly Changing Dimension (SCD) table. Learn to track user activity changes using lag functions and identify active and inactive player seasons. We also tackle common errors in query optimization, providing practical solutions for a real-world understanding. Watch the video for step-by-step guidance on implementing SCDs in your data pipelines. ### Advance Snowflake Day 2 Lab URL: https://www.dataexpert.io/lesson/advancesnowflakeday2-jan2026-p735-l1912 Module: Week 4: Advanced Snowflake · Enrolled students In this lab, Zach dives into chunking, RAG, and parsing, focusing on processing documents with Cortex. He reviews how to efficiently parse filings and transcripts, emphasizing the importance of filtering out short, unhelpful chunks. He also discusses the need to optimize their queries and explore indexing options for faster results. ### Monetizing Part 1: Brands & Coaching by Stephanie Nuesi URL: https://www.dataexpert.io/lesson/linkedin-masterclass-4 Module: Making Content on LinkedIn · Enrolled students Monetizing Part 1: Brands & Coaching by Stephanie Nuesi. This will be filmed on May 31st, 2024. ### Career Dev Session 2 URL: https://www.dataexpert.io/lesson/careerdevsession2-jan2026 Enrolled students Learn how to effectively leverage LinkedIn for career advancement in this session focused on building a strong professional profile and enhancing networking skills. Discover strategies for optimizing your LinkedIn presence, including crafting an engaging headline, obtaining endorsements, and connecting with industry professionals. This lesson is ideal for job seekers and professionals looking to improve their visibility and attract recruiters in the competitive job market. ### Test 4 URL: https://www.dataexpert.io/lesson/5a2bcb8a-8840-41b5-9247-c9e33afdef73test4 Enrolled students A discussion with Jason Reed, co-founder of Tabular, explores the evolution of data engineering, focusing on the Iceberg table format and its implications for data lakes and warehouses. The conversation covers the challenges of data correctness, productivity, and performance, as well as the future of data engineering in the context of AI. Jason shares insights on the importance of community contributions and the dynamics of open-source projects, making this lesson valuable for aspiring data engineers and industry professionals alike. ### Apache Iceberg Day 3 Lab URL: https://www.dataexpert.io/lesson/apache-iceberg-day-3-lab-jan2025 Enrolled students In this lab, Zach explores creating tables, performing data quality checks, and leveraging PySpark for data processing. He demonstrates how to ensure data integrity while discussing potential areas for improvement. ### Monitoring, scaling, experimenting URL: https://www.dataexpert.io/lesson/monitoring-scaling-experimenting Module: Advanced Backend Development · Enrolled students Monitoring, scaling, and experimenting are crucial for maintaining robust backend systems. This lesson covers key techniques such as implementing health checks, utilizing metrics for performance monitoring, and establishing alert systems for error rates. You'll learn about vertical and horizontal scaling strategies, including AWS auto-scaling and Kubernetes, as well as the importance of feature toggling and A/B testing for continuous delivery. Ideal for backend developers looking to enhance application reliability and scalability. ### dbt basics: creating a full dbt project lecture URL: https://www.dataexpert.io/lesson/dbt-basics-creating-a-full-dbt-project-lecture-p735-l567 Module: Week 1: dbt Basics · Enrolled students In this lecture, Bruno delves into the world of DBT testing and snapshots, focusing on how DBT handles transformation tasks. He explains the significance of DBT as a tool for SQL-savvy individuals, particularly in the context of Snowflake warehouses. Throughout the lecture, he discusses the creation of materialized views, testing methodologies, and the importance of unit tests versus data tests. By the end, he touches upon the concept of snapshots and their role in maintaining historical data integrity within a database. ### Tech Talk - Joe URL: https://www.dataexpert.io/lesson/techtalk1-aug26 Enrolled students Joe Reese, author of "Fundamentals of Data Engineering," shares insights on the evolving landscape of data engineering in this engaging tech talk. He discusses the implications of new data modeling techniques, including LTAP and hybrid transactional analytical databases, and explores the role of AI in transforming data engineering practices. This session is ideal for data professionals looking to deepen their understanding of modern data architecture and the future of data engineering. ### dbt basics: How dbt works: sources, models, packages Lecture URL: https://www.dataexpert.io/lesson/dbt-basics-how-dbt-works-lecture Module: dbt basics · Enrolled students In this lecture, Bruno delves into the intricacies of dbt, focusing on its role in data engineering and best practices. He explains how dbt simplifies data transformation processes, emphasizing the importance of software engineering practices in data projects. Students are encouraged to grasp the significance of dbt in ETL pipelines and its integration with various platforms. He highlights key features like models, seeds, and sources, offering insights into how dbt streamlines data workflows. ### Career Dev Session 1 URL: https://www.dataexpert.io/lesson/careerdevsession1-jan2026 Enrolled students Enhance your job search skills with insights on crafting effective resumes and navigating the job application process. This session covers essential topics such as resume formatting, the importance of metrics and action words, and strategies for intentional job applications. Ideal for job seekers in engineering and tech, you'll learn how to tailor your resume for different roles and improve your chances of landing interviews by showcasing your skills and experiences effectively. ### dbt basics: How dbt works: sources, models, packages Lab URL: https://www.dataexpert.io/lesson/dbt-basics-how-dbt-works-lab Module: dbt basics · Enrolled students In this lab, Bruno delves into DBT best practices, emphasizing the importance of one-to-one staging and the connection of staging stripe payments to payment sources. He discusses the use of prefixes like FACT for clarity and the creation of intermediate models to simplify complex code. ### Delta Table Day 2 Lecture URL: https://www.dataexpert.io/lesson/delta-table-day-2-lecture-may2025 Enrolled students In this video, Yared dives into advanced features of Delta Lake, focusing on liquid clustering and optimization techniques. He explains how liquid clustering improves query performance by dynamically organizing data and how the optimize command compacts smaller files for better read efficiency. He also touches on versioning and the importance of maintaining historical data. ### Special Guest, Vin Vashishta [Nov 17, 2023] URL: https://www.dataexpert.io/lesson/guest-speaker-2023-11-17-v3 Enrolled students In this video, we have a special guest, Vin Vashishta, who will be sharing insights on fundamental topics in data engineering. Vin discusses the importance of understanding data generating processes and systems, the future of communication in the industry, and the impact of smaller companies in the market. He also provides valuable advice on career growth, effective communication, and being forward-thinking. Dont miss out on this informative session! [Recorded Nov 17, 2023] ### Tech talk with Carly Taylor URL: https://www.dataexpert.io/lesson/tech-talk-with-carly-taylor Module: Tech Talk · Enrolled students Carly Taylor, a director at Activision, shares her journey from chemist to analytics engineer, emphasizing the importance of boot camps and community learning for career transitions. She discusses the evolving landscape of data roles, the significance of machine learning in driving business value, and the balance between technical skills and effective communication. This session is ideal for those considering a career change or seeking insights into the analytics and gaming industries. ### Tech Talk - Stephanie Nuesi URL: https://www.dataexpert.io/lesson/v4-tech-talk-stephanie-nuesi-5355 Enrolled students In this tech talk, Stephanie reviews Zach's resume and provide feedback on what's working and what's not. She emphasizes the importance of content over formatting and highlight the need to tailor the resume to the desired expertise. She also discusses the significance of including specific details, such as software used and results achieved.[Recorded on May22nd, 2024] ### Monetizing Part 3: Products & Funnels by Zach Wilson URL: https://www.dataexpert.io/lesson/linkedin-masterclass-6 Module: Making Content on LinkedIn · Free Monetizing Part 3: Products & Funnels by Zach Wilson. This will be filmed on June 14th, 2024. ### Snowflake Lecture URL: https://www.dataexpert.io/lesson/snowflake-lecture-jan2025-p736-l786 Enrolled students In this lecture, Bruno dives into the concept of Change Data Capture (CDC) and how it can be effectively implemented in Snowflake. He explains the importance of logging updates, deletes, and inserts, and how CDC can help manage high-volume data without slowing down the pipeline. He also touches on the use of streams and merging data to avoid duplication. ### TA Office hours 8 URL: https://www.dataexpert.io/lesson/taofficehours8-22bc5 Enrolled students ### dbt basics Day2 Lecture URL: https://www.dataexpert.io/lesson/dbtbasicsday2lecture-2be25 Enrolled students ### Supplemental lab session on GuardrailsAI URL: https://www.dataexpert.io/lesson/supplemental-lab-session-on-guardrailsai-july25-p737-l1389 Module: Week 4: MLOps, Guardrails & Business Value · Enrolled students In this session, Zach walks through setting up AI Guardrails, similar to our Vercel deploy example from week two. He installs various validators, including detect PII, competitor check, and toxic language, to ensure our AI behaves appropriately. He emphasizes the importance of explicit error handling and demonstrated how to validate user inputs and outputs effectively. He also discusses the need to implement guardrails in multiple areas, including document input and retrieval. ### Write a comprehensive pipeline spec document URL: https://www.dataexpert.io/lesson/write-a-comprehensive-pipeline-spec-document-lecture Module: Pipeline Spec Building + Airflow Fundamentals · Enrolled students Zach discusses the importance of data modeling and metrics in building a successful pipeline. He emphasizes that conceptual data modeling is accessible to everyone and highlight the need to consider the right people for feedback and collaboration. He also touches on the challenges of understanding stakeholders' true requirements and share his experience with choosing the right database solution. Throughout the video, he emphasizes the importance of metrics-driven approaches and the potential impact of well-designed dashboards. Overall, this lecture provides valuable insights and guidance for data engineers looking to optimize their pipeline. ### Friday Quiz URL: https://www.dataexpert.io/lesson/fridayquiz-8d36d Enrolled students ### Tech Talk - Max URL: https://www.dataexpert.io/lesson/techtalk5-aug26 Enrolled students Max, the creator of Apache Airflow and Superset, shares insights from his extensive career in data engineering during this engaging tech talk. He discusses the evolution of open-source tools, emphasizing the importance of community contributions and the role of AI in enhancing data workflows. Attendees will benefit from his perspectives on managing AI agents, fostering collaboration, and leveraging tools like AGOR for improved team productivity in data engineering and analytics. ### Spark Fundamentals (Day 1 Lecture) URL: https://www.dataexpert.io/lesson/spark-batch-day-1-lecture-v4 Module: Week 4: Batch Pipelines with Apache Spark · Enrolled students In this lecture, Zach discusses the importance of learning Spark and its role as a resilient framework for distributed compute. He explains how Spark can process large amounts of data and its advantages over other technologies like Java, MapReduce, and Hive. He also touches on the considerations for choosing Spark in complex pipelines and provide tips for optimizing Spark jobs. [Recorded on May28th, 2024]. Note - This is not the edited version. we will be replacing it with the edited and refined version soon. ### End-to-End AI Applications Day 1 Lab URL: https://www.dataexpert.io/lesson/using-mcp-to-build-more-powerful-agents-oct2025 Enrolled students In the lab, Zach walks through an N8N workflow that integrates AI agents to automate researching and generating LinkedIn shout‑out posts for creators in the data‑engineering space. He explains how the workflow calls the Google Search API and an OpenAI model to gather up‑to‑date information about creators, ensuring that each month the system highlights new voices without repeating previous mentions. ### Test Lesson URL: https://www.dataexpert.io/lesson/a4859c01-6e2f-4d69-b0ed-b86960deaf7ftestlesson-p737-l1378 Module: Career Development · Enrolled students ### Apache Iceberg: Big Data History Lecture URL: https://www.dataexpert.io/lesson/apache-iceberg-lecture-day-1-apr2025-1-p735-l1192 Module: Week 5: Data Modeling on Iceberg · Enrolled students In this lecture, Zach dives into the intricacies of Iceberg, a technology that emerged from Netflix, and how it revolutionizes data processing. He shares his experiences working with large datasets, particularly at Airbnb, and discuss the challenges we faced with traditional tools like Hive and Spark. He also highlights the importance of time travel in data management and how Iceberg addresses some of the limitations of previous technologies. ### TA Office Hour 4 (2025-01-21) URL: https://www.dataexpert.io/lesson/ta-office-hour-4-jan2025-p736-l767 Enrolled students Office hours provide a platform for students to ask questions and clarify concepts related to their assignments. This session focused on building a data pipeline using Apache Airflow, incorporating the right audit published pattern. Key topics included creating a daily partitioning table, loading data from the Polygon API into staging tables, and performing data quality checks before transitioning to production. Students also discussed various tools like PyIceberg and Trino for different parts of the assignment. ### MLOPs and Guardrails Day 2 Lecture URL: https://www.dataexpert.io/lesson/ml-ops-day2-lecture-oct25-p737-l1524 Module: Week 4: MLOps, Guardrails & Business Value · Enrolled students In this lecture, Zach dives into the critical topics of evaluations and reliability in AI, emphasizing that AI agents are not magical black boxes but statistical models that need careful management. He highlights how shifts in user behavior, code updates, and data changes can break AI systems and shares strategies for monitoring these issues effectively. Zach walks through various evaluation methods such as ROUGE and cosine distance and shows how they can be used to optimize prompts with DSPY. ### Fact Data Modeling: Compact Tables for Efficient Data Representation Day 2 Lab URL: https://www.dataexpert.io/lesson/fact-data-modeling-day-2-lab-yt-p736-l665 Enrolled students This lab guides you through the process of analyzing user activity using compact tables. The lab demonstrates how to create a table that efficiently captures user IDs and their corresponding activity status. ### Youtube URL: https://www.dataexpert.io/lesson/youtube-7367 Enrolled students testing updating ### Q&A with Zach 1 URL: https://www.dataexpert.io/lesson/qawithzach1-mar26 Enrolled students The Q&A session covers various topics related to setting up AI agents and addressing technical issues, particularly with proxies and API configurations. Participants learn about exporting URLs, configuring environment files, and the importance of understanding tools like LangChain for building agents. The session also encourages interaction, allowing attendees to ask questions about their projects and the boot camp, fostering a collaborative learning environment for both beginners and experienced individuals in AI and data engineering. ### Advance Snowflake Day 2 Lecture URL: https://www.dataexpert.io/lesson/advancesnowflakeday2lecture-04786 Enrolled students ### Three Pillars of Seniority Levels URL: https://www.dataexpert.io/lesson/threepillarsofsenioritylevels-126b7 Enrolled students Specific standards that distinguish Senior from Staff+ performance Hands-on: Personal Gap Analysis—map yourself against Staff+ criteria ### dbt Basics Day 2 Lab URL: https://www.dataexpert.io/lesson/dbt-basics-day-2-lab-jan2025 Enrolled students In this lab, Bruno walks you through the setup of our Airflow DBT project, focusing on the installation of dbt core, dbt snowflake, and sqlfluff. He also demonstrates how to define models and organize your project structure effectively. Please make sure to follow along and set up your environment as discussed. ### dbt basics Day2 Lecture URL: https://www.dataexpert.io/lesson/dbtbasicsday2lecture-jan2026 Enrolled students The DBT Basics Day 2 Lecture focuses on essential concepts for analytics engineers using DBT, including testing, snapshots, and packages. Participants will learn about data quality assurance through data tests and unit tests, and how to implement slowly changing dimensions using snapshots. The session also covers best practices for structuring DBT projects and introduces useful packages to enhance functionality. This lesson is ideal for those looking to deepen their understanding of DBT and improve their data transformation processes. ### Career Dev Session 4 URL: https://www.dataexpert.io/lesson/career-dev-session4-july25 Enrolled students Soft skills are crucial for success in job interviews, and this session focuses on mastering them, particularly during screening calls. Participants learn how to effectively communicate their skills, express genuine interest in roles, and prepare for common interview questions like "Tell me about yourself" and "Why do you want to work here?" The session also emphasizes the importance of being concise, energetic, and prepared to engage with recruiters, ensuring candidates present themselves as reliable and coachable team players. ### Q&A with Zach Week2 URL: https://www.dataexpert.io/lesson/qawithzachweek2-jan2026-p735-l1930 Module: Q&A with Zach · Enrolled students A live Q&A session covers various topics related to data engineering and analytics, focusing on upcoming lessons and homework. Key discussions include analytical patterns, change data capture, and the interaction between Snowflake and DBT. Participants ask questions about homework deadlines, data quality checks, and the roles of analytics engineers versus data engineers. The session also touches on the importance of understanding data modeling and the implications of using different data types in Snowflake. ### Build a Resume URL: https://www.dataexpert.io/lesson/careerdevsession1-jan2026-p735-l1947 Module: Career Development · Enrolled students Building an effective resume is crucial for job seekers looking to enhance their chances of landing interviews. This lesson covers essential topics such as resume formatting, the importance of metrics and action words, and strategies for tailoring applications to specific roles. Participants will learn how to present their skills and experiences effectively, understand the job search process, and identify relevant job platforms to maximize their opportunities in the competitive job market. ### Advanced dbt Day 3 Lecture URL: https://www.dataexpert.io/lesson/advanced-dbt-day-3-lecture-jan2025-p736-l796 Enrolled students In this lecture, Bruno dives into advanced techniques for running DBT projects, particularly focusing on integrating with Airflow. He discusses the importance of selecting specific models and the implications of running stale data. He also covers the benefits of using CI/CD for testing changes in isolated schemas to avoid merging bad code. ### Chat with Zach URL: https://www.dataexpert.io/lesson/week-4-zach-qa-v4 Enrolled students In this exclusive Q&A with Zach, he discusses various topics related to data engineering, including partitioning, data dependencies, and the use of Airflow and serverless functions. He shares his experiences working on a project before starting a bootcamp and provide insights into forming an LLC or an S Corp. The Q&A aims to provide valuable information and prompt viewers to consider different approaches to data engineering. [Recorded on May30th, 2024] ### TA Office hours 4 URL: https://www.dataexpert.io/lesson/taofficehours4-mar26-p737-l2132 Enrolled students Office hours provide an interactive platform for students to seek help with specific technical issues. This session covered troubleshooting Docker installation problems and resolving inactive API key errors for cloud services. Participants received guidance on uninstalling and reinstalling software, regenerating API keys, and integrating coding exercises with platforms like OpenAI, making it beneficial for learners facing similar challenges in their projects. ### Chat with Zach [Aug 24, 2023] URL: https://www.dataexpert.io/lesson/week-6-zach-qa-v2 Enrolled students Delve into the world of data engineering with our weekly Q&A / Fireside Chat series, featuring the founder of EcZachly Inc and lead instructor of the Advanced Data Engineering Bootcamp. Each session covers a different topic, providing invaluable insights and real-world experiences to empower data enthusiasts in their journey. (Recorded Aug 24, 2023) ### Book Club 3 URL: https://www.dataexpert.io/lesson/book-club3-july2025 Enrolled students The Book Club session focuses on evaluating large language models (LLMs) and the challenges associated with their assessment compared to traditional machine learning models. Participants discuss the complexities of LLM outputs, including their open-ended nature and the need for human evaluation in critical applications. The conversation also touches on the evolving roles of data engineers and domain experts in leveraging AI technologies, as well as the importance of maintaining and updating models to ensure relevance in rapidly changing fields. ### RAG optimisation + Reranking Day 1 Lab URL: https://www.dataexpert.io/lesson/ragoptimisation-reranking-day1-lab-oct2025-p737-l1573 Module: Week 2: Advanced RAG, Reranking & Dev Workflow · Enrolled students In this lab, Zach focuses on re-ranking and working with an external worker process, both crucial for optimizing their retrieval system. He implements metadata filtering and re-ranking to improve query results, ensuring accurate classification of document types and determining the optimal number of documents to retrieve. Zach demonstrates how to set up a PubNub channel for asynchronous processing, which allows the team to handle longer tasks without timing out. ### Analytical Patterns Recognizing Business Value Day 3 Lecture URL: https://www.dataexpert.io/lesson/analytical-patterns-recognizing-business-value-day-3-lecture-jan2025 Enrolled students Recognizing business value in data engineering is crucial for driving effective decision-making and maximizing project success. This lecture covers analytical patterns, the hierarchy of pipeline value, and the integration of analytics engineers in experimentation and machine learning. Participants will learn how to prioritize high-value projects, avoid common pitfalls in dashboard creation, and understand the significance of master data, experiments, and machine learning in delivering impactful business outcomes. ### Azure - Week 3 (Copy) URL: https://www.dataexpert.io/lesson/azure-week-3-2315 Enrolled students Understanding cloud databases in Azure is essential for data engineers and developers. This session covers key concepts such as structured, semi-structured, and unstructured data, and the appropriate database solutions for each type, including Azure SQL Database and NoSQL options like Cosmos DB. Learn about the creation and management of Azure SQL databases, including firewall settings, authentication through Azure Active Directory, and the differences between OLTP and OLAP systems, which are crucial for building scalable data solutions in the cloud. ### Fact Data Modeling Lab URL: https://www.dataexpert.io/lesson/fact-data-modeling-lab-p735-l579 Module: Bonus · Enrolled students In this hands-on lab session, we analyze NBA game data, unraveling the intricacies of data modeling and analysis. Breaking down the data into distinct columns, we elucidate the importance of each element. Through practical demonstrations, we showcase how to execute queries and perform joins, extracting meaningful insights from the dataset. Throughout the video, valuable tips are shared on data modeling, query optimization, and the significance of empathy when collaborating with stakeholders. Tune in to this video to acquire practical skills in NBA game data analysis and learn essential techniques for effective data modeling and analysis. ### Capstone Requirements URL: https://www.dataexpert.io/lesson/capstone-requirements-p735-l600 Module: Kickoff · Enrolled students Capstone projects require careful planning and execution, and this session outlines the essential components for success. Participants will learn about the importance of documentation, including specifications, data quality checks, and technology choices, while also exploring tools like Snowflake, Trino, and Airflow. This lesson is ideal for students looking to refine their capstone projects and understand the expectations for both individual and group submissions, ensuring they meet industry standards. ### Data Visualization and Impact: Hands-On with the CSV files Day 1 Lab URL: https://www.dataexpert.io/lesson/data-visualization-and-impact-day-1-lab-yt-p736-l691 Enrolled students Join this hands-on session, where you will dive into the practical realm of data visualization using visualization tools. ### Jason Reid - Co-Founder at Tabular (2.0) URL: https://www.dataexpert.io/lesson/techtalks5-feb26-p736-l2046 Module: Guest Speakers · Enrolled students Jason Reid, co-founder of Tabular and former head of product at Netflix, shares insights from his journey in data engineering and product development. He discusses the evolution of data modeling, the impact of AI on data engineering roles, and the importance of understanding business outcomes. This session is valuable for aspiring data engineers and entrepreneurs looking to navigate the changing landscape of data technologies and the integration of AI in business processes. ### Dimensional Data Modeling - Complex Data Types Arrays and Structs with Iceberg (Day 1 Lecture) - Pre-recorded from V3 URL: https://www.dataexpert.io/lesson/dimensional-data-modeling-day-1-lecture-v4 Module: Week 1: Dimensional Data Modeling in Iceberg · Enrolled students This is a pre-recorded lesson from V3. In this presentation, we explore the fundamental concepts essential for effective data modeling and database management. We dive into understanding your data consumer, the distinctions between OLTP and OLAP data modeling, the principles of Cumulative Table design, and the tradeoff between compactness and usability. We also address the challenges of temporal cardinality explosion and the potential pitfalls of run-length encoding compression. By the end of this lecture, you will have a strong foundation in the key concepts of data modeling and be prepared to apply these principles in practice. [Recorded Nov 7, 2023] ### Structured Streaming Kafka to Delta Live Table Day1 Lecture URL: https://www.dataexpert.io/lesson/structuredstreamingkafkatodeltalivetableday1lecture-feb26 Enrolled students In this lecture, Zach dives into the world of real-time data processing, focusing on the velocity problem and how technologies like Kafka, Iceberg, and Delta Lake fit into the picture. He explores the Kafka producer and consumer APIs, and shares insights on when to choose streaming over batch processing. He also highlights the advantages of Delta Live Table for making streaming pipelines more accessible. ### Q&A Week 4 URL: https://www.dataexpert.io/lesson/qa-week-4-jan2025-p736-l800 Module: Q&A with Zach · Enrolled students This Q&A session covers a wide range of topics related to data engineering, Spark, and the boot camp experience. Participants ask questions about Spark job structuring, the use of Snowflake and DBT, and strategies for handling real-time data from APIs like Stripe. The discussion also touches on personal experiences in the tech industry, including insights into interview preparation and the importance of using live data for capstone projects. This session is beneficial for anyone looking to deepen their understanding of data engineering concepts and tools. ### Everything you need to know about Vector databases URL: https://www.dataexpert.io/lesson/everythingyouneedtoknowaboutvectordatabases-aug26 Enrolled students In this lab, Eumar explains how to build the knowledge layer for the multi-persona student support agent Zachy by focusing on chunking strategies and vector search in Databricks. He emphasizes that the number one mistake in RAG is adopting the wrong chunking strategy, since naive fixed-size character splits can mix topics and lead to vague or incorrect answers. He covers the flow from transforming text into embeddings using Databricks models like GTE-Large-N, storing chunked content in a Delta table, and synchronizing it to an AI Search vector index with dense, sparse, or hybrid (BM25) retrieval. It also introduces Agent Bricks and the Knowledge Assistant as faster options, noting that Agent Bricks automates chunking, embeddings, and governance, while the Knowledge Assistant can turn documents into an agent by leveraging AI parse document for layout-aware extraction. ### LLM-Driven Data Engineering Day 2 Lecture URL: https://www.dataexpert.io/lesson/large-language-models-day-2-lecture Free In this LLM presentation, we explore building applications with LLMs, emphasizing LangChain as a versatile integration point for different LLMs and addressing privacy and security concerns. ### Managing Unstructured Data Day1 Lab URL: https://www.dataexpert.io/lesson/managingunstructureddataday1lab-feb26 Enrolled students In this video, Eumar walks you through the four notebooks focused on Unity Catalog, synthetic data generation, and using Databricks AI functions to parse receipts. He discusses the importance of handling PII data and how to effectively process data in parallel using Spark. He demonstrates how to create schemas, tables, and volumes, and encourages everyone to follow along and try the exercises at home. Additionally, he highlights the use of the Databricks AI DevKit for generating code and managing data. ### Local Video URL: https://www.dataexpert.io/lesson/local-video-175 Enrolled students ### LLMs Day 3 Part1 URL: https://www.dataexpert.io/lesson/llms-day-3-part-1-jan2025-p736-l830 Module: Bonus - LLMs · Enrolled students In this class, Li Yin shares her experiences and insights from working in AI for the past decade. She walks through the training process, discuss performance improvements, and highlight the importance of reasoning models. She also touches on various tools and libraries that can enhance our AI applications. ### TA Office Hour 7 URL: https://www.dataexpert.io/lesson/ta-office-hour7-july2025 Enrolled students The session focuses on creating an evaluation framework for a dependency scanner, guiding participants through the process of generating synthetic data and defining evaluation metrics such as precision and recall. Attendees will learn how to create dummy repositories, simulate vulnerabilities, and compile reports to assess the effectiveness of their scanners. This is particularly beneficial for those looking to enhance their understanding of data evaluation techniques in software development. ### Managing Unstructured Data Day1 Lecture URL: https://www.dataexpert.io/lesson/managingunstructureddataday1lecture-feb26 Enrolled students In this video, Eumar talks about building an enterprise AI agent named Zachy, designed to assist students and employees with various queries related to the bootcamp and expense reports. He focus on setting up their environment and understanding the foundational aspects of unstructured data, including the importance of Unity Catalog and AI functions for parsing documents. He emphasises the need for context engineering and governance on unstructured data, introducing Unity Catalog Volumes as a solution. ### Wednesday Quiz URL: https://www.dataexpert.io/lesson/wednesdayquiz-795e0 Enrolled students ### Managing Unstructured Data Day2 Lecture URL: https://www.dataexpert.io/lesson/managingunstructureddataday2lecture-feb26 Enrolled students In this video, Eumar dives deep into chunking strategies, Databricks vector search, and how to integrate the link chain in Databricks, emphasizing the importance of chunking for working with LLMs. He encourages everyone to become experts in chunking strategies, as they will be building a reg-based solution that can answer questions about the bootcamp, such as costs and content. ### Brian Pulliam URL: https://www.dataexpert.io/lesson/brian-pulliam-jan2025 Enrolled students Brian Pulliam shares invaluable insights on navigating the tech job market, focusing on interview preparation, career transitions, and personal branding. He emphasizes the importance of storytelling in interviews, highlighting how memorable narratives can set candidates apart. This session is particularly beneficial for mid-level data engineers, analysts transitioning to engineering roles, and software engineers seeking to pivot, providing strategies to enhance their job search and interview success. ### Data Pipelines Maintenance (Day 1 Lab) URL: https://www.dataexpert.io/lesson/pipeline-maintenance-day-1-lab-v4 Module: Week 6: Data Pipeline Maintenance (V4) · Enrolled students This is a pre-recorded class from V2. We did not have a lab class for this day in V3. ### OpenClaw Safeguards & Measuring business value for AI Agents Day2 Lecture URL: https://www.dataexpert.io/lesson/openclawsafeguardsmeasuringbusinessvalueforaiagentsday2lecture-mar26-p737-l2151 Module: Week 4: MLOps, Guardrails & Business Value · Enrolled students Understanding what tasks are worth automating is crucial for maximizing efficiency in your workflow. This lecture covers frameworks for evaluating automation opportunities, highlighting common pitfalls like over-automation and the hidden costs associated with it. Participants will engage in a lab session to identify specific tasks they can automate, using tools like OpenClaw and Notion, while considering factors such as frequency, cognitive load, and the potential risks involved. ### TA Office Hour 8 (2025-10-09) URL: https://www.dataexpert.io/lesson/ta-office-hour8-oct2025-p737-l1543 Enrolled students ### Authentication and Advanced techniques URL: https://www.dataexpert.io/lesson/authentication-and-advanced-techniques Module: Advanced Backend Development · Enrolled students Authentication and authorization are critical components of modern web applications, and this lesson delves into advanced techniques for implementing them effectively. Participants will learn about various authentication methods, including password-based and token-based systems, as well as the differences between authentication and authorization. The session also covers the use of JSON Web Tokens (JWT), OAuth 2.0, and best practices for securing APIs, making it ideal for backend developers looking to enhance their security knowledge and skills. ### TA Office Hour 3 URL: https://www.dataexpert.io/lesson/ta-office-hour3-oct2025 Enrolled students ### Q&A with Zach Week2 URL: https://www.dataexpert.io/lesson/qawithzachweek2-jan2026 Enrolled students A week two Q&A session covers various topics related to data engineering and analytics. Participants discuss upcoming lessons on analytical patterns and change data capture, along with tools like Snowflake and DBT. Key points include the importance of data quality checks, the role of analytics engineers versus data engineers, and the integration of AI in data pipelines. The session also addresses homework deadlines and project requirements, providing insights into best practices for managing data in cloud environments. ### dbt Basics Day 3 Lecture URL: https://www.dataexpert.io/lesson/dbt-basics-day-3-lecture-jan2025-p736-l777 Enrolled students In this lecture, Bruno dives into the functionalities of DBT, particularly focusing on its role in data transformation and testing. He discusses the importance of unit tests and data tests, explaining how they can help ensure data integrity and business logic. He also touches on the challenges DBT aims to solve, such as documentation and testing chaos. ### LinkedIn like a Pro URL: https://www.dataexpert.io/lesson/careerdevsession2-jan2026-p735-l1948 Module: Career Development · Enrolled students Mastering LinkedIn is essential for effective networking and job searching in today's competitive landscape. This session covers how to optimize your LinkedIn profile, including crafting an impactful headline, securing endorsements, and connecting with industry professionals. Participants will learn strategies for increasing visibility, leveraging LinkedIn Premium, and effectively engaging with recruiters, making this lesson valuable for anyone looking to enhance their career opportunities through LinkedIn. ### Capstone Project Day 1 Lecture URL: https://www.dataexpert.io/lesson/capstone-project-day-1-lecture-april2025 Enrolled students In this lecture, Zach shares his experiences with replicating Heroku and working with Postgres, emphasizing the importance of maintaining data integrity while querying. He discusses the significance of starting with dimensions in your capstone projects and the challenges of handling sensitive information in data models. He also touches on the need for real-time updates for student progress and the use of tools like DBT and Astronomer for data orchestration. ### Special Guests, Jitender Aswani and Bill Inmon [August 23, 2023] URL: https://www.dataexpert.io/lesson/v2-special-guests-jitender-aswani-and-bill-inmon-august-23-2023-5854 Enrolled students Join us each week as we sit down with visionary speakers from diverse fields, engaging in enlightening conversations that explore cutting-edge insights, trends, and their wealth of expertise. (Recorded August 23, 2023) ### Carly Taylor - Field CTO at Databricks URL: https://www.dataexpert.io/lesson/tech-talk-5-carly-taylor-april2025-p735-l1007 Module: Tech Talks · Enrolled students Carly Taylor, Field CTO at Databricks, shares insights from her diverse career journey, transitioning from chemistry to data engineering. This session explores the evolving landscape of data roles, emphasizing the importance of integrating data engineering with business insights. Carly discusses how AI tools can enhance productivity in data engineering, the significance of mentorship, and strategies for personal growth in tech careers. This talk is beneficial for aspiring data professionals looking to navigate their paths in a rapidly changing industry. ### Hard Orchestration Lessons Day 2 Lab URL: https://www.dataexpert.io/lesson/hard-orchestration-lessons-day-2-lab-jan2025-p736-l726 Module: Week 6: Airflow & Pipeline Specs with Trino · Enrolled students In this lab, Zach takes you through the optimization of a data pipeline, with a focus on making it idempotent and predictable. He demonstrates steps such as setting up dependencies, handling data waits, and ensuring task success. Zach also emphasizes the importance of scheduling and avoiding duplicates. ### TA Office Hour 6 URL: https://www.dataexpert.io/lesson/ta-office-hour-6-april2025-p735-l1012 Enrolled students This office hours session covers various topics related to homework and project queries for students in the data engineering bootcamp. Participants discuss troubleshooting issues with DBT, including schema errors and setting up environment variables. The session also addresses capstone project proposals, API data sourcing, and the use of DBT for building models and testing. Students benefit from real-time support and clarification on assignments, enhancing their understanding of data transformation processes. ### Test 3 URL: https://www.dataexpert.io/lesson/ad5f98c4-fc06-4bdb-922e-504cd90c25betest3 Enrolled students Learn how to create and manage a metrics table using SQL, focusing on daily aggregates and array manipulations. This lesson covers the creation of an array metrics table, data types, and the use of functions like COALESCE and ARRAY_FILL to handle null values. Ideal for data engineers and analysts, the session also emphasizes best practices for data modeling and aggregation techniques, ensuring a solid understanding of working with time-series data in PostgreSQL. ### Apache Iceberg Lecture Day3 URL: https://www.dataexpert.io/lesson/apacheiceberglectureday3-6fb59-p735-l1995 Module: Week 5: Data Modeling on Iceberg · Enrolled students Understanding data contracts is crucial for data engineers and analytics professionals, as they define the expectations between data producers and consumers. This lecture explores the concept of data contracts, their implementation in big tech, and the importance of the WAP pattern for ensuring data quality. Additionally, it covers Iceberg's branching model and time travel features, highlighting their implications for data governance and privacy. Participants will gain insights into best practices for managing data quality and the significance of effective communication in data engineering. ### TA Office hour 7 URL: https://www.dataexpert.io/lesson/taofficehour7-9b936-p736-l2053 Enrolled students ### How to write non-boring content by Carly Taylor URL: https://www.dataexpert.io/lesson/linkedin-masterclass-3 Module: Making Content on LinkedIn · Enrolled students How to write non-boring content by Carly Taylor. In this class, Carly shares her insights on how to write compelling content on LinkedIn that captures the attention of your audience. She discusses the importance of creating engaging hooks, sharing personal stories, and embracing authenticity. She also provides tips on using humor and evoking emotions to make your content more impactful. [Recorded on May24th, 2024] ### Fact Data Modeling - Minimizing Shuffle and Reducing Facts with Iceberg (Day 3 Lecture) - Pre-recorded from V3 URL: https://www.dataexpert.io/lesson/fact-data-modeling-day-3-lecture-v4 Module: Week 2: Fact Data Modeling in Iceberg · Enrolled students This is a pre-recorded lesson from V3. The lecture emphasizes the significance of minimizing shuffle when dealing with fact data. It explains how reducing shuffle can enhance the efficiency of data analysis and pipelines. Furthermore, the concept of reducing facts is explored, illustrating how it empowers analytics and data scientists to experiment and make informed, data-driven decisions. Students will be guided through various frameworks and strategies for handling big data, with a focus on making data more accessible and efficient for extended timeframes.[Recorded Nov 16, 2023] ### TA Office Hour 5 (2025-06-17) URL: https://www.dataexpert.io/lesson/ta-office-hour-5-p736-l1152 Enrolled students This office hour session focuses on guiding students through an assignment related to Databricks and advanced Spark techniques. Participants learn about processing a tweets dataset, including data cleaning, creating fact and dimension tables, and performing broadcast joins. The discussion also covers defining tweet counts and managing historical data, making it beneficial for students seeking clarification on practical applications of data engineering concepts. ### Tech Talk - Joe Reis URL: https://www.dataexpert.io/lesson/guest-speaker-8-v4 Enrolled students In this Tech Talk with Joe Reis, he discusses the importance of data modeling and its impact on data management. He also explains the different levels of data modeling (conceptual, logical, and physical) and emphasize the need for a well-designed data model to ensure data accuracy, clarity, and ease of use. He touches on the drawbacks of not implementing proper data modeling and provide real-life examples to illustrate the concepts. [Recorded on May29th, 2024] ### New Lesson URL: https://www.dataexpert.io/lesson/5510476d-f616-4132-b94f-4be285d4e1aenewlesson Enrolled students The session focuses on career coaching for data engineers, emphasizing interview skills, job transitions, and personal branding. Participants will learn how to navigate the job market, improve their storytelling abilities for interviews, and understand the importance of networking and self-presentation. The discussion includes insights on leveraging personal experiences and the significance of aligning career goals with personal values, making it beneficial for those seeking clarity and confidence in their career paths. ### Snowflake Basics Day 1 Lecture URL: https://www.dataexpert.io/lesson/snowflake-basics-day-1-lecture-jan2025-p736-l769 Enrolled students In this lecture, Bruno introduces Snowflake and DBT, focusing on basic concepts like time travel, fail safe, and data types. He also discusses the differences between Snowflake and Databricks, emphasizing the ease of scaling in Snowflake. ### Advanced Snowflake Data Modeling and UDFs Lecture Day 3 URL: https://www.dataexpert.io/lesson/advanced-snowflake-data-modeling-and-udfs-lecture-day-3-april2025-p735-l986 Module: Week 3: Data Modeling on Snowflake · Enrolled students In this lecture, Zach dives into some advanced data modeling techniques, particularly focusing on how Snowflake and other data structures interact. He shares insights from my experience with large datasets, including how to optimize data storage using arrays and objects. He also discusses the importance of understanding the strengths and weaknesses of different data types. ### Building functional AI agents using LangChain Day1 Lecture URL: https://www.dataexpert.io/lesson/buildingfunctionalaiagentsusinglangchainday1lecture-mar26 Enrolled students In this lesson, Zach dives into the concept of AI agents, exploring what they are and how they differ from simple prompts and tools. He emphasises the importance of understanding the AI pyramid, where most people are still at the bottom, and encouraged everyone to feel confident in their learning journey. Zach also introduces the idea of using tools and middleware to enhance AI capabilities, and the class discussed the need for human-in-the-loop validation for risky actions. ### Special Guest, Madison Schott [Recorded Dec 22, 2023] URL: https://www.dataexpert.io/lesson/v3-5-special-guest-madison-schott-recorded-dec-22-2023-4112 Enrolled students In this session, Madison discusses data engineering, analytics, and tech industry trends. Madison shares her unique journey as an analytics engineer at Convert Kit. They explore differences between data and analytics engineering, emphasize software engineering role in analytics, and touch on industry shifts. The conversation also touches on data quality, AI, and effective metric management. They conclude by sharing experiences with role transitions and career growth. This conversation offers valuable insights into the evolving world of data and analytics.[Recorded on Dec 22,2023] ### Langchain Starter Repo Setup URL: https://www.dataexpert.io/lesson/langchain-starter-repo-p737-l2226 Module: Kickoff & Setup · Enrolled students Setting up the Langchain starter repository is essential for those looking to dive into AI development. This lesson guides you through cloning the repository, creating a Python virtual environment, and installing necessary libraries using PIP. You'll also learn how to configure your OpenAI API key and run both the backend and frontend of the application, enabling you to interact with Langchain's chat functionalities. Perfect for beginners eager to start building AI applications. ### How to use snowpark and Snowflake optimization Lab URL: https://www.dataexpert.io/lesson/how-to-use-snowpark-and-snowflake-optimization-lab-p735-l652 Module: Week 7: Analytical Patterns & CDC · Enrolled students In this lab, Zach dives into cost management and data handling processes. He explores loading data into the Polygon API, connecting to Snowflake, and managing super secret credentials. He also touches on Capstone project deadlines and demonstrate querying data from Trino. ### Q&A with Zach Week 4 (2025-10-09) URL: https://www.dataexpert.io/lesson/qa-with-zach-week4-oct2025-p737-l1547 Module: Q&A with Zach · Enrolled students ### Data Lakes with Delta Table Day2 Lecture URL: https://www.dataexpert.io/lesson/datalakeswithdeltatableday2lecture-feb26-p736-l2023 Module: Week 3: Data Lakes with Delta Table · Enrolled students In this lecture, Zach dives into Unity Catalog, explaining how it serves as the central control plane for their Lakehouse, preventing data swamps and providing fine-grained access control. He explores the evolution of data architectures, comparing Lambda and Kappa architectures, and discuss how Databricks is evolving to resemble Snowflake with features like liquid clustering and enhanced governance. ### Graph Data Modeling Lab URL: https://www.dataexpert.io/lesson/graph-data-modeling-lab Module: Bonus · Enrolled students In this lab, Zach discusses the importance of data metrics and effective communication strategies. He highlights various data sources, such as banks, rental properties, and Excel sheets, and explain how they can be used to track metrics related to health, career, and recreation. He also emphasizes the significance of considering context and building relationships when communicating with stakeholders. The information shared can help them gain valuable insights and improve their communication skills. ### Tech Talk - Shwetha URL: https://www.dataexpert.io/lesson/techtalks5-mar26 Enrolled students The session features Shwetha, a data engineer at OpenAI, who shares insights from her experience in data engineering, particularly in handling unstructured data and building detection pipelines for safety classifiers. She discusses the differences between structured and unstructured data roles, the importance of logging, and the evolving landscape of data engineering, emphasizing the need for data quality and understanding user requirements. This lesson is beneficial for aspiring data engineers and those interested in AI safety systems. ### Special Guests, Sundar Velayutham and Aishwarya Srinivasan [August 9, 2023] URL: https://www.dataexpert.io/lesson/week-4-speaker-v2 Enrolled students Join us each week as we sit down with visionary speakers from diverse fields, engaging in enlightening conversations that explore cutting-edge insights, trends, and their wealth of expertise. (Recorded August 9, 2023) ### Supplemental lab session on GuardrailsAI URL: https://www.dataexpert.io/lesson/supplemental-lab-session-on-guardrailsai-july25 Enrolled students In this session, Zach walks through setting up AI Guardrails, similar to our Vercel deploy example from week two. He installs various validators, including detect PII, competitor check, and toxic language, to ensure our AI behaves appropriately. He emphasizes the importance of explicit error handling and demonstrated how to validate user inputs and outputs effectively. He also discusses the need to implement guardrails in multiple areas, including document input and retrieval. ### Data modeling with Date lists Lab URL: https://www.dataexpert.io/lesson/data-modeling-with-date-lists-lab Module: Bonus · Enrolled students This lab guides you through the process of analyzing user activity using compact tables. The lab demonstrates how to create a table that efficiently captures user IDs and their corresponding activity status. We address the intricacies of handling date calculations and utilizing bitwise operations to represent user activity. The lab captures the presenter encountering and troubleshooting various challenges, providing valuable insights and explanations. By the end of the lab, you will have gained a clear understanding of how to design and leverage compact tables for the purpose of analyzing user activity. ### Data Structures and Algorithms Interview URL: https://www.dataexpert.io/lesson/data-structures-and-algorithms-interview Module: Interview Skills · Enrolled students Data structures and algorithms interviews can be challenging, often requiring candidates to demonstrate their problem-solving skills under pressure. This lesson covers essential strategies for success, including understanding common data structures like stacks, queues, and trees, as well as mastering algorithmic concepts such as time and space complexity. Ideal for aspiring software engineers and data engineers, it emphasizes the importance of clear communication and optimization techniques during coding interviews. ### Chat with Zach [Aug 17, 2023] URL: https://www.dataexpert.io/lesson/week-5-zach-qa-v2 Enrolled students Delve into the world of data engineering with our weekly Q&A / Fireside Chat series, featuring the founder of EcZachly Inc and lead instructor of the Advanced Data Engineering Bootcamp. Each session covers a different topic, providing invaluable insights and real-world experiences to empower data enthusiasts in their journey. (Recorded Aug 17, 2023) ### Structured Streaming Kafka to Delta Live Table Day2 Lecture URL: https://www.dataexpert.io/lesson/structuredstreamingkafkatodeltalivetableday2lecture-feb26-p736-l2031 Module: Week 4: Structured Streaming with Spark & Kafka · Enrolled students In this lecture, Zach dives deeper into Delta Live Table and its evolution from traditional streaming architectures like Lambda and Kappa. He discusses the challenges of Python UDFs, emphasizing their scalability issues, and highlights the advantages of Delta Tables, which simplify the transition from streaming to batch processing. He explores the Medallion Architecture and its implications for data quality, and shares insights on managing bad data in their pipelines. ### Tech Talk - Steve Huynh URL: https://www.dataexpert.io/lesson/guest-speaker-2-v4 Enrolled students In this tech talk, Steve discusses different ways to grow your career in the tech industry based on his 18 years of experience at Amazon. He touches on the benefits of switching companies, the importance of asking the right questions, and the value of defining problem spaces. He also shares insights on how to navigate ambiguity and provide clarity in your work. Watch this video to gain valuable career advice and actionable strategies for professional growth. ### Build community, not content by Zach Wilson URL: https://www.dataexpert.io/lesson/linkedin-masterclass-2 Module: Making Content on LinkedIn · Enrolled students Build community, not content by Zach Wilson. This will be filmed on May 17th, 2024. ### Youtube URL: https://www.dataexpert.io/lesson/youtube-9367 Enrolled students testing updating ### Vaishali Macwan - Senior Data Scientist at Amazon URL: https://www.dataexpert.io/lesson/tech-talk-vaishali-may2025-p736-l1166 Module: Guest Speakers · Enrolled students Vaishali Macwan, a Senior Data Scientist at Amazon, shares her extensive experience in data science and analytics, emphasizing the importance of collaboration between data engineers and data scientists. She discusses how AI tools can enhance efficiency in data workflows and the necessity of understanding both technical and business aspects to drive impactful changes. This session is valuable for aspiring data professionals looking to navigate the complexities of tech roles and improve their collaboration skills. ### Day 1: Array model & indexing URL: https://www.dataexpert.io/lesson/array-model-indexing-p913-d1 Enrolled students Understand arrays as ordinal, indexable sequences with O(1) access by index, how iteration works, and when to prefer arrays over key-value stores. ### TA Office Hour 7 (2025-08-06) URL: https://www.dataexpert.io/lesson/ta-office-hour7-july2025-p737-l1377 Enrolled students The session focuses on the final week's assignment, which involves creating an evaluation framework for a dependency scanner. Participants will learn to generate synthetic data, simulate vulnerabilities, and define evaluation metrics such as precision and recall. This lesson is beneficial for those looking to enhance their skills in data generation, metrics evaluation, and report creation, particularly in the context of software dependency management. ### Day 2 Lab: Vibe Coding Full Stack application in 45 minutes URL: https://www.dataexpert.io/lesson/prompt-engineering-day2-lab-jul2025-p737-l1172 Module: Week 1: Prompt Engineering & Basic RAG · Enrolled students In this video, Zach walks through building a FastAPI server that features a chat GPT-like interface. He starts with a simple "Hello World" endpoint and gradually enhances the application by adding an HTML interface and integrating a Milvus vector database for retrieval-augmented generation (RAG). Zach also addresses some common issues, such as dependency management and connection errors, while emphasizing the importance of being skeptical of AI-generated code. ### Ryan Brandt - CEO at VundaAI (2.0) URL: https://www.dataexpert.io/lesson/tech-talk3-oct2025-p737-l1551 Module: Tech Talks · Enrolled students ### Data modeling with Date lists Lecture URL: https://www.dataexpert.io/lesson/data-modeling-with-date-lists-lecture-p735-l585 Module: Bonus · Enrolled students In this lecture, we cover the fundamental concepts of fact and dimension data structures. The lecture delves into the significance of fact data, highlighting its role as the core element in data modeling. Additionally, we explore dimension data and its crucial function in grouping and categorizing information. The discussion emphasizes the importance of capturing changes within dimensions and introduces the benefits of employing slowly changing dimensions. The lecture provides examples and insights, offering viewers a comprehensive understanding of fact and dimension data structures in the context of data engineering. ### Capstone Presentation URL: https://www.dataexpert.io/lesson/capstonepresentation-aug26 Enrolled students The Capstone Presentation showcases a variety of innovative projects developed by participants in the Data Brick State Engineering bootcamp. Each project highlights unique applications of data engineering and AI, including a Polymarket monitor that analyzes real-time trading data, a job market tracker that connects AI skills with job opportunities, and a healthcare app designed to optimize patient care coordination. Attendees will gain insights into practical implementations of data pipelines, machine learning models, and user-centric applications across diverse industries. ### Chat with Zach [Aug 3, 2023] URL: https://www.dataexpert.io/lesson/v2-3-chat-with-zach-aug-3-2023-4204 Enrolled students Delve into the world of data engineering with our weekly Q&A / Fireside Chat series, featuring the founder of EcZachly Inc and lead instructor of the Advanced Data Engineering Bootcamp. Each session covers a different topic, providing invaluable insights and real-world experiences to empower data enthusiasts in their journey. (Recorded Aug 3, 2023) ### OpenClaw and Automating business processes Day1 Lab URL: https://www.dataexpert.io/lesson/openclawandautomatingbusinessprocessesday1lab-mar26-p737-l2146 Module: Week 3: Agentic AI, MCP & Automation · Enrolled students In this video, Drew walks through setting up OpenClaw, an open-source project, using Docker. He covers the installation process, security considerations, and how to create your first agent, the Chief of Staff. He emphasises the importance of understanding security risks and using best practices. ### RAG and LLMs Day2 Lab URL: https://www.dataexpert.io/lesson/rag-and-llms-day-2-lab-jan2025-p736-l829 Module: Bonus - LLMs · Enrolled students In this lab, Zach walks you through some key concepts in natural language processing, particularly focusing on lemmatization and its importance in our projects. He also discusses how to manage environment variables and the integration of OpenAI keys into our repository. ### Day 2 Lab: Enhancing RAG with Agentic AI URL: https://www.dataexpert.io/lesson/advanced-rag-and-agentic-ai-day2-lab-july2025 Enrolled students Enhancing Retrieval-Augmented Generation (RAG) with Agentic AI focuses on building a multi-modal research agent using Langchain and OpenAI tools. Participants will learn to deploy a system that conducts web searches, analyzes YouTube transcripts, and synthesizes outputs into reports and podcasts. This lesson is ideal for data engineers and AI practitioners looking to improve their skills in creating intelligent workflows that integrate various AI capabilities. ### TA Office hour 8 URL: https://www.dataexpert.io/lesson/taofficehour8-jan2026 Enrolled students Office hours provide an opportunity for students to ask questions and clarify doubts regarding their coursework and capstone projects. Key topics discussed include submission deadlines for homework and capstone, scheduling additional support sessions, and specific project requirements related to analytics engineering and data pipelines. Students benefit from personalized guidance on their assignments, including data modeling and using APIs, while also sharing experiences about job applications in the field. ### TA Office Hour 2 URL: https://www.dataexpert.io/lesson/ta-office-hour2-oct2025 Enrolled students ### Future of Data Engineering with Archita URL: https://www.dataexpert.io/lesson/futureofdataengineeringwitharchita-0f130 Module: Tech Talks · Enrolled students ### Career Development - Strategic Networking URL: https://www.dataexpert.io/lesson/career-development-strategic-networking-jan2025 Enrolled students Strategic networking is essential for career growth and visibility in the data and AI engineering fields. This lesson covers various networking strategies, including leveraging LinkedIn, attending meetups, and engaging with industry influencers. Participants will learn how to identify key contacts, optimize their online presence, and effectively communicate to enhance their professional relationships, ultimately accessing hidden job markets and boosting their credibility. ### Chat with Zach URL: https://www.dataexpert.io/lesson/v4-3-chat-with-zach-7982 Enrolled students Exclusive Q&A for homework doers. [Recorded on May23rd, 2024] ### Advanced SQL and Applying Analytical patterns (Day 1 Lab) URL: https://www.dataexpert.io/lesson/analytical-patterns-day-1-lab-v4 Module: Week 5: Analytical Patterns & Analysis in Iceberg · Enrolled students This is a pre-recorded class. In this hands-on lab session, we embark on a practical journey of growth accounting. The video guides you through the step-by-step process, illustrating how to create a partition column, calculate active dates, and adeptly track state transitions. Common challenges encountered in this process are addressed, accompanied by effective solutions. By the conclusion of the video, you will have gained a lucid understanding of analyzing user growth and retention. This session equips you with practical skills to navigate and excel in the intricacies of growth accounting, providing valuable insights applicable to real-world scenarios. [Recorded on Nov 27, 2023] ### TA Office Hour 8 URL: https://www.dataexpert.io/lesson/ta-office-hour-8-may2025-p736-l1208 Enrolled students Office Hour 8 provides a platform for students to ask questions and clarify doubts regarding their capstone projects and assignments. Key topics discussed include the use of Databricks for pipeline orchestration, project requirements, and troubleshooting issues related to AWS Glue and S3 buckets. Participants benefit from personalized guidance on project implementation, tool selection, and certification requirements, fostering a collaborative learning environment. ### Day 2 Lab: Building an asynchronous vs synchronous agent URL: https://www.dataexpert.io/lesson/building-end-to-end-ai-application-day2-lab-july25-p737-l1191 Module: Week 5: End-to-End AI Applications · Enrolled students In this lab, Zach walks viewers through the powerful capabilities of NAN for creating AI pipelines—task workflows analogous to data pipelines. He explores various triggers, including manual activation, app events, form submissions, and scheduled triggers, emphasizing the importance of efficient automation. Zach demonstrates a workflow that scrapes LinkedIn profiles and evaluates candidates for a data engineering bootcamp, while highlighting the need to improve email formatting for clearer communication. He also addresses how to minimize API calls by using Google Sheets as a cache for storing LinkedIn data. ### Guest Speakers / Panel Discussion (The reality vs. expectations of Staff+ work) URL: https://www.dataexpert.io/lesson/guestspeakerspaneldiscussion-b4f9d Enrolled students What I wish I knew before pursuing Staff+ The reality vs. expectations of Staff+ work Biggest surprises about operating at this level How the role differs across companies (Meta vs Google vs Amazon) Managing the tension between deep technical work and broad influence When to pursue Staff+ vs. management track ### Fact Data Modeling - Practical Guide to Formatting and Aggregating Data with Iceberg (Day 3 Lab) - Pre-recorded from V3 URL: https://www.dataexpert.io/lesson/fact-data-modeling-day-3-lab-v4 Module: Week 2: Fact Data Modeling in Iceberg · Enrolled students This is a pre-recorded lesson from V3. This lab demonstration provides a practical guide on working with daily metrics. The lab walks through the process of formatting data, highlighting key steps such as partitioning and grouping by date and metric name. The benefits of utilizing a monthly array web metrics table are explained, and the video demonstrates how to create monthly aggregates. The lab aims to equip viewers with hands-on skills for effectively handling and organizing daily metrics, showcasing a step-by-step approach to enhance data management and analysis.[Recorded Nov 16, 2023] ### Data Communication and Impact (Day 1 Lab) URL: https://www.dataexpert.io/lesson/visual-impact-day-1-lab-v4 Module: Week 6: Data Impact Communication & Visualization with Tableau · Enrolled students This is a pre-recorded class. Join this hands-on session, where you will dive into the practical realm of data visualization using SuperSet or alternative visualization tools. Through step-by-step guidance, you will explore the intricacies of working with CSV files, mastering the creation of sets to elevate your visualizations. Delve into discussions on diverse chart patterns and discover strategies to enhance their impact. This lab promises valuable techniques for mastering the art of effective data visualization. Tune in and elevate your visualization skills! [Recorded on Dec 18th 2023] ### TA Office Hour 7 URL: https://www.dataexpert.io/lesson/ta-office-hour-7-may2025-p736-l1167 Enrolled students This office hours session focuses on deploying ETL pipelines using Databricks and the Medallion architecture. Participants discuss creating and managing notebooks for raw, silver, and gold data layers, as well as troubleshooting issues related to pipeline execution. The session is beneficial for learners looking to refine their understanding of data pipeline deployment and dashboard creation within Databricks, particularly those working on assignments or projects involving synthetic data generation. ### Local Video URL: https://www.dataexpert.io/lesson/local-video-6276 Enrolled students ### Day 7: Linked list basics URL: https://www.dataexpert.io/lesson/linked-list-basics-p913-d7 Enrolled students Learn how singly linked lists work, how to traverse/search them, do O(1) inserts/deletes at the head or after a node, and when to choose lists vs arrays. ### Chat with Zach [Aug 3, 2023] URL: https://www.dataexpert.io/lesson/week-3-zach-qa-v2 Enrolled students Delve into the world of data engineering with our weekly Q&A / Fireside Chat series, featuring the founder of EcZachly Inc and lead instructor of the Advanced Data Engineering Bootcamp. Each session covers a different topic, providing invaluable insights and real-world experiences to empower data enthusiasts in their journey. (Recorded Aug 3, 2023) ### Tuesday Quiz URL: https://www.dataexpert.io/lesson/tuesdayquiz-ba9b6 Enrolled students ### Docker Vercel and Optimization URL: https://www.dataexpert.io/lesson/docker-vercel-and-optimization Module: Advanced Frontend Development · Enrolled students Learn how to deploy Next.js applications using Docker and Vercel while exploring optimization techniques for performance enhancement. This lesson covers essential deployment strategies, including environment configurations, security considerations, and CI/CD practices. You'll also delve into optimization methods such as code splitting and image management to improve load times and user experience, making it ideal for developers looking to enhance their web applications. ### TA Office hour URL: https://www.dataexpert.io/lesson/taofficehour0-feb26 Enrolled students This office hours session provides an opportunity for students to clarify course material, ask questions about homework, and discuss capstone project ideas. Key topics include navigating course resources, understanding homework formats, and forming study groups based on time zones. Participants can also seek assistance with technical issues related to their projects, ensuring they stay on track with their learning objectives. ### Fact Data Modeling: Core Elements in Data Modeling Day 2 Lecture URL: https://www.dataexpert.io/lesson/fact-data-modeling-day-2-lecture-yt-p736-l664 Enrolled students In this lecture, we cover the fundamental concepts of fact and dimension data structures. The lecture delves into the significance of fact data, highlighting its role as the core element in data modeling. ### Test URL: https://www.dataexpert.io/lesson/27716d5f-1bdf-470d-a6b5-86e883b7363atest Enrolled students A guest speaker shares insights on career advancement in tech, focusing on job searching, interviewing, and negotiation strategies. With over 26 years of experience, he discusses the importance of understanding the hiring landscape, the value of storytelling in interviews, and how to navigate the challenges of transitioning roles. This session is ideal for data engineers, analysts, and software engineers looking to enhance their job search skills and gain clarity on their career paths. ### You must learn Databricks in 2026 - The DataExpert.io Roadmap Lecture URL: https://www.dataexpert.io/lesson/youmustlearndatabricksin2026thedataexpertioroadmap-lecture Enrolled students ### Data Visualization and Impact: Exploring Data Visualization and Aggregation Techniques Day 2 Lab URL: https://www.dataexpert.io/lesson/data-visualization-and-impact-day-2-lab-yt-p736-l693 Enrolled students Welcome to a hands-on lab where we explore diverse data aggregation and visualization techniques using a sample dataset. The video takes you through the intricacies of aggregating data at different levels, such as host and source, and demonstrates the creation of meaningful visualizations. ### Advanced dbt: Production Pipelines and dbt with Airflow Lecture URL: https://www.dataexpert.io/lesson/advanceddbtday2lecture-jan2026-p735-l1923 Module: Week 2: Advanced dbt · Enrolled students Advanced dbt production pipelines are explored through the integration of dbt with Airflow in this comprehensive lecture. Participants will learn how to automate data transformation processes, manage dependencies, and implement efficient scheduling using dbt commands like dbt build and Airflow's task management. This lesson is ideal for data engineers and analysts looking to enhance their workflow efficiency and ensure robust data pipeline management in production environments. ### Optimize your dev workflow with Claude Code and Codex Day2 Lab URL: https://www.dataexpert.io/lesson/optimizeyourdevworkflowwithclaudecodeandcodexday2lab-mar26-p737-l2144 Module: Week 2: Advanced RAG, Reranking & Dev Workflow · Enrolled students In this lab, Kevin walk throughs building a composable testing and validation pipeline using Claude skills. He starts a test server, then create a lint and test skill, and finally build Playwright based smoke tests that record browser actions and send videos to Telegram via MCP, then wrap everything in a validate skill. He then creates a top level feature skill that plans, builds, and adds a new favourites feature to the smoke tests, so validation runs end to end. ### Apache Spark programming with Databricks Day 1 Lecture URL: https://www.dataexpert.io/lesson/apache-spark-programming-with-databricks-day-1-lecture-may2025-1 Enrolled students Gain a comprehensive understanding of Apache Spark programming with Databricks in this introductory lecture. Learn about Spark's architecture, including the roles of drivers and executors, and explore various programming languages supported by Spark, such as Python and SQL. This lesson also covers key concepts like transformations, actions, and the differences between Spark and Hadoop, making it ideal for data engineers and data scientists looking to enhance their big data processing skills. ### Capstone Showcase with Jesse Charbneau URL: https://www.dataexpert.io/lesson/capstone-showcase-with-jesse-charbneau-7173 Enrolled students The Capstone Showcase presents Jesse Charbonneau's project, "Rosie Economics," which focuses on visualizing economic data with impact analysis overlays. Participants will learn how to leverage tools like React, FastAPI, and OpenAI's REST interface to create interactive charts that simplify complex economic indicators for casual users and DIY investors. The session also covers the integration of predictive analytics and AI-generated insights, making it beneficial for those interested in data visualization and economic analysis. ### Prompt Engineering + Basic RAG Day 2 Lecture URL: https://www.dataexpert.io/lesson/prompt-engineering-day2-lecture-oct2025 Enrolled students In this video, Zach explores the world of vectors, retrieval-augmented generation (RAG), and fine-tuning, underscoring the significance of understanding vectors in AI engineering. He examines the two types of vectors—dense and sparse—elaborating on their differences and advising when to use each. Additionally, he discusses how to select the appropriate embedding model, with a particular emphasis on OpenAI's options, along with the trade-offs between RAG and fine-tuning in relation to data dynamics and access patterns. ### Q&A Week 1 URL: https://www.dataexpert.io/lesson/qa-week-1-jan2025-p736-l717 Module: Q&A with Zach · Enrolled students The Week 1 Q&A session covers essential topics in data engineering, focusing on tools like Apache Iceberg, Hive, Trino, and Spark. Participants can gain insights into the practical applications of these technologies, including when to use Trino versus Spark for data processing and how to integrate Iceberg with AWS Glue. The discussion also addresses common challenges and strategies for capstone projects, making it beneficial for both aspiring and current data engineers looking to enhance their skills and knowledge in the field. ### End-to-End AI Applications Day 2 Lecture URL: https://www.dataexpert.io/lesson/building-end-to-end-ai-application-day2-lecture-oct2025 Enrolled students In this lecture Zach discusses about AI applications and stresses the critical importance of understanding business requirements before diving into any project. He recounts a cautionary tale from his time at Netflix, where a $25 million initiative collapses because the team over‑promised and under‑delivered, underscoring that 80 % of AI projects fail. Zach urges the students to invest time in the foundational layers of development, warning that rushing leads to costly mistakes. He introduces the concept of “human in the loop,” explaining when it becomes essential—particularly in high‑risk scenarios. ### Career Dev Session 1 - Win with soft skills! URL: https://www.dataexpert.io/lesson/career-dev-session1-oct2025 Enrolled students ### TA Office Hour 1 URL: https://www.dataexpert.io/lesson/ta-office-hour-1-april2025-p735-l981 Enrolled students Office hours provide a platform for students to ask questions and clarify doubts regarding their assignments. This session, led by Sandeep, covers the first assignment of the boot camp, which involves fetching data from the Polygon API and creating a partitioned table using PyIceberg or PySpark. Key topics include handling API responses, managing AWS credentials, and understanding the assignment's requirements. Students can benefit from the interactive discussion and demonstrations related to their coursework. ### Snowflake Basics Day 1 Lab URL: https://www.dataexpert.io/lesson/snowflake-basics-day-1-lab-7680 Module: Week 3: Data Modeling on Snowflake · Enrolled students Welcome to day one of Snowflake Basics Lab! In this lab, Bruno demonstrates how to create tables and views in Snowflake, highlighting the ease and flexibility compared to traditional methods. He showcases features like dark theme, tables, views, UDFs, AI, ML, and project organization. ### Azure - Week 4 URL: https://www.dataexpert.io/lesson/azure-week-4-may2025-p736-l1162 Enrolled students Understanding Azure Data Factory (ADF) is essential for anyone looking to create data pipelines in the Azure cloud environment. This session covers the foundational concepts of ADF, including its components like link services, datasets, and activities, and how they work together to facilitate data movement and transformation. Participants will learn to set up ADF for ETL processes, making it accessible even for those with minimal coding experience, and explore its similarities with other tools like SSIS and AWS Glue. ### TA Office Hour 6 URL: https://www.dataexpert.io/lesson/ta-office-hour6-oct2025 Enrolled students ### Career Dev Session 4 URL: https://www.dataexpert.io/lesson/career-dev-session4-oct2025 Enrolled students ### Q&A with Zach Week 1 URL: https://www.dataexpert.io/lesson/qa-with-zach-week-1-april2025 Enrolled students The Week 1 Q&A session covers a range of topics relevant to data engineering and analytics, including the impact of AI on the field, effective data governance practices, and strategies for managing data environments. Participants gain insights into deploying tools like Iceberg on-premise, optimizing data masking in Snowflake, and the importance of conceptual data modeling for capstone projects. This session is ideal for data engineers and analysts looking to deepen their understanding and enhance their skills in a collaborative environment. ### dbt Basics Day 3 Lab URL: https://www.dataexpert.io/lesson/dbt-basics-day-3-lab-jan2025-p736-l778 Enrolled students In this lab, Bruno walks you through the second day of our DBT Basics series, focusing on adding tests and snapshots to our DBT project. He covers various types of tests, including data tests and unit tests, and demonstrate how to implement them effectively. Additionally, he introduces the concept of snapshots and how to set them up in your project. ### Tuesday Quiz URL: https://www.dataexpert.io/lesson/tuesdayquiz-b98b1 Enrolled students ### Data Lakes with Delta Table Day1 Lab URL: https://www.dataexpert.io/lesson/datalakeswithdeltatableday1lab-feb26 Enrolled students In this lab, Zach walks you through setting up your Git repository in Databricks and creating a Delta Table with synthetic data. He explores schema generation, data manipulation, and how to query the table's history, including operations like deletes and updates. Zach also highlights the importance of time travel in Delta Tables for data recovery. ### Advance Snowflake Day 2 Lecture URL: https://www.dataexpert.io/lesson/advancesnowflake day2lecture-jan2025-p735-l1911 Module: Week 4: Advanced Snowflake · Enrolled students In this lecture, Zach dives into the exciting realm of context engineering and its evolution from traditional data engineering. He explores the nine different types of context that data engineers must understand, emphasizing the importance of business context, organizational context, and semantic context. Zach shares insights on how to create rich contexts from unstructured data and the role of vectors in this process. ### TA Office Hour 1 URL: https://www.dataexpert.io/lesson/taofficehour1-aug26 Enrolled students The TA Office Hour session provides an interactive platform for students to seek help with their lab assignments and technical issues related to data engineering concepts. Participants discuss troubleshooting steps for schema naming conventions, synchronization tables, and resolving errors in their projects. This session is beneficial for students looking to clarify doubts, understand practical applications of the tools discussed in the course, and enhance their learning experience through peer interaction and guidance from teaching assistants. ### Advanced SQL Patterns Day 2 Lab URL: https://www.dataexpert.io/lesson/advanced-sql-patterns-day-2-lab-jan2025 Enrolled students Master advanced SQL techniques with a focus on grouping sets, window functions, and the qualify keyword in this lab session. Participants will learn how to effectively manage data cardinality, create OLAP cubes, and utilize window functions to analyze user sessions and event timestamps. This lesson is ideal for data engineers and analysts looking to enhance their SQL skills for real-world applications, particularly in environments like Snowflake and Starburst. ### Processing data with Apache Flink (Day 2 Lab) URL: https://www.dataexpert.io/lesson/flink-streaming-day-2-lab-v4 Module: Week 5: Kafka-Flink Streaming Pipelines · Enrolled students In this lab, we delve into data processing capabilities of Flink. We will explore components like Kafka, Postgres, and Spark, witnessing how they collaborate in efficient data pipelines. The concept of parallelism and its impact on Flink jobs will be demystified. Throughout the session, we will highlight key features and considerations for managing sizable Flink pipelines. To top it off, we will run a live popularity contest, showcasing Flink in action. Gain a swift understanding of Flink capabilities and optimization strategies in this hands-on lab.[Recorded June 6th, 2024] ### Spark Structured Streaming Day 2 Lab URL: https://www.dataexpert.io/lesson/spark-structured-streaming-day-2-lab-jan2025 Enrolled students Explore the practical application of Spark Structured Streaming in this lab session, focusing on real-time data ingestion from Kafka to Databricks. Participants will learn how to set up a streaming pipeline, create bronze, silver, and gold tables, and utilize JSON schemas for data transformation. This lesson is ideal for data engineers and analysts looking to enhance their skills in building data pipelines and dashboards for insightful decision-making. ### Lesson 4 URL: https://www.dataexpert.io/lesson/lesson-4-2335 Enrolled students ### Introduction to Containers URL: https://www.dataexpert.io/lesson/introduction-to-containers-5d238 Module: Q&A with Dan · Enrolled students ### Day 1 Lecture: Leveraging MCP and tools to integrate effectively URL: https://www.dataexpert.io/lesson/building-end-to-end-ai-application-day1-lecture-july25 Enrolled students In this session , Zach discusses the Model Context Protocol (MCP) and how it enhances AI capabilities beyond simple prompts. He explores the five levels of AI engagement, from basic prompting with ChatGPT to advanced AI agent work, highlighting the importance of tools in enabling AI to take meaningful actions. Zach emphasizes that without tools, AI is limited to providing information rather than executing tasks. ### advanced dbt Day 2 Lecture URL: https://www.dataexpert.io/lesson/advanceddbtday2lecture-a6b62 Enrolled students ### Creating a full dbt Project Lab URL: https://www.dataexpert.io/lesson/dbtbasicsday2lab-jan2026-p735-l1927 Module: Week 1: dbt Basics · Enrolled students Learn how to create a complete dbt project by exploring essential features such as snapshots, tests, and packages. This lab focuses on building a slowly changing dimension to track customer data changes over time, ensuring accurate historical analysis. Ideal for analytics engineers, this lesson provides practical insights into using dbt for data transformation and quality assurance, enhancing your data modeling skills. ### Lesson 4 URL: https://www.dataexpert.io/lesson/lesson-4-7281 Enrolled students ### Apache Iceberg: Data Lake Architectures Lecture URL: https://www.dataexpert.io/lesson/apache-iceberg-lecture-day-2-april2025-p735-l957 Module: Week 5: Data Modeling on Iceberg · Enrolled students In this lecture, Zach dives into the intricacies of data pipelines, focusing on the differences between warm and cold storage, and how they impact data architecture. He also discusses the importance of pre-aggregating data for low-latency dashboards and the role of Kafka in managing data in motion. ### Apache Iceberg Lab Day3 URL: https://www.dataexpert.io/lesson/apacheiceberglabday3-9e57d-p735-l1996 Module: Week 5: Data Modeling on Iceberg · Enrolled students The Apache Iceberg lab focuses on implementing the right audit published pattern using the Polygon API to create and manage tables in Iceberg. Participants will learn how to ingest stock data using PySpark, handle paginated API responses, and manage data quality through various checks. This lesson is ideal for data engineers looking to enhance their skills in data modeling and API integration with Iceberg and Spark. ### Test 8 URL: https://www.dataexpert.io/lesson/cd7568fa-3ccf-44bc-9baa-e2ba676b3b07test8 Enrolled students Explore the financial trajectory of a data engineering career through personal insights and salary milestones. This lesson details the speaker's journey from a data analyst earning $30,000 to a peak salary of $575,000 at Airbnb, highlighting key career moves and promotions at major tech companies like Facebook and Netflix. Ideal for aspiring data engineers and professionals seeking to understand potential earnings and career growth in the field. ### Applying Advanced SQL for Analytical Insights - Recursive CTEs, Window Functions, and Practical Insights (Day 2 Lecture) URL: https://www.dataexpert.io/lesson/analytical-patterns-day-2-lecture-v3 Module: Week 3: Analytical Patterns & Analysis in Iceberg · Enrolled students In this presentation, Zach explores a range of topics related to Advanced SQL Analytical Patterns. The discussion covers the application of recursive CTEs, count case when, grouping sets, cross join unnest, cube, rollup, and window functions. Drawing from Zachs experience in conducting interviews and working with these concepts, the presentation offers insights and practical advice. [Recorded on Nov 29, 2023] ### TA Office Hour 6 (2025-10-09) URL: https://www.dataexpert.io/lesson/ta-office-hour6-oct2025-p737-l1541 Enrolled students ### Lab 1 URL: https://www.dataexpert.io/lesson/advanced-api-lab1 Module: Advanced Backend Development · Enrolled students Authentication is a key focus in this lab, where participants learn to implement user authentication using Supabase, a backend-as-a-service platform similar to Firebase. The lesson covers setting up authentication, creating a REST API for a to-do list application, and managing user sessions with middleware. Additionally, it introduces caching with Redis to optimize data retrieval, making it ideal for developers looking to enhance their backend skills and application performance. ### Building Expertise: STAR format and Impact Story Frameworks URL: https://www.dataexpert.io/lesson/buildingexpertisestarformatandimpactstoryframeworks-8cf2a Enrolled students When and why STAR works: interviews, quick examples, demonstrating execution Crafting tight, compelling STAR stories How Impact Stories build on STAR for promotion contexts How the same accomplishment scales from STAR → Impact Story ### TA Office Hour 4 (2025-07-24) URL: https://www.dataexpert.io/lesson/ta-office-hour4-july2025-p737-l1302 Enrolled students Office hours provide a platform for participants to ask questions and clarify concepts related to their coursework. This session covered various topics, including homework submissions, testing criteria for systems, and the integration of vector databases like Milvus and Pinecone. Attendees sought guidance on specific homework components, AWS key management, and the use of GitHub for data loading, ensuring they are well-prepared for their projects and assignments. ### Capstone Showcase Jan 2025 URL: https://www.dataexpert.io/lesson/capstone-showcase-jan2025 Enrolled students The Capstone Showcase features a diverse array of projects from aspiring data engineers, showcasing their skills in real-world applications. Presentations cover topics such as real-time train delay monitoring using APIs, eSports voice analytics leveraging unstructured data, and stock market indicators analytics for actionable insights. Participants demonstrate their proficiency in tools like Kafka, Snowflake, and DBT, while addressing challenges in data integration, visualization, and real-time processing. ### Apache Iceberg Lecture Day 2 URL: https://www.dataexpert.io/lesson/apache-iceberg-lecture-day-2-april2025 Enrolled students In this lecture, Zach dives into the intricacies of data pipelines, focusing on the differences between warm and cold storage, and how they impact data architecture. He also discusses the importance of pre-aggregating data for low-latency dashboards and the role of Kafka in managing data in motion. ### dbt basics Day 2 Lab URL: https://www.dataexpert.io/lesson/dbt-basics-day-2-lab-april2025 Enrolled students The second day of the DBT lab session provided participants with valuable insights into managing data models, testing methodologies, and the use of snapshots. The session emphasized the importance of structured environments and the need for effective testing strategies to ensure data integrity and accuracy. ### Shubham Srivastava - Senior Data Engineer at Amazon (2025-06-24) URL: https://www.dataexpert.io/lesson/tech-talk-shubham-may2025-p736-l1165 Module: Guest Speakers · Enrolled students Shubham Srivastava, a Senior Data Engineer at Amazon, shares insights from his extensive career in data engineering and analytics. He discusses his journey from starting at Deloitte to navigating various roles at Amazon, emphasizing the importance of adaptability and strategic career growth. Attendees will benefit from his experiences in managing data infrastructure, leveraging AI tools, and understanding the dynamics of team management and project prioritization in a fast-paced tech environment. ### Advanced SQL Patterns Day 2 Lab URL: https://www.dataexpert.io/lesson/advanced-sql-patterns-day-2-lab-jan2025-p736-l819 Enrolled students Mastering advanced SQL patterns is essential for data engineers looking to enhance their analytical skills. This lab focuses on practical applications of grouping sets, window functions, and the qualify keyword, using real-world data from bootcamp web events. Participants will learn how to reduce cardinality, create OLAP cubes, and effectively utilize window functions to analyze user sessions, making it invaluable for anyone involved in data analysis and reporting. ### Capstone Project Brainstorming URL: https://www.dataexpert.io/lesson/capstoneprojectbrainstorming-feb2026-p736-l2094 Module: Kickoff · Enrolled students Capstone Project Brainstorming focuses on generating innovative ideas for capstone projects in data and AI engineering. Participants discuss project requirements, such as building data pipelines, implementing quality controls, and utilizing multiple data sources. The session encourages collaboration and feedback on project concepts, including real-time analytics, AI-driven applications, and integrating various data feeds, making it ideal for students looking to refine their project ideas and enhance their technical skills. ### How to make AI that actualy provides Value Lecture URL: https://www.dataexpert.io/lesson/leveraging-mcp-and-tools-to-integrate-effectively-oct2025-p737-l1519 Module: Week 5: End-to-End AI Applications · Enrolled students In this lecture Zach walks the class through what AI excels at—prediction, detection, and optimization—while pointing out common barriers such as vague business objectives and user resistance to change. Zach argues for stronger frameworks and highlights N8N as a superior choice to the OpenAI Agent Kit for constructing AI workflows. ### TA Office Hour 4 URL: https://www.dataexpert.io/lesson/ta-office-hour-4-april2025 Enrolled students This office hours session covers a range of topics related to homework assignments and project proposals in data engineering. Participants discuss specific questions about SQL queries, UDF implementations, and the expectations for submitting assignments. Key points include the importance of clear comments in code, the interpretation of assignment requirements, and strategies for handling data in Snowflake. The session also addresses troubleshooting issues with project setups and the use of external APIs for capstone projects. ### Apache Spark Core Day 3 Lab URL: https://www.dataexpert.io/lesson/apache-spark-core-day-3-lab-jan2025-p736-l780 Module: Week 1: Databricks Basics · Enrolled students In this lab, Yared walks you through the basics of Spark SQL and the DataFrame API, highlighting how to create and interact with DataFrames. He explains the difference between transformations and actions, and how they affect job creation in Spark. He also emphasizes the importance of understanding narrow and wide transformations, as well as schema inference. ### TA Office Hour 1 URL: https://www.dataexpert.io/lesson/ta-office-hour1-oct2025 Enrolled students ### Managing Unstructured Data - Day 1 Lecture URL: https://www.dataexpert.io/lesson/managing-unstructured-data-day-1-lecture-may2025 Enrolled students In the video Zach focuses on unstructured data, particularly in the context of Retrieval Augmented Generation (RAG) and Large Language Models (LLMs). Zach explores how LLMs are impacting data engineering and vice versa, as well as discuss practical ways data engineers can leverage unstructured data. ### How to build job orchestration that matters URL: https://www.dataexpert.io/lesson/howtobuildjoborchestrationthatmatters-fc963 Enrolled students ### KPIs and Experimentation: Setting up and Analysing Experiments Day 1 Lab URL: https://www.dataexpert.io/lesson/kpis-and-experimentation-day-1-lab-yt-p736-l695 Free Observe how data is collected in this process, gaining hands-on experience in defining KPIs and conducting experiments for measurement and improvement. ### Q&A with Zach Week 4 URL: https://www.dataexpert.io/lesson/qa-with-zach-week4-oct2025 Enrolled students ### Javascript lab part 2 URL: https://www.dataexpert.io/lesson/ec0ac5ab-2b67-419d-aa13-090dc779ce05javascriptlabpart2 Module: Web Dev Fundamentals · Enrolled students Learn how to enhance your JavaScript skills by creating interactive web elements such as checkboxes and buttons. This lesson covers building a to-do list application, focusing on event listeners, DOM manipulation, and code refactoring for better organization. Ideal for web developers looking to improve their JavaScript proficiency and debugging techniques, this lesson provides practical examples and insights into effective coding practices. ### Day 1 Lecture: Trusting your workflows (evaluation metrics and guardrails) URL: https://www.dataexpert.io/lesson/ml-ops-day1-lecture-july25-p737-l1183 Module: Week 4: MLOps, Guardrails & Business Value · Enrolled students Understanding evaluation metrics and guardrails is crucial for ensuring the quality of machine learning applications in production. This lecture covers the common pitfalls of deploying large language models, including issues like hallucination and prompt injection, and discusses strategies to prevent these problems using techniques such as multi-tenant architecture and input/output guardrails. Data scientists and AI engineers will benefit from real-world examples and practical approaches to safeguard their applications. ### Chat with Zach [Nov 8, 2023] URL: https://www.dataexpert.io/lesson/zach-qa-2023-11-08-v3 Enrolled students Delve into the world of data engineering with our weekly Q&A / Fireside Chat series, featuring the founder of EcZachly Inc and lead instructor of the Advanced Data Engineering Bootcamp. Each session covers a different topic, providing invaluable insights and real-world experiences to empower data enthusiasts in their journey. (Recorded Nov 9, 2023) ### Kafka Fundamentals, consumer APIs with Confluent (Day 1 Lab) URL: https://www.dataexpert.io/lesson/flink-streaming-day-1-lab-v4 Module: Week 5: Kafka-Flink Streaming Pipelines · Enrolled students In this lab, Zach demonstrates how to process data from Kafka using Confluent. He explains the setup of Kafka queues, struct types, and data frames. He showcases batch processing and discuss the transition to streaming data. Students are encouraged to explore different processing strategies and understand the importance of real-time data handling. [Recorded on June 5th, 2024] ### Deep Dive On Workflows Day 3 Lecture URL: https://www.dataexpert.io/lesson/deep-dive-on-workflows-day-3-lecture-jan2025-p736-l822 Module: Week 4: Structured Streaming with Spark & Kafka · Enrolled students Explore the intricacies of orchestration in Databricks with a focus on workflows, particularly for structured streaming using Spark and Kafka. This session covers the advantages of Databricks workflows over traditional orchestration tools, emphasizing cost efficiency and unified management. Learn how to implement Delta Live Tables for data quality tracking and monitor task execution, making it ideal for data engineers and data scientists looking to streamline their data processing pipelines. ### Applying KPIs and Experimentation - Tim Chan's Guide to Statsig (Day 2 Lecture/Lab) URL: https://www.dataexpert.io/lesson/kpi-metrics-day-2-lecture-v3 Module: Week 4: KPIs and Experimentation with Statsig · Enrolled students Information about the recording will be available here soon. [Recorded on Dec 6, 2023] ### Day 1 Lecture: Prompt Engineering Theory URL: https://www.dataexpert.io/lesson/prompt-engineering-day1-lecture-jul2025 Enrolled students In this lecture, Zach will dive into prompt engineering, a concept that emerged with ChatGPT in November 2022. He will explore how prompt engineering is reshaping the tech landscape, effective prompting techniques, and the various tasks suited for different LLMs like Claude and ChatGPT. Zach will also address whether prompt engineering is a waste of time, emphasizing the importance of understanding algorithms and data structures even in this new AI-driven world. ### RAG and LLMs Day1 Lab URL: https://www.dataexpert.io/lesson/rag-and-llms-day-1-lab-jan2025-p736-l827 Module: Bonus - LLMs · Enrolled students In this lab, Zach walks you through the process of building your own RAG model, including how to set up an OpenAI API key. He explains the importance of how your vector database matches data and provide a practical example using the RAG Example Index. He also discusses how to filter queries effectively and the significance of context in your database. ### TA Office hours 5 URL: https://www.dataexpert.io/lesson/taofficehours5-mar26-p737-l2133 Enrolled students This office hours session provides an interactive platform for students to ask questions and troubleshoot issues related to bot creation and Docker container management. Participants discuss the setup of a Telegram bot, connectivity challenges with Open Claw, and the importance of API tokens. This session is beneficial for learners seeking hands-on assistance with their projects and understanding the integration of various tools in data engineering workflows. ### Apache Iceberg Lab Day2 URL: https://www.dataexpert.io/lesson/apacheiceberglabday2-031e0 Enrolled students Explore the intricacies of Apache Iceberg in this lab session focused on data compaction techniques. Participants will learn how to create and manipulate tables, apply run-length encoding for data compression, and execute Glue jobs for data rewriting. The session also covers automatic compaction strategies and the integration of Iceberg with Snowflake, providing valuable insights for data engineers looking to optimize data storage and retrieval processes. ### KPIs and Experimentation: Decoding Business Success: Metrics, Growth Strategies and Collaborative Approaches Day 1 Lecture URL: https://www.dataexpert.io/lesson/kpis-and-experimentation-day-1-lecture-yt-p736-l694 Enrolled students In this insightful video, we unravel the critical landscape of business success. The first part delves into the core of metrics, elucidating their correlation with outcomes and distinguishing leading from lagging metrics, offering a nuanced understanding of their role in driving success. ### Mastering Change Data Capture (CDC) URL: https://www.dataexpert.io/lesson/masteringchangedatacapturecdc-ba0fb Enrolled students ### Day 2 Lecture: Optimizing Dev Workflows with Cursor and Windsurf URL: https://www.dataexpert.io/lesson/prompt-engineering-day2-lecture-jul2025 Enrolled students In this lecture, Zach dives into how AI is transforming workflow productivity and the importance of adapting skills to stay relevant. He discusses the different categories of work and how AI will impact tasks, particularly highlighting that routine tasks done solo are at risk, while collaborative and creative tasks will thrive. Zach shares insights on the best practices for using AI in coding and the various tools available, emphasizing the need for engineers to become more technical and socially adept. ### Tech Talk - Joe URL: https://www.dataexpert.io/lesson/tech-talk-joe-may2025 Enrolled students Joe discusses the evolving landscape of data engineering, emphasizing the importance of data modeling and the integration of AI and machine learning. He highlights the shift left phenomenon, where data engineers are increasingly adopting software engineering practices, and vice versa. This session benefits aspiring data engineers and industry professionals looking to understand the future of data modeling, the significance of multimodal data, and the convergence of various engineering disciplines in a rapidly changing environment. ### TA Office Hour 1 URL: https://www.dataexpert.io/lesson/ta-office-hour-1-may2025 Enrolled students A live Q&A session focused on the first assignment for a data engineering boot camp. Participants discussed building an end-to-end data pipeline using Apache Airflow and Iceberg, including setting up DAGs, managing environment variables, and implementing data quality checks. Key topics included troubleshooting code issues, understanding branching in data workflows, and strategies for creating cumulative tables. This session is beneficial for learners looking to clarify assignment requirements and enhance their practical skills in data engineering. ### TA Office Hour 1 (2025-07-15) URL: https://www.dataexpert.io/lesson/ta-office-hour1-jul2025-p737-l1289 Enrolled students The office hour session provides guidance on the first assignment of the boot camp, focusing on prompt engineering and the development of a simple FastAPI app. Participants learn how to manually craft prompts to identify countries with repeated characters in their names and how to create an API endpoint that integrates with ChatGPT. This session is beneficial for students looking to clarify assignment requirements and improve their understanding of prompt optimization and API integration techniques. ### 7 URL: https://www.dataexpert.io/lesson/7-556e9 Enrolled students ### Fact Data Modeling: Minimizing Shuffle and Reducing Facts Day 3 Lecture URL: https://www.dataexpert.io/lesson/fact-data-modeling-day-3-lecture-yt-p736-l668 Enrolled students The lecture emphasizes the significance of minimizing shuffle when dealing with fact data. It explains how reducing shuffle can enhance the efficiency of data analysis and pipelines. ### Data Impact Communication & Visualization - Hands-On with SuperSet and CSV Files (Day 1 Lab) URL: https://www.dataexpert.io/lesson/visual-impact-day-1-lab-v3 Module: Week 6: Data Impact Communication & Visualization with Tableau · Enrolled students Join this hands-on session, where you will dive into the practical realm of data visualization using SuperSet or alternative visualization tools. Through step-by-step guidance, you will explore the intricacies of working with CSV files, mastering the creation of sets to elevate your visualizations. Delve into discussions on diverse chart patterns and discover strategies to enhance their impact. This lab promises valuable techniques for mastering the art of effective data visualization. Tune in and elevate your visualization skills! [Recorded on Dec 18th 2023] ### Thursday Quiz URL: https://www.dataexpert.io/lesson/thursdayquiz-a6cdb Enrolled students ### Special Guest, Vin Vashishta [Nov 17, 2023] URL: https://www.dataexpert.io/lesson/v3-1-special-guest-vin-vashishta-nov-17-2023-9209 Enrolled students In this video, we have a special guest, Vin Vashishta, who will be sharing insights on fundamental topics in data engineering. Vin discusses the importance of understanding data generating processes and systems, the future of communication in the industry, and the impact of smaller companies in the market. He also provides valuable advice on career growth, effective communication, and being forward-thinking. Dont miss out on this informative session! [Recorded Nov 17, 2023] ### Q&A with Zach Week 3 URL: https://www.dataexpert.io/lesson/q-a-with-zach-week-3-may2025-p736-l1154 Module: Q&A with Zach · Enrolled students A live Q&A session with Zach focused on various topics related to data engineering, including the use of Databricks, Spark, and Iceberg. Participants asked questions about capstone projects, technology choices, and best practices for SQL and Spark performance. Zach provided insights into effective querying techniques, the importance of using Delta Lake for data management, and strategies for handling workload distribution in Spark jobs. This session is beneficial for data engineering students seeking practical advice and clarification on their projects and technology stack. ### Advanced snowflake: Deploying the Capstone dag Lecture URL: https://www.dataexpert.io/lesson/advanced-snowflake-deploying-the-capstone-dag-lecture Module: Advanced Snowflake · Enrolled students In this lecture, Zach delves into the key aspects of creating impactful capstone projects to maximize your bootcamp experience. He discusses the importance of longevity, impact, quality, and proof in project development. I share insights on project complexity, data pipeline management, and showcasing your work effectively. ### Azure - Week 4 (Copy) URL: https://www.dataexpert.io/lesson/azure-week-4-4081 Enrolled students Understanding Azure Data Factory (ADF) is crucial for anyone involved in data engineering within the Azure ecosystem. This session covers the foundational concepts of ADF, including its components like link services, datasets, and activities, which are essential for creating data pipelines. Participants will learn how to move data from various sources to destinations, utilizing ADF's low-code interface, making it accessible for users without extensive coding knowledge. ### Managing Unstructured Data Day2 Lab URL: https://www.dataexpert.io/lesson/managingunstructureddataday2lab-feb26-p736-l2087 Module: Week 5: Managing Unstructured Data · Enrolled students In this video, Eumar explores document parsing and chunking strategies using AI tools. Hw successfully upload a PDF syllabus, parse it, and save the content into a database, while also discussing advanced web crawling techniques. He encourages everyone to review the 10 chunking strategies he outlined in Lab 2, as understanding these will be crucial for their upcoming projects. ### Day 1 Lab: Setting up CI/CD for an AI application URL: https://www.dataexpert.io/lesson/ml-ops-day1-lab-july25 Enrolled students Setting up CI/CD for an AI application involves integrating various guardrails to ensure safe and effective model performance. This lesson covers key concepts such as toxicity detection, PII injection prevention, and content filtering using tools like OpenAI's moderation API and regex. Participants will learn how to implement these techniques in their applications, making it ideal for AI engineers and developers looking to enhance their application's reliability and safety. ### Wednesday Quiz URL: https://www.dataexpert.io/lesson/wednesdayquiz-95a12 Enrolled students ### Snowflake Basics Day 1 Lecture URL: https://www.dataexpert.io/lesson/snowflake-basics-day-1-lecture-9688 Module: Week 3: Data Modeling on Snowflake · Enrolled students In this lecture, Bruno introduces Snowflake and DBT, focusing on basic concepts like time travel, fail safe, and data types. He also discusses the differences between Snowflake and Databricks, emphasizing the ease of scaling in Snowflake. ### TA Office Hour URL: https://www.dataexpert.io/lesson/taofficehour-oct2025-p737-l1572 Enrolled students This office hour session provides an interactive platform for students to ask questions and troubleshoot issues related to their boot camp setup. Participants discuss various topics, including connecting to PubNub, managing Python versions, and configuring GitHub repositories. The TAs assist students with specific technical challenges, ensuring they are equipped to proceed with their coursework effectively. This session is ideal for anyone seeking personalized support in their data engineering journey. ### Apache Iceberg Day 2 Lecture URL: https://www.dataexpert.io/lesson/apache-iceberg-day-2-lecture-may2025 Enrolled students In this lecture, we explore the evolution of big data processing technologies, starting with Java MapReduce, the impact of Hive, and the rise of Spark. Zach shares experiences from adopting Scala and Spark at Airbnb, discuss the power of Iceberg in data engineering, and how object storage technologies like S3 have changed the game. We also look at the benefits of Iceberg, its role in data restoration, and why partitioning strategies are so important for data engineers. ### TA Office Hour 9 URL: https://www.dataexpert.io/lesson/taofficehour9-4f028 Enrolled students ### TA Office Hour 4 URL: https://www.dataexpert.io/lesson/ta-office-hour-4-jan2025 Enrolled students Office hours provide an interactive space for students to discuss their weekly assignment focused on building a data pipeline using Airflow. Participants will learn to create a daily partitioning table, load data from the Polygon API into a staging table, and perform data quality checks before transitioning to production. This session also covers various approaches using tools like Trino and PyIceberg, ensuring students can adapt their solutions based on their preferences and requirements. ### Book Club 3 URL: https://www.dataexpert.io/lesson/bookclub3-feb26 Enrolled students The Book Club session focuses on Chapter 12, discussing the future of data systems, particularly the tension between real-time and batch processing. Participants share experiences related to building action-oriented architectures, emphasizing the importance of understanding business needs versus technical capabilities. ### Prompt Engineering Day 1 Lecture URL: https://www.dataexpert.io/lesson/prompt-engineering-day1-lecture-oct2025-p737-l1506 Module: Week 1: Prompt Engineering & Basic RAG · Enrolled students In this video, Zach discusses about prompt engineering, highlighting its evolution from being overhyped to recognizing its true value alongside retrieval augmented generation. He emphasizes that mastering these two concepts is crucial for success as an AI engineer, even if participants forget everything else. He covers the differences between structured and unstructured tasks, the importance of context, and how to effectively prompt large language models (LLMs). ### Vaishali Macwan - Senior Data Scientist URL: https://www.dataexpert.io/lesson/tech-talk-with-vaishali-macwan-p735-l595 Module: Tech Talks · Enrolled students This is the tech talk with Vaishali Macwan. ### Streaming Pipelines: Kafka, Postgres, Spark Integrations and Parallelism Day 2 Lab URL: https://www.dataexpert.io/lesson/streaming-pipelines-day-2-lab-yt-p736-l677 Enrolled students In this lab, we delve into data processing capabilities of Flink. We will explore components like Kafka, Postgres, and Spark, witnessing how they collaborate in efficient data pipelines. The concept of parallelism and its impact on Flink jobs will be demystified. ### Office Hour 2 URL: https://www.dataexpert.io/lesson/office-hour-2 Module: TA Office Hours · Enrolled students Office Hour 2 provides an interactive session focused on addressing homework-related queries and clarifying concepts in data engineering. Participants discuss SQL techniques, including handling duplicates, using window functions, and constructing slowly changing dimensions (SCD) for actor data. The session also covers practical insights on using Airflow for data pipelines, including task dependencies and partitioning strategies. This is beneficial for learners seeking to enhance their understanding of SQL and data engineering workflows. ### Apache Iceberg Lecture Day 1 URL: https://www.dataexpert.io/lesson/apache-iceberg-lecture-day-1-apr2025-1 Enrolled students In this lecture, Zach dives into the intricacies of Iceberg, a technology that emerged from Netflix, and how it revolutionizes data processing. He shares his experiences working with large datasets, particularly at Airbnb, and discuss the challenges we faced with traditional tools like Hive and Spark. He also highlights the importance of time travel in data management and how Iceberg addresses some of the limitations of previous technologies. ### Q&A with Zach 4 URL: https://www.dataexpert.io/lesson/qawithzach4-mar26 Enrolled students Week 4's Q&A session features an engaging discussion led by Zach, focusing on key skills for data engineers and AI engineers. Participants gain insights into essential tools like Kafka, Spark, and Airflow, as well as the importance of storytelling in interviews. The session also covers feedback mechanisms for instructors and highlights the significance of community engagement in the learning process. This is a valuable opportunity for those looking to enhance their technical skills and job readiness in the data and AI fields. ### TA Office hours 2 URL: https://www.dataexpert.io/lesson/taofficehours2-mar26-p737-l2130 Enrolled students Office hours provide an interactive platform for participants to discuss technical challenges and clarify concepts related to AI and data engineering. Key topics include troubleshooting API connection issues, understanding cloud environments, and managing session data in applications. Participants benefit from real-time support as they navigate their assignments and projects, gaining insights into best practices for coding and data management. ### TA Office Hour 1 URL: https://www.dataexpert.io/lesson/ta-office-hour-1-jan2025 Enrolled students Office Hour 1 provides a platform for students to ask questions and seek clarification on course material, particularly around homework and project expectations. Key topics discussed include data modeling, the use of .env files for environment variables, and the importance of group collaboration for project success. Students also share insights on navigating homework challenges and the tools needed for their capstone projects, fostering a supportive learning environment. ### Build your first pipeline with dbt URL: https://www.dataexpert.io/lesson/buildyourfirstpipelinewithdbt-jan2026-p735-l1913 Module: Week 1: dbt Basics · Enrolled students Learn to build your first data transformation pipeline using dbt in this foundational lesson. You'll explore key concepts such as models, sources, and seeds, while gaining insights into the role of analytics engineers. This lesson is ideal for beginners and those looking to enhance their data engineering skills, leveraging dbt's capabilities to create clean, reliable datasets for analysis. ### Fact Data Modeling - Core Concepts, Deduplication Techniques, and Retention Considerations with Iceberg (Day 1 Lecture) - Pre-recorded from V3 URL: https://www.dataexpert.io/lesson/fact-data-modeling-day-1-lecture-v4 Module: Week 2: Fact Data Modeling in Iceberg · Enrolled students This is a pre-recorded lesson from V3. In this lecture, we delve into the realm of fact data modeling, focusing on the core concept of facts and addressing the challenges associated with handling duplicates. Our exploration underscores the pivotal role of facts in data engineering, shedding light on their significance. Practical insights into deduplication techniques are provided, and we discuss the ramifications of retention on fact data. Join us in this video to gain a comprehensive understanding of fact data modeling and its crucial implications in the field of data engineering.[Recorded Nov 14, 2023] ### Capstone Showcase with Jesse Charbneau URL: https://www.dataexpert.io/lesson/capstone-showcase-with-jesse-charbneau Enrolled students The capstone showcase presents "Rosie Economics," a project focused on visualizing economic data with impact analysis overlays. Designed for casual users and DIY investors, this tool simplifies complex economic indicators and integrates AI-generated insights using OpenAI's REST interface. The lesson covers the technology stack, including React, FastAPI, and DBT, and demonstrates how to analyze economic trends through interactive visualizations, making it accessible for those without a deep financial background. ### Apache Spark Core Day 3 Lecture URL: https://www.dataexpert.io/lesson/apache-spark-core-day-3-lecture-jan2025-p736-l779 Module: Week 1: Databricks Basics · Enrolled students In this lecture, Yared dives into the Query Execution Engine and explore the different types of transformations in Spark SQL, specifically narrow and wide transformations. He provides examples to illustrate these concepts and discuss the importance of schema inference for performance. He also touches on how to configure clusters for optimal data processing. ### Special Guest, Ben Rogajan [Dec 1, 2023] URL: https://www.dataexpert.io/lesson/v3-2-special-guest-ben-rogajan-dec-1-2023-9712 Enrolled students Join us for this installment of the Spotlight Series with Ben Rogajan, as we explore the landscape of data engineering and consulting. He presents insights and experiences in data engineering and consulting. The discussion covers industry challenges and opportunities, emphasizing the significance of building a strong personal brand. Practical tips on effective outreach and recommended resources for continuous learning are provided. Watch the video to gain insights and elevate your career in these dynamic fields.[Recorded Dec 1st, 2023] ### MLOPs and Guardrails Day 1 Lab URL: https://www.dataexpert.io/lesson/ml-ops-day1-lab-oct25-p737-l1523 Module: Week 4: MLOps, Guardrails & Business Value · Enrolled students In this video, Zach walks through the setup of guardrails for his MLOPS project, focusing on the database agent. He explores how to manage toxic inputs and ensure that the AI remains respectful during user interactions. Zach demonstrates the middleware functionalities, including toxicity detection and on-topic checks, while emphasizing the importance of keeping the agent aligned with relevant data tables. He also highlights the need for the middleware to adapt dynamically as new tables are created in the database. ### Apache Iceberg Lecture Day 3 URL: https://www.dataexpert.io/lesson/apache-iceberg-lecture-day-3-april2025 Enrolled students In this lecture, Zach dives into the concept of data contracts and their importance in analytics engineering. He discusses how data contracts help define the relationship between data producers and consumers, ensuring clarity on data usage and expectations. He also highlights the significance of quality checks and the role of tools like Iceberg in managing data effectively. ### Building Rich Context for AI Agents Day1 Lab URL: https://www.dataexpert.io/lesson/buildingrichcontextforaiagentsday1lab-aug26 Enrolled students In this lab, Eumar explains how to build agent-ready foundations using Unity Catalog, Genie, and SQL-based unstructured document processing. He demonstrates Genie as a text to SQL capability that generates queries and charts, and explains how agents are exposed via MCP endpoints while access is governed by Unity Catalog. He reviews Unity Catalog governance with tables, volumes, and models, including permissions, lineage, and converting Delta tables into AI search vector indexes. He creates a user-specific schema and three volumes (receipts, bootcamp docs, code) via the Databricks SDK, then generate synthetic receipts using parallel Pandas UDFs, and finally parse images and extract structured fields using AIParseDocuments and AIExtract to populate raw and parsed receipt tables. ### TA Office Hour 3 (2025-10-09) URL: https://www.dataexpert.io/lesson/ta-office-hour3-oct2025-p737-l1538 Enrolled students ### Q&A with Zach Week 1 (2025-10-09) URL: https://www.dataexpert.io/lesson/qa-with-zach-week1-oct2025-p737-l1544 Module: Q&A with Zach · Enrolled students A live Q&A session with Zach covers essential topics for setting up a local server and application environment for the boot camp. Participants learn how to clone repositories, configure environment variables, and address common setup issues. Zach also discusses the legality of web scraping, particularly concerning LinkedIn, and emphasizes best practices for using APIs. This session is beneficial for students needing guidance on technical setup and understanding data engineering concepts. ### Managing Unstructured Data - Day 1 Lecture URL: https://www.dataexpert.io/lesson/managing-unstructured-data-day-1-lecture-may2025-p736-l1193 Module: Week 5: Managing Unstructured Data · Enrolled students In the video Zach focuses on unstructured data, particularly in the context of Retrieval Augmented Generation (RAG) and Large Language Models (LLMs). Zach explores how LLMs are impacting data engineering and vice versa, as well as discuss practical ways data engineers can leverage unstructured data. ### Office Hour 4 URL: https://www.dataexpert.io/lesson/office-hour-4 Module: TA Office Hours · Enrolled students Office Hour 4 focuses on discussing homework assignments related to querying NBA player data. Participants learn how to track state changes of players across seasons using the bootcamp NBA players table, as well as alternative approaches using the NBA player seasons table for better historical insights. The session also covers aggregation techniques with grouping sets to analyze player performance metrics, including total points scored by players for specific teams and seasons. ### test 1 URL: https://www.dataexpert.io/lesson/test1-d0517 Enrolled students ### Snowflake Lecture URL: https://www.dataexpert.io/lesson/snowflake-lecture-jan2025 Enrolled students In this lecture, Bruno dives into the concept of Change Data Capture (CDC) and how it can be effectively implemented in Snowflake. He explains the importance of logging updates, deletes, and inserts, and how CDC can help manage high-volume data without slowing down the pipeline. He also touches on the use of streams and merging data to avoid duplication. ### Advanced snowflake: Deploying the Capstone dag Lab URL: https://www.dataexpert.io/lesson/advanced-snowflake-deploying-the-capstone-dag-lab Module: Advanced Snowflake · Enrolled students In this lab, Zach covers key aspects of our Data Engineering Bootcamp. He discusses setting up Airflow DAGs, managing secrets with AWS Secrets Manager, and working with Glue Spark. He also highlights the importance of not storing secrets in code and provide guidance on project deadlines. ### Chat with Zach [Aug 10, 2023] URL: https://www.dataexpert.io/lesson/v2-4-chat-with-zach-aug-10-2023-413 Enrolled students Delve into the world of data engineering with our weekly Q&A / Fireside Chat series, featuring the founder of EcZachly Inc and lead instructor of the Advanced Data Engineering Bootcamp. Each session covers a different topic, providing invaluable insights and real-world experiences to empower data enthusiasts in their journey. (Recorded Aug 10, 2023) ### TA Office hour 7 URL: https://www.dataexpert.io/lesson/taofficehour7-jan2026 Enrolled students Office hours provide a platform for students to ask questions and clarify doubts related to their capstone projects in the analytics engineering boot camp. Key topics discussed include submission deadlines, handling API keys, and deploying projects using branches in version control systems. Students benefit from real-time guidance on best practices for managing environment variables and troubleshooting local development issues. ### Prompt Engineering Day1 Lab URL: https://www.dataexpert.io/lesson/prompt-engineering-day1-lab-oct2025-p737-l1507 Module: Week 1: Prompt Engineering & Basic RAG · Enrolled students In this video, Zach dives into prompt engineering exercises, focusing on the system prompt and how to enhance chat and resume parsing applications. He emphasizes the importance of setting up environment variables, particularly the OpenAI key, and demonstrates how to interact with the FastAPI backend. He also highlights the significance of the system prompt in shaping AI responses and shares insights on managing response times for API calls. ### Friday Quiz URL: https://www.dataexpert.io/lesson/fridayquiz-7e3bd Enrolled students ### Career Dev Session 4 (2025-10-09) URL: https://www.dataexpert.io/lesson/career-dev-session4-oct2025-p737-l1531 Module: Career Development · Enrolled students ### lesson-poetry URL: https://www.dataexpert.io/lesson/1bad2cac-eab5-4ad4-9bff-1fa5360b2828lessonpoetry Enrolled students ### Applying Analytical Patterns: Exploring SQL, Scaling Projects and Aggregation Analysis Day 1 Lecture URL: https://www.dataexpert.io/lesson/applying-analytical-patterns-day-1-lecture-yt-p736-l670 Enrolled students In this video lecture, we explore fundamental concepts in data engineering, with a primary focus on the pivotal role of SQL as the foundational element in this field. The discussion extends to the patterns and structures essential for scaling data engineering projects, providing a comprehensive understanding of their implementation. ### Spark Batch Processing - User-Defined Functions (UDFs) and Broadcast Join (Day 2 Lab) URL: https://www.dataexpert.io/lesson/spark-batch-day-2-lab-v3 Module: Week 4: Batch Pipelines with Apache Spark · Enrolled students In this Spark lab, Zach explains the concept of User-Defined Functions (UDFs) and their application within the Spark execution environment. Through a practical code example, he guides viewers on schema updates and emphasizes the significance of caching. The tutorial extends to showcasing the power of broadcast join, featuring a query with partitioning. Zach concludes by highlighting the advantages of leveraging PySpark UDFs. [Recorded on Dec 7, 2023] ### Advanced JavaScript Concepts URL: https://www.dataexpert.io/lesson/advanced-javascript-concepts-dcccb Module: Web Dev Fundamentals · Enrolled students Advanced JavaScript concepts are essential for web developers looking to deepen their understanding of the language. This lesson covers topics such as block-scoped variables with let and const, function hoisting, template literals for string interpolation, destructuring assignments for cleaner code, and the spread/rest operator for merging arrays and objects. Additionally, it introduces classes and asynchronous programming with promises, providing practical insights into modern JavaScript syntax and best practices. ### Youtube URL: https://www.dataexpert.io/lesson/youtube-1167 Enrolled students testing updating ### Structured Streaming Kafka to Delta Live Table Day1 Lab URL: https://www.dataexpert.io/lesson/structuredstreamingkafkatodeltalivetableday1lab-feb26 Enrolled students In this video, Zach walks you through setting up the Spark Streaming Lab, focusing on creating and managing secrets in Databricks. He also demonstrates how to run a simple Spark streaming job that processes data from Kafka, highlighting the importance of triggers and checkpoints. ### TA Office Hour 2 (2025-06-05) URL: https://www.dataexpert.io/lesson/ta-office-hour-2-may2025-p736-l1098 Enrolled students Office hours provide a platform for students to clarify doubts and discuss technical challenges related to data engineering tasks. Key topics covered include the creation of main and audit branches in table management, the use of Trino for table creation, and the implementation of data quality checks. Students also engage in discussions about API usage for data retrieval and the requirements for their capstone project proposals, ensuring they understand the necessary components for successful project completion. ### Tech Talk - Joe URL: https://www.dataexpert.io/lesson/techtalksweek1-jan2026 Enrolled students A tech talk featuring Joe Reese, author of "Fundamental Data Engineering," explores the evolving landscape of data engineering and AI. Joe discusses the importance of foundational skills in data orchestration, data quality, and the integration of AI into data pipelines. He emphasizes the need for engineers to understand both traditional practices and emerging technologies, highlighting the shift towards action-oriented architectures that enable real-time decision-making. This session benefits aspiring data engineers and those interested in the intersection of data engineering and AI. ### Enhancements and Implementations in MLflow URL: https://www.dataexpert.io/lesson/enhancementsandimplementationsinmlflow-feb26-p736-l2163 Module: Week 6: Building AI Agents with Databricks · Enrolled students In this video, Eumar provides an update on the GitHub repository enhancements for the Day 2 labs in the agentic module, focusing on the latest MLflow 3 implementation. He demonstrates how to leverage this new version for improved model tracing, observability, and deployment, including A-B testing capabilities. He encourages everyone to run the updated labs, as they now include features like human and LLM evaluations, synthetic dataset generation, and a streamlined user experience for feedback. He reminds everyone to check the changes in the catalogs and engage with the new evaluation sessions, emphasizing that their feedback is crucial as the team refines the models and processes. ### TA Office hour 1 URL: https://www.dataexpert.io/lesson/taofficehour1-feb26 Enrolled students This office hour session focuses on addressing student concerns regarding the alignment of lecture content, homework, and lab exercises in a data engineering boot camp. Participants discuss the importance of having a foundational understanding of Python and SQL to successfully complete assignments. The session also emphasizes the availability of teaching assistants for support and encourages students to engage with the course materials and ask questions for clarification. ### TA Office Hour 7 URL: https://www.dataexpert.io/lesson/ta-office-hour7-oct2025 Enrolled students ### QnA with Zach - Jun7th 2026 - Lab URL: https://www.dataexpert.io/lesson/weeklyqnawithzachjun7th2026lab Enrolled students A live Q&A session with Zach focused on AI engineering and project development. Participants discussed various topics, including the use of AI tools like Atle and cloud services, strategies for building personal projects, and the importance of understanding metadata in AI applications. Zach emphasized the significance of practical experience and encouraged attendees to actively engage in building projects to demonstrate their skills in AI-focused roles. ### Tech Talk 2 - Madison URL: https://www.dataexpert.io/lesson/tech-talk-2-madison-april2025 Enrolled students A live Q&A session featuring Madison, a senior analytics engineer at KIT and author of the Learn Analytics Engineering newsletter. Madison discusses the nuances of analytics engineering, including its distinction from data engineering and data analysis, stakeholder interactions, and project management challenges. She shares insights on effective communication with stakeholders, the importance of setting boundaries, and the role of documentation in analytics projects. ### Delta Table Day 2 Lab URL: https://www.dataexpert.io/lesson/delta-table-day-2-lab-may2025-p736-l1131 Module: Week 3: Data Lakes with Delta Table · Enrolled students In this video, Yared walks viewers through various data transformation techniques using SQL, focusing on data profiling, cleaning, and complex transformations. He highlights the importance of understanding data distribution and how to handle JSON data effectively. He also discusses the creation of silver tables and the significance of defining schemas for efficient data processing. ### Li Yin - Founder at Adal (2.0) URL: https://www.dataexpert.io/lesson/tech-talk5-oct2025-p737-l1553 Module: Tech Talks · Enrolled students ### Introduction to Spark Day 2 Lab URL: https://www.dataexpert.io/lesson/introduction-to-spark-day-2-lab-jan2025 Enrolled students In this lab, Yared walks you through the process of running Spark jobs in Databricks, focusing on the differences between shared and personal clusters. He highlights the importance of understanding job IDs, stages, and tasks when executing jobs. He also requests everyone to switch to their personal clusters for better visibility of job details. Please make sure to destroy your cluster after use to avoid unnecessary costs. ### Data Visualization with tableau (Day 2 Lecture) URL: https://www.dataexpert.io/lesson/visual-impact-day-2-lecture-v4 Module: Week 6: Data Impact Communication & Visualization with Tableau · Enrolled students This is a pre-recorded class. In this insightful lecture, we delve into the crucial realm of dashboard metrics. Exploring the importance of various metric types, including total aggregates, time and entity-based aggregates, dimensional mix, and retention and survivorship numbers, the video provides a comprehensive understanding. The discussion extends to common pitfalls to steer clear of when crafting dashboards. This video serves as a valuable guide to assist you in selecting the most suitable metrics for your dashboard, ultimately enhancing its overall performance. [Recorded on Dec 20, 2023] ### Day 2: Dynamic arrays & amortized append URL: https://www.dataexpert.io/lesson/dynamic-arrays-amortized-append-p913-d2 Enrolled students Learn how dynamic arrays grow, why appends are amortized O(1), and why middle inserts cost O(n). ### Saturday SQL Quiz URL: https://www.dataexpert.io/lesson/saturdayquiz-289d8 Enrolled students ### Lesson 1 URL: https://www.dataexpert.io/lesson/lesson1-e7368 Enrolled students Explore the fundamentals of Apache Airflow in this lesson focused on backfilling practices and DAG (Directed Acyclic Graph) creation. Participants will learn how to set up and run DAGs using PySpark, manage AWS Glue jobs, and handle backfills effectively. This lesson is ideal for data engineers and analysts looking to enhance their workflow automation skills and gain practical experience with Airflow and Spark in a cloud environment. ### Data Expert.io Architecture URL: https://www.dataexpert.io/lesson/dataexpert-architecture-jan2026-p735-l1958 Module: Kickoff · Enrolled students ### Fact Data Modeling - Core Elements in Data Modeling with Iceberg (Day 2 Lecture) - Pre-recorded from V3 URL: https://www.dataexpert.io/lesson/fact-data-modeling-day-2-lecture-v4 Module: Week 2: Fact Data Modeling in Iceberg · Enrolled students This is a pre-recorded lesson from V3. In this lecture, we cover the fundamental concepts of fact and dimension data structures. The lecture delves into the significance of fact data, highlighting its role as the core element in data modeling. Additionally, we explore dimension data and its crucial function in grouping and categorizing information. The discussion emphasizes the importance of capturing changes within dimensions and introduces the benefits of employing slowly changing dimensions. The lecture provides examples and insights, offering viewers a comprehensive understanding of fact and dimension data structures in the context of data engineering.[Recorded Nov 15, 2023] ### Databricks and Advanced Spark Day1 Lab URL: https://www.dataexpert.io/lesson/databricksandadvancedsparkday1lab-feb26 Enrolled students In this lab, Zach walks you through how to work with the Massive API, to extract stock prices for various companies like Apple, Netflix, and MSFT. Zach also highlights the importance of understanding Spark's explain plans to optimize their queries and avoid performance issues, particularly regarding shuffle operations. ### Grouping sets and Window functions Lab URL: https://www.dataexpert.io/lesson/grouping-sets-and-window-functions-lab Module: Applying Analytical Patterns · Enrolled students In this lab session, Zach demonstrates various queries and aggregations that can be applied to the bootcamp web events table. The talk emphasizes the significance of reducing cardinality and showcases the use of functions like coalesce and rollup. Zach also touches on the concept of self-joins and how they can aid in analyzing a users journey on the website. ### Day 2 Lab: Integrating RAG into your full stack app URL: https://www.dataexpert.io/lesson/understanding-rag-lab-day2-july2025 Enrolled students In this lab, Zach walks through deploying a RAG project on Vercel, highlighting the importance of setting environment variables like the OpenAI API key and Milvus credentials. He discusses the differences between using PyMilvus and querying Milvus via REST, which allows for a lighter serverless deployment. He also covers how to implement re-ranking in our RAG model for better search results and the significance of testing our system through CI/CD practices. ### Analytics Data Quality - Strategies and Insights from Zach's Airbnb Experience (Day 1 Lecture) URL: https://www.dataexpert.io/lesson/analytics-data-quality-day-1-lecture-v3 Module: Week 5: DBT and Pipeline Spec Building · Enrolled students In this lecture, Zach explores the crucial theme of data quality and its significance. Covering diverse aspects such as misunderstood and incomplete data definitions, he underscores the importance of timely data and effective communication with stakeholders. Throughout the session, Zach draws on his experience at Airbnb, sharing valuable insights and practical tips to ensure top-notch data quality. [Recorded on Dec 11, 2023] ### TA Office Hour 7 URL: https://www.dataexpert.io/lesson/taofficehour7-20fc1 Enrolled students ### TA Office hour 8 URL: https://www.dataexpert.io/lesson/taofficehour8-feb26-p736-l2054 Enrolled students This office hours session addresses common issues faced by students in the boot camp, particularly related to permissions for creating tables and inserting data in Spark. Participants discuss troubleshooting steps, including checking folder paths and permissions, as well as clarifying assignment expectations and capstone project proposals. This session is beneficial for students needing support with practical coding challenges and project guidance. ### dbt basics Day1 Lab URL: https://www.dataexpert.io/lesson/dbtbasicsday1lab-0deb6 Enrolled students ### Day 4: Dummy/sentinel nodes URL: https://www.dataexpert.io/lesson/dummy-sentinel-nodes-p913-d4 Enrolled students Simplify linked list insert/delete by using dummy (sentinel) head/tail nodes, with clear invariants, code, and trade-offs. ### Shachar Meir URL: https://www.dataexpert.io/lesson/shachar-meir-jan2025 Enrolled students Career development in data engineering is the focus of this session, where Shachar Meir shares insights from his extensive experience. He emphasizes the importance of understanding business outcomes over mere technical tasks, fostering resilience in the face of challenges, and the value of soft skills for career advancement. Attendees will benefit from practical advice on navigating their career paths, building support systems, and making informed decisions about job changes, all aimed at enhancing their professional growth in the data field. ### TA Office Hour 4 URL: https://www.dataexpert.io/lesson/ta-office-hour-4-april2025-p735-l999 Enrolled students This office hour session provides an interactive platform for students to discuss their assignments and clarify doubts related to SQL queries and Python UDFs. Participants engage in conversations about homework expectations, including how to handle missing years for actors using SQL and Python, and best practices for commenting code. The session also addresses technical issues with assignments and project proposals, fostering a collaborative learning environment for data engineering concepts. ### advanced dbt Day 1 Lecture URL: https://www.dataexpert.io/lesson/advanceddbtday1lecture-jan2026 Enrolled students Advanced Jinja, macros, and incremental models are key topics in this lesson, designed for data engineers looking to enhance their DBT skills. Participants will learn how to leverage Jinja for writing dynamic SQL, utilize macros for reusable code snippets, and implement incremental models to optimize data processing. The lesson covers practical examples and strategies, including append, merge, and delete-plus-insert methods, making it ideal for those seeking to improve efficiency in data transformation workflows. ### Q&A with Zach Week5 URL: https://www.dataexpert.io/lesson/qawithzachweek5-feb26 Enrolled students This session features a live Q&A with Zach, focusing on the upcoming AI boot camp and the introduction of new instructors, including Carly Taylor and Kevin Notten. Participants will learn about setting up OpenClaw agents, prompt engineering, and the integration of various AI tools and techniques. The discussion also touches on the importance of data engineering skills in the evolving job market and the challenges of the capstone project timeline. ### Capstone Requirements URL: https://www.dataexpert.io/lesson/capstone-requirements Module: Bootcamp Orientation · Enrolled students The capstone project requirements are outlined, focusing on essential criteria for a successful submission. Participants will learn about the importance of documentation, including specifications, schemas, and data quality checks, while utilizing tools like Snowflake and Trino for data management. This lesson is beneficial for students looking to enhance their data engineering skills and create impactful capstone projects that meet industry standards. ### RAG optimisation + Reranking Day 2 Lecture URL: https://www.dataexpert.io/lesson/rag-optimisation-reranking-day2-lecture-oct-2025 Enrolled students In this lecture, Zach focuses on maximizing relevance in AI, exploring various types of RAG, and learning how to offload heavy AI tasks to worker tasks. He emphasizes the importance of using Markdown for data input and the significance of metadata filtering and re-ranking in improving retrieval accuracy. Zach also dives into hands-on lab work, including updating the chatbot and creating an asynchronous email-generating agent. ### Sai Bysani - Lead Data Engineer URL: https://www.dataexpert.io/lesson/techtalks3-mar26-p737-l2116 Module: Tech Talks · Enrolled students Insights into leveraging LinkedIn for career advancement are shared by Sai Bysani, a Lead Data Engineer with a background in data analytics. He emphasizes the importance of building a personal brand through consistent posting and networking, highlighting how visibility can lead to job opportunities and promotions. Additionally, Sai discusses the evolving role of data professionals in the age of AI, stressing the need for both technical and non-technical skills to thrive in the industry. ### Databricks Platform Overview Day 1 Lab URL: https://www.dataexpert.io/lesson/databricks-platform-overview-day-1-lab-jan2025 Enrolled students In this lab, we go through the databricks platform. ### Executive Presence for Technical Leaders URL: https://www.dataexpert.io/lesson/executivepresencefortechnicalleaders-4f30f Enrolled students What executive presence means for Staff+ ICs (it's different from management) Communication patterns that signal Staff+ thinking Presenting technical strategy to senior leadership Handling pushback and disagreement at higher levels Interactive: Communication Workshop—practice articulating impact in executive language ### Q&A with Zach Week3 URL: https://www.dataexpert.io/lesson/qawithzachweek3-jan2026 Enrolled students The Week 3 Q&A session covers a range of topics related to data engineering challenges and solutions. Participants discuss complex data modeling, the use of Trino for data processing, and the importance of understanding legacy code. The session also addresses practical issues like optimizing data pipelines, managing metadata with Iceberg, and strategies for effective data storage and retrieval. This interactive session benefits data engineers seeking to enhance their skills and tackle real-world problems in their projects. ### Local Video URL: https://www.dataexpert.io/lesson/local-video-1001 Enrolled students ### Tech talk with Gourav Khanijoe URL: https://www.dataexpert.io/lesson/tech-talk-with-gourav-khanijoe Module: Tech Talk · Enrolled students A tech talk featuring Gourav Khanijoe, a staff software engineer at HubSpot, offers insights into the journey from senior to staff engineer. Gourav discusses the importance of scaling oneself, effective delegation, and tackling critical organizational problems. He emphasizes building relationships with leadership to secure impactful projects and shares strategies for simplifying complex solutions. This session is valuable for engineers aspiring to advance their careers in tech and leadership roles. ### Office Hour 3 URL: https://www.dataexpert.io/lesson/office-hour-3 Module: TA Office Hours · Enrolled students This TA Office hours was hosted by Sundeep and Weeks 1 and 2 homework was discussed. ### Apache Iceberg Lab Day2 URL: https://www.dataexpert.io/lesson/apacheiceberglabday2-031e0-p735-l1994 Module: Week 5: Data Modeling on Iceberg · Enrolled students Explore the practical applications of Apache Iceberg in this lab focused on data modeling and compaction techniques. Participants will learn how to create and manage tables, perform data compaction using AWS Glue, and understand the implications of run-length encoding on data storage efficiency. This session is ideal for data engineers and analysts looking to enhance their skills in optimizing data storage and retrieval in Iceberg environments. ### Day 2 Lecture: How to link your AI agent to value URL: https://www.dataexpert.io/lesson/building-end-to-end-ai-application-day2-lecture-july25-p737-l1190 Module: Week 5: End-to-End AI Applications · Enrolled students In this lecture Zach dives into end-to-end AI applications, discusses what AI excels at, the roadblocks that hinder its value, and strategies to overcome those challenges. He shares that around 60–70% of AI projects fail, slightly better than the 80% failure rate often cited for data science projects. Zach explores key AI use cases—prediction, detection, and optimization—while emphasizing that not every problem requires large language models. ### Building AI Agents with Databricks Day2 Lecture URL: https://www.dataexpert.io/lesson/buildingaiagentswithdatabricksday2lecture-feb26-p736-l2035 Module: Week 6: Building AI Agents with Databricks · Enrolled students In this lecture, Eumar discusses the critical transition from demo to production for AI agents, emphasizing that 80% of GenAI projects fail to reach production due to issues like data quality and user trust. He explores essential technologies such as Databricks Model Serving, Guardrails, and MLflow Evaluation, which can help ensure successful deployment. He highlights the importance of observability and monitoring in maintaining agent performance post-launch. ### Cumulative DAGs in Production Day 3 Lecture URL: https://www.dataexpert.io/lesson/cumulative-dags-in-production-day-3-lecture-jan2025-p736-l760 Module: Week 6: Airflow & Pipeline Specs with Trino · Enrolled students Cumulative DAGs are a complex yet powerful concept in data engineering, particularly when using Apache Airflow. This lecture explores the debate between functional and efficient data engineering, focusing on cumulative table design and its implications for slowly changing dimensions (SCDs). Participants will learn about the trade-offs involved in maintaining simplicity versus efficiency in data pipelines, as well as techniques for effectively managing large datasets while minimizing duplication. ### Advanced dbt Lab Day 2 URL: https://www.dataexpert.io/lesson/advanced-dbt-lab-day-2-april2025 Enrolled students Advanced DBT Lab Day 2 focuses on implementing the right audit publish pattern in DBT, teaching participants how to manage data integrity through auditing critical models like fact orders. The session covers selection methods for running specific models, integrating DBT with Airflow using Bash and Cosmos operators, and optimizing the audit process for efficiency. Ideal for data engineers looking to enhance their DBT skills and streamline their data pipelines. ### Managing Your Promotion Strategically URL: https://www.dataexpert.io/lesson/managingyourpromotionstrategically-9f1ee Enrolled students "Managing your promotion timeline and key stakeholders Building visibility without self-promotion Finding and activating sponsors (not just mentors) Navigating calibration and promotion committees Hands-on: Stakeholder Mapping—identify your promotion sponsors and influencers" ### Databricks Platform Overview Day 1 Lecture URL: https://www.dataexpert.io/lesson/databricks-platform-overview-day-1-lecture-jan2025 Enrolled students In this lecture, Yared delves into the intricacies of data architecture, focusing on the concept of lake house architecture, Delta Lake, and Databricks Data Intelligence. He breaks down the evolution from data warehousing to the innovative lake house architecture, highlighting the benefits and applications in real-world scenarios. We dive into the world of structured and unstructured data, the role of data lakes, and the integration of advanced analytics for a comprehensive understanding of modern data management. ### RAG and LLMs Day1 Lab URL: https://www.dataexpert.io/lesson/rag-and-llms-day-1-lab-jan2025 Enrolled students In this lab, Zach walks you through the process of building your own RAG model, including how to set up an OpenAI API key. He explains the importance of how your vector database matches data and provide a practical example using the RAG Example Index. He also discusses how to filter queries effectively and the significance of context in your database. ### End-to-End AI Applications Day1 Lab URL: https://www.dataexpert.io/lesson/endtoendaiapplicationsday1lab-mar26 Enrolled students In this lab, Zach walks through the ML Ops for LangChain agents, focusing on the new Tests section and how to run them in CI with nightly evals and PR critical runs. He shows how PyTest filters critical tests with marks, and warn that eval tests call OpenAI and can burn tokens fast, so they are gated. He explains how seeded cases and judge rubrics work, and highlights that judge quality can miss real artifact quality unless we pass the DAG properly. Finally, he demonstrates DSPy based autoprompt optimization. ### Fact Data Modeling - Compact Tables for Efficient Data Representation with Iceberg (Day 2 Lab) - Pre-recorded from V3 URL: https://www.dataexpert.io/lesson/fact-data-modeling-day-2-lab-v4 Module: Week 2: Fact Data Modeling in Iceberg · Enrolled students This is a pre-recorded lesson from V3. This lab guides you through the process of analyzing user activity using compact tables. The lab demonstrates how to create a table that efficiently captures user IDs and their corresponding activity status. We address the intricacies of handling date calculations and utilizing bitwise operations to represent user activity. The lab captures the presenter encountering and troubleshooting various challenges, providing valuable insights and explanations. By the end of the lab, you will have gained a clear understanding of how to design and leverage compact tables for the purpose of analyzing user activity.[Recorded Nov 15, 2023] ### Processing terabytes of data with Spark URL: https://www.dataexpert.io/lesson/processingterabytesofdatawithspark-aug26 Enrolled students In this lesson, Zach focuses on how Apache Spark joins and shuffle behavior affect real-world large-scale data pipelines, using Netflix as the main case study. He explains that shuffle and sort are the most expensive Spark operations and compares join strategies: sort-merge join, shuffled hash join, and broadcast join, noting broadcast join avoids shuffle but requires the lookup table. ### Building AI agents that provide business value URL: https://www.dataexpert.io/lesson/buildingaiagentsthatprovidebusinessvalue-aug26 Enrolled students In this lesson, Eumar focuses on making an agent production-ready, particularly in the context of unstructured data and AI with Databricks. Key topics include deploying the agent, monitoring its performance, and ensuring it adheres to best practices with LLM operations and governance layers like the Unity AI Gateway. Additionally, he discusses the significance of preventively addressing potential issues such as inconsistent results and ensuring sensitive data protection. ### Shubham Srivastava - Senior Data Engineer at Amazon (02-01-2026) URL: https://www.dataexpert.io/lesson/techtalks1-feb26-p736-l2042 Module: Guest Speakers · Enrolled students Shubham Srivastava, a Principal Data Engineer at Amazon, shares insights from his extensive career in data engineering during this engaging guest speaker session. He discusses his journey from India to Seattle, the challenges of transitioning from legacy systems to modern architectures, and the importance of ownership and critical thinking in engineering roles. This lesson is beneficial for aspiring data engineers looking to understand career progression, the impact of AI on data engineering, and how to effectively advocate for change within their organizations. ### Intro to System Design URL: https://www.dataexpert.io/lesson/intro-to-system-design-b5027 Module: Q&A with Dan · Enrolled students ### Office Hour 1 URL: https://www.dataexpert.io/lesson/office-hour-1-p735-l601 Enrolled students In this office hour, we go over the airflow lab and the homework 1 requirements. ### Advanced Snowflake Data Modeling and UDFs Lecture Day 3 URL: https://www.dataexpert.io/lesson/advanced-snowflake-data-modeling-and-udfs-lecture-day-3-april2025 Enrolled students In this lecture, Zach dives into some advanced data modeling techniques, particularly focusing on how Snowflake and other data structures interact. He shares insights from my experience with large datasets, including how to optimize data storage using arrays and objects. He also discusses the importance of understanding the strengths and weaknesses of different data types. ### Streaming Pipelines: Setting up Streaming Pipelines Day 1 Lab URL: https://www.dataexpert.io/lesson/streaming-pipelines-day-1-lab-yt-p736-l675 Enrolled students In this lab session, you will learn the ropes of setting up and configuring streaming pipelines with Apache Flink. Write a job that connects to Kafka, filter out unwanted events, and write the relevant data to Postgres. ### Spark Basics URL: https://www.dataexpert.io/lesson/spark-basics-may2025 Enrolled students ### TA Office Hour 6 URL: https://www.dataexpert.io/lesson/ta-office-hour6-july2025 Enrolled students This office hours session focuses on addressing student questions related to homework and capstone project requirements. Key topics include best practices for file management in coding assignments, the use of graph databases for dependency scanning, and strategies for vulnerability analysis. Students also discuss project proposals and seek clarification on submission expectations, making this session beneficial for those looking to enhance their understanding of software engineering practices and project development. ### LLMs - Using LLMs to provide business value Day 1 Lecture URL: https://www.dataexpert.io/lesson/machine-learning-day-1-lecture-v4 Module: Week 6: Machine Learning · Free In this lecture, Zach simplifies the concept of LLM to its basics, emphasizing the importance of starting with simple examples for better understanding. He discussed the impact of LLM on data engineering, highlighting the significance of feedback, soft skills, and best practices in adapting to this evolving technology. Recorded on June 20th, 2024. ### The SQL Interview URL: https://www.dataexpert.io/lesson/the-sql-interview Module: Interview Skills · Free This video covers the main things you need to remember to pass the SQL interview ### Career Dev Session 5 URL: https://www.dataexpert.io/lesson/career-dev-session5-oct2025-p737-l1571 Module: Career Development · Enrolled students ### How to use Snowpark and Snowflake Optimizations Lecture Day 2 URL: https://www.dataexpert.io/lesson/howtousesnowparkandsnowflakeoptimizationslabday2-2dceb Enrolled students Explore the advanced features of Snowpark and Snowflake optimizations in this comprehensive lecture. Learn how to effectively use Snowpark for managing external data, understand the differences between Snowflake and Iceberg for data storage, and discover techniques for cost management within Snowflake. This lesson is ideal for data engineers and analytics professionals looking to enhance their skills in data modeling, clustering, and performance optimization in cloud environments. ### Analytics Engineering Bootcamp Kickoff URL: https://www.dataexpert.io/lesson/analyticsengineeringbootcampkickoff-jan2026 Enrolled students ### Office Hour 2 URL: https://www.dataexpert.io/lesson/office-hour-2-p735-l602 Enrolled students Office Hour 2 features a live Q&A session focused on addressing homework-related questions and clarifying concepts around SQL queries and data pipelines. Participants discuss various topics, including handling duplicates in data, using SQL techniques like GROUP BY and window functions, and best practices for implementing Slowly Changing Dimensions (SCD) in data engineering tasks. This session is beneficial for learners seeking guidance on practical applications of SQL and data management in their projects. ### Test 5 URL: https://www.dataexpert.io/lesson/6961b548-3aee-40b7-a0da-afb50b378cactest5 Enrolled students ### How to make AI that actualy provides Value Lab URL: https://www.dataexpert.io/lesson/using-mcp-to-build-more-powerful-agents-oct2025-p737-l1520 Module: Week 5: End-to-End AI Applications · Enrolled students In the lab, Zach walks through an N8N workflow that integrates AI agents to automate researching and generating LinkedIn shout‑out posts for creators in the data‑engineering space. He explains how the workflow calls the Google Search API and an OpenAI model to gather up‑to‑date information about creators, ensuring that each month the system highlights new voices without repeating previous mentions. ### Advanced Data Pipeline Maintenance - Strategies for Maintenance and Dock Building (Day 2 Lecture) URL: https://www.dataexpert.io/lesson/pipeline-maintenance-day-2-lecture-v3 Module: Week 6: Data Pipeline Maintenance · Enrolled students In this video, Zach delves into the crucial realm of maintenance and dock building within data engineering. Unveiling the impact these often-overlooked tasks can have on our work, the discussion provides valuable insights. He shares examples and proffers tips on how to optimize compute, reduce costs, and elevate data quality. For those keen on enhancing their data engineering skills, this video is a goldmine of actionable steps and knowledge. [Recorded on Dec 21st 2023] ### Tech Talk - Jaco URL: https://www.dataexpert.io/lesson/techtalk3-aug26 Enrolled students A dynamic discussion with Jaco, founder of D-Typed and former data engineering lead at IKEA, delves into the evolution of data engineering, particularly with DataBricks. Jaco shares insights on his journey, the importance of specialization in tools like DataBricks, and the role of data engineering in impactful projects, including advancements in AI and healthcare. This session is valuable for aspiring data engineers and those interested in building a focused career in data technologies. ### Applying Analytical Patterns: Mastering Growth Accounting and Retention Analysis Day 1 Lab URL: https://www.dataexpert.io/lesson/applying-analytical-patterns-day-1-lab-yt-p736-l671 Free In this hands-on lab session, we embark on a practical journey of growth accounting. The video guides you through the step-by-step process, illustrating how to create a partition column, calculate active dates, and adeptly track state transitions. ### Snowflake and Glue Catalog Integration Lab URL: https://www.dataexpert.io/lesson/snowflakeandgluecatalogintegrationlab-c5e3c Enrolled students ### Apache Spark Shuffle Joins Day 1 Lab URL: https://www.dataexpert.io/lesson/apache-spark-shuffle-joins-day-1-lab-jan2025 Enrolled students In this lab, Zach walks through the process of joining two datasets related to NBA games and players, focusing on the performance implications of different join strategies. He demonstrates how to disable broadcast joins and analyze the resulting execution plans to better understand the underlying processes. He also highlights the importance of using formatted modes for clarity in job execution. ### Day 2 Lecture: Types of Agents (reflection, plan-and-execute, rewoo) URL: https://www.dataexpert.io/lesson/advanced-rag-and-agentic-ai-day2-lecture-july2025 Enrolled students Understanding AI agents is crucial for anyone involved in data and AI engineering. This lecture covers the components and types of AI agents, including reflection, plan-and-execute, and rewoo architectures. Participants will learn how agents utilize tools like LLMs and retrieval-augmented generation to enhance reasoning and problem-solving capabilities, while also addressing challenges such as latency and cost. The session provides insights into practical applications and frameworks for developing effective AI agents. ### Dimensional Data Modeling - Slowly-changing Dimensions and idempotent queries with Trino (Day 2 Lab) URL: https://www.dataexpert.io/lesson/dimensional-data-modeling-day-2-lab-v3 Module: Week 1: Dimensional Data Modeling in Iceberg · Enrolled students In this lab, we reinforce lecture concepts with a hands-on approach. We dive into implementing a Slowly Changing Dimension (SCD) table. Learn to track user activity changes using lag functions and identify active and inactive player seasons. We also tackle common errors in query optimization, providing practical solutions for a real-world understanding. Watch the video for step-by-step guidance on implementing SCDs in your data pipelines. [Recorded Nov 8, 2023] ### V3 Boot Camp Kick-Off URL: https://www.dataexpert.io/lesson/boot-camp-kick-off-v3 Module: Boot Camp Orientation · Enrolled students Join us for the exciting launch of Zach Wilsons Data Engineering Boot Camp, now in its third and most improved version. In this kickoff, we go over the topics we will be covering, the weekly schedule, group projects, certification process, policies, and provide answers to your most burning questions. [Recorded on Nov 6, 2023] ### Career Development - LinkedIn Optimization URL: https://www.dataexpert.io/lesson/career-development-linkedin-optimization-jan2025-p736-l720 Module: Career Development · Enrolled students Learn how to optimize your LinkedIn profile for career advancement and job opportunities in this comprehensive lesson. Discover techniques for enhancing your profile, including the importance of a professional photo, effective headlines, and showcasing skills and endorsements. The session also covers strategies for job searching, networking with recruiters, and the benefits of content creation on LinkedIn, making it ideal for professionals looking to elevate their online presence and connect with potential employers. ### TA Office Hour 2 URL: https://www.dataexpert.io/lesson/taofficehour2-aug26 Enrolled students This office hour session addresses technical issues related to a platform problem affecting assignment submissions and table creation in a data engineering context. Participants discuss the need for updates to code and schemas, specifically regarding the use of a single schema for student tables. The session provides clarity on deadlines and troubleshooting steps, benefiting students who are currently facing challenges with their assignments. ### dbt Basics Day 2 Lecture URL: https://www.dataexpert.io/lesson/dbt-basics-day-2-lecture-jan2025 Enrolled students In this lecture, Bruno dives into the importance of DBT for data transformation within cloud data warehouses like Snowflake and BigQuery. He discusses how the shift from ETL to ELT has changed the landscape, making it more cost-effective to store and process data. He also highlights best practices for using DBT and the challenges that come with it. ### Fact Data Modeling - Core Elements in Data Modeling with Iceberg (Day 2 Lecture) URL: https://www.dataexpert.io/lesson/fact-data-modeling-day-2-lecture-v3 Module: Week 2: Fact Data Modeling in Iceberg · Enrolled students In this lecture, we cover the fundamental concepts of fact and dimension data structures. The lecture delves into the significance of fact data, highlighting its role as the core element in data modeling. Additionally, we explore dimension data and its crucial function in grouping and categorizing information. The discussion emphasizes the importance of capturing changes within dimensions and introduces the benefits of employing slowly changing dimensions. The lecture provides examples and insights, offering viewers a comprehensive understanding of fact and dimension data structures in the context of data engineering.[Recorded Nov 15, 2023] ### Day 2 Lab: Vibe Coding Full Stack application in 45 minutes URL: https://www.dataexpert.io/lesson/prompt-engineering-day2-lab-jul2025 Enrolled students In this video, Zach walks through building a FastAPI server that features a chat GPT-like interface. He starts with a simple "Hello World" endpoint and gradually enhances the application by adding an HTML interface and integrating a Milvus vector database for retrieval-augmented generation (RAG). Zach also addresses some common issues, such as dependency management and connection errors, while emphasizing the importance of being skeptical of AI-generated code. ### Advanced dbt Day 3 Lab URL: https://www.dataexpert.io/lesson/advanced-dbt-day-3-lab-jan2025-p736-l797 Enrolled students In this lab, Bruno walks you through the integration of DBT with Airflow, focusing on how to run and test data models effectively. He emphasizes the importance of testing small amounts of data and provide a live demonstration of setting up the environment. ### Growth Accounting Lab URL: https://www.dataexpert.io/lesson/growthaccountinglab-jan2026-p735-l1905 Module: Week 7: Analytical Patterns & CDC · Enrolled students In this lab, Zach walks through building a state transition tracking system to analyze user behavior, including churn and retention rates. He works with several datasets, specifically focusing on platform users and website events, and he demonstrates how to create and manipulate tables using SQL. He also highlights the importance of understanding user activity over time and how to set up a growth accounting DAG with data quality checks. ### TA Office Hour 8 URL: https://www.dataexpert.io/lesson/ta-office-hour-8-may2025 Enrolled students Office Hour 8 provides a platform for students to ask questions and clarify doubts regarding their projects, particularly focusing on the capstone project and homework assignments. Key discussions include the use of Databricks for pipeline orchestration, the requirements for certification, and troubleshooting issues with AWS Glue jobs and S3 bucket arguments. This session is beneficial for learners seeking guidance on project implementation and requirements in data engineering and AI. ### Saturday Quiz URL: https://www.dataexpert.io/lesson/saturdayquiz-12e15 Enrolled students ### Advanced dbt Day 3 Lab URL: https://www.dataexpert.io/lesson/advanced-dbt-day-3-lab-jan2025 Enrolled students In this lab, Bruno walks you through the integration of DBT with Airflow, focusing on how to run and test data models effectively. He emphasizes the importance of testing small amounts of data and provide a live demonstration of setting up the environment. ### Day 1 Lecture: Trusting your workflows (evaluation metrics and guardrails) URL: https://www.dataexpert.io/lesson/ml-ops-day1-lecture-july25 Enrolled students Understanding how to trust your workflows is crucial for ensuring the quality of applications in production. This lecture covers evaluation metrics, guardrails, and the importance of preventing issues like hallucinations and prompt injections in AI systems. Participants will learn about multi-tenant architecture for sensitive data, explore real-world examples of failures, and discover tools like OpenAI's moderation API and Guardrails AI to enhance application reliability and security. ### Thursday SQL Quiz URL: https://www.dataexpert.io/lesson/thursday-quiz-a8b81 Enrolled students ### Growth Accounting Models Lab URL: https://www.dataexpert.io/lesson/growth-accounting-models-lab Module: Applying Analytical Patterns · Enrolled students In this hands-on lab session, we embark on a practical journey of growth accounting. The video guides you through the step-by-step process, illustrating how to create a partition column, calculate active dates, and adeptly track state transitions. Common challenges encountered in this process are addressed, accompanied by effective solutions. By the conclusion of the video, you will have gained a lucid understanding of analyzing user growth and retention. This session equips you with practical skills to navigate and excel in the intricacies of growth accounting, providing valuable insights applicable to real-world scenarios. ### Apache Spark Unit Testing Day 3 Lab URL: https://www.dataexpert.io/lesson/apache-spark-unit-testing-day-3-lab-may2025-p736-l1148 Module: Week 2: Databricks & Advanced Spark · Enrolled students In this lab, Zach walks you through the process of creating unit tests in PySpark, including how to handle bad data and ensure your tests pass. He demonstrates the importance of testing logic without impacting datasets and share tips on using named tuples and data frames effectively. He also discusses the significance of CI/CD practices and how to properly deploy changes. ### Data Expert.io Architecture URL: https://www.dataexpert.io/lesson/dataexpert-architecture-jan2026 Enrolled students ### Capstone Project Day 1 Lecture URL: https://www.dataexpert.io/lesson/capstone-project-day-1-lecture-april2025-p735-l1051 Module: Week 8: End-to-End Project Hacking · Enrolled students In this lecture, Zach shares his experiences with replicating Heroku and working with Postgres, emphasizing the importance of maintaining data integrity while querying. He discusses the significance of starting with dimensions in your capstone projects and the challenges of handling sensitive information in data models. He also touches on the need for real-time updates for student progress and the use of tools like DBT and Astronomer for data orchestration. ### test 6 URL: https://www.dataexpert.io/lesson/test6-a1197 Enrolled students ### Backfilling with Airflow URL: https://www.dataexpert.io/lesson/backfillingwithairflow-3483a Enrolled students Backfilling with Airflow focuses on optimizing data workflows using Apache Airflow and AWS Glue. This lesson teaches data engineers how to effectively manage memory usage during backfill operations, highlighting best practices for orchestrating ETL processes without overloading resources. Participants will learn to implement efficient data handling techniques, including using Glue for heavy lifting, and gain insights into managing large datasets while maintaining system performance. ### Q&A with Zach Week 5 URL: https://www.dataexpert.io/lesson/qa-with-zach-week5-july25 Enrolled students A live Q&A session with Zach covers a variety of topics including AI engineering, the importance of fundamentals, and the evolving landscape of data engineering. Participants engage in discussions about tools like NAN and Ollama, the relevance of prompt engineering, and the significance of networking for career growth. Zach emphasizes the necessity of adapting to AI advancements and shares insights on leveraging LinkedIn for job hunting and personal branding. This session is ideal for those looking to deepen their understanding of AI and data engineering practices. ### The 2026 Context Engineering Roadmap URL: https://www.dataexpert.io/lesson/the2026contextengineeringroadmap Enrolled students ### Analytics Engineering Bootcamp Kickoff URL: https://www.dataexpert.io/lesson/analytics-engineering-bootcamp-kickoff-p735-l589 Module: Kickoff · Enrolled students The Analytics Engineering Bootcamp Kickoff introduces participants to the exciting world of analytics engineering, focusing on tools like Snowflake, DBT, and Airflow. Over six weeks, attendees will engage in a mix of live and recorded lectures, hands-on projects, and Q&A sessions with industry leaders, all designed to equip them with the skills needed to excel in analytics engineering roles. This bootcamp is ideal for those looking to enhance their technical capabilities and network within a supportive community. ### Q&A with Zach Week4 URL: https://www.dataexpert.io/lesson/qawithzachweek4-feb26-p736-l2040 Module: Q&A with Zach · Enrolled students Week 4's Q&A session dives into streaming pipelines, Spark Streaming, and Delta Live Tables, providing insights for data engineers and analysts. Participants engage in discussions about the cost and performance comparisons between Snowflake and Databricks, as well as considerations for setting up data ingestion pipelines from various sources. The session also covers the importance of team skill sets and the implications of technology choices on project outcomes, making it valuable for professionals navigating data architecture decisions. ### How to do Growth Accounting like Meta URL: https://www.dataexpert.io/lesson/howtodogrowthaccountinglikemeta-jan2026 Enrolled students In this video, Zach shares a crash course on applying analytical patterns based on the two years at Meta in growth analytics. He explores three main types of analyses: aggregation-based, state change tracking, and survivorship analysis, emphasizing the importance of repeatable analyses to reduce cognitive load. He also highlights the significance of understanding dimensions and their additive properties to avoid common pitfalls in data interpretation. ### Optimize your dev workflow with Claude Code and Codex Day1 Lecture URL: https://www.dataexpert.io/lesson/optimizeyourdevworkflowwithclaudecodeandcodexday1lecture-mar26 Enrolled students In this video, Kevin introduces to AI Coding and Context Engineering course, exploring the evolution of programming and the importance of embracing AI as a natural progression in the field. He highlights the need for context engineering to maximize AI's effectiveness in coding. Kevin discusses the significance of planning, executing, and validating work with AI, emphasizing that coding tools are here to stay and are rapidly improving. ### Test Loom URL: https://www.dataexpert.io/lesson/test-loom-6831 Module: test module · Free ### LLMs - Vector database for RAG applications Day 2 Lecture URL: https://www.dataexpert.io/lesson/machine-learning-day-2-lecture-v4 Module: Week 6: Machine Learning · Free This is new content and will be recorded on June11th 2024. ### Conceptual Data Modeling Lecture Day 1 URL: https://www.dataexpert.io/lesson/conceptualdatamodelinglectureday1-5e255 Enrolled students Conceptual data modeling is a crucial skill for aspiring senior or staff engineers, focusing on how to determine what data to build before diving into technical implementations like SQL or data pipelines. This lesson emphasizes the importance of cross-functional collaboration, stakeholder engagement, and understanding business value to create effective data models. Participants will learn to navigate the complexities of requirements, identify false requirements, and develop a structured approach to brainstorming and designing data solutions that drive business impact. ### Friday SQL Quiz URL: https://www.dataexpert.io/lesson/fridayquiz-41956 Enrolled students ### Introduction to databases and relational SQL URL: https://www.dataexpert.io/lesson/introduction-to-databases-and-relational-sql Module: Fundamentals of Backend Development · Enrolled students Understanding databases is essential for backend development, and this lesson covers the fundamentals of relational SQL databases. You'll learn about the structure and purpose of databases, the importance of SQL for querying data, and how to create and manage tables, relationships, and constraints. This lesson is beneficial for aspiring backend developers looking to build robust applications that efficiently store and retrieve data using tools like PostgreSQL and Docker. ### Onboard to DataExpert Platform URL: https://www.dataexpert.io/lesson/subscription-onboarding-lesson Free The onboarding session for the DataExpert platform covers essential setup steps for new users, including creating access credentials, joining the Discord community, and linking GitHub accounts. Participants will learn how to utilize AWS for ETL tasks, generate API keys for OpenAI and Anthropic, and access various learning paths tailored to data engineering and analytics. This lesson is ideal for anyone looking to effectively navigate the DataExpert platform and maximize their learning experience. ### Analytics Data Quality - Mastering DBT Projects with Bruno: Troubleshooting, Profiles, Snapshots, and Testing (Day 2 Lab) URL: https://www.dataexpert.io/lesson/analytics-data-quality-day-2-lab-v3 Module: Week 5: DBT and Pipeline Spec Building · Enrolled students In this lab, we delve into the intricacies of a DBT (Data Build Tool) project, offering a comprehensive overview and practical demonstration. The discussion kicks off with insights on running the dbt debug command and adeptly troubleshooting potential issues. Emphasis is placed on key elements such as profiles and DBT project files, with guidance on how to verify their existence. The session extends to cover advanced topics including snapshots, slowly changing dimensions, and the implementation of tests. Join us in this video to enhance your comprehension of DBT and unlock its powerful functionalities.[Recorded on Dec 13th, 2023] ### Advanced NextJS Features URL: https://www.dataexpert.io/lesson/advanced-nextjs-features Module: Advanced Frontend Development · Enrolled students Advanced NextJS features focus on enhancing data fetching and server-side functionality for web applications. Learn about managing server components, caching strategies, and the new server actions that simplify backend interactions. This lesson is ideal for developers looking to optimize their NextJS applications by integrating direct database calls, implementing form submissions, and leveraging built-in caching mechanisms to improve performance and user experience. ### Exploring UDFs and SQL Benchmarks in Spark Streaming URL: https://www.dataexpert.io/lesson/exploringudfsandsqlbenchmarksinsparkstreaming-feb26 Enrolled students In this video, Zach dives into the differences between UDFs and built-in SQL functions in Spark Streaming, using a benchmark that processes 5 million random numbers. He explores how UDFs can be slower, with results showing about a 10% speed-up when using built-in functions, but this can vary based on caching and the complexity of operations. Zach also experiments with increasing the row count to 100 million and even 1 billion to see how performance changes. He encourages everyone to run similar benchmarks and observe the results for themselves, as there are nuances that can affect performance. ### Advanced SQL Patterns Day 2 Lecture URL: https://www.dataexpert.io/lesson/advanced-sql-patterns-day-2-lecture-jan2025 Enrolled students Advanced SQL techniques are essential for data engineers looking to enhance their querying skills and excel in technical interviews. This lecture covers critical topics such as the differences between SQL interview scenarios and real-world applications, including the use of window functions, recursive CTEs, and advanced grouping techniques like grouping sets, roll-up, and cube. Participants will learn how to optimize their SQL for efficiency and clarity, ensuring they can tackle complex data modeling challenges effectively. ### Change Data Capture (CDC) and Snowflake basics Lab URL: https://www.dataexpert.io/lesson/change-data-capture-and-snowflake-basics-lab Module: Applying Analytical Patterns · Enrolled students In this lab, Zach guides you through setting up a Change Data Capture Worksheet. He recommends duplicating the worksheet with your name for individual streams. You work on separate streams to run code together efficiently. He demonstrates updating data and explain the concept of inserts, updates, and deletes. ### Databricks DeltaTable Q&A URL: https://www.dataexpert.io/lesson/databricksdeltatableqaandlab Enrolled students A live Q&A session focused on Databricks and Delta Tables, where participants can ask questions and discuss various topics related to data engineering. The session includes insights on using Databricks for Spark optimization, data modeling, and handling skewed data. Attendees will also prepare for an upcoming hands-on lab, making it an excellent opportunity for data engineers looking to deepen their understanding of Databricks and improve their skills in a collaborative environment. ### Advanced dbt: Understanding incremental models Lab URL: https://www.dataexpert.io/lesson/advanced-dbt-understanding-incremental-models-lab Module: Advanced dbt · Enrolled students In this lab, Bruno demonstrates the implementation of incremental models using DBT. He showcases the process of setting up a new staging model to display incremental models, emphasizing the importance of defining conditions for merging logic. Throughout the lab, he explains the concept of incremental materialization and the use of full refresh flags. ### Advanced Data Pipeline Maintenance - Navigating the Complexities of Data Engineering (Day 1 Lecture) URL: https://www.dataexpert.io/lesson/pipeline-maintenance-day-1-lecture-v3 Module: Week 6: Data Pipeline Maintenance · Enrolled students Explore the dynamic world of data engineering in this insightful video. Zach shares a wealth of experiences and insights, addressing key challenges faced in the field. Dive into discussions on the delicate balance between depth and breadth, emphasizing the significance of domain knowledge and effective communication. Uncover the impact of ownership and accountability on data engineering projects. Gain practical tips on pipeline optimization, navigating tech debt, and handling migrations.[Recorded on Dec 19th 2023] ### TA Office hour 6 URL: https://www.dataexpert.io/lesson/taofficehour6-feb26-p736-l2052 Enrolled students This office hours session primarily focused on addressing student questions and troubleshooting issues related to assignments and projects. Key topics included running data pipelines, resolving permission errors in databases, and clarifying requirements for the capstone project. Participants discussed the use of tools like Snowflake and Databricks, as well as strategies for managing data access and project submissions. This session is beneficial for students seeking guidance on practical challenges in their data engineering coursework. ### Apache Spark Unit Testing Day 3 Lecture URL: https://www.dataexpert.io/lesson/apache-spark-unit-testing-day-3-lecture-jan2025 Enrolled students In this lecture, Zach discusses the critical aspects of data engineering, focusing on the importance of maintaining data quality and minimizing customer pain. He highlights the challenges we face in balancing speed and quality, especially when dealing with machine learning models. He also emphasizes the need for best practices and frameworks to ensure efficient data pipelines. ### LLMs Day 3 Part1 URL: https://www.dataexpert.io/lesson/llms-day-3-part-1-jan2025 Enrolled students In this class, Li Yin shares her experiences and insights from working in AI for the past decade. She walks through the training process, discuss performance improvements, and highlight the importance of reasoning models. She also touches on various tools and libraries that can enhance our AI applications. ### Q&A with Zach Week 2 URL: https://www.dataexpert.io/lesson/qa-with-zach-week2-oct2025 Enrolled students Week 2 features a live Q&A session focused on addressing participant questions about the curriculum and capstone projects. Topics discussed include strategies for staying updated in the AI field, overcoming tutorial hell, and the importance of building practical projects. Participants also explore the role of AI engineers, industry trends, and the impact of foundational models on career paths. This session is ideal for learners seeking guidance on navigating their AI education and career development. ### Wednesday Quiz URL: https://www.dataexpert.io/lesson/wednesdayquiz-2119f Enrolled students ### Q&A with Zach Week 2 URL: https://www.dataexpert.io/lesson/qa-with-zach-week-2-april2025 Enrolled students The Week 2 Q&A session focuses on addressing students' questions related to their capstone projects and the analytics engineering boot camp. Participants discuss project ideas, budget considerations for cloud tools, and the importance of real-time data versus static datasets. The conversation emphasizes the value of creating impactful projects that demonstrate technical skills and business relevance, while also encouraging a mindset shift towards embracing challenges and continuous learning in data engineering. ### Q&A with Zach Week 1 URL: https://www.dataexpert.io/lesson/qa-with-zach-week-1-april2025-p735-l959 Module: Q&A with Zach · Enrolled students The Week 1 Q&A session with Zach covers a range of topics relevant to data engineering and analytics, including the impact of AI on the industry, data governance challenges, and effective strategies for data anonymization in Snowflake. Participants gain insights into deploying Iceberg on-premise, the importance of coding skills, and the value of conceptual data modeling for capstone projects. This interactive session is ideal for those looking to deepen their understanding of data engineering practices and tools. ### Q&A with Zach 2 URL: https://www.dataexpert.io/lesson/qawithzach2-mar26-p737-l2120 Module: Q&A with Zach · Enrolled students A live Q&A session addresses various topics related to the bootcamp, including group dynamics, API limits, and homework deadlines. Participants discuss issues with homework reviews and the importance of consistent feedback. The session also covers technical aspects such as agentic architecture, data pipelines, and the use of tools like LangChain and Snowflake. Attendees are encouraged to ask questions about their projects, share experiences, and prepare for upcoming lessons, fostering a collaborative learning environment. ### Master Advanced Snowflake Data Modeling with the VARIANT data type URL: https://www.dataexpert.io/lesson/masteradvancedsnowflakedatamodelingwiththevariantdatatype-jan2026-p735-l1909 Module: Week 4: Advanced Snowflake · Enrolled students In this lesson, Zach dives deep into unstructured data, data modeling, and some exciting features of Snowflake. He explores the exponential growth of unstructured data, driven by the fact that nearly five billion people now have smartphones, creating massive amounts of content daily. Zach also introduces Cortex, their AI tool, and how it can help them process and model unstructured data effectively. ### Office Hour 1 URL: https://www.dataexpert.io/lesson/office-hour-1 Module: TA Office Hours · Enrolled students In this office hour, we go over the airflow lab and the homework 1 requirements. ### Structured Streaming Kafka to Delta Live Table Day1 Lab URL: https://www.dataexpert.io/lesson/structuredstreamingkafkatodeltalivetableday1lab-feb26-p736-l2030 Module: Week 4: Structured Streaming with Spark & Kafka · Enrolled students In this video, Zach walks you through setting up the Spark Streaming Lab, focusing on creating and managing secrets in Databricks. He also demonstrates how to run a simple Spark streaming job that processes data from Kafka, highlighting the importance of triggers and checkpoints. ### Monday Big O Notation URL: https://www.dataexpert.io/lesson/mondaybigonotation-24a31 Enrolled students ### Tech talk with Andrew Madson URL: https://www.dataexpert.io/lesson/tech-talk-with-andrew-madson Module: Tech Talk · Enrolled students A discussion with Andrew Madsen, a tech evangelist and analytics expert, explores the evolving role of analytics engineers in bridging technical skills and business needs. Madsen shares insights on effective data strategy, the importance of aligning technical outcomes with business results, and the necessity of prioritizing data requests. This session is valuable for professionals looking to enhance their analytics engineering skills and understand the impact of data on decision-making processes. ### Fact Data Modeling - Practical Guide to Formatting and Aggregating Data with Iceberg (Day 3 Lab) URL: https://www.dataexpert.io/lesson/fact-data-modeling-day-3-lab-v3 Module: Week 2: Fact Data Modeling in Iceberg · Enrolled students This lab demonstration provides a practical guide on working with daily metrics. The lab walks through the process of formatting data, highlighting key steps such as partitioning and grouping by date and metric name. The benefits of utilizing a monthly array web metrics table are explained, and the video demonstrates how to create monthly aggregates. The lab aims to equip viewers with hands-on skills for effectively handling and organizing daily metrics, showcasing a step-by-step approach to enhance data management and analysis.[Recorded Nov 16, 2023] ### LinkedIn Like Pro! URL: https://www.dataexpert.io/lesson/linkedin-like-pro-april2025 Enrolled students Maximize your LinkedIn presence to enhance your job search and personal branding. This lesson covers essential strategies for building a standout profile, including optimizing your headline, summary, and experience sections. Learn how to connect with the right people, engage with recruiters, and utilize LinkedIn features like endorsements and skill assessments to increase your visibility and appeal to potential employers. Perfect for job seekers and professionals looking to elevate their networking game. ### How to use Snowpark and Snowflake Optimizations Lab Day 2 URL: https://www.dataexpert.io/lesson/howtousesnowparkandsnowflakeoptimizationslabday2-e63c5 Enrolled students Cost optimization techniques in Snowflake are explored through practical applications involving Snowpark and Iceberg tables. Participants will learn how to configure warehouse settings for efficiency, create and manage Iceberg tables, and understand the integration between Snowflake and external data sources like S3. This lesson is ideal for data engineers and analysts looking to enhance their skills in optimizing data storage and processing within the Snowflake environment. ### Book Club 3 URL: https://www.dataexpert.io/lesson/book-club-3-oct2025 Enrolled students ### Sundas Khalid - Principal Analytics Lead at Google URL: https://www.dataexpert.io/lesson/sundas-khalid-jan2025-p736-l784 Module: Guest Speakers · Enrolled students Sundas Khalid, Principal Analytics Lead at Google, shares her extensive journey in data engineering, data science, and analytics during this engaging guest speaker session. With experience at both Google and Amazon, she discusses her career progression, insights on salary negotiation, and the importance of financial literacy for tech professionals. This lesson is ideal for aspiring data engineers and analysts looking to learn from a seasoned expert and gain practical tips for career advancement in the tech industry. ### Streaming Pipelines: Exploring Data Collection and Processing Day 2 Lecture URL: https://www.dataexpert.io/lesson/streaming-pipelines-day-2-lecture-yt-p736-l676 Enrolled students In this presentation, the focus is on elucidating how data is collected and processed utilizing Kafka. The step-by-step walkthrough covers the process of making a request and the seamless passage of data to a Kafka producer. ### Tech Talk - Chip URL: https://www.dataexpert.io/lesson/tech-talk-chip-july25 Enrolled students A dynamic conversation with Chip, a best-selling author and AI expert, explores the evolution of AI and its implications for engineering. Chip shares insights from her experiences at leading companies like Nvidia and Snorkel AI, discussing the transition from traditional machine learning to generative AI. This session is ideal for engineers and data scientists looking to understand the current landscape of AI, best practices for implementation, and how to navigate the rapidly changing technology environment. ### Capstone Showcase with Billy Switzer URL: https://www.dataexpert.io/lesson/capstone-showcase-with-billy-switzer Enrolled students Join us as we welcome Billy Switzer, a standout student from our V4 cohort, who’s here to showcase his impressive capstone project! In this session, Billy walks us through his project, sharing the strategies and skills that helped him succeed. He covers the challenges he faced, the solutions he crafted, and how he leveraged core concepts to build a project that truly stands out. The session wraps up with an open Q&A, where Billy answered questions from the audience, providing valuable insights on capstone creation, project execution, and making an impact in the field. ### Fact Data Modeling - Compact Tables for Efficient Data Representation with Iceberg (Day 2 Lab) URL: https://www.dataexpert.io/lesson/fact-data-modeling-day-2-lab-v3 Module: Week 2: Fact Data Modeling in Iceberg · Enrolled students This lab guides you through the process of analyzing user activity using compact tables. The lab demonstrates how to create a table that efficiently captures user IDs and their corresponding activity status. We address the intricacies of handling date calculations and utilizing bitwise operations to represent user activity. The lab captures the presenter encountering and troubleshooting various challenges, providing valuable insights and explanations. By the end of the lab, you will have gained a clear understanding of how to design and leverage compact tables for the purpose of analyzing user activity.[Recorded Nov 15, 2023] ### Special Guest, Joe Reis [Recorded Dec 8, 2023] URL: https://www.dataexpert.io/lesson/v3-3-special-guest-joe-reis-recorded-dec-8-2023-6263 Enrolled students In this captivating speaker series, Joe Reis delivers an insightful talk on the pivotal role of data engineering in facilitating informed decision-making. With a profound understanding of the challenges data engineering addresses, Joe shares valuable suggestions drawn from his extensive experience. Delving into the historical evolution of the data industry, he underscores the critical importance of navigating the vast landscape of data. Whether you are a seasoned data professional or someone keen on understanding the impact of data engineering on decision-making, this talk offers a wealth of knowledge and expertise. Dive into this special session for an enriching exploration of the dynamic intersection between data, engineering, and strategic decision-making.[Recorded on Dec 8th, 2023] ### Book Club 1 URL: https://www.dataexpert.io/lesson/book-club-1-oct2025 Enrolled students The discussion centers on prompt engineering, exploring key concepts from the relevant chapters of AI engineering. Participants share insights on the evolution of best practices, emphasizing the importance of systematic approaches to prompt design and the balance between providing examples and maintaining clarity. This session benefits those interested in AI and data engineering, particularly in refining their skills in creating effective prompts for large language models. ### Kick Off URL: https://www.dataexpert.io/lesson/kickoff-91390 Enrolled students ### dbt basics Day1 Lab URL: https://www.dataexpert.io/lesson/dbtbasicsday1lab-jan2026 Enrolled students DBT Basics Day 1 Lab introduces essential concepts of dbt, focusing on models, sources, and seeds within a Snowflake environment. Participants will learn how to set up a dbt project, configure environment variables, and create their first dbt models while understanding best practices for defining sources in YAML files. This lab is ideal for beginners looking to grasp the foundational elements of dbt and its integration with data transformation workflows. ### How to build job orchestration that matters URL: https://www.dataexpert.io/lesson/howtobuildjoborchestrationthatmatters-jan2026 Enrolled students In this lecture, Zach dives into the concept of orchestration and its critical role in data pipelines, emphasizing that orchestration is not just about tools like Airflow, but about the entire process from data collection to ROI. He shares insights on common pitfalls in orchestration, such as bad logging and schema changes, and discussed best practices like the right audit publish pattern. He also highlights the importance of idempotency in pipelines to avoid inconsistencies. ### Advanced Spark Optimization Techniques Day 1 Lecture URL: https://www.dataexpert.io/lesson/advanced-spark-optimization-techniques-day-1-lecture-jan2025 Enrolled students Advanced Spark optimization techniques are essential for data engineers looking to enhance the performance of their Spark applications. This lecture covers key topics such as query optimization using Catalyst Optimizer, Adaptive Query Execution (AQE), and effective memory management strategies. Participants will learn how to apply filters efficiently, manage data partitions, and utilize Spark's execution plans to optimize performance in real-world scenarios, such as airline data processing. ### TA Office Hour 3 URL: https://www.dataexpert.io/lesson/ta-office-hour-3-april2025-p735-l994 Enrolled students Office hours provide an opportunity for students to ask questions and clarify concepts related to their assignments. This session focused on the current week's assignment involving SQL queries in Snowflake, specifically using Common Table Expressions (CTEs) and User Defined Functions (UDFs) to analyze an Actor films dataset. Students received guidance on generating sequences of years for inactive actors, creating functions in Python, and leveraging Snowflake's capabilities for data processing. ### Evaluations and Reliability MLOps Lab URL: https://www.dataexpert.io/lesson/endtoendaiapplicationsday1lab-mar26-p737-l2154 Module: Week 5: End-to-End AI Applications · Enrolled students In this lab, Zach walks through the ML Ops for LangChain agents, focusing on the new Tests section and how to run them in CI with nightly evals and PR critical runs. He shows how PyTest filters critical tests with marks, and warn that eval tests call OpenAI and can burn tokens fast, so they are gated. He explains how seeded cases and judge rubrics work, and highlights that judge quality can miss real artifact quality unless we pass the DAG properly. Finally, he demonstrates DSPy based autoprompt optimization. ### Fact Data Modeling Lab URL: https://www.dataexpert.io/lesson/fact-data-modeling-lab Module: Bonus · Enrolled students In this hands-on lab session, we analyze NBA game data, unraveling the intricacies of data modeling and analysis. Breaking down the data into distinct columns, we elucidate the importance of each element. Through practical demonstrations, we showcase how to execute queries and perform joins, extracting meaningful insights from the dataset. Throughout the video, valuable tips are shared on data modeling, query optimization, and the significance of empathy when collaborating with stakeholders. Tune in to this video to acquire practical skills in NBA game data analysis and learn essential techniques for effective data modeling and analysis. ### Special Guest, Joseph Machado [Dec 15,2023] URL: https://www.dataexpert.io/lesson/v3-4-special-guest-joseph-machado-dec-15-2023-3516 Enrolled students In this video, Joseph Matado, a data engineer at LinkedIn, shares insights on data engineering, discusses the Iceberg data format, and provides career tips. He emphasizes the importance of integrating multiple systems and being cautious with running multiple inserts. Joseph also touches on the popularity of Python and the benefits of preparing for interviews. The video offers valuable information for data engineers and those interested in the field.[Recorded on Dec 15th, 2023] ### Optimize your dev workflow with Claude Code and Codex Day2 Lab URL: https://www.dataexpert.io/lesson/optimizeyourdevworkflowwithclaudecodeandcodexday2lab-mar26 Enrolled students In this lab, Kevin walk throughs building a composable testing and validation pipeline using Claude skills. He starts a test server, then create a lint and test skill, and finally build Playwright based smoke tests that record browser actions and send videos to Telegram via MCP, then wrap everything in a validate skill. He then creates a top level feature skill that plans, builds, and adds a new favourites feature to the smoke tests, so validation runs end to end. ### Day 2 Lecture: RAG Tradeoffs URL: https://www.dataexpert.io/lesson/understanding-rag-lecture-day2-july2025 Enrolled students In this lecture Zach dives deeper into identifying trigger points for implementing AI in our businesses, discussing how to make our RAG chunks more relevant, and exploring cloud deployment and unification of applications. He covers four key trigger points: chatbots, CICD, cron jobs, and event queues, highlighting their risks and benefits. He will also address how to improve the relevance of our RAG results and the importance of proper data chunking. ### OpenClaw and Automating business processes Day1 Lecture URL: https://www.dataexpert.io/lesson/openclawandautomatingbusinessprocessesday1lecture-mar26-p737-l2145 Module: Week 3: Agentic AI, MCP & Automation · Enrolled students In this lecture, Carly dives into what OpenCLAW is. She discuss its rapid growth to 250,000 GitHub stars in just two months and set it up in a safe Docker environment ### Day 2 Lecture: Understanding LLM Reasoning traceability URL: https://www.dataexpert.io/lesson/ml-ops-day2-lecture-july25 Enrolled students Understanding LLM reasoning traceability is crucial for evaluating and improving AI applications. This session covers the importance of tracing and evaluating large language models (LLMs), focusing on techniques for assessing their performance in production. Participants will learn about tools like Lang Smith for evaluation, the significance of guardrails, and production challenges such as accuracy and latency, ensuring they can effectively deploy and monitor AI applications. ### TA Office hour 2 URL: https://www.dataexpert.io/lesson/taofficehour2-feb26-p736-l2048 Enrolled students The TA office hour session focuses on guiding students through the capstone project selection process for the data engineering boot camp. Participants discuss project ideas, including the integration of AI and machine learning for forecasting in an auto manufacturing context, and seek advice on project proposals and deadlines. The session emphasizes collaboration among students and the importance of reaching out for support within the boot camp community. ### Tech Talk - Conor URL: https://www.dataexpert.io/lesson/tech-talk-conor-july2025 Enrolled students Gain insights from a tech talk featuring Connor, founder of Hyperspell, who shares his expertise on the context layer for AI and retrieval-augmented generation (RAG). This session is ideal for data engineers and aspiring AI engineers, covering topics such as handling unstructured data, optimizing embeddings, and the importance of metadata. Learn about the challenges of building AI systems, the significance of context in AI applications, and best practices for multimodal data processing. ### Course 1 updating URL: https://www.dataexpert.io/lesson/course-1-updating-6028 Enrolled students testing updating ### Supplemental Session on Auto-prompt Optimization URL: https://www.dataexpert.io/lesson/supplemental-session-on-auto-prompt-optimization-jul2025-p737-l1297 Module: Week 1: Prompt Engineering & Basic RAG · Enrolled students In this video, Zach provides an overview of the Autoprompt optimization session, where he discusses the key components of Adallflow pipelines, including the system prompt, examples, and input strings. He emphasizes the importance of having a ground truth for effective auto-optimization and explains how we can iteratively improve our system prompts through a structured training process. ### Apache Iceberg Day 2 Lab URL: https://www.dataexpert.io/lesson/apache-iceberg-day-2-lab-may2025 Enrolled students In this lab, we explore the key concepts of partitioning and metadata in data management. Zach explains the importance of creating a schema, using dollar files for queries, and demonstrate how partitions work. We also discuss the value of low cardinality number columns and the idea of partition pruning. ### Tuesday Quiz URL: https://www.dataexpert.io/lesson/tuesdayquiz-1895d Enrolled students ### Cumulative DAGs in Production Day 3 Lab URL: https://www.dataexpert.io/lesson/cumulative-dags-in-production-day-3-lab-jan2025-p736-l761 Module: Week 6: Airflow & Pipeline Specs with Trino · Enrolled students Cumulative table design and concurrent writes are the focus of this lab, which is ideal for data engineers looking to enhance their skills with Airflow and Trino. Participants will learn how to create and manage cumulative DAGs, handle staging and production tables, and address race conditions during concurrent writes. The session also covers practical techniques for data quality checks and utilizing array transformations, making it a valuable resource for those working with data pipelines. ### Apache Spark Unit Testing Day 3 Lecture URL: https://www.dataexpert.io/lesson/apache-spark-unit-testing-day-3-lecture-may2025 Enrolled students In this lecture, Zach discusses the critical aspects of data engineering, focusing on the importance of maintaining data quality and minimizing customer pain. He highlights the challenges we face in balancing speed and quality, especially when dealing with machine learning models. He also emphasizes the need for best practices and frameworks to ensure efficient data pipelines. ### Career Development - LinkedIn Optimization URL: https://www.dataexpert.io/lesson/career-development-linkedin-optimization-jan2025 Enrolled students Optimizing your LinkedIn profile can significantly enhance your career development by showcasing your skills and expanding your professional network. This lesson covers essential strategies such as creating a compelling profile, leveraging job search tools, and engaging in content creation to attract recruiters. Ideal for job seekers and professionals looking to advance their careers, you'll learn how to effectively utilize LinkedIn features, including endorsements and recommendations, to stand out in a competitive job market. ### dbt basics Day 2 Lecture URL: https://www.dataexpert.io/lesson/dbt-basics-day-2-lecture-april2025 Enrolled students The second day of the DBT (Data Build Tool) lecture focused on enhancing the understanding of DBT as a transformation tool, its functionalities, and its integration with data warehouses. The session aimed to provide a faster-paced lecture to allow for a more extensive demonstration of DBT's capabilities. ### Building functional AI agents using LangChain Day2 Lab URL: https://www.dataexpert.io/lesson/buildingfunctionalaiagentsusinglangchainday2lab-mar26-p737-l2140 Module: Week 1: Prompt Engineering & Basic RAG · Enrolled students In this lab, Zach dives into deep research agents and MCP tools. He explores the GitHub agent and its functionality, including how to manage pull requests and create issues on Linear. He demonstrates how to refine the agent's prompts to ensure it uses the correct tools and avoids confusion. He also introduces the deep research agent that utilizes web searches and saves findings to Supabase, emphasizing the importance of semantic matching to avoid duplicate research. ### QnA with Zach 1 URL: https://www.dataexpert.io/lesson/qnawithzach1-aug26 Enrolled students A live Q&A session focused on interview preparation for data engineering roles. Key topics discussed include coding challenges in SQL and Python, data modeling, system design, and behavioral interviews. Participants can gain insights into effective preparation strategies, the importance of practice questions, and the role of AI in interview feedback. This session is ideal for anyone looking to enhance their skills and confidence before entering the job market. ### Orchestration and Airflow Fundamentals Lab URL: https://www.dataexpert.io/lesson/orchestration-and-airflow-fundamentals-day-1-lab-may2025 Enrolled students In this video, Zach walks through the first lab of the Spring Data Engineering Bootcamp, focusing on idempotent pipelines, performance optimizations, and Airflow DAGs. The structure of the AirflowDBT project is explained, along with effective methods for creating and managing DAGs. Additionally, the video demonstrates how to handle data processing using AWS Glue and Spark, highlighting the significance of developing idempotent pipelines. ### Optimize your dev workflow with Claude Code and Codex Day1 Lab URL: https://www.dataexpert.io/lesson/optimizeyourdevworkflowwithclaudecodeandcodexday1lab-mar26 Enrolled students In this video, Kevin walks viewers through building an RSS feed reader using Cloud and React with TypeScript. He begins with a vague prompt to demonstrate how context affects the output, then refines the approach by planning additional features like tagging and marking articles as read. Kevin emphasizes the importance of validating code and following best practices in development. ### Q&A with Zach Week 3 URL: https://www.dataexpert.io/lesson/qa-with-zach-week-3-april2025 Enrolled students A live Q&A session with Zach covers various topics related to the boot camp, including mentorship, capstone projects, and data engineering techniques. Participants ask questions about project requirements, data sourcing, and transitioning from data analytics to engineering roles. Zach emphasizes the importance of completing assignments on time and discusses strategies for effective project execution. The session also touches on the evolving landscape of data tools and the significance of building a strong portfolio through capstone projects. ### Databricks, Git, Discord, AI Keys Setup URL: https://www.dataexpert.io/lesson/databricksgitdiscordaikeyssetup-aug26 Enrolled students Setting up essential tools for data engineering is crucial for a successful bootcamp experience. This session covers the setup process for Databricks, GitHub, and Discord, ensuring participants can access necessary resources and collaborate effectively. Learn how to generate API keys for AI tools, configure GitHub access, and troubleshoot common setup issues, making it ideal for anyone looking to streamline their onboarding process in data engineering and AI projects. ### Fact Data Modeling - Minimizing Shuffle and Reducing Facts with Iceberg (Day 3 Lecture) URL: https://www.dataexpert.io/lesson/fact-data-modeling-day-3-lecture-v3 Module: Week 2: Fact Data Modeling in Iceberg · Enrolled students The lecture emphasizes the significance of minimizing shuffle when dealing with fact data. It explains how reducing shuffle can enhance the efficiency of data analysis and pipelines. Furthermore, the concept of reducing facts is explored, illustrating how it empowers analytics and data scientists to experiment and make informed, data-driven decisions. Students will be guided through various frameworks and strategies for handling big data, with a focus on making data more accessible and efficient for extended timeframes.[Recorded Nov 16, 2023] ### Apache Iceberg: Partitioning, time travel for RCA and metadata Lab URL: https://www.dataexpert.io/lesson/apache-iceberg-lab-day-1-april2025-p735-l956 Module: Week 5: Data Modeling on Iceberg · Enrolled students In this lab, Zach dives into the features of Apache Iceberg, focusing on partitioning, snapshotting, and the importance of metadata. He also discusses how Iceberg provides more control compared to other systems like Snowflake. ### Madison Schott - Analytics Engineer (2.0) URL: https://www.dataexpert.io/lesson/tech-talk-with-madison-schott-p735-l594 Module: Tech Talks · Enrolled students This is the tech talk with Madison Schott. ### Building functional AI agents using LangChain Day2 Lecture URL: https://www.dataexpert.io/lesson/buildingfunctionalaiagentsusinglangchainday2lecture-mar26 Enrolled students In this lecture, Zach builds on prior discussions, focuses on why agents fail, how to provide the right context, and when to use multi-agent architectures. He introduces new endpoints to the Data Expert API that help participants create a functioning version of Zach GPT. He encourages attendees to consider the difference between false positives and false negatives and to share their thoughts in the chat. ### Day 2 Lab: Tracing LLM reasoning with LangChain + LangSmith URL: https://www.dataexpert.io/lesson/ml-ops-day2-lab-july25-p737-l1185 Module: Week 4: MLOps, Guardrails & Business Value · Enrolled students Learn how to trace the reasoning of large language models (LLMs) using LangChain and LangSmith in this lab session. Participants will set up a simple Python application that utilizes the Tali web search API to gather information and generate explainable responses, while implementing tracing for performance monitoring. This lesson is ideal for data engineers and AI practitioners looking to enhance their understanding of LLMs and improve application efficiency through effective tracing and evaluation techniques. ### SQL URL: https://www.dataexpert.io/lesson/sql-intereview Module: Passing Analytics Engineer Interviews · Enrolled students Zach discusses key insights from SQL interviews, highlighting common mistakes to avoid and emphasizing the importance of problem-solving skills over just getting the right answer. He also touches on the evolving nature of interview processes at different companies, offering tips on how to stand out and optimize your performance. ### Tech Talk - Joe Reis URL: https://www.dataexpert.io/lesson/v4-tech-talk-joe-reis-3502 Enrolled students In this Tech Talk with Joe Reis, he discusses the importance of data modeling and its impact on data management. He also explains the different levels of data modeling (conceptual, logical, and physical) and emphasize the need for a well-designed data model to ensure data accuracy, clarity, and ease of use. He touches on the drawbacks of not implementing proper data modeling and provide real-life examples to illustrate the concepts. [Recorded on May29th, 2024] ### Event driven microservices URL: https://www.dataexpert.io/lesson/event-driven-microservices Module: Advanced Backend Development · Enrolled students Event-driven microservices are a powerful architectural pattern that enhances system scalability and resilience. This lesson covers the fundamentals of event-driven architecture, including asynchronous communication, loose coupling, and the benefits of using event buses like Kafka. Participants will learn how to design microservices that can scale independently, handle high traffic loads, and improve fault tolerance through effective event management and service discovery techniques. Ideal for backend developers looking to deepen their understanding of modern microservice architectures. ### Apache Iceberg Lab Day 1 URL: https://www.dataexpert.io/lesson/apache-iceberg-lab-day-1-april2025 Enrolled students In this lab, Zach dives into the features of Apache Iceberg, focusing on partitioning, snapshotting, and the importance of metadata. He also discusses how Iceberg provides more control compared to other systems like Snowflake. ### Book Club 1 URL: https://www.dataexpert.io/lesson/book-club-1-july2025 Enrolled students A guided discussion on prompt engineering unfolds in this interactive book club session, where participants engage in breakout groups to explore key concepts and share insights from their readings. Attendees will benefit from collaborative discussions on best practices, challenges, and personal experiences with prompt engineering techniques, enhancing their understanding of this critical area in AI development. Key topics include the definition of prompt engineering, intuitive practices, and the impact of context in prompts. ### Airflow Data Quality (Day 1 Lab) URL: https://www.dataexpert.io/lesson/airflow-data-quality-day-1-lab-v4 Module: Week 3: Airflow and Airflow Data Quality · Enrolled students In this lab, Zach will provide an overview of airflow backfilling practices and discuss the pipeline code using pyspark. He will explain the importance of including the username in the DAG name and demonstrate how to run glue jobs and backfill data. He will also show a table that was backfilled and discuss the comparison of tables using union all. [Recorded on May 22nd, 2024] ### Apace Iceberg Lab Day 2 URL: https://www.dataexpert.io/lesson/apache-iceberg-lab-day-2-april2025 Enrolled students In this lab, Zach walks through data compression techniques, demonstrating how to reduce data size and enhance efficiency. He explores concepts like run-length encoding and bin-pack compression, focusing on optimizing data storage. ### Monday Quiz URL: https://www.dataexpert.io/lesson/mondayquiz-54719 Enrolled students ### Apache Iceberg Lecture Day 1 URL: https://www.dataexpert.io/lesson/apache-iceberg-lecture-day-1-april2025 Enrolled students In this lecture, Zach dives into the intricacies of Iceberg, a technology that emerged from Netflix, and how it revolutionizes data processing. He shares his experiences working with large datasets, particularly at Airbnb, and discuss the challenges we faced with traditional tools like Hive and Spark. He also highlights the importance of time travel in data management and how Iceberg addresses some of the limitations of previous technologies. ### Your story matters and how to tell it by Stephanie Nuesi URL: https://www.dataexpert.io/lesson/linkedin-masterclass-1 Module: Making Content on LinkedIn · Enrolled students Your story matters and how to tell it by Stephanie Nuesi. This was filmed on May 10th, 2024. ### How to do Growth Accounting like Meta URL: https://www.dataexpert.io/lesson/howtodogrowthaccountinglikemeta-2eec4 Enrolled students ### Q&A with Zach Week 3 URL: https://www.dataexpert.io/lesson/qa-with-zach-week3-july2025 Enrolled students A live Q&A session covers a variety of topics related to data engineering and AI, including the use of tools like Cursor for coding, the differences between graph-based and regular retrieval-augmented generation (RAG) systems, and practical applications for AI in business. Participants gain insights on project ideas, including AI-powered applications, and discuss the importance of foundational skills in data engineering. This session is beneficial for those looking to deepen their understanding of AI technologies and their applications in real-world scenarios. ### Change Data Capture Lab URL: https://www.dataexpert.io/lesson/changedatacapturelab-jan2026 Enrolled students Change Data Capture is a crucial technique for tracking changes in data, and this lab focuses on implementing it using database triggers. Participants will learn how to create an audit table to monitor time zone changes in a PostgreSQL database, utilizing tools like DataGrip and Airflow. By the end of the session, attendees will understand how to capture and analyze time zone updates effectively, providing insights into user behavior and data integrity. ### Career Development - Data Modeling Interview URL: https://www.dataexpert.io/lesson/career-development-data-modeling-interview-jan2025 Enrolled students Mastering data modeling interviews is crucial for aspiring data engineers and architects. This session covers the intricacies of the data modeling round, emphasizing the importance of asking clarifying questions, understanding access patterns, and designing effective schemas. Participants will learn how to approach common scenarios, such as modeling for a ride-sharing app, while discussing trade-offs, indexing strategies, and the relationships between tables. This lesson is ideal for those preparing for technical interviews in data engineering roles. ### TA Office Hour 4 URL: https://www.dataexpert.io/lesson/ta-office-hour4-july2025 Enrolled students This office hour session provides an interactive platform for students to ask questions and clarify doubts regarding their homework and projects. Key topics discussed include homework requirements for loading data from GitHub, testing criteria for a rag system, and AWS credential management. Participants also explore technical challenges related to vector databases and integration with data processing tools, making this session beneficial for students seeking guidance on practical applications in data engineering. ### Chat with Zach URL: https://www.dataexpert.io/lesson/week-2-zach-qa-v4 Enrolled students Recorded on May16th, 2024. ### TA Office Hour URL: https://www.dataexpert.io/lesson/taofficehour-c0408 Enrolled students ### RAG optimisation + Reranking Day 2 Lecture URL: https://www.dataexpert.io/lesson/rag-optimisation-reranking-day2-lecture-oct-2025-p737-l1511 Module: Week 2: Advanced RAG, Reranking & Dev Workflow · Enrolled students In this lecture, Zach focuses on maximizing relevance in AI, exploring various types of RAG, and learning how to offload heavy AI tasks to worker tasks. He emphasizes the importance of using Markdown for data input and the significance of metadata filtering and re-ranking in improving retrieval accuracy. Zach also dives into hands-on lab work, including updating the chatbot and creating an asynchronous email-generating agent. ### Day 6: Merge two sorted lists URL: https://www.dataexpert.io/lesson/merge-two-sorted-lists-p913-d6 Enrolled students Iteratively merge two sorted singly linked lists into one sorted, stable list in O(n+m) time using a dummy tail with O(1) extra space. ### MLOPs and Guardrails Day 1 Lecture URL: https://www.dataexpert.io/lesson/ml-ops-day1-lecture-oct25 Enrolled students In this lesson, Zach dives into the critical topic of guardrails for ensuring LLM quality in production. He explores the various risks associated with LLMs, such as jail-breaking, prompt injection, and handling sensitive data, while emphasizing the importance of both input and output guardrails. Zach discusses effective implementation strategies for these guardrails, including the use of middleware and moderation APIs. ### Advanced Snowflake Data Modeling and UDFs Lab Day 3 URL: https://www.dataexpert.io/lesson/advancedsnowflakedatamodelingandudfslabday3-5fc67 Enrolled students Advanced Snowflake Data Modeling focuses on utilizing User-Defined Functions (UDFs), structs, and objects to analyze player performance data. Participants will learn to implement complex SQL queries to identify streaks in player scoring, leveraging both declarative and imperative programming approaches. The session also covers integrating OpenAI for enhanced data processing, making it ideal for data engineers and analysts looking to deepen their Snowflake expertise and streamline data analysis workflows. ### Q&A with Zach Week 4 URL: https://www.dataexpert.io/lesson/qa-with-zach-week-4-april2025 Enrolled students A live Q&A session covers various topics related to the ongoing boot camp, including discussions on project proposals, the use of paid APIs, and insights into data engineering roles. Participants ask questions about specific tools like DBT and Snowflake, and share experiences regarding their capstone projects. The session also touches on the importance of understanding data definitions and operational pain points in analytics engineering, providing valuable insights for students looking to enhance their skills and project outcomes. ### Managing Unstructured Data Day2 Lecture URL: https://www.dataexpert.io/lesson/managingunstructureddataday2lecture-feb26-p736-l2086 Module: Week 5: Managing Unstructured Data · Enrolled students In this video, Eumar dives deep into chunking strategies, Databricks vector search, and how to integrate the link chain in Databricks, emphasizing the importance of chunking for working with LLMs. He encourages everyone to become experts in chunking strategies, as they will be building a reg-based solution that can answer questions about the bootcamp, such as costs and content. ### Tech Talk with Li URL: https://www.dataexpert.io/lesson/tech-talk5-oct2025 Enrolled students ### Tech talk with Madison Schott URL: https://www.dataexpert.io/lesson/tech-talk-with-madison-schott Module: Tech Talk · Enrolled students This is the tech talk with Madison Schott. ### TA Office hour 2 URL: https://www.dataexpert.io/lesson/taofficehour2-feb26 Enrolled students The TA office hour session focuses on addressing student questions related to capstone projects and assignments in the data engineering boot camp. Participants discuss project ideas, including the integration of AI and machine learning for forecasting and chatbot development, while also seeking guidance on submission processes and deadlines. The session encourages collaboration among students and provides insights into project feasibility and support resources available within the program. ### HTML & CSS LAB URL: https://www.dataexpert.io/lesson/html-css-lab Module: Web Dev Fundamentals · Enrolled students HTML and CSS LAB focuses on building a responsive one-page website using HTML5 and CSS. Participants will learn to create semantic HTML structures, including headers, navigation bars, and forms, while utilizing tools like Visual Studio Code. The lesson emphasizes best practices for accessibility and responsive design, guiding learners through styling techniques and layout strategies to achieve a polished final product. Ideal for beginners in web development, this lab provides hands-on experience in crafting modern web pages. ### Day 2 Lab: Integrating RAG into your full stack app URL: https://www.dataexpert.io/lesson/understanding-rag-lab-day2-july2025-p737-l1177 Module: Week 1: Prompt Engineering & Basic RAG · Enrolled students In this lab, Zach walks through deploying a RAG project on Vercel, highlighting the importance of setting environment variables like the OpenAI API key and Milvus credentials. He discusses the differences between using PyMilvus and querying Milvus via REST, which allows for a lighter serverless deployment. He also covers how to implement re-ranking in our RAG model for better search results and the significance of testing our system through CI/CD practices. ### Infrastructure Data Quality - Elevating Data Quality in Analytics Engineering: Importance, Challenges, and Leadership Perspectives (Day 1 Lecture) URL: https://www.dataexpert.io/lesson/infra-data-quality-day-1-lecture-v3 Module: Week 5: Unit Testing Pipelines & Write-audit-publish · Enrolled students In this presentation, we delve into the pivotal role of data quality in the realm of analytics engineering, unraveling its profound impact on the success of data pipelines. The discussion underscores the necessity for automated data quality checks, shedding light on the challenges encountered by data engineers in their quest for ensuring data accuracy. Exploring the leadership dimension, we examine the crucial role of management in prioritizing data quality and the far-reaching consequences of subpar data quality on business outcomes. Join us in this insightful video to glean a deeper understanding of the critical role that data quality plays in analytics engineering, unraveling its implications for informed decision-making. [Recorded on Dec 12, 2023] ### Career Dev Session 3 URL: https://www.dataexpert.io/lesson/career-dev-session3-oct2025 Enrolled students ### Vikram Chennai - Founder at Ardent AI URL: https://www.dataexpert.io/lesson/tech-talk-vikram-july2025-p737-l1299 Module: Tech Talks · Enrolled students Vikram, founder of Ardent AI, discusses the innovative concept of the world's first AI data engineer, aimed at automating complex data engineering tasks. This session delves into the challenges of data lineage, the importance of metadata, and the development of self-healing pipelines. Data engineers and AI enthusiasts will benefit from insights on leveraging AI to streamline data workflows, enhance efficiency, and reduce manual intervention in data management processes. ### TA Office hour 6 URL: https://www.dataexpert.io/lesson/taofficehour6-feb26 Enrolled students This office hours session focuses on addressing student questions and troubleshooting issues related to assignments and projects in the boot camp. Key topics include data injection in pipelines, permission errors in accessing databases, and clarifications on using technologies like Snowflake and DBT for capstone projects. Students benefit from real-time support and guidance on resolving technical challenges, ensuring they stay on track with their learning objectives. ### Building Rich Context for AI Agents Day2 Lab URL: https://www.dataexpert.io/lesson/buildingrichcontextforaiagentsday2lab-aug26 Enrolled students In this lab, Eumar focuses on chunking strategies, embeddings, and building knowledge assistants for RAG. He starts with document ingestion by using AI parse document to extract text from a Syllabus PDF and a Databricks bootcamp website, saving results into a RAW bootcamp docs Delta table in the Bootcamp Docs volume. He demonstrates multiple LangChain chunking approaches such as fixed size with overlap, punctuation and sentence aware splitters, structure aware chunking for markdown and HTML, semantic chunking, and code aware chunking to avoid breaking code blocks. He then connects chunking to vector search and retrieval, explains dense versus hybrid retrieval for keyword queries, and closes by showing the low code Knowledge Assistant workflow that automatically handles parsing, chunking, embeddings, and citations. ### Friday Quiz URL: https://www.dataexpert.io/lesson/fridayquiz-1e809 Enrolled students ### dbt basics Day 1 Lab URL: https://www.dataexpert.io/lesson/dbt-basics-day-1-lab-april2025 Enrolled students In this lab, Bruno walks you through the basics of DBT, including how to set up your environment and connect to Snowflake. He covers essential files like the dbt project YAML and provide instructions for both Mac and Windows users. Please make sure to follow the steps carefully, and if you encounter any issues, reach out in the DBT channel. ### TA Office Hour 6 (2025-07-31) URL: https://www.dataexpert.io/lesson/ta-office-hour6-july2025-p737-l1347 Enrolled students Office hours provide an interactive platform for students to seek clarification on homework and capstone projects. Key topics discussed include best practices for file management in coding assignments, strategies for dependency scanning using graph databases, and handling GitHub API limitations. Students also explore vulnerability analysis and how to effectively utilize AI tools for coding assistance while emphasizing the importance of understanding the underlying concepts. ### Advanced dbt: Understanding incremental models Lecture URL: https://www.dataexpert.io/lesson/advanced-dbt-understanding-incremental-models-lecture Module: Advanced dbt · Enrolled students In this lecture, Bruno delves into the intricacies of incremental models in dbt. He explains how to define and implement incremental strategies, showcasing the use of unique keys and filters for efficient data processing. Additionally, he touches upon the significance of materialization types and the impact on data modeling. The explanation of the is_incremental() function got a bit messed up. Go to the hands-on lab to watch a better explanation of it. ### TA Office hour 1 URL: https://www.dataexpert.io/lesson/taofficehour1-jan2026 Enrolled students Office hours provide a platform for students to ask questions related to the boot camp, covering topics such as homework deadlines, capstone project guidelines, and troubleshooting issues with tools like Trino and Airflow. Participants engage in discussions about schema permissions, project ideas, and the submission process for assignments. This session is particularly beneficial for students seeking clarification on course expectations and technical challenges they encounter during the boot camp. ### Local Video URL: https://www.dataexpert.io/lesson/local-video-5132 Enrolled students ### Understanding Capstone Requirements URL: https://www.dataexpert.io/lesson/understanding-capstone-requirements Module: Bootcamp Orientation · Enrolled students Capstone project requirements are clarified in this open Q&A session, focusing on the necessity of utilizing at least two data sources. Participants learn about various data source combinations, including APIs, and how to effectively structure their projects for maximum business value. The session also addresses common concerns regarding project approval, the importance of conceptual models, and strategies for transitioning to data engineering roles, making it beneficial for both current students and those considering a career change. ### Capstone May 2025 URL: https://www.dataexpert.io/lesson/capstone-may-2025 Enrolled students ### Chat with Zach [Aug 17, 2023] URL: https://www.dataexpert.io/lesson/v2-5-chat-with-zach-aug-17-2023-8857 Enrolled students Delve into the world of data engineering with our weekly Q&A / Fireside Chat series, featuring the founder of EcZachly Inc and lead instructor of the Advanced Data Engineering Bootcamp. Each session covers a different topic, providing invaluable insights and real-world experiences to empower data enthusiasts in their journey. (Recorded Aug 17, 2023) ### Day 1 Lab: Vectorizing Inputs into Milvus URL: https://www.dataexpert.io/lesson/understanding-rag-day1-lab-july2025-p737-l1175 Module: Week 1: Prompt Engineering & Basic RAG · Enrolled students Learn how to vectorize inputs into Milvus by creating and managing collections for YouTube video transcripts. This lab covers essential tasks such as setting up a Milvus account, installing necessary packages, and using FastAPI to connect to Milvus. Participants will gain hands-on experience in downloading video data, generating transcripts using OpenAI, and integrating these transcripts into a retrieval-augmented generation (RAG) system, making it ideal for data engineers and AI practitioners looking to enhance their skills in vector databases and AI applications. ### Day 1 Lab: Using MCP to build more powerful agents URL: https://www.dataexpert.io/lesson/building-end-to-end-ai-application-day1-lab-july25-p737-l1188 Module: Week 5: End-to-End AI Applications · Enrolled students In this lab, Zach walks through the MCP lab where he explores building AI agents using LangChain and FastAPI. He covers the necessary Python versions, dependencies like LangChain and Pydantic, and how to set up the environment. Zach demonstrates how to interact with the MCP server, including creating and managing endpoints, as well as using the GitHub MCP server for various tasks. ### dbt day 2 lecture URL: https://www.dataexpert.io/lesson/dbt-day-2-lecture Module: DBT · Enrolled students In this video, Bruno provides a quick recap of the previous discussion on models and introduce the concept of dbt optimization. He explains how dbt sends queries to run your data warehouse and discuss the use of incremental models to transform small amounts of data. He also touches on the difference between merge and delete plus insert, as well as the importance of query performance optimization. ### Applying KPIs and Experimentation - Decoding Business Success: Metrics, Growth Strategies, and Collaborative Approaches (Day 1 Lecture) URL: https://www.dataexpert.io/lesson/kpi-metrics-day-1-lecture-v3 Module: Week 4: KPIs and Experimentation with Statsig · Enrolled students In this insightful video, we unravel the critical landscape of business success. The first part delves into the core of metrics, elucidating their correlation with outcomes and distinguishing leading from lagging metrics, offering a nuanced understanding of their role in driving success. Shifting gears, the discussion seamlessly transitions to growth strategies, emphasizing the significance of signups and user experience. From A/B testing to impactful ad creatives, customer referrals, and optimizing call-to-action on signup pages, this segment encapsulates essential strategies for continuous improvement. Throughout, we stress the importance of collaboration between data engineers and data scientists, along with the art of prioritizing tasks, providing a compact yet comprehensive guide for a holistic business approach. [Recorded on Dec 4th 2023] ### Q&A with Zach Week3 URL: https://www.dataexpert.io/lesson/qawithzachweek3-feb26 Enrolled students A live Q&A session featuring Zach and Umar, where they discuss various topics related to data engineering and the upcoming lesson on unstructured data. Participants can ask questions about tools like Data Bricks, Spark, and Lake Base, and the conversation touches on the importance of governance, scalability, and the evolving landscape of data solutions. This session is beneficial for anyone looking to deepen their understanding of data engineering practices and technologies. ### Building AI Agents with Databricks Day1 Lecture URL: https://www.dataexpert.io/lesson/buildingaiagentswithdatabricksday1lecture-feb26 Enrolled students In this session, Eumar walks you through the process of transforming Zachy into a real agent capable of answering questions about code, course syllabus, and employee expenses using tools like Unity catalog functions, Langgraph, and Databricks. He discusses the differences between RAGs and agents, emphasizing how agents can perform multi-step actions and interact with various tools. He also highlights the importance of using the Supervisor Agent for orchestrating multiple tools and shares his excitement about the capabilities of Langgraph for simplifying this process. ### Airflow Day2 Lab URL: https://www.dataexpert.io/lesson/airflowday2lab-jan2026-p735-l1928 Module: Week 6: Airflow & Pipeline Specs with Trino · Enrolled students The Airflow Day 2 Lab focuses on enhancing data quality in ETL processes using the Pie Iceberg API and Trino. Participants will learn how to identify and resolve common data quality issues, such as duplicates and incorrect currency representations, while working with stock ticker data. This hands-on session is ideal for data engineers looking to improve their skills in data validation and pipeline orchestration within Airflow. ### Integration Testing URL: https://www.dataexpert.io/lesson/integration-testing-73eb5 Module: Office Hours · Enrolled students ### Data Impact Communication & Visualization - Mastering Data Engineering: Challenges, Solutions, and Career Insights (Day 1 Lecture) URL: https://www.dataexpert.io/lesson/visual-impact-day-1-lecture-v3 Module: Week 6: Data Impact Communication & Visualization with Tableau · Enrolled students Join this insightful video as it delves into the world of data engineering. In this session, common challenges encountered in data engineering are explored, accompanied by practical solutions and best practices for constructing efficient data pipelines. The discussion extends beyond technical aspects, offering valuable insights into career progression, shedding light on the journey to becoming a staff engineer, and emphasizing the pivotal role of mentorship. Enhance your data engineering skills, unlock career growth strategies, and gain actionable tips to propel your professional journey. [Recorded on Dec 18th 2023] ### Apache Iceberg Lecture Day2 URL: https://www.dataexpert.io/lesson/apacheiceberglectureday2-bc90f-p735-l1993 Module: Week 5: Data Modeling on Iceberg · Enrolled students Understanding data lake architectures is crucial for data engineers aiming to transition from building simple pipelines to designing comprehensive data solutions. This session covers key concepts such as agnostic storage, Lambda versus Kappa architecture, and Iceberg's file compaction and row-level updates. Participants will also explore the implications of hot, warm, and cold storage, along with strategies for optimizing data retrieval and managing small files, making it valuable for engineers looking to enhance their data modeling skills. ### Day 1 Lab: Building Graph RAG in production URL: https://www.dataexpert.io/lesson/advanced-rag-and-agentic-ai-day1-lab-july2025-p737-l1179 Module: Week 2: Advanced RAG, Reranking & Dev Workflow · Enrolled students In this lab, Ashwani walks you through the process of setting up a knowledge graph using Neo4j and OpenAI. He demonstrates how to clone the repository, configure environment variables, and create a graph with entities like person, organization, and location. He also explores how to perform hybrid searches and generate responses using a language model. ### dbt day 1 lab URL: https://www.dataexpert.io/lesson/dbt-day-1-lab Module: DBT · Enrolled students In this lab, Bruno provides a step-by-step walkthrough of DBT, covering topics such as debugging, documentation generation, package installation, and more. He also demonstrates how to create chapters, run commands, and customize SQL code. Throughout the video, he offers helpful tips and tricks to make working with DBT easier ### Advanced pipelines: Blue-green and WAP with Airflow Lecture URL: https://www.dataexpert.io/lesson/advanced-pipelines-blue-green-and-wap-with-airflow-lecture Module: Advanced dbt · Enrolled students In this lecture,Bruno delves into the intricacies of DBT pipelines, focusing on structuring pipelines, selecting project components, and understanding downstream dependencies. He explains the importance of organizing models in folders for efficient maintenance and debugging. Additionally, he highlights the significance of manifest files in tracking model changes and ensuring data integrity. ### Change Data Capture (CDC) and Snowflake basics Lab URL: https://www.dataexpert.io/lesson/change-data-capture-and-snowflake-basics-lab-p735-l613 Module: Week 7: Analytical Patterns & CDC · Enrolled students In this lab, Zach guides you through setting up a Change Data Capture Worksheet. He recommends duplicating the worksheet with your name for individual streams. You work on separate streams to run code together efficiently. He demonstrates updating data and explain the concept of inserts, updates, and deletes. ### Intro to Infrastructure as Code URL: https://www.dataexpert.io/lesson/intro-to-infrastructure-as-code-a90c6 Module: Q&A with Dan · Enrolled students ### Tech Talk 3 - Ben Rogojan URL: https://www.dataexpert.io/lesson/tech-talk-3-ben-rogojan-april2025 Enrolled students This session features Ben Rogojan, a seasoned data consultant with experience at Meta and various startups. He shares insights on transitioning from big tech to consulting, emphasizing the importance of relationship-building and trust with clients. The discussion covers strategies for effective client engagement, pricing models, and the challenges of managing expectations around data tools and technologies. Ideal for aspiring consultants and data engineers, this talk provides practical advice on navigating the consulting landscape. ### Special Guests, Jitender Aswani and Bill Inmon [August 23, 2023] URL: https://www.dataexpert.io/lesson/v2-special-guests-jitender-aswani-and-bill-inmon-august-23-2023-8743 Enrolled students Join us each week as we sit down with visionary speakers from diverse fields, engaging in enlightening conversations that explore cutting-edge insights, trends, and their wealth of expertise. (Recorded August 23, 2023) ### Day 2 Lab: Building an asynchronous vs synchronous agent URL: https://www.dataexpert.io/lesson/building-end-to-end-ai-application-day2-lab-july25 Enrolled students In this lab, Zach walks viewers through the powerful capabilities of NAN for creating AI pipelines—task workflows analogous to data pipelines. He explores various triggers, including manual activation, app events, form submissions, and scheduled triggers, emphasizing the importance of efficient automation. Zach demonstrates a workflow that scrapes LinkedIn profiles and evaluates candidates for a data engineering bootcamp, while highlighting the need to improve email formatting for clearer communication. He also addresses how to minimize API calls by using Google Sheets as a cache for storing LinkedIn data. ### LLM-Driven Data Engineering Day 1 Lab URL: https://www.dataexpert.io/lesson/large-language-models-day-1-lab Free Learn the different roles you can play using the ChatGPT Python API, how to interact with GPT-4 using different API configurations, and how to make the most of LLMs to streamline data engineering tasks. In this live lab session, we put GPT-4 to the test and see how well it does at generating an Airflow DAG, writing SQL queries, and crafting dataset documentation. ### Special Guest, Joe Reis [Recorded Dec 8, 2023] URL: https://www.dataexpert.io/lesson/v3-3-special-guest-joe-reis-recorded-dec-8-2023-9386 Enrolled students In this captivating speaker series, Joe Reis delivers an insightful talk on the pivotal role of data engineering in facilitating informed decision-making. With a profound understanding of the challenges data engineering addresses, Joe shares valuable suggestions drawn from his extensive experience. Delving into the historical evolution of the data industry, he underscores the critical importance of navigating the vast landscape of data. Whether you are a seasoned data professional or someone keen on understanding the impact of data engineering on decision-making, this talk offers a wealth of knowledge and expertise. Dive into this special session for an enriching exploration of the dynamic intersection between data, engineering, and strategic decision-making.[Recorded on Dec 8th, 2023] ### Prasad Rao - Principal Solutions Architect URL: https://www.dataexpert.io/lesson/prasad-rao-jan2025-p736-l870 Module: Guest Speakers · Enrolled students Prasad Rao, a Principal Solutions Architect at AWS, shares insights on navigating behavioral interviews, particularly for roles in big tech companies. He emphasizes the importance of articulating past experiences effectively, showcasing both technical and consulting skills. This session is invaluable for aspiring solutions architects and professionals preparing for interviews, as it covers strategies for presenting oneself, handling common interview questions, and understanding the nuances of behavioral assessments across various roles. ### Special Guests, Jitender Aswani and Bill Inmon [August 23, 2023] URL: https://www.dataexpert.io/lesson/week-6-speaker-v2 Enrolled students Join us each week as we sit down with visionary speakers from diverse fields, engaging in enlightening conversations that explore cutting-edge insights, trends, and their wealth of expertise. (Recorded August 23, 2023) ## Company - [Pricing](https://www.dataexpert.io/pricing): Plans and subscription options - [Reviews](https://www.dataexpert.io/reviews): Student reviews and outcomes - [About the instructor](https://www.dataexpert.io/about) - [Scholarships](https://www.dataexpert.io/scholarships) - [Contact](https://www.dataexpert.io/contact) - [Blog](https://www.dataexpert.io/blog): Articles and newsletter