Data Pipeline Maintenance: Navigating the Complexities of Data Engineering Day 1 Lecture

Data pipeline maintenance is an inevitable part of data engineering. In this 2 hour course we're going to be covering all of the things like how to do a good data migration, how to set up run books for your on-call rotation, what are different ownership models that you can do so that your data engineers do not become burnt out. And we'll also cover common ownership problems and maintenance problems that come up that I've seen throughout my career at Facebook, Netflix and Airbnb. I hope you enjoy the course today and if you want to do more of this stuff in the cloud, you should check out the Data Expert Academy in the description below where I can get you 20% off. If you're a high performing data engineer, this, uh, is something that is a fear that you will have as you grow deeper and deeper into your career, is, um, Every time you build a pipeline, It's not like you just build it and it's done. Right? If you build it and then you have to maintain it. There's like this continual cost. So like in some ways, like, there's this, I, I, I, I've always had this fear where it's like, uh doing data engineering is inherently unsustainable because every pipeline you write, Has it uh an added cost. So that means that like as you build more pipelines, eventually you're gonna have too many pipelines that you can't maintain them all. Like that's just like the nature of the beast, that like literally there's like, it's an inherently unsustainable pattern because of that.