The life of a Silicon Valley big data engineer: 3 critical soft skills for success

<p>Big data - the two words that sound like a huge pile of insights waiting to be unlocked if you’re a corporation with lots of data, or a source of anxiety about privacy and automation if you’re an average human. Sentiments about big data are definitely varied. Regardless of the sentiments, working with big data is challenging, dynamic, and exciting career opportunity. </p> <br/> <p> When I was in college, I wanted to be the person who could finally gather insights and value from the mountains of data that companies collect. I started my big data journey in 2015. I’ve learned a lot about data engineering and analytics while working in big data for the past four years at four different companies, in four cities, and on both coasts, including data behemoths Netflix, Facebook, and the US military,. This is the first of a multipart blog about what I’ve learned from this journey so far and suggestions for how to be successful as a big data engineer. The first focus area will be soft skills, an area that people often don’t associate with data engineers. </p> <br/> <h3><b>Three Essential Data Engineering Soft Skills</b></h3>

<h4>#1 Build Relationships with Team Members and Managers</h4>
<p>One of the assumptions I made after moving to the valley was that I could be successful purely on my technical abilities. I learned just how wrong that assumption was when I was trying to move up and get promoted. My exceptional technical skills were not enough. Even if I was capable of completing projects on my own to make an impact as a data engineer, I needed to be aware of what my team members were working on and coordinate with them to reach project milestones. This relationship building and communication gets more important as you increase your impact and advance in your career.</p>
<h4>#2 Prioritize, be flexible, set aside ad-hoc time</h4>
<p>Most data engineering roles today are at companies that don’t have very much data infrastructure. With so much work to do, it is essential to correctly prioritize and order the tasks and projects that must be completed. It’s also important to stay focused on those priorities and not say yes to everything someone wants you to do. If you never say no, you will get buried by your workload quickly. Setting aside time to respond to ad-hoc requests and operational work is absolutely critical for maintaining work-life balance as a data engineer.</p>
<h4>#3 An Ounce of Negotiation is Worth a Pound of Engineering</h4>
<p>As a data engineer, you are expected to be the data expert. Most of your users will want as much data as possible. Choosing whether to optimize, deprecate, or sample can be a hard decision and often a hard negotiation. Knowing your users and their data needs makes it much more possible to deprecate things. Being cognizant of the fact that optimization is the most time-consuming option and only should be applied to long-lived data pipelines, deprecation or sampling is usually the next best choice.</p>
<br/>
<p>There’s a myth out there that big data engineering requires mostly just a hard technical skill set and is mostly just building fancy data pipelines. If they collaborate, communicate clearly and find ways to cooperate with both team members and stakeholders, rockstar data engineers can demonstrate tremendous business value from their data solutions. </p>