Welcome to my journey through 17 months of being a Data Engineer at Walmart, where every day brought new challenges and opportunities for growth. In this blog, I'll unravel the fascinating world where data plays a crucial role in uncovering insights, solving mysteries, and ensuring the safety and integrity of one of the world's largest retailers.
The beginning of my tenure at Walmart was marked by the realization that as a Data Engineer, I would be more than just a technical expert. I would be a detective, leveraging data to investigate and resolve a variety of issues, from fraud prevention to ensuring compliance.
In the grand orchestra of product development, where diverse talents harmonize to create a masterpiece, one group often operates backstage, fine-tuning the data infrastructure that powers the show. But that's just one component of the show, we haven't started to talk about the musicians or the ticket sales people or the venue hosters or even the audience. Join me as we explore the diverse landscape of roles that contribute to the success of a product.
Biggest learning from the 1st full time corporate job: Understanding the roles of different Engineers
=> The PMs
While thinking about a product, it is super crucial to think about the audience. You can't cater a grand orchestra for pop audience, you might wanna throw a concert for them instead. Understanding this is a job of your Product Managers. Product managers are responsible for understanding the product and for laying out a vision and for making sure that the product is moving in the right direction.
=> The UX
Once the PM has set the vision and expectation of the product, we need someone who can visualise the product for us. Here comes the UX team who design the application and gives a form to it. This design also gives a clear idea of the product requirements and the support needed to achieve it.
=> The SDEs
Now given that we have the physical form of the product, we need to actually materialise it. This is where software engineers are required. Software engineers basically does 2 different jobs
Back End
Front End
And thus they are also called as Full Stacks because they build the application from scratch to the screen.
Back End developers focus on the logic and the information the application actually needs.
Front End developers focus on how the data get displayed on the screen based on user's easy readability and interpretability.
=> The QA
Here comes the most critical pack of developers. The PMs, the UX and the SDEs can afford to make some mistakes but Quality Assurance Engineers absolutely can't. Once, with contributions from all the engineers, the application is ready, QA engineers have the responnsiblity to test the application for the truth of information and to verify if the application actually serves the purpose of creation. After QA team's approval, the application is launched for the users to explore.
=> The TLs
The Team Leads are responsible for all the communications between serveral group of engineer packs and the business. The Team Leads also has the responsibility to motivate all the engineers and to make sure that the product is delivered within the given the time frame with utmost quality and the developer's happiness.
So where are the data engineers in this process....
=> The DEs
So here is me, Data Engnineer. What work do I do? How am I contributing to the product delivery?
Well, a huge retailer company like Walmart doesn't have all data ready in one place. Even if you figure out the data availability, it is not going to be structured or super reliable. The data is always prone to a lot of bias and noise. Even if you find a clean source of data, you need to take the challenge of figuring out where and how to store the data, think about the amount of data a retail company like Walmart gets every single day.
This is where a Data Engineer is needed. Data Engineer creates automated solutions to build data pipelines to organise, preprocess, store, retrieve, structurise the data from its raw form to the information form which can be extracted by software engineers for their developent purpose. Data Engineers take up the duty of managing real time flow of data from multiple sources and also on scheduling when the data pipelines should be triggered. Data Engineers holds the base for the entire product's foundation and growth.
=> The complete picture
As you can see now, there are so many divisions of engineers working from different aspects to successfully deliver a product. It is never a single engineer's victory. A coherent work of all these engineers together makes a product. Developers from all these categories come together as a team to build a product. If the product wins, the team wins else the entire team fails. It is absolute magic to see how these developers work together without competting with each other for credits to create huge successful products that helps multiple people in unprecedented ways possible.
It is crucial for anyone to undertstand the perpective of other engineers to provide them with the best for a successful product.
Whether you're a data enthusiast, a business leader, or simply curious about the inner workings of the digital world, I invite you to appreciate the silent brilliance of engineering and how multiple pieces of engineering are put together to solve a puzzle. Share your thoughts, questions, or experiences in the comments below, and let's continue unraveling the mysteries of the data universe together. Until next time!