Top Work Duties of a Data Engineer Working at a Large Company
Personal Account and Frequency Analysis of Typical Work Tasks
--
In this article, I will summarize and describe my top work duties as a data engineer. I work at a large bank and my team specializes in bringing machine learning models to production. I’ve been in this role for about 2.5 years and this is the beginning of my career as a data engineer. This study was meant to be a robust analysis where I record all the time I spent on various tasks at work. Unfortunately, I did not record what I was doing every minute so instead I’m giving a detailed account and frequency analysis of tasks as an approximation of a data engineer’s role.
Process:
At work I keep a list of tasks in the notes app on a daily basis. I took the entirety of these tasks from October 2020 to December 2021 (roughly my first year) and cleaned and processed them into distinct categories. I then analyzed the frequency of these tasks by category. Although I was not able provide and analyze the time spent on these tasks, I believe this list will be illuminating.
Findings:
This frequency analysis shows the most frequent work tasks I had as a first-year data engineer. The most frequent types of tasks were learning and coding.
Explanation of Findings:
In order to make sense of the tasks, I categorized them into buckets. This includes learning, coding, troubleshooting (including pipeline, code or integration), meetings, administrative (e.g. emails, submitting hours, etc), documentation, compliance tasks (including JIRA, mandatory training, and performance/career planning tasks), infrastructure requests, and visualization (e.g. building a dashboard).
Here’s a small example of raw data, i.e. actual tasks.
Things like coding, learning and troubleshooting are self-evident and probably what most people expect for a data engineering role. However, a great deal of my time was spent on administrative, documentation, compliance and…
