Guidelines to Get a Data Engineer Job Against the Odds

Data engineers are responsible for managing big data sets and building applications to move, transform and analyze them. They work closely with business analysts, data analysts, data scientists, and project managers to build solutions that help businesses make sense of their data. If you want to understand deeper what the data engineer functions there are several articles and videos explaining it but I advise you to read this article with a video from Seattle Data Guy and this article from Coursera. But be aware of not mixing concepts by falling into the common cliché of saying that data engineers are similar to plumbers because both work with pipelines (see my article about this).

For someone without an IT background (software engineering, computer science technical courses, or similar), getting a data engineer job may not be an easy task and straightforward. Even for someone with an engineering master’s degree or another technical background, it can be difficult to be accepted by companies that need data engineers.

In this article, I am going to present the following:

• Why is it so hard to get a data engineer job without an IT background?
• Five guidelines to get a data engineer job

1. Why is it so hard to get a data engineer job without an IT background?

There is a shortage of data engineers but several companies need this professional leading to a mismatch between the number of data engineers needed and the available candidates. Even with the job market unbalanced, it is weird to think why a professional without an IT background can’t easily get a data engineer job.

Why is this happening? For several reasons:

• A data engineer is not typically an entry-level position. Many employers look for data engineers with two or more years of experience in analytics, IT management, computer programming, or a role-related field. And getting into an IT job without a computer science background is difficult;

• Because data engineering is the intersection of software engineering and data science, some employers prefer candidates with at least a bachelor’s degree in a related field such as computer science or data science;

• There are not many specific specializations or pos-degrees in data engineering. Only now colleges and universities are attracting more interest in the study of data engineering by offering specializations and bachelor’s degrees in the field. Here are some exceptions online, Data Engineering Zoomcamp and Learn Data Engineering.

• Data engineer jobs are relatively new, and the industry is rapidly changing, with new technologies being developed all the time. This is excellent for someone already familiar with the area, but it will be quite perplexing for a newcomer. (“The programming language A is better than the B”, “The cloud provider X is the best” 😨)

• Many companies give preference to hard skills rather than soft skills. So even if a professional is very dedicated and willing to learn it will be set aside for someone with some knowledge in IT;

• Prejudice and bias. 😐 It is weird to think about it but there is still some bias among IT professionals against non-IT background professionals. It is strange to happen this in 2022 but it still happens (It occurred to me more than once…);

I will explain in the next section how you can try to overcome all these problems and get a data engineer job against the odds.

2. Some guidelines to get a data engineer job

2.1. Education, education, education!!!

As previously said, one barrier to obtaining a position as a data engineer for someone without IT experience is the lack of IT hard skills/knowledge. So the solution is “simple,” but laborious: you must complete a large number of courses, boot camps, or other sorts of education that any data engineer requires. A comprehensive and well-designed course/specialization or Bootcamp will provide you with the essential tools you need to become a data engineer. This is a great option if you don’t want to go to college (and honestly, it’s expensive).

“But which topics should I learn to work as a data engineer?” you may ask. There are numerous articles, websites, and publications that detail the courses of study required to become a data engineer. One great article is this Data Engineering Roadmap For 2021 from Ben Rogojan (AKA Seattle Data Guy) or if you want something more visual you also have this Modern Data Engineer Roadmap by data stack. tv.

In my humble point of view, you should first learn:

• Coding: This is a very important skill in data engineering. My favorite programming language is Python but Scala and Java are also widely used in data engineering. Choose one and learn the basics, the more important modules, and something of programming object-oriented;
• SQL and database fundamentals: SQL is the most important database language and it is essential in data engineering. First, you should learn the basics of DML and DDL, then understand how a relational database is organized, and in an advanced way you should learn data warehousing;
• Tools provided in clouds: To be a good data engineer, you should be able to work with tools available from cloud providers (AWS, Azure, Google Cloud Platform). Nowadays it is important to know how to work with cloud storage (eg. AWS S3, Google Storage), with databases on the cloud (eg. Redshift, BigQuery), with tools for data orchestration (eg. Azure Data factory) or with tools regarding Spark (eg. Databricks). I advise you to know, at least, which tools the main three cloud providers have to offer;
• Some other software knowledge: You should also know some bash commands, how to work with version control, and some knowledge regarding the basics of IT.

To learn all these contents you can access the multiple free MOOC platforms available. If you don’t know which one to choose from see my favorites here.

2.2. Show yourself with data projects

Like in the meme presented before you know it is harder to get a job without experience but you will get experience with a job. So, how do we break this negative cycle? By doing some personal projects and creating a portfolio.

Data engineering job applicants are frequently asked about their projects. If you’ve never worked as a data engineer, you can explain a project you worked on for a course or submitted to GitHub after doing some IT challenges like HackerRank or LeetCode.

You can also volunteer for some associations that do IT jobs pro-bono. For example, I did a web scraper for the Data Science For Social Good Portugal.

Also, a good idea is to “show yourself” with certificates and certifications. In this article, I explained which are the best.

2.3. Be the best friend with networking

We have a popular saying in Portugal “Who has friends will not die in prison!” that says that if you have a strong network you will have great advances in life (unfortunately some people take that saying to the extreme😒 ).

How to build and maintain a solid network?

• First of all, you need to follow Linkedin the best professionals in data science and data engineering. If you don’t know who they are check my article here.;

• Then you should use several platforms to show yourself. For example, after an achievement writes on Twitter: “Hello everyone, today I finished my 100 HackerRank challenge”;
• You must connect to recruiters or professionals in companies you like. After the first connection it is important to keep in touch with them;
• Last but not least, In platforms with professional purposes like Linkedin or Landing. jobs you must always (!!!!) keep your CV updated.

2.4. Learn from failure

This is a very important guideline because if you think like this you will always win, it is a win-win situation. Just remember that learning from failure means that sometimes you win and sometimes you learn something that will lead to winning in job seeking.

How can this be applied in job searching?

When looking for a new job and you fail you have to take the time to reflect on the “WHY” you were not successful in a particular job search and to use that information to make changes and improvements in your approach. This might involve analyzing the feedback you received from employers, assessing your job search strategy, and identifying areas where you can improve your skills or qualifications.

In my case, I used this principle, especially in the technical interviews. When, during the technical interview, I was asked one theme of data engineering that I didn’t know how to answer I went to study that theme for the next technical interview.

2.5 Be resilient, persistent, patient, and never give up

I finished my Master’s in Engineering in 2007, and only in 2017, I got my first good and stable engineering job. During those 10 years, I’ve been unemployed, in a foreign country, working in a non-technical area, etc… You can see the history of my professional experience in my article Every Time I am Cleaning the Dishes I Remember Norway …

If you read my article you may be thinking I am an optimistic and positive person but… “Au Contraire My Friend”, I am a very pessimistic person and always with negative thoughts but I am not a quitter!! 😁I was able to achieve my current position because I was very persistent, I had to be patient and resilient.

Resilience and persistence are important when looking for a new job because the process can be challenging and unpredictable. It will be common for you to face rejection or to have to apply to many jobs before finding the right fit. Being resilient can help you stay motivated and keep trying, even when faced with setbacks.

Overall, being resilient and persistent can be a valuable asset in the job search process and in your career.

Summary

Getting a data engineer job may not be an easy task and straightforward, especially for someone without an IT background.

In this article, I presented to you the main reasons why getting a data engineer position is hard and how you can surpass some difficulties by following these five pieces of advice:

1. Education, education, education!!!

2. Show yourself with data projects

3. Be the best friend with networking

4. Learn from failure

5. Be resilient, persistent, patient, and never give up

And you, are you trying to get a data engineer position?

Are you following these pieces of advice?

Good luck 😉

Did you like this article? Follow me for more articles on Medium.