DataTalksClub Week 1: Data engineering essentials

Introduction

I have recently enrolled in the DataTalksClub Data Engineering zoom camp so will be posting a series of blogs detailing what I have learned and the journey along the way - enjoy!

In this post, I'll explain why I decided to start this course and share my experiences from the first week.


Why this course?

  • Completely free: This course is entirely free. Which is great for those starting out or who are budget conscious.
  • Beginner-friendly: The course assumes no prior knowledge of data engineering, although you are expected to know the basics of python.
  • Hands-on learning: You get the opportunity to complete an example project plus weekly homework exercises and code-alongs. Great for building up your Github profile!
  • Community support: DataTalksClub has a large and active community of learners and professionals. This is so handy if you are stuck on a problem or are even if you are looking for career advice.
  • Up to date: The content covers modern data engineering tools and the content is updated regularly.
  • Flexible schedule: The course is self-paced, so you can fit it around your existing commitments - ideal for those working full time. (Although there are deadlines for the (optional) homework.)


Week 1 overview: building the basics

The first week focused on setting up essential environments and becoming familiar with the fundamental tools involved in data engineering. The primary areas of focus were:

  • Docker: A platform designed to simplify the creation, deployment, and running of applications by using containers. Containers allow developers to package an application with all its dependencies, ensuring consistency across various environments. Please see a previous blog post I did all about Docker!
  • PostgreSQL: An open-source relational database system known for its robustness and versatility.
  • pgAdmin: An easy to use web based interface that facilitates the management and visualisation of PostgreSQL databases, making it easier to interact with and query data.
  • SQL: The standard language for querying and manipulating relational databases.
  • Terraform: An infrastructure as code tool that allows for the automation of infrastructure management. It enables the definition and provisioning of data centre infrastructure using a high-level configuration language.


Key takeaways from week 1

  • The Power of Containerisation with Docker: Docker's ability to create isolated environments ensures that applications run uniformly across different systems. Docker provides consistency for developers, which anyone who has ever worked on a team project will know is so important.
  • Managing Data with PostgreSQL and pgAdmin: Setting up a PostgreSQL database and interacting with it through pgAdmin provided hands-on experience in data management. The web based system of pgAdmin made the whole thing feel really similar to something like MySQL Workbench.
  • Attention to Detail is Key: Setting up the environments needed for the course took me a longer time than expected. The delays were mostly self-made, where I was rushing ahead or not paying attention to small details. Working within the terminal makes these mistakes feel even more frustrating when trying to solve them. A lesson learned for next week is to take my time!
Screenshot of pgAdmin SQL code
An example of the SQL queries done on the pgAdmin web tool




Reflections from a beginner

As is obvious from my other blog posts, I am a beginner in the tech world trying to up-skill as much as I can. Most of my past experience has been in the data analysis and visualisation space, but I am keen to go back a few steps and look at the data engineering side of things in 2025.

I struggled with some of the set up this week, mostly because of my lack of patience. Looking back now whilst writing this blog, I feel like I learned a lot from fixing the errors; it forced me to research more into what I was doing, rather than blindly copying. That is the great thing about coding/data - there is always something to learn and investigate. The feeling of having all systems connected and querying the database correctly was such a relief!


As I progress through the Data Engineering ZoomCamp, I look forward to delving deeper into more advanced topics and sharing my insights. Stay tuned!


For those interested in exploring the course further, more information is available on the DataTalks.Club website.



Comments

Post a Comment

Popular posts from this blog

Top tips for your first hackathon

Image
So you have taken the plunge and have signed up to your first hackathon or tech challenge! This blog post will provide you with the tips you need to make your first hackathon run as smoothly as possible, and will *hopefully* help lead you to a win! What is a hackathon? A hackathon is an event where developers, and other technical professionals, work together on a project over a relatively small timeframe. The specifics vary from hackathon to hackathon: Some last 24 hours, others two months. Some are team based whilst others are individual. Some allow prep time and others do not allow any prior preparation. Taking part in a hackathon is a great way to develop your technical and soft skills and build a portfolio of projects to show off in the future. Hackathons are also a great way to practice working collaboratively, which is an essential skill for any developer. So on to the tips… Before the hackathon Read over the rules and requirements of the hackathon properly.  Some hackathons...
Read more

Microsoft Certifications: Advice on Achieving Exam Success

Image
Welcome back to The Tech Dive! It has been a while since the last blog as I spent the last few months of 2024 studying for two Microsoft certifications. I am now back in the blog space and thought my experience of studying for, and passing, two certifications in short succession would make an interesting first blog back. Introduction So what are Microsoft certifications? In a nutshell, they are qualifications offered by Microsoft that show that the holder is proficient in certain Microsoft technologies and technical skills. There are many different certifications and certification pathways (groups of similar certifications) for all different skill points. Microsoft certifications are a great way to evidence your technical skills to your current or potential future employers. In fact many employers will fund the cost of Microsoft certifications for employees! It is definitely worth investigating this with your employer; you could save on the entry cost of between £60 and + £200. This bl...
Read more