Posts

Showing posts from August 7, 2024

[Day 219] Fundamentals of Data Eng and LLM data preprocessing pipelines in Mage

Image
 Hello :) Today is Day 219! A quick summary of today: starting 'Fundamentals of Data Engineering' covered module 5: orchestration of the LLM zoomcamp When I woke up today I saw that DeepLearning.AI is launching a new course - DE Professional Certificate at the end of August. The instructor - Joe Reis, is one of the writers of the infamous 'holy book' of DE - Fundamentals of Data Engineering. Thankfully, I found the book officially published for free by Redpand . Below is a summary of what I read today (Chapter 1). What is Data Engineering? There is a lot of definitions of the term, but all have a similar idea. The book combines it all in this one: Data engineering is the development, implementation, and maintenance of systems and processes that take in raw data and produce high-quality, consistent information that supports downstream use cases, such as analysis and machine learning. Data engi‐ neering is the intersection of security, data management, DataOps, data arch