50+ days of Machine Learning

 Hello :) Today is Day 160! A quick summary of today: simple data engineering pipeline with prefect tons of trouble learning about orchestration with mage.ai After yesterday's journey with prefect the youtube algorithm recommended me another tutorial for prefect  - this time for creating data pipelines with prefect. So I decided to give it a go.  What is data engineering? data scientists can do data engineering, but in specific cases where the two jobs cannot or are not needed to be separate data engineers build databases, they build lots of data pipelines and manage infrastructure (also care about cost, security) What are data pipelines? ETL(ELT)/batch pipelines that move data from A to B databases, APIs, files streaming pipelines - as data comes in, we consume that data and send it wherever it needs to go  message queues, polled data The main github repo used is here . After some basic setup, when we run 'pipeline' in the terminal which runs the main.py file: In pr...

 Hello :) Today is Day 159! A quick summary of today: did Module 3 of the MLOps zoom camp from the 2023 cohort  that uses Prefect Before everything else, I finally got the notebook expert title on Kaggle ! :party: As for prefect ~ Github repo from today's study is here . A common MLOps workflow Where we take data from a database into pandas, save it (checkpoint), reload with parquet, then maybe use sklearn for feature engineering and or running models, mlflow is there to track experiments, and finally the model is served.  However, we might have failure points at any of the arrows (connections) between steps.  Prefect comes in when we give an engineer the following tasks: could you just set up this pipeline to train this model? could you set up logging? could you do it every day? could you make it retry if it fails? could you send me a message when it succeeds? could you visualise the dependencies? could you add caching? could you add collaborators to run ad hoc - wh...