Posts

Showing posts from June 8, 2024

[Day 159] Learning and using prefect for MLOps orchestration

Image
 Hello :) Today is Day 159! A quick summary of today: did Module 3 of the MLOps zoom camp from the 2023 cohort  that uses Prefect Before everything else, I finally got the notebook expert title on Kaggle ! :party: As for prefect ~ Github repo from today's study is here . A common MLOps workflow Where we take data from a database into pandas, save it (checkpoint), reload with parquet, then maybe use sklearn for feature engineering and or running models, mlflow is there to track experiments, and finally the model is served.  However, we might have failure points at any of the arrows (connections) between steps.  Prefect comes in when we give an engineer the following tasks: could you just set up this pipeline to train this model? could you set up logging? could you do it every day? could you make it retry if it fails? could you send me a message when it succeeds? could you visualise the dependencies? could you add caching? could you add collaborators to run ad hoc - who don't cod