50+ days of Machine Learning

 Hello :) Today is Day 162! A quick summary of today: yes! I finally completed the 03-orchestration part from the MLOps zoomcamp downloaded the NYC taxi dataset and started learning how to turn it into a graph dataset Firstly, regarding mage.ai I cannot express how glad I am, to finish *completely* module 3. Here is my repo . So many errors and deviations from what is shown on the videos (which were recorded 2 weeks ago...) I am speechless. Nevertheless, I finally did it all. Today I figured out some of the errors related to AWS authentication, and managed to deploy an instance to AWS using github actions. I hope I can use it, or what is probably better - use what I learned today as a template for an actual production-ready deployment. Because right now some things like database info and passwords are on github.  I played around with deployments ~. Next will be module 4 - deployment.  Secondly, about what I did in the lab Firstly, today I read a nice paper:  PGCN: Pr...

 Hello :) Today is Day 161! A quick summary of today: explored how Generative Adversarial Networks can be used for OD demand matrix generation found that the original graph attention network has limitations and read the paper for GATv2 First paper - A GAN framework-based dynamic multi-graph convolutional network for origin–destination-based ride-hailing demand prediction [ ScienceDirect ] Introduction OD-based prediction focuses on estimating travel demand between specific origin and destination regions, which is useful for understanding inter-regional travel patterns. This approach requires consideration of complex spatio-temporal correlations, making it more challenging than region-level prediction. Although some progress has been made in OD-level prediction, several key issues remain: Practical OD Demand Forecasting: Previous studies often only consider relationships between OD pairs, rather than between OD regions. Accurate OD demand matrices and inter-regional relationshi...

 Hello :) Today is Day 160! A quick summary of today: simple data engineering pipeline with prefect tons of trouble learning about orchestration with mage.ai After yesterday's journey with prefect the youtube algorithm recommended me another tutorial for prefect  - this time for creating data pipelines with prefect. So I decided to give it a go.  What is data engineering? data scientists can do data engineering, but in specific cases where the two jobs cannot or are not needed to be separate data engineers build databases, they build lots of data pipelines and manage infrastructure (also care about cost, security) What are data pipelines? ETL(ELT)/batch pipelines that move data from A to B databases, APIs, files streaming pipelines - as data comes in, we consume that data and send it wherever it needs to go  message queues, polled data The main github repo used is here . After some basic setup, when we run 'pipeline' in the terminal which runs the main.py file: In pr...

 Hello :) Today is Day 159! A quick summary of today: did Module 3 of the MLOps zoom camp from the 2023 cohort  that uses Prefect Before everything else, I finally got the notebook expert title on Kaggle ! :party: As for prefect ~ Github repo from today's study is here . A common MLOps workflow Where we take data from a database into pandas, save it (checkpoint), reload with parquet, then maybe use sklearn for feature engineering and or running models, mlflow is there to track experiments, and finally the model is served.  However, we might have failure points at any of the arrows (connections) between steps.  Prefect comes in when we give an engineer the following tasks: could you just set up this pipeline to train this model? could you set up logging? could you do it every day? could you make it retry if it fails? could you send me a message when it succeeds? could you visualise the dependencies? could you add caching? could you add collaborators to run ad hoc - wh...