50+ days of Machine Learning

 Hello :) Today is Day 169! A quick summary of today: started writing introduction for my paper attended day 2 of IEUK Firstly, in regards to my paper I decided to give overleaf a go. It is a popular latex editor so I went with it. I will share with white covering most of the part I bit. I managed to write a whole page. Not sure if that is a lot, but I am quick to recall what I have read. Every time I wrote a university assignment or previous short research proposals, I end up changing the sections a lot. It is pretty normal (for me at least) that the initially written version is never the final version. Nevertheless, I also learned about bibtex and how its used for citations and referencing. At the moment I am thinking to give myself time until the end of June to write this intro + literature review + data plan + expected results, and then submit to my professor for review. And it would be good to do it at the end of June since it is exam season now for him.  (I am also a lit...

 Hello :) Today is Day 168! A quick summary of today: attended day 1 (of 4) of Internship Experience UK : Technology by Bright Network did MLOps zoomcamp's homework 5 on model monitoring discovering LLM zoomcamp For completeness: here is the full schedule, and there are various sessions by top employers in the UK on building up our careers. Below are some notes from today's sessions Session on leadership skills with Deloitte creative problem-solving identify challenges generate solutions making mistakes skills in practice empowering your team and embracing responsibility encourage diverse perspectives build trust accountability and reliability there was a task where we had to think about being a leader in a 2-week deadline merger&acquisition case - how to organise, manage communication, strategy and deadlines using the Eisenhower matrix for task management Session on transferable skills by ICAEW teamwork people skills decision making & problem solving professionalism c...

 Hello :) Today is Day 167! A quick summary of today: started Module 5: Monitoring of the MLOps zoomcamp read a bit of Multivariate statistical methods: a primer (which I bought yesterday) Firstly, about ML model monitoring ML monitoring involves looking at things like service health, model performance, data quality and integrity, data and concept drift, performance by segment, model bias/fairness, outliers, explainability.  Here are the materials over which I studied today. Using evidently's python library we can great nice looking reports and dashboards like the below.  In this, 2 created reports can be seen.  Below is one of the default suggested dataset summary tables Along with variable info And this is an example of a simple dashboard we can use to monitor specific items from our reports Secondly, some things from the book examples of multivariate data matrix algebra (vectors, matrices, eigenvalues/vectors) visualising multivariate data (draftsman's plot - sc...

 Hello :) Today is Day 166! A quick summary of today: attendend the 8th pseudocon by pseudolabs in Seoul went to a big book store and bought a cool little book  created a space on huggingface for fine-tuned whisper on our Glaswegian dataset Firstly, about the pseudocon It was decently big - many people. And was organised in the same Microsoft office area as the Microsoft Azure hackathon I attended a few weeks ago.  Some interesting bits from the sessions which covered a wide range - open source, genAI, AWS, stable diffusion and games.  Found out that firebase has free db hosting (again?). I used it back in 2018 when I was initially learning about webapp dev so I might give 'an old friend' a try again Found a nice community of AWS learners - startAWS There was a session on using AI to generate characters for games. It is a touchy topic among gamers and even more among artists, but it is something that as presented today 'can massively decrease cost' Also in the fin...

 Hello :) Today is Day 165! A quick summary of today: started using mlflow to track experiments for my research in the lab completed MLOps zoomcamp Module 4 and its homework Firstly, about experiment tracking with mlflow I got a feeling that I will run lots of models from now on with different parameters, and hey, I have been learning and using this cool library called mlflow in my studies - this would be a great opportunity to use it in a real scenario.  So I set it up and just started running models. The training is a bit slow because my GPU is not that strong, but I found ways to get free GPU hours. I also want to find a way to host a simple sqlite db to use for the experiment trackings/artifact store.  My initial thoughts on sharing code that I will use for my research - I will definitely do it. I do not like that many papers do not share it for one reason or another, but for now I will keep it closer to my chest while I am still replicating base models.  Second...

 Hello :) Today is Day 164! A quick summary of today: started Module 4: deployment from the MLOps zoomcamp re-creating a baseline OD demand matrix model - ConvLSTM Firstly, about module 4: deployment Code from today is on my github . The first part was about simple model packaging with pickle and serving the model through a flask api Second part was about using mlflow and saving the ran experiments to S3 on AWS The code is pretty much like the one in the pic above, but with the below added above it: RUN_ID = "" AWS_S3_BUCKET = "" logged_model = f's3://{AWS_S3_BUCKET}/1/{RUN_ID}/artifacts/model' model = mlflow.pyfunc.load_model(logged_model) And this is when I realised I am getting charged by AWS. It is only $9, but I cannot continue using it. I thought I am under some 'free tier' but nope... I started deleting all EC2, RDS, Load balancers, Users, everything I could see. But because I am not completely sure, I wanted to use some cloud-nuke tool - and...

 Hello :) Today is Day 163! A quick summary of today: started concentrating more on OD demand matrix papers trying to recreate baseline models for OD demand matrix prediction Firstly, the papers I read Deep Multi-View Spatiotemporal Virtual Graph Neural Network for Significant Citywide Ride-hailing Demand Prediction [ link ] Introduction In spatial-temporal deep learning, two main spatial data representation methods are used: image-based and graph-based. The image-based approach grids urban areas by latitude and longitude, with statistical data as pixel values for CNN models. This approach struggles with data sparsity at high granularity and loss of detail at low granularity. The graph-based approach, used for defined networks like roads, captures dynamics via GCN models but has limited structured data access and transferability. The paper proposes a method using high-granularity grids of urban areas, discarding sparse regions, and retaining significant demand signals to create vir...

 Hello :) Today is Day 162! A quick summary of today: yes! I finally completed the 03-orchestration part from the MLOps zoomcamp downloaded the NYC taxi dataset and started learning how to turn it into a graph dataset Firstly, regarding mage.ai I cannot express how glad I am, to finish *completely* module 3. Here is my repo . So many errors and deviations from what is shown on the videos (which were recorded 2 weeks ago...) I am speechless. Nevertheless, I finally did it all. Today I figured out some of the errors related to AWS authentication, and managed to deploy an instance to AWS using github actions. I hope I can use it, or what is probably better - use what I learned today as a template for an actual production-ready deployment. Because right now some things like database info and passwords are on github.  I played around with deployments ~. Next will be module 4 - deployment.  Secondly, about what I did in the lab Firstly, today I read a nice paper:  PGCN: Pr...

 Hello :) Today is Day 161! A quick summary of today: explored how Generative Adversarial Networks can be used for OD demand matrix generation found that the original graph attention network has limitations and read the paper for GATv2 First paper - A GAN framework-based dynamic multi-graph convolutional network for origin–destination-based ride-hailing demand prediction [ ScienceDirect ] Introduction OD-based prediction focuses on estimating travel demand between specific origin and destination regions, which is useful for understanding inter-regional travel patterns. This approach requires consideration of complex spatio-temporal correlations, making it more challenging than region-level prediction. Although some progress has been made in OD-level prediction, several key issues remain: Practical OD Demand Forecasting: Previous studies often only consider relationships between OD pairs, rather than between OD regions. Accurate OD demand matrices and inter-regional relationshi...