50+ days of Machine Learning

 Hello :) Today is Day 217! A quick summary of today: talking about the KB project reading papers related to bank telemarketing models Today my KB project partner and I met to catch up. We looked over what he had done during the past week. In addition, we talked about what happens after developing a model in a jupyter notebook - creating a script that takes a new row of features, preprocessing them, and then putting that through the model to get a prediction for that new case. Then we went over the below graph Giving him an overview of what each technology does. He is going on vacation from tomorrow to Friday, so the tasks left are to write up the project ppt that is needed for the submission in Korean.  In addition, for reproducability - I noticed that the kafka-producer uses local data to send transactions. For better reproducability, I uploaded the dataset to HuggingFace and used in the code I am reading the csv from the HF url I also read some papers related to bank term d...

 Hello :) Today is Day 195! One of my lab mates sent me a few papers to skim through to help for his team's project related to predicting bank deposit subscriptions. Thanks to ChatGPT skimming is very easy now. Below are the outputs from ChatGPT on the five papers I got. Predictive Analytics and Machine Learning in Direct Marketing for Anticipating Bank Term Deposit Subscriptions Introduction: Direct marketing is essential for personalized client communication in banking. Predictive analytics and machine learning offer new opportunities for refining marketing strategies. The research aims to enhance direct marketing's effectiveness by applying sophisticated analytical models. Literature Review: Examines eight studies on machine learning and data mining in banking. Highlights methodologies like the S_Kohonen network, Improved Whale Optimization Algorithm, META-DES-AAP, and various machine learning models. Emphasizes the importance of time deposits, customer credit products, and ...

 Hello :) Today is Day 194! A quick summary of today: I finished the paper for which I read many papers and posted them throughout May/June/July Since May I have been talking about reading research related to predicting OD demand matrix using either graph neural networks or next-frame (video) prediction models. Well fast forward to today ~ and I finished it. Everything is on my github repo . I ran data through 3 models: historical average, ConvLSTM, PredRNN - HA the most common baseline, and ConvLSTM and PredRNN - two of the best next-frame prediction models. Here is the abstract of the paper: Predicting taxi demand is essential for managing urban transportation effectively. This study explores the application of next-frame prediction models—ConvLSTM and PredRNN—to forecast Origin-Destination (OD) taxi demand matrices using a concatenated dataset of NYC taxi data from early 2024. ConvLSTM achieved an RMSE of 1.27 with longer training times, while PredRNN achieved 1.59 with faster t...

 Hello :) Today is Day 170! A quick summary of today: decided to give mage.ai another go and now use it alongside GCP Well... I obviously did not suffer enough with the countless problems I experienced when I was first learning about mage from DataTalksClub's MLOps zoomcamp, so I decided to do a cool-looking data engineering project [ youtube ]. The caveat is that I will use GCP. And I just hope I do not incur any major costs. I checked, and I have 21 days left on plenty of free credits.  Getting to the project It uses the infamous NYC taxi dataset.  Using lucid I learned a bit amout data deminsion modelling Then using python, some basic preprocessing was done on the raw data, to convert it into the top 8 tables. I actually put everything so far on my github , and plan on doing a nice readme documentation once everything is finished. Even though I started it after work today, I did not finish it because of *again* mage problems.  Before I get to the mage problems, a ...

 Hello :) Today is Day 169! A quick summary of today: started writing introduction for my paper attended day 2 of IEUK Firstly, in regards to my paper I decided to give overleaf a go. It is a popular latex editor so I went with it. I will share with white covering most of the part I bit. I managed to write a whole page. Not sure if that is a lot, but I am quick to recall what I have read. Every time I wrote a university assignment or previous short research proposals, I end up changing the sections a lot. It is pretty normal (for me at least) that the initially written version is never the final version. Nevertheless, I also learned about bibtex and how its used for citations and referencing. At the moment I am thinking to give myself time until the end of June to write this intro + literature review + data plan + expected results, and then submit to my professor for review. And it would be good to do it at the end of June since it is exam season now for him.  (I am also a lit...

 Hello :) Today is Day 165! A quick summary of today: started using mlflow to track experiments for my research in the lab completed MLOps zoomcamp Module 4 and its homework Firstly, about experiment tracking with mlflow I got a feeling that I will run lots of models from now on with different parameters, and hey, I have been learning and using this cool library called mlflow in my studies - this would be a great opportunity to use it in a real scenario.  So I set it up and just started running models. The training is a bit slow because my GPU is not that strong, but I found ways to get free GPU hours. I also want to find a way to host a simple sqlite db to use for the experiment trackings/artifact store.  My initial thoughts on sharing code that I will use for my research - I will definitely do it. I do not like that many papers do not share it for one reason or another, but for now I will keep it closer to my chest while I am still replicating base models.  Second...

 Hello :) Today is Day 163! A quick summary of today: started concentrating more on OD demand matrix papers trying to recreate baseline models for OD demand matrix prediction Firstly, the papers I read Deep Multi-View Spatiotemporal Virtual Graph Neural Network for Significant Citywide Ride-hailing Demand Prediction [ link ] Introduction In spatial-temporal deep learning, two main spatial data representation methods are used: image-based and graph-based. The image-based approach grids urban areas by latitude and longitude, with statistical data as pixel values for CNN models. This approach struggles with data sparsity at high granularity and loss of detail at low granularity. The graph-based approach, used for defined networks like roads, captures dynamics via GCN models but has limited structured data access and transferability. The paper proposes a method using high-granularity grids of urban areas, discarding sparse regions, and retaining significant demand signals to create vir...

 Hello :) Today is Day 162! A quick summary of today: yes! I finally completed the 03-orchestration part from the MLOps zoomcamp downloaded the NYC taxi dataset and started learning how to turn it into a graph dataset Firstly, regarding mage.ai I cannot express how glad I am, to finish *completely* module 3. Here is my repo . So many errors and deviations from what is shown on the videos (which were recorded 2 weeks ago...) I am speechless. Nevertheless, I finally did it all. Today I figured out some of the errors related to AWS authentication, and managed to deploy an instance to AWS using github actions. I hope I can use it, or what is probably better - use what I learned today as a template for an actual production-ready deployment. Because right now some things like database info and passwords are on github.  I played around with deployments ~. Next will be module 4 - deployment.  Secondly, about what I did in the lab Firstly, today I read a nice paper:  PGCN: Pr...

 Hello :) Today is Day 161! A quick summary of today: explored how Generative Adversarial Networks can be used for OD demand matrix generation found that the original graph attention network has limitations and read the paper for GATv2 First paper - A GAN framework-based dynamic multi-graph convolutional network for origin–destination-based ride-hailing demand prediction [ ScienceDirect ] Introduction OD-based prediction focuses on estimating travel demand between specific origin and destination regions, which is useful for understanding inter-regional travel patterns. This approach requires consideration of complex spatio-temporal correlations, making it more challenging than region-level prediction. Although some progress has been made in OD-level prediction, several key issues remain: Practical OD Demand Forecasting: Previous studies often only consider relationships between OD pairs, rather than between OD regions. Accurate OD demand matrices and inter-regional relationshi...

 Hello :) Today is Day 158! A quick summary of today: created a better dataset preprocessing 'pipeline' for new audio files read a bit about Mixture density networks and their application Firstly, about the Scottish dataset The latest dataset has ~50 minutes worth of Glaswegian (Scottish) accent clips. Amazing ^^ [ huggingface link ] I also finetuned microsoft's SpeechT5 on this latest data, but I am still getting a bit robotic outputs. I need to play more around with the trainer setup.  As for the 'pipeline' ~  It starts with renaming the audio files (so that we have some kind of tracking), I rename them by adding the preprocess date. Second is a bit longer, so I will just paste the execute function: create_audio_metadata_csv(transcriptions_csv, filenames_df, audio_files_path, output_csv) It takes a transcription csv that has 1 column with transcriptions, a 2nd csv with file names, path to the audio files, and where to output the csv which contains file_name, trans...