50+ days of Machine Learning

 Hello :) Today is Day 197! A quick summary of today: finally decided to learn about kafka and streamed data I started the day by watching and replicating this video on  Building a Real-Time Data Streaming Pipeline using Kafka,Postgres and Streamlit Kafka, and in general streaming (data/models) have been in my zone of interest for a while, and this video finally gave me a good starting point. The video helped me set up a Kafka service, a Zookeeper (which manages metadata), a producer (which reads data from a streaming source like a live weather feed and sends it to Kafka), and a consumer (which reads data from Kafka).  The whole thing was around sentiment of generated sentences. First we generate sentences, send them to kafka using the producer, read them from kafka using the consumer and use a sentence sentiment analyser to get a score -> then upload the sentence + score to postgres -> then as each new data comes in, show it in a UI (streamlit). I took some pics b...

 Hello :) Today is Day 196! A quick summary of today: Today I found this great resource for MLOps. Below I will summarise the posts I read A better pic of the above is on my repo . I learned about the above 'ML Canvas' concept from the below resources. Motivating MLOps Why MLOps? Machine learning (ML) models are increasingly being used in production environments, but their development and deployment are often disconnected from traditional software development and operations practices. This disconnection leads to various pain points, such as: Lack of collaboration: Data scientists, engineers, and operators work in silos, leading to inefficiencies and errors. Inconsistent workflows: Ad-hoc processes and manual interventions hinder reproducibility, scalability, and maintainability. Inadequate infrastructure: Insufficient infrastructure and tools lead to difficulties in deploying, monitoring, and updating ML models. Motivation for MLOps To address these challenges, MLOps aims to b...

 Hello :) Today is Day 195! One of my lab mates sent me a few papers to skim through to help for his team's project related to predicting bank deposit subscriptions. Thanks to ChatGPT skimming is very easy now. Below are the outputs from ChatGPT on the five papers I got. Predictive Analytics and Machine Learning in Direct Marketing for Anticipating Bank Term Deposit Subscriptions Introduction: Direct marketing is essential for personalized client communication in banking. Predictive analytics and machine learning offer new opportunities for refining marketing strategies. The research aims to enhance direct marketing's effectiveness by applying sophisticated analytical models. Literature Review: Examines eight studies on machine learning and data mining in banking. Highlights methodologies like the S_Kohonen network, Improved Whale Optimization Algorithm, META-DES-AAP, and various machine learning models. Emphasizes the importance of time deposits, customer credit products, and ...

 Hello :) Today is Day 194! A quick summary of today: I finished the paper for which I read many papers and posted them throughout May/June/July Since May I have been talking about reading research related to predicting OD demand matrix using either graph neural networks or next-frame (video) prediction models. Well fast forward to today ~ and I finished it. Everything is on my github repo . I ran data through 3 models: historical average, ConvLSTM, PredRNN - HA the most common baseline, and ConvLSTM and PredRNN - two of the best next-frame prediction models. Here is the abstract of the paper: Predicting taxi demand is essential for managing urban transportation effectively. This study explores the application of next-frame prediction models—ConvLSTM and PredRNN—to forecast Origin-Destination (OD) taxi demand matrices using a concatenated dataset of NYC taxi data from early 2024. ConvLSTM achieved an RMSE of 1.27 with longer training times, while PredRNN achieved 1.59 with faster t...

 Hello :) Today is Day 193! A quick summary of today: covered chapter 5,6, and 7 from Effective Data Science Infrastructure Chapter 5: Practicing scalability and performance Effective infrastructure must accommodate a wide range of projects. Rather than adopting a one-size-fits-all approach, it should offer a versatile toolbox of robust methods to achieve adequate scalability and performance. To enhance organizational scalability and ensure projects are comprehensible to the largest audience, our primary strategy is simplicity. Given that people's understanding is limited overengineering and overoptimizing can cause extra costs. Vertical scalability it refers to the idea of handling more compute and larger datasets just by using larger instances To start things, we begin with a skeleton flow, and then keep adding new things till we get to the final solution.  The model uses Yelp review data and the goal is to group reviews together to find what kind of reviews are general...

 Hello :) Today is Day 192! A quick summary of today: set up AWS resources and configured metaflow with aws (finally) What is Scalability? Scalability refers to a system's ability to handle increasing amounts of work by adding resources. This concept is distinct from performance, which measures how well a system functions under a fixed workload. Scalability involves: 1. Growth: It’s relevant only when discussing systems that need to handle more work. 2. Efficient Resource Use: Adding resources like more computers or memory should proportionally increase the system's capacity. 3. Different Measures: The dimensions of scalability (e.g., volume, velocity, validity, and variety) must be defined based on specific needs. When building infrastructure, it’s crucial to ensure scalability across all layers, supporting a large number of applications and users, and enabling quick development and deployment of data science projects. Culture of Experimentation Modern data science organizatio...

 Hello :) Today is Day 191! A quick summary of today: today I started a book from Manning - Effective Data Science Infrastructure Below is a summary of the topics covered up to and including chapter 3. To enhance data scientists' productivity, prioritize a well-structured development environment over solely focusing on scalable production infrastructure. Many companies overlook this, managing code development, debugging, and testing haphazardly. Effective development environments, integral to infrastructure, enhance productivity by focusing on ergonomics, defined as optimizing efficiency in the working environment. Key activities to optimize include: 1. Prototyping - Translating expertise into functional code and models. 2. Interaction with Production Deployments - Connecting code and models to systems to generate business value. The prototyping loop (similar to the REPL loop in software engineering) involves developing, evaluating, and analyzing code iteratively. To boost pro...