50+ days of Machine Learning

 Hello :) Today is Day 193! A quick summary of today: covered chapter 5,6, and 7 from Effective Data Science Infrastructure Chapter 5: Practicing scalability and performance Effective infrastructure must accommodate a wide range of projects. Rather than adopting a one-size-fits-all approach, it should offer a versatile toolbox of robust methods to achieve adequate scalability and performance. To enhance organizational scalability and ensure projects are comprehensible to the largest audience, our primary strategy is simplicity. Given that people's understanding is limited overengineering and overoptimizing can cause extra costs. Vertical scalability it refers to the idea of handling more compute and larger datasets just by using larger instances To start things, we begin with a skeleton flow, and then keep adding new things till we get to the final solution.  The model uses Yelp review data and the goal is to group reviews together to find what kind of reviews are general...

 Hello :) Today is Day 192! A quick summary of today: set up AWS resources and configured metaflow with aws (finally) What is Scalability? Scalability refers to a system's ability to handle increasing amounts of work by adding resources. This concept is distinct from performance, which measures how well a system functions under a fixed workload. Scalability involves: 1. Growth: It’s relevant only when discussing systems that need to handle more work. 2. Efficient Resource Use: Adding resources like more computers or memory should proportionally increase the system's capacity. 3. Different Measures: The dimensions of scalability (e.g., volume, velocity, validity, and variety) must be defined based on specific needs. When building infrastructure, it’s crucial to ensure scalability across all layers, supporting a large number of applications and users, and enabling quick development and deployment of data science projects. Culture of Experimentation Modern data science organizatio...

 Hello :) Today is Day 191! A quick summary of today: today I started a book from Manning - Effective Data Science Infrastructure Below is a summary of the topics covered up to and including chapter 3. To enhance data scientists' productivity, prioritize a well-structured development environment over solely focusing on scalable production infrastructure. Many companies overlook this, managing code development, debugging, and testing haphazardly. Effective development environments, integral to infrastructure, enhance productivity by focusing on ergonomics, defined as optimizing efficiency in the working environment. Key activities to optimize include: 1. Prototyping - Translating expertise into functional code and models. 2. Interaction with Production Deployments - Connecting code and models to systems to generate business value. The prototyping loop (similar to the REPL loop in software engineering) involves developing, evaluating, and analyzing code iteratively. To boost pro...