50+ days of Machine Learning

 Hello :) Today is Day 220! A quick summary of today: read chapter 2 from the book 'Fundamentals of Data Engineering' transferred more posts onto the new blog UI What Is the Data Engineering Lifecycle? The data engineering lifecycle by getting data from source systems and storing it. Next, we transform the data and then proceed to our central goal, serving data to analysts, data scientists, ML engineers, and others. In reality, storage occurs throughout the lifecycle as data flows from beginning to end. There are 5 stages: Generation Storage Ingestion Transformation  Serving data Generation: Source Systems Sources produce data consumed by downstream systems, including humangenerated spreadsheets, IoT sensors, and web and mobile applications. Each source has its unique volume and cadence of data generation. A data engineer should know how the source generates data, including relevant quirks or nuances. Data engineers also need to understand the limits of the source systems they...

 Hello :) Today is Day 193! A quick summary of today: covered chapter 5,6, and 7 from Effective Data Science Infrastructure Chapter 5: Practicing scalability and performance Effective infrastructure must accommodate a wide range of projects. Rather than adopting a one-size-fits-all approach, it should offer a versatile toolbox of robust methods to achieve adequate scalability and performance. To enhance organizational scalability and ensure projects are comprehensible to the largest audience, our primary strategy is simplicity. Given that people's understanding is limited overengineering and overoptimizing can cause extra costs. Vertical scalability it refers to the idea of handling more compute and larger datasets just by using larger instances To start things, we begin with a skeleton flow, and then keep adding new things till we get to the final solution.  The model uses Yelp review data and the goal is to group reviews together to find what kind of reviews are general...

 Hello :) Today is Day 192! A quick summary of today: set up AWS resources and configured metaflow with aws (finally) What is Scalability? Scalability refers to a system's ability to handle increasing amounts of work by adding resources. This concept is distinct from performance, which measures how well a system functions under a fixed workload. Scalability involves: 1. Growth: It’s relevant only when discussing systems that need to handle more work. 2. Efficient Resource Use: Adding resources like more computers or memory should proportionally increase the system's capacity. 3. Different Measures: The dimensions of scalability (e.g., volume, velocity, validity, and variety) must be defined based on specific needs. When building infrastructure, it’s crucial to ensure scalability across all layers, supporting a large number of applications and users, and enabling quick development and deployment of data science projects. Culture of Experimentation Modern data science organizatio...

 Hello :) Today is Day 191! A quick summary of today: today I started a book from Manning - Effective Data Science Infrastructure Below is a summary of the topics covered up to and including chapter 3. To enhance data scientists' productivity, prioritize a well-structured development environment over solely focusing on scalable production infrastructure. Many companies overlook this, managing code development, debugging, and testing haphazardly. Effective development environments, integral to infrastructure, enhance productivity by focusing on ergonomics, defined as optimizing efficiency in the working environment. Key activities to optimize include: 1. Prototyping - Translating expertise into functional code and models. 2. Interaction with Production Deployments - Connecting code and models to systems to generate business value. The prototyping loop (similar to the REPL loop in software engineering) involves developing, evaluating, and analyzing code iteratively. To boost pro...