50+ days of Machine Learning

 Hello :) Today is Day 176! A quick summary of today: finished Module 4 : analysis engineering and using dbt from the data engineering zoomcamp A preview of what I created in the end Continuing from yesterday with dbt ~ First I learned about testing and documenting dbt models We need to make sure the data we deliver to the end user is correct, and how do we make sure that we are not building our dbt models on top of incorrect data? dbt tests assumptions that we make about our data tests in dbt are essentially a select sql query these assumptions get compliled to sql that returns the amount of failing records tests are defined on a column in the .yml file dbt provides basic tests to check if the column values are: unique, not null, accepted values, a foreign key to another table we can create custom tests as queries Before writing tests, to ensure our data's schema is correct we can autogenerate it using a package First, include the package in packages.yml (and run dbt deps to insta...

 Hello :) Today is Day 175! A quick summary of today: basic SQL in BigQuery continued Module 4: analytics engineering and dbt  from the data eng camp and learned about dbt There were some bits from Module 3 that I did not finish. About using BigQuery to create a partitioned and clustered dataset and see the impact it has on processed data. Using the infamous nyc taxi dataset, I executed some simple queries on creating external datasets in BigQuery and could see the effect of partitioning and clustering First create a non-partitioned and partitioned table in BigQuery The dataset is by dates so we are using one of the datetime columns to partition by it. Below we can see a significant drop in processed data when doing a simple WHERE between dates on the non-partitioned vs partitioned tables. I faced an issue here because when creating the partitioned data, in BigQuery there is a table details section where it says 0 partitions. And I was really confused because the data I was pa...