https://ivanstudyblog.github.io/ Hello :) After 220 days posting here, I am moving my blog to https://ivanstudyblog.github.io/ Please head onto there for the latest days The UI there is a bit more flexible and customisable so I will continue my learning journey there. That is all for today :) See you in the new blog.
Hello :) Today is Day 173! A quick summary of today: learned more about terraform and how to set up a GCP VM and connect to it locally used mage for some data engineering pipelines with GCP Last videos from Module 1: terraform variables , GCP set up Turns out there is a bit more of terraform from the data eng zoomcamp, and today I covered it. After learning how to connect to gcp using terraform and create a storage bucket, the first thing today was creating a bigquery dataset Adding the above to main.tf which now looks like: terraform apply, creates a demo_dataset as well Then I learned about variables in terraform Create a variables.tf file and put a variable like: and in main.tf we can directly use the created variables like: Great intro to terraform - being able to define infrastructure code, create resources, and destroy resources. The next part was an instruction on setting up GCP (cloud VM + SSH access) First was creating an ssh key locally And add it to the metadata in GCP'...
Hello :) Today is Day 186! A quick summary of today: using FastAPI for model serving, and running it in Docker starting to think about model monitoring All code, pictures are on my github repo . Firstly, yesterday I was having version issues between prefect and prefect-gcp - fixed that with the help of Prefect's community slack, and now I can run one file which runs both flows - upload_to_gcs flow and train_model flow. I also realised that for solo (and mini project) devs Prefect cloud is free, so I uploaded my flows to there just to be a bit safer - not having everything locally only. The UI is the same, but it is in the cloud. Next, serving the model for inference I decided to use FastAPI as it seems very easy to create and run a server. Also when I run a fastapi server, there is a nice /docs page where I can quickly and easily test my endpoints. It is pretty basic for now, and I just need to improve the prefict_fraud function. I also set up the server to run i...