[Day 173] Terraform, GCP, virtual machines, data pipelines

 Hello :)
Today is Day 173!


A quick summary of today:
  • learned more about terraform and how to set up a GCP VM and connect to it locally
  • used mage for some data engineering pipelines with GCP


Last videos from Module 1: terraform variables, GCP set up

Turns out there is a bit more of terraform from the data eng zoomcamp, and today I covered it.

After learning how to connect to gcp using terraform and create a storage bucket, the first thing today was creating a bigquery dataset

Adding the above to main.tf which now looks like:

terraform apply, creates a demo_dataset as well

Then I learned about variables in terraform

Create a variables.tf file and put a variable like:

and in main.tf we can directly use the created variables like:

Great intro to terraform - being able to define infrastructure code, create resources, and destroy resources.

The next part was an instruction on setting up GCP (cloud VM + SSH access)
First was creating an ssh key locally
And add it to the metadata in GCP's compute engine (hiding the username just in case)
Then create a VM, and connect to it locally using that ssh key, using `ssh -i ~/.ssh/gcp username@gcp_vm_external_ip`
For a quick connection to the VM, I set up a config which includes Host, HostName, User and IdentityFile, so now I can just run ssh `Host` and I am connected to the VM through my terminal. Nice.

Also set up vs code to connect to the created ssd


Then, I installed anaconda. 
And docker
Then docker-compose
And make it executable from anywhere by adding the below to .bashrc
`export PATH="${HOME}/bin:${PATH}"`
And now we have it 

Then installed pgcli with conda
(random note - I am using a vm from my local terminal like this for the 1st time and its kind of cool)

And just like before(2 days ago, first part of the data eng zoomcamp), I can run docker-compose up -d and then pgcli to connect to mt db
In VS code that is connected to the VM, we can forward the port to the db
And now when I run pgcli from my own PC's terminal, I can connect to it too.
By adding port 8080 as well in VS code, I now can access pgadmin too from my browser (even tho it is all running on that GCP VM)
Same for jupyter - added port 8888 in vs code, then I can run jupyter notebook in the VM, and access in my browser. 
Next, I installed terraform for linux

When I was setting up terraform, I created a my-creds.json with the credentials from GCP. Now using sftp I transferred the json file from my local to the VM (sftp - another tool I am using for the 1st time)
And then I could run the same terraform apply and destroy to create and destroy resources.

And if I want to stop and restart the instance I can do it through the terminal (`sudo shutdown now`) or the GCP console. And when I start it again in order for the quick ssh connection command to work, I need to edit the config file's HostName that I created earlier. 
I found that if I restart the VM and want to use terraform, I need to set my credentials and gcloud auth using (and also just saving the commands for later):
`export GOOGLE_APPLICATION_CREDENTIALS=~/.gc/my-creds.json`
`gcloud auth activate-service-account --key-file $GOOGLE_APPLICATION_CREDENTIALS`

Next onto Module 2: workflow orchestration

My good old friend mage.ai. Let's hope for at least less errors than when I covered it in the MLOps zoomcamp. 

The first bit was to establish a connection with the postgres database which is ran alongside mage in docker-compose.yml

Creating a block in a new pipeline to test the connection:

So far so good.

Next is writing a simple ETL pipeline - loading data from an API to postgres, where I just load taxi data in the first block using data type checking, do a little bit of preprocessing in the 2nd block and then make a connection to my db and load the data there in the 3rd block

Next is connecting my gcp service account to mage (using the creds.json file), and a connection is made. 

Loading data to google cloud storage is very easy - just adding a google cloud storage (GCS) data exporter block, and putting my info down and its done

Can be seen in gcs
However, with larger files we should not load data into a single parquet file. We should partition it.

I learned how to use pyarrow (as it abstracts chunking logic) for that as in the below block:

And the partitioned data is in gcs now. Awesome

And then load it into BigQuery


My experience using mage in this course compared to the MLOps zoomcamp is completely different haha. Now, the teacher Matt Palmer did a great job ^^


That is all for today!

See you tomorrow :)

Popular posts from this blog

[Day 198] Transactions Data Streaming Pipeline Porject [v1 completed]

Image
 Hello :) Today is Day 198! A quick summary of today: data streaming pipeline project [v1 done] Here is a link to the project's repo. Well ... I did not know I can do it in a day (~14 hours) after yesterday's issues but here we are. Turns out in order to insert the full (~70 variables with nested/list structure), I need the proper pyspark schema. And yesterday I did not have that and that is why when I was reading data in the kafka producer I was getting NULL in the columns - my schema was wrong. Well today I not only fixed the schema for the 4 variables I had yesterday, but included *all* the variables that come from the Stripe API ~ 70 (for completeness).  When I run docker-compose, the data streams and is input into the postgres db (and is still running). Unfortunately, the free Stripe API for creating realistic transactions has a limit of 25, so every 3 seconds, 25 new transactions are sent to the db. It has been running half the day (since I got that set up) and as I am wr
Read more

[Day 107] Transforming natural language to charts

Image
 Hello :) Today is Day 107! A quick summary of today: created text2chart - transforming natural language to charts [ github ] [ webapp ] I am super sick, but in the around 3 hours of my fever not being high, I managed to study a bit. I was again thinking about creating some nlp project but was stuck on the idea of how to host a model. After looking around I found this article where they create a similar app called chat2vis  and they use codellama. I know codellama is an open source model that I can get from huggingface and I wondered, how did they host it. Turns out - they ask the user to input a huggingface api key, which then allows for querying and generating content. (AWESOME! this is applicable to some of the other PDF chat apps that I have, but I will look to make an app of them later).  My fever eased a bit, and I sat on my chair to do some work ^^ Here is a breakdown of the app. I used streamlit. Also, 2 default datasets during dev: Financial Statements of Major Companies(2009-
Read more

[Day 54] I became a backprop ninja! (woohoo)

Image
 Hello :) Today is Day 54! A quick summary of today: Read through cs231n's neural network lectures [1] [2] [3] Spent 8 hours going over Andrej Karpathy's backprop ninja tutorial After yesterday's confidence boost in doing backprop, I came in today with the intention of becoming of, as Andrej Karpathy calls it, a backprop ninja.  The cs231n NN lectures were the usual about how NN became to be, param initialization, activation functions, loss functions, regularization and optimizers. Nothing out of the ordinary, but still was good to reassure myself of that knowledge. But that was definitely not the main even of today. As I mentioned, I wanted to become (or at least attempt to become) a backprop ninja today. I have watched that particular tutorial twice, the 1st time I had no idea what is going on, the 2nd time I could follow, and now it was the 3rd time - and now, I wanted to participate and actually see the why and the how of the things we have to do, when we do manual back
Read more