[Day 173] Terraform, GCP, virtual machines, data pipelines

 Hello :)
Today is Day 173!


A quick summary of today:
  • learned more about terraform and how to set up a GCP VM and connect to it locally
  • used mage for some data engineering pipelines with GCP


Last videos from Module 1: terraform variables, GCP set up

Turns out there is a bit more of terraform from the data eng zoomcamp, and today I covered it.

After learning how to connect to gcp using terraform and create a storage bucket, the first thing today was creating a bigquery dataset

Adding the above to main.tf which now looks like:

terraform apply, creates a demo_dataset as well

Then I learned about variables in terraform

Create a variables.tf file and put a variable like:

and in main.tf we can directly use the created variables like:

Great intro to terraform - being able to define infrastructure code, create resources, and destroy resources.

The next part was an instruction on setting up GCP (cloud VM + SSH access)
First was creating an ssh key locally
And add it to the metadata in GCP's compute engine (hiding the username just in case)
Then create a VM, and connect to it locally using that ssh key, using `ssh -i ~/.ssh/gcp username@gcp_vm_external_ip`
For a quick connection to the VM, I set up a config which includes Host, HostName, User and IdentityFile, so now I can just run ssh `Host` and I am connected to the VM through my terminal. Nice.

Also set up vs code to connect to the created ssd


Then, I installed anaconda. 
And docker
Then docker-compose
And make it executable from anywhere by adding the below to .bashrc
`export PATH="${HOME}/bin:${PATH}"`
And now we have it 

Then installed pgcli with conda
(random note - I am using a vm from my local terminal like this for the 1st time and its kind of cool)

And just like before(2 days ago, first part of the data eng zoomcamp), I can run docker-compose up -d and then pgcli to connect to mt db
In VS code that is connected to the VM, we can forward the port to the db
And now when I run pgcli from my own PC's terminal, I can connect to it too.
By adding port 8080 as well in VS code, I now can access pgadmin too from my browser (even tho it is all running on that GCP VM)
Same for jupyter - added port 8888 in vs code, then I can run jupyter notebook in the VM, and access in my browser. 
Next, I installed terraform for linux

When I was setting up terraform, I created a my-creds.json with the credentials from GCP. Now using sftp I transferred the json file from my local to the VM (sftp - another tool I am using for the 1st time)
And then I could run the same terraform apply and destroy to create and destroy resources.

And if I want to stop and restart the instance I can do it through the terminal (`sudo shutdown now`) or the GCP console. And when I start it again in order for the quick ssh connection command to work, I need to edit the config file's HostName that I created earlier. 
I found that if I restart the VM and want to use terraform, I need to set my credentials and gcloud auth using (and also just saving the commands for later):
`export GOOGLE_APPLICATION_CREDENTIALS=~/.gc/my-creds.json`
`gcloud auth activate-service-account --key-file $GOOGLE_APPLICATION_CREDENTIALS`

Next onto Module 2: workflow orchestration

My good old friend mage.ai. Let's hope for at least less errors than when I covered it in the MLOps zoomcamp. 

The first bit was to establish a connection with the postgres database which is ran alongside mage in docker-compose.yml

Creating a block in a new pipeline to test the connection:

So far so good.

Next is writing a simple ETL pipeline - loading data from an API to postgres, where I just load taxi data in the first block using data type checking, do a little bit of preprocessing in the 2nd block and then make a connection to my db and load the data there in the 3rd block

Next is connecting my gcp service account to mage (using the creds.json file), and a connection is made. 

Loading data to google cloud storage is very easy - just adding a google cloud storage (GCS) data exporter block, and putting my info down and its done

Can be seen in gcs
However, with larger files we should not load data into a single parquet file. We should partition it.

I learned how to use pyarrow (as it abstracts chunking logic) for that as in the below block:

And the partitioned data is in gcs now. Awesome

And then load it into BigQuery


My experience using mage in this course compared to the MLOps zoomcamp is completely different haha. Now, the teacher Matt Palmer did a great job ^^


That is all for today!

See you tomorrow :)

Popular posts from this blog

[Day 198] Transactions Data Streaming Pipeline Porject [v1 completed]

Image
 Hello :) Today is Day 198! A quick summary of today: data streaming pipeline project [v1 done] Here is a link to the project's repo. Well ... I did not know I can do it in a day (~14 hours) after yesterday's issues but here we are. Turns out in order to insert the full (~70 variables with nested/list structure), I need the proper pyspark schema. And yesterday I did not have that and that is why when I was reading data in the kafka producer I was getting NULL in the columns - my schema was wrong. Well today I not only fixed the schema for the 4 variables I had yesterday, but included *all* the variables that come from the Stripe API ~ 70 (for completeness).  When I run docker-compose, the data streams and is input into the postgres db (and is still running). Unfortunately, the free Stripe API for creating realistic transactions has a limit of 25, so every 3 seconds, 25 new transactions are sent to the db. It has been running half the day (since I got that set up) and as I am w...
Read more

[미리 공부] 기초 통계 복습 (Day 1는 1월2일)

Image
모험을 시작하기 전에 기초 지식을 복습하고자 했다. 오늘 SPSS 대안 프로그램을 찾아보려고 해서 JASP에 대해 알게 되었다. 유용한 프로그램인 것 같아서 선회귀와 기술통계를 내려고 했는데 재미있었다. 그런데 JASP에 대해 더 알기 전에 '기초 통계 지식을 좀 복습을 하고자 하면 좋을 것 같아'란 생각을 들었다. 다행히, Coursera에서 Stanford University의 Guenther Walther 교수님께서 진행된  Introduction to Statistics  무료 강좌가 있다.  좀 부족한 부분은 다양한 검정통계 하는 거고 (F test, t-test, chi-square 등) 이제 JASP 아니면 다른 통계 프로그램 사용하게 되어도 이런 부분을 좀 더 자세히 집중하여 공부하면 된다.  특히 homoscedasticity 및 heteroscedasticity 개념을 기억에 남았다. Homoscedasticity (선): Definition: In a homoscedastic dataset, the variance of the errors (residuals) is constant across all levels of the independent variable(s). In simpler terms, the spread of the residuals is the same throughout the range of predictor values. Heteroscedasticity (악): Definition: Heteroscedasticity occurs when the variance of the errors is not constant across all levels of the independent variable(s). In other words, the spread of residuals changes as the values of the independe...
Read more

[Day 61] Stanford CS224N (NLP with DL): Machine translation, seq2seq + a side CDCGAN mini project

Image
 Hello :) Today is Day 61! A quick summary of today: Covered Lecture 7 : machine translation, seq2seq, attention from Stanford CS224N Tried to make a conditional DCGAN to generate MNIST numbers ( colab ) ( kaggle ) I will first cover the GAN story (then will share my notes from the lecture) So... while watching and taking notes today, I started thinking, what if I can use my notes as data to a model and afterwards, when I want, I can give it raw string text and it will output text in the format of my notes (with my handwriting). Well I started looking around and actually the first model architecture that came to my mind was the GAN (specifically conditional GAN) - I remembered there was a GAN architecture that alongside the pictures, we can give it the labels, and then on-demand generate. In retrospect, there are of course others, but I decided to go with GAN.  For maybe 2 hours I busted my head trying to make a simple model with the EMNIST dataset (english characters), and I ...
Read more