50+ days of Machine Learning

 Hello :) Today is Day 47! A quick summary of today: Looking at GANs from DeepLearning.AI's perspective Trying to find more KAIST courses 1) GANs with DeepLearning.AI GANs... a war between a generator and a discriminator Given X, the disc tries to discriminate between dogs and cats, and the generator, given random noise and a class, it tries to create the X and trick the discriminator How do we train GANs? We start with the discriminator We give it real and fake examples. It gives its prediction on the fake images, calculate the loss on the fakes, then give it the real images, and calculate the loss on the real images. And the disc loss in this case is the avg of both.  Next, Then we train the generator,  we make random noise again, make fake images, get the discriminator to evaluate the fake images, and we get the generator loss.  Also the reason we use random noise is so that we dont get the same image each time.  Next - problems with traditional GANs Mode col...

 Hello :) Today is Day 46! Understanding Transformers with Professor Choi from KAIST The first time I learned about transformer was Day 32, it was a simple intro, but I did not understand exactly what is happening. I felt like, I was just made aware of their existance in the NLP world. This img is from Andrew Ng's Deep learning course.  In a transformer, the data goes through encoder-decoder network. In the encoder: for each token its attention is calculated according to the other tokens. And This attention mechanism allows the model to weigh the importance of each token in the context of the entire sequence. This information is put through a feed forward network that extracts deeper features.  In the decoder, we start to predict words. For example we start with an <start of sentence> token, then we pass that at the bottom, then from the encoder we take the K(key) and V(value) and with the Q(query) from the decoder input, we try to predict the next item in the ...

 Hello :) Today is Day 45! A quick summary of today: Learning the theory behind VAEs with Professor Choi from KAIST Implementing a VAE from scratch with  Aladdin Persson  (now probably my 2nd favourite DL youtuber after Andrej Karpathy) 1) Theory behind Variational Auto Encoders (VAEs) we begin with autoencoders given an input, they compress it into a lower-dimensional representation, and there is a decoder, that reconstructs the original input from this compressed representation. What VAEs want to do is with that space in the middle, sample from it and generate new samples.  But that comes with its challenges.  The idea behind VAEs is that we want to estimate the posterior distribution. We have X, and we want to know the distribution of Z in reality, estimating the posterio dist is extremely hard, so what we do instead is we approximate P(Z|X) to be pretty similar to a Q(Z) which follows a gaus distribution. But P(Z|X) doesnt always follow a gaus dist, so in th...

 Hello! :) Today is day 44 A quick summary of today: Found KAIST Professor Choi's Programming for AI lectures  Discovered that there is Layer and Group norm (not only Batch norm) Learned about GANs 1) Batch vs Layer vs Group normalization methods In one of Professor Choi's lectures, he explained about the above three norm layers (I have not heard of the 2nd 3rd 4th), plus some searches online, I found the difference. Firstly, batch norm: given a batch of activations for a specific layer, the mean and std for the batch is calculated. Then, it subtracts the mean and divides by the std to normalize the values. (+ an epsilon is added to the standard deviation for numerical stability) following that, a scale factor "gamma" and shift factor "beta" which are learnable parameters are applied. Secondly, layer norm: proposed in 2016, layer norm operates over the feature dimension (i.e., it calculates the mean and variance for each instance separately, over all the fe...