50+ days of Machine Learning

 Hello :) Today is Day 46! Understanding Transformers with Professor Choi from KAIST The first time I learned about transformer was Day 32, it was a simple intro, but I did not understand exactly what is happening. I felt like, I was just made aware of their existance in the NLP world. This img is from Andrew Ng's Deep learning course.  In a transformer, the data goes through encoder-decoder network. In the encoder: for each token its attention is calculated according to the other tokens. And This attention mechanism allows the model to weigh the importance of each token in the context of the entire sequence. This information is put through a feed forward network that extracts deeper features.  In the decoder, we start to predict words. For example we start with an <start of sentence> token, then we pass that at the bottom, then from the encoder we take the K(key) and V(value) and with the Q(query) from the decoder input, we try to predict the next item in the ...

 Hello :) Today is Day 45! A quick summary of today: Learning the theory behind VAEs with Professor Choi from KAIST Implementing a VAE from scratch with  Aladdin Persson  (now probably my 2nd favourite DL youtuber after Andrej Karpathy) 1) Theory behind Variational Auto Encoders (VAEs) we begin with autoencoders given an input, they compress it into a lower-dimensional representation, and there is a decoder, that reconstructs the original input from this compressed representation. What VAEs want to do is with that space in the middle, sample from it and generate new samples.  But that comes with its challenges.  The idea behind VAEs is that we want to estimate the posterior distribution. We have X, and we want to know the distribution of Z in reality, estimating the posterio dist is extremely hard, so what we do instead is we approximate P(Z|X) to be pretty similar to a Q(Z) which follows a gaus distribution. But P(Z|X) doesnt always follow a gaus dist, so in th...

 Hello! :) Today is day 44 A quick summary of today: Found KAIST Professor Choi's Programming for AI lectures  Discovered that there is Layer and Group norm (not only Batch norm) Learned about GANs 1) Batch vs Layer vs Group normalization methods In one of Professor Choi's lectures, he explained about the above three norm layers (I have not heard of the 2nd 3rd 4th), plus some searches online, I found the difference. Firstly, batch norm: given a batch of activations for a specific layer, the mean and std for the batch is calculated. Then, it subtracts the mean and divides by the std to normalize the values. (+ an epsilon is added to the standard deviation for numerical stability) following that, a scale factor "gamma" and shift factor "beta" which are learnable parameters are applied. Secondly, layer norm: proposed in 2016, layer norm operates over the feature dimension (i.e., it calculates the mean and variance for each instance separately, over all the fe...

 Hello :) Today is Day 43! Quick summary of today: write LeNet from scratch write VGG from scratch write InceptioNet from scratch write UNet from scratch (again) Wanting to understand the popular models a bit more, I decided to do the above.  1) Let's begin with LeNet .  A basic framework developed in the 1990s, basic but set the groundwork for networks like AlexNet, VGG and ResNet. consists of 2 conv layers, each followed by a maxpool, and then ending with 2 fully connected (linear) layer.  2) Next is VGG The paper proposes numerous versions, VGG11, VGG13, VGG16, VGG19 but from a google search VGG16 seems most popular (version D in the pic). It is deeper than the earlier LeNet, consisting of multiple conv+maxpool layers, each increasing the amount of filters, and decreasing the size of the image.  instead of 1 version, a general model was created so that it can adapt to the desired VGG architecture below is the implementation. I think this is a nice set-up...