50+ days of Machine Learning

 Hello :) Today is Day 71! A quick summary of today: Finished up the manual backprop code to make it more clear Read some research papers Gaussian Error Linear Units (GELUs) Using Hallucinations to Bypass RLHF Filters Stealing Part of a Production Language Model Firstly, the manual backprop code cleanup. To finish it up (for now atleast), I decided to get more data. Original was ~32000 names, on kaggle I found with around ~90000 names.  As for the architecture - I settled on the original input > flatten > batch norm > activation > output, I added more even up to 2 hidden layers and batch norm for each, but the training time got longer, and the results were not even better than the simple original one.  Even though I stayed with the original model structure, I tried to modify embedding size, context length, hidden layers, just to play around with them and see the result. At the end the generated names look decent. But I was not able to achieve a much better re...

 Hello :) Today is Day 70! Today, I saw someone on X mentioning backprop and I decided to go back to my 'backprop ninja' code and re-do it to practice, and make sure I know the code. I initially did the 'become a backprop ninja tutorial' by Andrej Karpathy on Day 54 , but since then I have not looked at it in depth, and I felt that I was not satisfied with how I did it. Though I learned a lot about how backprop works, for the code... I did not feel like it was my own. Though I wrote the manual backprop by myself, the rest was pretty much copied from Andrej's code. Well today I wanted to fix that, and fully make the code 'my own' but also it would give me a chance to test my knowledge. Firstly, I started with proper naming for the vars, to make the code more readable. For instance,  original code: updated: The full new version is on this github repo . Besides variable renaming to make the code more readable, I wanted to add something else, so I decided to add...