Natural Language Processing

- What is it? - What kind of tasks exist? - Where is the field now? This site explains in detail much where the field is currently and what kinds of tasks are being explored.

Transformers are the building blocks for the language models we see today. They are built on the concept of attention and consist of encoder and decoder stacks. Attention allows for models to remember the context that a word was found in. This is what BERT and other popular language models are built on top of. The notes below come from The Illustrated Transformer. These notes are better used as a supplement to the blog post than a replacement. If you are trying to implement a transformer based model, look into Hugging Face and their Transformers library. Additionally, this Twitter thread contains a thread of useful resources to fully understand the concept from a high-level idea to the implimentation details. Alammar, Jay (2018). The Illustrated Transformer [Blog post]. Retrieved from https://jalammar.github.io/illustrated-transformer/

Large language models like BERT and GPT have taken over the field of NLP and are seen on many of the state of the art models for a variety of tasks. These language models are trained on HUGE amounts of data at large companies (Google and OpenAI). Below, I've attached my notes from reading The Illustrated BERT. I would recommend reading his blog post and use these notes as a supplement since he explains the math behind this model much better than my notes can. Alammar, Jay (2018). The Illustrated BERT [Blog post]. Retrieved from http://jalammar.github.io/illustrated-bert/