NLP & LLMs — How Machines Read

What you'll learn

• How computers turn words into numbers
• Tokens, embeddings, and what makes an LLM tick
• Why ChatGPT is "just" predicting the next word
• You'll build a mini tokenizer and word counter

Turning language into math 🔤

Computers only do math on numbers — but language is words. NLP (Natural Language Processing) is the bridge.

Step 1: Tokenization. Break text into pieces called tokens (roughly words or word-parts). "I love AI" → ["I", "love", "AI"]. This is the first thing every language model does to your input.

Step 2: Embeddings. Each token becomes a vector of numbers (remember vectors?) that captures its meaning. The magic: words with similar meanings get similar vectors. So "king" and "queen" land near each other, and famously king - man + woman ≈ queen in vector-math. Meaning becomes geometry.

Step 3: The model. An LLM (Large Language Model) like GPT is a giant neural network trained on a huge chunk of the internet with one deceptively simple goal: predict the next token. Given "The capital of France is", it learned that "Paris" is overwhelmingly likely.

That's the big secret: ChatGPT is a spectacularly good next-word predictor. Ask it a question and it predicts, word by word, what a helpful answer would look like — because it saw millions of helpful answers during training. Scale that up massively (billions of weights, most of the internet) and next-word prediction becomes something that looks a lot like understanding.

Terms you'll hear: a prompt is your input; tokens are the billing/length unit (that's why long chats cost more); context window is how many tokens the model can "see" at once; hallucination is when it confidently predicts something false (it optimizes for plausible, not true).

Attention: how a model knows what to focus on 🔦

Older language models read text strictly left to right, one word at a time, which made it hard to connect words that were far apart in a sentence. Modern LLMs are built on an architecture called the Transformer, and its core idea is attention.

Attention lets the model, for every word it processes, ask: "which other words in this sentence matter most for understanding me right now?" and weigh them accordingly. Take the sentence:

"The trophy didn't fit in the suitcase because it was too big."

What does "it" refer to — the trophy or the suitcase? A human resolves this instantly from context. Attention gives the model a mechanism to do the same: when processing the word "it," the model learns to assign a high attention weight to "trophy" and a low weight to "suitcase," effectively looking back at the most relevant earlier word instead of just the most recent one.

Crucially, attention lets every word look at every other word simultaneously (not just the previous one), and it does this with plain weighted sums — the same weights-and-dot-products math from the math-for-ML lesson, just applied cleverly. This parallelism is also why Transformers can be trained so much faster than older step-by-step architectures: instead of processing one word at a time in sequence, a GPU can compute attention across an entire sentence at once.

Stack many layers of attention on top of each other, scale up to billions of parameters and a huge chunk of internet text, and you get the GPT-style LLMs from earlier in this lesson.

Real-world case study: Google Translate goes neural 🔍

For years, machine translation services (including Google Translate) relied on statistical machine translation — systems that translated phrase-by-phrase using probabilities learned from huge bodies of paired text, without much sense of the whole sentence's meaning. Translations were often clunky, especially for languages with very different grammar from English.

In 2016, Google switched Translate to Google Neural Machine Translation (GNMT), an approach built on deep neural networks that processed entire sentences at once rather than fragment by fragment. The jump in quality was large enough that Google's own researchers described the improvement as comparable to several years of the old system's incremental progress happening overnight. Translations became noticeably more fluent and better at handling context, word order, and idioms.

This transition — from phrase-based statistical methods to end-to-end neural networks reading whole sentences — is part of the same broader wave that led to the attention-based Transformer architecture you just learned about (the original Transformer paper itself was framed around improving machine translation). It's a clear, everyday example of how the shift from older NLP techniques to deep neural approaches translated directly into a better experience for hundreds of millions of daily users, almost overnight.

Summary

• NLP turns language into numbers: tokenize → embed → model
• Embeddings put meaning into vector space (king − man + woman ≈ queen)
• LLMs are giant next-token predictors; watch for hallucinations
• Attention lets Transformers weigh every word against every other word at once, resolving context and enabling fast, parallel training
• You built a real tokenizer + word counter, and a simple next-word predictor
• Case study: Google Translate's 2016 switch to neural machine translation showed this shift's real-world impact overnight

Next: Computer Vision — teaching machines to see.