Deep Learning — Neural Networks

What you'll learn

• What a neural network is, using the brain analogy
• Neurons, layers, and why "deep"
• How networks learn (forward + backward)
• You'll implement a single neuron

A team of tiny decision-makers 🧠

The human brain has billions of neurons, each doing one tiny thing, connected into a network that produces intelligence. Deep learning copies this idea in software.

An artificial neuron is simple. It:

1. Takes several inputs (numbers),
2. Multiplies each by a weight (importance — remember the weighted sum!),
3. Adds them up,
4. Passes the result through an activation function that decides "should I fire, and how strongly?"

One neuron is weak. But stack them in layers — the outputs of one layer feed the next — and something magical happens:

• The first layers learn simple things (edges in an image).
• Middle layers combine those into shapes (an eye, a wheel).
• Later layers combine those into concepts (a face, a car).

"Deep" just means many layers. More layers = ability to learn more complex patterns. This is why deep learning powers image recognition, voice assistants, and ChatGPT.

How does it learn? Two steps repeated millions of times:

1. Forward pass — push an example through the network, get a prediction.
2. Backpropagation — measure the error, then walk backward adjusting every weight a tiny bit to reduce that error (gradient descent again!).

Do that over huge datasets and the network gradually tunes millions of weights until its predictions are astonishingly good. You'll almost always use libraries like PyTorch or TensorFlow that handle the math — your job is designing and training, not doing calculus by hand.

Activation functions, and the trap of overfitting ⚡

The step activation you just coded (fire 1 or 0) is the simplest possible activation, but real networks almost always use smoother ones:

• ReLU (Rectified Linear Unit): max(0, x) — output the input unchanged if positive, otherwise 0. Dead simple, cheap to compute, and the most common activation in modern hidden layers.
• Sigmoid: squashes any number into a range between 0 and 1 — handy for outputs that represent a probability.
• Softmax: turns a list of raw scores into probabilities that all add up to 1 — used for "which of these 10 categories is this?" type outputs.

Without a nonlinear activation function like these, stacking layers would be pointless — a stack of purely linear steps collapses mathematically into a single linear step, no matter how many layers you add. The nonlinearity is what lets deep networks bend and curve to fit complex patterns instead of just straight lines.

The trap: overfitting. A large network with millions of weights is powerful enough to simply memorize every training example, including its noise and quirks, rather than learning the general pattern. The symptom is a model that scores nearly perfectly on training data but performs poorly on new, unseen data. Common fixes:

• More training data — harder to memorize, easier to generalize.
• Dropout — randomly "switch off" some neurons during training so the network can't over-rely on any single one.
• Early stopping — stop training once performance on held-out validation data stops improving, even if training accuracy keeps climbing.

A model that aces its training set but flops in the real world hasn't actually learned — it's cheated by memorizing the answer key.

Real-world case study: AlphaGo 🔍

For decades, the ancient board game Go was considered a grand challenge for AI — far harder for a computer than chess, because the number of possible positions is so vast that brute-force search doesn't work. Experts predicted it would take decades more before a computer could beat a top human player.

In 2016, DeepMind's AlphaGo played a five-game match against Lee Sedol, one of the greatest Go players in history — and won four games to one. AlphaGo combined deep neural networks (trained on both human game records and self-play) with a search algorithm, letting it develop an intuition for the game similar in spirit to how the neurons in this lesson combine simple weighted sums into increasingly abstract judgments about "how good is this position?"

One moment from the match became famous: in game two, AlphaGo played a move so unusual that commentators initially thought it was a mistake — it later proved brilliant, a move no human professional would likely have played, showing the network had learned genuinely novel strategic ideas rather than just copying human style.

The result stunned the AI research community and the Go world alike, and is widely seen as a landmark moment showing deep learning could master domains requiring intuition and long-term strategic judgment, not just pattern recognition on static images or text.

Summary

• Neural networks = layers of simple neurons (weighted sum + activation)
• "Deep" = many layers; early layers learn simple features, later layers learn concepts
• Learning = forward pass + backpropagation (gradient descent)
• Nonlinear activations (ReLU, sigmoid, softmax) are what let depth matter; overfitting is the trap of memorizing instead of generalizing
• You implemented a working neuron and a ReLU activation layer
• Case study: AlphaGo's 2016 win over Lee Sedol showed deep learning could master intuition-heavy strategic games

Next: NLP & LLMs — how machines understand language, the tech behind ChatGPT.