Convolutional networks
For images. Small filters slide across the picture looking for the same feature anywhere it appears, which is why a cat in the corner is still a cat.
Neural networks and deep learning
Layers, weights, backpropagation, and why transformers changed what language models can do.
A neuron is a weighted sum and a threshold
An artificial neuron takes several numbers in, multiplies each by a weight, adds them up, and passes the total through a function that decides how strongly to fire. That is the complete description. There is nothing biological happening.
The power comes from stacking them. One layer can only draw a straight line through the data. Two layers can draw curves. Many layers can represent relationships nobody could write down — which is exactly the case for "what makes this photograph a cat".
What the layers do
Each layer receives the previous layer's output and builds something slightly more abstract from it. The labels below are human interpretations; the network has no names for anything.
| Layer | Units | Shape | What it ends up representing |
|---|---|---|---|
| Input | 6 | The picture, as numbers. One value per pixel. | |
| Hidden 1 | 8 | Edges and corners. Nobody told it to look for these. | |
| Hidden 2 | 8 | Combinations: wheels, windows, eyes. | |
| Output | 3 | One score per possible answer. |
Where "deep" comes from
Purely from the number of layers. A network with three layers is shallow; the image models of the 2010s ran to over a hundred. The word carries no claim about depth of thought.
Backpropagation, in one paragraph
When the network gets an answer wrong, the error is measured at the output and then pushed backwards through the layers. Each weight receives a share of the blame proportional to how much it contributed, and moves a small step in the direction that would have reduced the error. Doing this efficiently, layer by layer, is backpropagation — and it is the reason deep networks are trainable at all.
The step size is called the learning rate. Too large and the model overshoots and never settles; too small and training takes a month. Choosing it well is a craft, not a formula.
Four architectures worth knowing by name
The structure of a network is chosen to match the shape of the data going into it.
Recurrent networks
For sequences. They carry a memory from one step to the next. Largely superseded for language, still used where data arrives strictly in order.
Transformers
For almost everything now. Attention lets every element of the input weigh every other one directly, in parallel — which is what made training on the scale of the whole web practical.
Diffusion models
For generating images. Start with pure noise and remove a little of it at a time, guided by the prompt, until a picture is left behind.
Why a language model can write but cannot count
A language model is trained on one objective: predict the next token. Everything it appears to do — translating, summarising, arguing, coding — is that single ability applied to text where the next token happens to be a translation, a summary, an argument or a line of code.
It explains the strengths and the odd gaps in the same breath. Prose is exactly the thing next-token prediction is good at. Counting the letters in a word is not, because the model never sees letters — it sees tokens, which are chunks of several characters at once.