Neural Network

What is a neural network?

A neural network is a machine learning model built from many simple processing units, called neurons, arranged in layers. Each neuron takes several numbers as input, multiplies them by adjustable values called weights, adds them up, and passes the result through a nonlinear activation function. The network turns an input (an image, a sentence, a row of numbers) into an output (a label, a score, the next word) by passing signals forward through these layers.

The name comes loosely from biology. Real brain neurons fire when their inputs cross a threshold, and early researchers built a mathematical caricature of that behavior. The resemblance is shallow and the comparison is mostly historical, but it gives the right mental picture: many tiny units, each doing trivial arithmetic, combine into a system that can recognize faces, translate languages, or steer a car.

What makes a neural network useful is that the weights are not programmed by hand. They start as random numbers and are tuned automatically during training so that the network's outputs match known correct answers. This is why neural networks sit at the center of modern machine learning and underpin almost every deep learning system in use today.

• Built from layers of neurons that each compute a weighted sum plus a nonlinearity.
• Maps inputs to outputs by passing signals forward through the layers.
• Learns by adjusting weights automatically rather than being hand-coded.

The artificial neuron: weights, bias, and activation

A single artificial neuron is the smallest piece of the network, and it does two things. First it computes a weighted sum: it multiplies each input by a weight, adds a constant called the bias, and totals the result. The weights say how much each input matters, and the bias shifts the threshold at which the neuron responds. This first step is purely linear.

Second, the neuron passes that sum through an activation function, which adds a nonlinear bend. The activation is what lets the network represent curved relationships instead of straight lines. The most common choice today is ReLU, which keeps positive values and zeroes out negatives, but sigmoid, tanh, and GELU are also used depending on the layer and the task.

A single neuron on its own can only draw a straight dividing line through its inputs, which is why one neuron solves very little. The power appears when you connect thousands of them. Each neuron learns to respond to a slightly different pattern, and stacking them lets the network compose those patterns into something far richer.

• Weight: how strongly one input influences the neuron's output.
• Bias: a constant offset that shifts when the neuron activates.
• Activation: the nonlinear step (often ReLU) applied after the weighted sum.

Layers: input, hidden, and output

Neurons are organized into layers, and the layers do different jobs. The input layer receives the raw data, with one unit per feature: one neuron per pixel for an image, or one per measurement for tabular data. The output layer produces the final answer, such as a probability for each class. Everything in between is a hidden layer, so called because its values are intermediate and never observed directly.

In the most basic design, a fully connected (or dense) network, every neuron in one layer feeds every neuron in the next. Information flows in one direction from input to output, which is why this arrangement is called a feedforward network. Each hidden layer transforms its input into a new representation, and the network gradually reshapes the raw data into a form where the final answer becomes easy to read off.

The intuition behind stacking layers is hierarchy. In a vision network, early hidden layers tend to detect edges and simple textures, middle layers assemble those into shapes and parts, and later layers recognize whole objects. Each layer builds on the abstractions discovered by the one before it, which is exactly the property that makes deep networks powerful.

• Input layer: one neuron per input feature; holds the raw data.
• Hidden layers: intermediate transformations that build progressively abstract features.
• Output layer: produces the prediction, such as class probabilities or a numeric value.

Why depth and nonlinearity matter

Depth and nonlinearity together are what give neural networks their reach. Without a nonlinear activation, every layer would just be a linear transformation, and stacking linear transformations produces another single linear transformation. A hundred-layer network with no activations would be mathematically identical to one layer and could only separate data with a straight boundary.

The activation function breaks that collapse. With nonlinearity in place, adding layers genuinely increases what the network can represent. The universal approximation theorem, first proven by George Cybenko in 1989 for a single hidden layer with a suitable activation, formalizes this: a network with even one hidden layer can approximate any continuous function to arbitrary accuracy, given enough neurons.

In practice, depth is usually more efficient than width. A deep network can represent some functions with far fewer neurons than a shallow one would need, because it reuses features layer by layer instead of memorizing everything in a single enormous layer. That efficiency, made practical by better hardware and training methods, is what the term deep learning describes.

• Stacking layers without nonlinearity collapses to a single linear model.
• Cybenko (1989) proved one hidden layer can approximate any continuous function in principle.
• Depth is often more parameter-efficient than width for the same accuracy.

How a neural network learns: training, loss, and backpropagation

A neural network learns by repeatedly comparing its outputs to known correct answers and nudging its weights to reduce the gap. Training starts with random weights and runs in a loop. The forward pass feeds an example through the network to produce a prediction. A loss function then measures how wrong that prediction is, turning the error into a single number to minimize.

The key step is figuring out which weights to change and by how much. Backpropagation answers this by applying the chain rule of calculus to compute the gradient of the loss with respect to every weight, working backward from the output layer to the input. The gradient points in the direction that increases error, so the network moves the opposite way.

Gradient descent does the moving: each weight is adjusted by a small step proportional to its gradient, scaled by a learning rate. Repeat this over many examples and many passes through the data (epochs), and the loss falls while the network's predictions improve. This is supervised learning in its most common form, where every training example carries a correct label.

• Forward pass: run an input through the network to get a prediction.
• Loss function: measures prediction error as a single number to minimize.
• Backpropagation + gradient descent: compute gradients and step the weights downhill.

A minimal example in code

Modern frameworks hide the math behind a few lines. The snippet below defines a small feedforward network in PyTorch: an input layer feeding a hidden layer of 64 ReLU units, then an output layer with 10 values for a 10-class problem. The framework handles backpropagation automatically once you compute the loss and call backward.

Reading the code top to bottom mirrors the forward pass. Data enters the first linear layer, gets a nonlinear ReLU bend, then passes through the second linear layer to produce the output scores. Nothing here is specific to images or text; the same pattern scales up to the much larger networks behind today's models.

import torch.nn as nn
import torch.optim as optim

net = nn.Sequential(
    nn.Linear(784, 64),   # input layer -> hidden
    nn.ReLU(),            # nonlinearity
    nn.Linear(64, 10),    # hidden -> output (10 classes)
)

loss_fn = nn.CrossEntropyLoss()
opt = optim.SGD(net.parameters(), lr=0.01)

out = net(x)              # forward pass
loss = loss_fn(out, y)
opt.zero_grad()
loss.backward()           # backpropagation
opt.step()                # gradient-descent update

• nn.Linear is the weighted-sum layer; nn.ReLU is the activation between layers.
• loss.backward() runs backpropagation; the optimizer applies the gradient-descent step.
• The same forward/backward loop scales from this toy net to billion-parameter models.

Common architectures beyond the basic network

The fully connected feedforward network is the starting point, not the destination. Different data shapes call for different wiring. Convolutional neural networks add filters that slide across grid data such as images, sharing weights across positions so the same pattern can be detected anywhere. This makes them efficient and well suited to vision and document scanning.

Recurrent networks process sequences one step at a time and carry a hidden state forward, which once made them the default for text and speech. They have largely been displaced by the transformer, which uses an attention mechanism to relate every element of a sequence to every other in parallel. Transformers are the architecture behind large language models and most current generative AI.

These are all neural networks: they share neurons, weights, activations, and training by backpropagation. What changes is how the neurons are connected, which encodes assumptions about the data. Choosing an architecture means choosing which structure fits your problem rather than reinventing the underlying learning machinery.

• Convolutional networks: weight-shared filters for images and grid data.
• Recurrent networks: step-by-step processing of sequences with a carried state.
• Transformers: attention-based, parallel, and the basis of modern language models.

A short history and where neural networks stand in 2026

Neural networks are old. The McCulloch-Pitts model of an artificial neuron dates to 1943, and Frank Rosenblatt's perceptron, a trainable single-layer network, arrived in 1958. A 1969 critique by Marvin Minsky and Seymour Papert showed that single-layer perceptrons could not solve simple problems like XOR, which helped cool interest for years.

The field revived when backpropagation made training multi-layer networks practical in the 1980s, and again, decisively, around 2012 when deep convolutional networks plus GPUs crushed prior records on image recognition. That moment kicked off the deep learning era that still defines the field. The core ideas, weighted sums, nonlinear activations, and gradient-based learning, never changed; the scale and the hardware did.

As of 2026, neural networks are the dominant approach to machine learning, and the largest are transformers with hundreds of billions of parameters. They still inherit the same limitations: they need large amounts of data, their internal reasoning is hard to interpret, and they can confidently produce wrong answers. Understanding the neuron, the layer, and the training loop is the foundation for understanding everything built on top of them.

• 1943 McCulloch-Pitts neuron and 1958 Rosenblatt perceptron set the foundations.
• Backpropagation (1980s) and GPU-trained deep nets (around 2012) drove the breakthroughs.
• In 2026, transformers dominate, but the core neuron-layer-training ideas are unchanged.