What a training loop repeats

TensorFlow and neural networks · applies to: Supervised training, any framework · last updated 2016-06-15

Training is a loop with four fixed steps, and every framework spells the same four steps differently.

The four steps

A batch of examples goes forward through the model and produces predictions. Those predictions and the true labels go into a loss function, which collapses the batch down to one number describing how wrong the model currently is. The derivative of that number with respect to every parameter is computed backwards through the same path. Finally each parameter moves a small distance against its derivative.

That is the whole loop. Learning rate schedules, momentum, dropout and batch normalisation are all refinements of the fourth step or additions to the first; none of them change the order.

Why the batch exists

Computing the derivative over the entire dataset gives the truest direction but one update per pass, which is slow. Computing it over a single example gives many updates but a direction that jumps around. A batch is the compromise: enough examples to average out the noise, few enough to fit in memory and to update often.

Batch size therefore interacts with learning rate. Larger batches give a steadier direction, which tolerates and often needs a larger step.

What an epoch is

An epoch is one full pass over the training set, made of many batches. It is a bookkeeping unit, not a mechanism: the model does not know when an epoch ends. It matters only because shuffling between epochs and evaluating at epoch boundaries are the conventional places to do those things.

One iteration, in order

1. Draw the next batch of inputs and labels from the shuffled training set.
2. Run the forward computation to get predictions.
3. Reduce predictions and labels to a single loss value.
4. Compute the gradient of that loss with respect to every parameter.
5. Step each parameter against its gradient, scaled by the learning rate.
6. Record the loss, and at epoch boundaries evaluate on data the loop never trains on.

for epoch in range(epochs):
    shuffle(training_set)
    for batch in batches(training_set, batch_size):
        predictions = forward(batch.inputs)
        cost = loss(predictions, batch.labels)
        gradients = backward(cost)
        step(parameters, gradients, learning_rate)

The loop written out plainly

Return to TensorFlow and neural networks