TensorFlow graphs and sessions

TensorFlow and neural networks · applies to: TensorFlow, graph construction · last updated 2016-06-18

A TensorFlow program is written in two stages: first a graph of operations is described, then that graph is run.

The two stages

Nothing computes while the graph is being built. Creating a constant, a variable or a matrix multiply adds a node to a graph and hands back a handle to that node's future output. The handle prints as a tensor with a shape and a data type, not as a number, and that surprises almost everyone once.

The second stage opens a session, which owns the actual memory and the actual devices, and asks it to evaluate one or more of those handles. The session walks backwards from what was asked for, runs only the nodes that contribute to it, and returns plain arrays.

Why the split exists

Describing the whole computation before running any of it lets the runtime see the entire dependency structure at once. It can drop nodes nobody asked for, place operations on whichever device suits them, and run independent branches in parallel without being told to.

The cost is that ordinary debugging habits stop working. Printing a handle shows the handle. To see a value, it has to be fetched through the session along with everything else being fetched in that same call.

Feeding values in

Placeholders are graph nodes with a declared shape and no value. Every call supplies the values through a feed mapping, which is how the same graph serves a training batch, a validation batch and a single prediction without being rebuilt.

Variables are different: they hold state across calls, they must be initialised once before first use, and forgetting that initialisation is the most common first error in a fresh program.

The shape of a minimal program

1. Describe the inputs as placeholders with fixed shapes and data types.
2. Create the variables that will hold the learned parameters.
3. Compose the forward computation from those inputs and variables.
4. Define a loss node that compares the forward output against a label placeholder.
5. Attach an optimiser node that minimises the loss.
6. Open a session, run the variable initialiser once, then run the optimiser node in a loop with fed batches.

# one call, one consistent view of the graph state
_, loss_value = session.run(
    [train_step, loss],
    feed_dict={x: batch_inputs, y: batch_labels},
)

Fetching two nodes in one call keeps them consistent

Return to TensorFlow and neural networks