Unit 6 exercises

The spatial size follows from the output-size formula \(H_{\mathrm{out}} = \lfloor (H_{\mathrm{in}} - K + 2P)/S \rfloor + 1\), where \(H_{\mathrm{in}}\) is the input height (or width), \(K\) the filter size, \(P\) the padding added on each side and \(S\) the stride. Here it gives

so the output is 32 by 32. The number of output channels equals the number of filters, 16. Each filter has \(3 \times 5 \times 5 = 75\) weights plus one bias, so the layer has \(16 \times (75 + 1) = 1{,}216\) parameters. The parameter count does not depend on the 64 by 64 input size.

The convolutional layer has \(12 \times (1 \times 3 \times 3 + 1) = 120\) parameters. A fully connected layer mapping the \(100 \times 100 = 10{,}000\) inputs to \(12 \times 100 \times 100 = 120{,}000\) outputs has

parameters, about ten million times as many. This saving comes from the local receptive fields and the weight sharing of Lesson 6.1 (local connections alone would leave \(120{,}000 \times 10 = 1{,}200{,}000\) parameters, and sharing removes the remaining factor of 10,000), and it is why the convolutional feature extractor is cheap while the dense classifier on top holds most of a small network's parameters.

The gates are \(f_t = \sigma(1.1) = 0.7503\), \(i_t = \sigma(-0.3) = 0.4256\), \(o_t = \sigma(0.5) = 0.6225\), and the candidate is \(\tilde{c}_t = \tanh(0.8) = 0.6640\). Then

and

The forget gate retains three quarters of the old cell state while the input gate admits part of the candidate.

With factor 0.6, the scale after 25 steps is \(0.6^{25} \approx 2.84 \times 10^{-6}\), so the gradient reaching the early state is about three millionths of its original size and carries almost no training signal, which is the vanishing-gradient case of Bengio et al. (1994). With factor 1.2, the scale is \(1.2^{25} \approx 95.4\), which amplifies the gradient and can destabilize training, the exploding case. Either way plain recurrent networks struggle to learn dependencies across 25 steps, which motivates the additive cell state of the LSTM in Lesson 6.2.

(a) The transition is terminal, so there is no successor term, and \(Q \leftarrow 0 + 0.3(1 - 0) = 0.3\). (b) The successor's best value is now 0.3, so

The reward has propagated one step back, discounted and scaled by the learning rate. (c) The transition is terminal with reward \(-1\), so \(Q \leftarrow 0 + 0.3(-1 - 0) = -0.3\) and the greedy policy will avoid that action. Repeating such updates over many episodes is how the agent in Lab 6c learns the safe route.

With all action-values equal to zero at the start, the greedy action from a first-index argmax is always the same action (index 0) in every state. At the start host, action 0 targets the start host itself and wastes a step, and from the start host's neighbors it leads straight back. The agent therefore repeats that action except on the epsilon fraction of steps that are random, so it explores very little. With a sparse reward that appears only after at least three moves (four on either safe route), it reaches the target in only a few episodes, too rarely for the reward to propagate back, and the action-values at the start host stay at zero. Two fixes other than random tie-breaking are optimistic initialization, setting all initial action-values above the largest achievable return so every untried action looks attractive and is tried (Sutton and Barto, 2018), and adding a small random perturbation to the initial values to break the symmetry. Raising epsilon to 0.5 gives the first-index agent much more random exploration, so it should begin to reach the reward and learn a usable policy, though at the cost of a lower return during the random steps; running the lab with epsilon 0.5 and the first-index rule confirms that it now learns, unlike at epsilon 0.1.

The claim is wrong because order can be encoded in features. A logistic regression can be given the identity of the previous one or two events as lag features, counts of each event within a sliding window, or indicators for specific n-grams of events, all of which carry order. What the LSTM adds is the ability to summarize a variable-length history in a learned hidden state and to generalize across histories it did not see exactly, where the logistic regression relies on a fixed, human-chosen set of window or lag features. The right comparison is between a model that learns its own summary of the past and one given a fixed summary, as Lesson 6.2 states. A reasonable experiment is to train a logistic regression next-event model on lag and window-count features, apply the same top-g rule to its predicted probabilities (flag a sequence when any of its events is not among the \(g\) most probable next events), and compare its precision and recall with those of the LSTM at each \(g\). In Lab 6b the workflow is first order, so the count-based n-gram model with its short history is already the correct model there, and its recall is at least as high as the LSTM's at every \(g\). At \(g = 4\), where neither model flags a normal sequence, the n-gram model keeps recall 1.00 while the LSTM's recall falls to 0.95, although at \(g = 2\) and \(g = 3\) the LSTM has the slightly higher precision (0.29 against 0.25 at \(g = 2\)). A fair test of what the LSTM adds needs dependencies that reach further back.