Unit 6. Convolutional, sequence and reinforcement learning models

This unit covers three families of methods and their security uses: convolutional networks, which share a local filter across positions and apply to malware byte-image classification and to image-based detectors for synthetic media; sequence models such as the long short-term memory (LSTM) network, which read system logs as sequences of event templates and flag surprising next events; and reinforcement learning, which fits a policy for acting in an environment and appears in penetration-testing and network-defense simulators. You will learn the mechanisms (the convolution and its arithmetic, the gates of an LSTM, the Q-learning update and the Bellman optimality equation), the practical pitfalls (only approximate shift invariance, scarce labeled log data, and the tie-breaking trap that can stop an agent from learning), and the realistic scope of each application.

After this unit you will be able to compute the output size and parameter count of a convolutional layer, explain the difference between equivariance and invariance, set up transfer learning with a frozen feature extractor, build an n-gram and an LSTM next-event detector and choose a top-g threshold, and implement tabular Q-learning with sound exploration and tie-breaking. The unit assumes Units 1 through 5, in particular neural network training and backpropagation from Unit 5 and the evaluation metrics of Units 1 and 2. Plan about 8 to 10 hours including the three labs and the exercises.

Lessons

1. 6.1 Convolutional networks and transfer learning
2. 6.2 Sequence models for logs
3. 6.3 Reinforcement learning

Labs

Practice

• Exercises with worked solutions
• Self-check quiz