Understanding Deep Neural Networks Training Course

Part 1 – Deep Learning and DNN Concepts

Introduction to AI, Machine Learning & Deep Learning

• The history of artificial intelligence, its basic concepts, common applications, and separation from popular myths
• Collective Intelligence: aggregating knowledge shared across multiple virtual agents
• Genetic algorithms: evolving populations of virtual agents through selection processes
• Standard Learning Machines: definition and scope.
• Task types: supervised learning, unsupervised learning, and reinforcement learning
• Action types: classification, regression, clustering, density estimation, and dimensionality reduction
• Examples of Machine Learning algorithms: Linear regression, Naive Bayes, and Random Trees
• Machine Learning vs. Deep Learning: identifying problems where Machine Learning remains the state-of-the-art (e.g., Random Forests & XGBoost)

Basic Concepts of a Neural Network (Application: multi-layer perceptron)

• Review of essential mathematical foundations.
• Definition of a neural network: classical architecture, activation functions, and
• Weighting of previous activations and network depth
• Defining the learning process: cost functions, back-propagation, Stochastic Gradient Descent, and maximum likelihood.
• Modelling neural networks: structuring input and output data based on problem type (regression, classification, etc.) and addressing the curse of dimensionality.
• Distinguishing between multi-feature data and signals. Selecting appropriate cost functions based on data characteristics.
• Approximating functions using neural networks: theoretical presentation and practical examples.
• Approximating distributions using neural networks: theoretical presentation and practical examples.
• Data Augmentation: strategies for balancing datasets.
• Generalising the results obtained from neural networks.
• Initialising and regularising neural networks: L1 / L2 regularization, and Batch Normalization.
• Optimization and convergence algorithms.

Standard ML / DL Tools

A brief overview of key tools, highlighting their advantages, disadvantages, position within the ecosystem, and typical use cases, is planned.

• Data management tools: Apache Spark, Apache Hadoop.
• Machine Learning libraries: Numpy, Scipy, Sci-kit.
• High-level DL frameworks: PyTorch, Keras, Lasagne.
• Low-level DL frameworks: Theano, Torch, Caffe, TensorFlow.

Convolutional Neural Networks (CNN).

• Overview of CNNs: fundamental principles and key applications.
• Basic CNN operations: convolutional layers, kernel usage,
• Padding & stride, feature map generation, and pooling layers. Extensions to 1D, 2D, and 3D.
• Presentation of various CNN architectures that have established the state-of-the-art in classification.
• Image Processing: LeNet, VGG Networks, Network in Network, Inception, and ResNet. Analysis of innovations introduced by each architecture and their broader applications (e.g., 1x1 convolutions or residual connections).
• Implementing attention models.
• Applying CNNs to common classification tasks (text or image).
• Generative CNNs: super-resolution and pixel-to-pixel segmentation. Overview of
• Primary strategies for enhancing feature maps for image generation.

Recurrent Neural Networks (RNN).

• Overview of RNNs: fundamental principles and applications.
• Basic RNN operations: hidden activations, back-propagation through time, and unfolded representations.
• Evolution towards Gated Recurrent Units (GRUs) and Long Short-Term Memory (LSTM) networks.
• Presentation of different states and the advancements brought by these architectures.
• Challenges related to convergence and vanishing gradients.
• Classical architectures: time series prediction, classification, and other tasks.
• RNN Encoder-Decoder architectures. Utilisation of attention models.
• NLP applications: word / character encoding and translation.
• Video Applications: predicting the next frame in a video sequence.

Generative models: Variational AutoEncoder (VAE) and Generative Adversarial Networks (GAN).

• Introduction to generative models and their relationship with CNNs.
• Auto-encoders: dimensionality reduction and limited generation capabilities.
• Variational Auto-encoders: generative models and distribution approximation. Definition and use of latent space, the reparameterization trick, applications, and observed limitations.
• Generative Adversarial Networks: fundamental concepts.
• Dual Network Architecture (Generator and Discriminator) with alternating learning and available cost functions.
• GAN convergence and associated difficulties.
• Improved convergence techniques: Wasserstein GAN, Began, and Earth Mover's Distance.
• Applications in image/photograph generation, text generation, and super-resolution.

Deep Reinforcement Learning.

• Introduction to reinforcement learning: controlling an agent within a defined environment
• Characterised by states and possible actions.
• Utilising neural networks to approximate state functions.
• Deep Q-Learning: experience replay and application to video game control.
• Optimising learning policies: On-policy & off-policy methods, Actor-Critic architecture, and A3C.
• Applications: controlling single video games or digital systems.

Part 2 – Theano for Deep Learning

Theano Basics

• Introduction.
• Installation and Configuration.

Theano Functions

• Managing inputs, outputs, updates, and givens.

Training and Optimization of a neural network using Theano

• Modelling Neural Networks.
• Logistic Regression.
• Hidden Layers.
• Training a network.
• Computation and Classification.
• Optimization.
• Log Loss.

Testing the model

Part 3 – DNN using TensorFlow

TensorFlow Basics

• Creating, initialising, saving, and restoring TensorFlow variables.
• Feeding, reading, and preloading data in TensorFlow.
• Leveraging TensorFlow infrastructure to train models at scale.
• Visualising and evaluating models using TensorBoard.

TensorFlow Mechanics

• Data Preparation.
• Downloading Data.
• Inputs and Placeholders.
• Building the Graphs
  • Inference.
  • Loss.
  • Training.
• Training the Model
  • The Graph.
  • The Session.
  • Training Loop.
• Evaluating the Model
  • Building the Eval Graph.
  • Eval Output.

The Perceptron

• Activation functions.
• The perceptron learning algorithm.
• Binary classification using the perceptron.
• Document classification using the perceptron.
• Limitations of the perceptron.

From the Perceptron to Support Vector Machines

• Kernels and the kernel trick.
• Maximum margin classification and support vectors.

Artificial Neural Networks

• Nonlinear decision boundaries.
• Feedforward and feedback artificial neural networks.
• Multilayer perceptrons.
• Minimising the cost function.
• Forward propagation.
• Back propagation.
• Improving neural network learning efficiency.

Convolutional Neural Networks

• Objectives.
• Model Architecture.
• Principles.
• Code Organisation.
• Launching and Training the Model.
• Evaluating a Model.

Basic Introductions to be provided for the following modules (Brief overviews offered subject to time availability):

TensorFlow - Advanced Usage

• Threading and Queues.
• Distributed TensorFlow.
• Writing documentation and sharing models.
• Customising Data Readers.
• Manipulating TensorFlow Model Files.

TensorFlow Serving

• Introduction.
• Basic Serving Tutorial.
• Advanced Serving Tutorial.
• Serving Inception Model Tutorial.