Deep Learning Advanced Learning Path - MAI4CAREU Master in AI
This Deep Learning (DL) learning path offers a structured curriculum that delves into Deep Learning, a subfield that has been associated with the revolution of AI in the last decade, with advancements in vision, natural language processing, attention and artificial general intelligence. The impact of DL extends in societal, technological, industrial, and financial sectors – and is poised to shape our future even more profoundly.
It is based on the elective advanced course offered at the Master in Artificial Intelligence of the University of Cyprus, which was developed with co-funding from the MAI4CAREU European project. The course offers a guided exploration of DL’s fundamental principles and diverse applications, and consists of twelve units organized in three parts as shown below, which are further arranged into Introductory (eight units), and Advanced (four units) according to their level of difficulty. The recommended order for studying all materials is the one shown below (from 1 to 12).
Part I: Introduction
- Introduction to Deep Learning
- Deep Learning Fundamentals
- Mathematics of Deep Learning
Part II: Deep Neural Networks
- Principles of Deep Neural Networks
- Convolutional Neural Networks
- Transfer Learning and Residual Networks
- Optimizing Deep Neural Networks
- Recurrent Deep Neural Networks
Part III: Advanced and Emerging Topics in Deep Learning
- Attention and Transformers
- Generative Adversarial Networks
- Deep Reinforcement Learning
- Emerging Topics in Deep Learning
MAI4CAREU - Deep Learning - Introduction to Deep Learning
Deep learning has evolved rapidly over the past decade, driven by advances in computing power, the availability of vast amounts of data, and breakthroughs in neural network architectures. Initially inspired by the structure and function of the human brain, deep learning algorithms, particularly neural networks with multiple layers, have demonstrated unprecedented performance in various tasks such as image recognition, natural language processing, and speech recognition. In the first introductory lecture, students are introduced to the foundational concepts and principles of this powerful subset of machine learning. The lecture begins by defining deep learning and its significance in modern artificial intelligence, highlighting its ability to automatically learn representations from data. Basic components of neural networks, including neurons, layers, and activation functions, are explained, and key training techniques like backpropagation and gradient descent are also introduced, emphasizing their role in optimizing neural network parameters. The unit also introduces basic structures of deep neural network models and architectures.
MAI4CAREU - Deep Learning - Fundamentals
Prior to understanding the basics of deep learning, students will be introduced to foundational concepts in mathematics, including calculus, linear algebra, and probability theory, as these form the basis of many deep learning algorithms. They will also reiterate data representation, data preparation, dimensions, and tensors. The unit also covers machine learning principles, such as supervised and unsupervised learning, as well as knowledge of optimization algorithms like gradient descent. Additionally, an overview of neural networks and their components, like activation functions and layers, are included to set the stage for deeper exploration in the course.
MAI4CAREU - Deep Learning - Mathematics of Deep Learning
This Unit introduces the students to the mathematics of deep learning, with fundamental concepts in calculus, linear algebra, and probability theory covered. This includes derivatives and gradients for optimization, matrix operations for handling data transformations, and probability distributions for understanding uncertainty in predictions. Additionally, students will learn about key mathematical operations within neural networks, such as activation functions and their derivatives. Overall, the Unit aims to equip students with the mathematical tools necessary to understand and implement deep learning algorithms effectively.
MAI4CAREU - Deep Learning - Principles of Deep Neural Networks
This unit, on the fundamentals of deep neural networks, begins by introducing students to shallow networks, emphasizing their limitations in capturing complex patterns in data due to their shallow architecture. By gradually increasing the depth of the network, students will learn how deeper architectures enable the model to learn hierarchical representations of the data, leading to better performance and generalization. This progression allows students to grasp the concept of feature hierarchies, where each layer learns increasingly abstract features from the previous layer's output. Through this approach, students gain insight into why deep neural networks are more powerful than shallow ones, enabling them to tackle more challenging tasks and datasets effectively. The principles of convolutional neural networks are also introduced.
MAI4CAREU - Deep Learning - Convolutional Neural Networks
This unit extends the introduction to convolutional neural networks (CNNs), and teaches students about specialized architectures tailored for processing grid-like data such as images. It covers the reasoning behind CNNs, highlighting their ability to leverage local connectivity and shared weights, which drastically reduces the number of parameters compared to fully connected networks. By using convolutional layers with filters, CNNs can efficiently extract hierarchical features from images, capturing spatial patterns while maintaining translation invariance. Deep convolutional neural networks (DCNNs) extend this concept by stacking multiple convolutional layers, allowing for the learning of increasingly complex and abstract features. Overall, CNNs and DCNNs are powerful tools for tasks like image classification and object detection due to their ability to automatically learn and extract meaningful features from raw input data.
MAI4CAREU - Deep Learning - Transfer Learning and Residual Neural Networks
This unit, mainly composed of two parts, introduces transfer learning for deep neural networks, where students are introduced to the concept of leveraging pre-trained models for new tasks, reducing the need for large labelled datasets and computational resources. Transfer learning involves fine-tuning a pre-trained model on a new dataset or using it as a feature extractor.
In the second part, residual networks (ResNets) are introduced as a breakthrough architecture that mitigates the vanishing gradient problem in very deep networks by introducing skip connections, allowing the model to learn residual mappings. The reasoning behind ResNets lies in their ability to facilitate the training of significantly deeper networks while maintaining manageable optimization and addressing the degradation problem. This unit equips students with strategies to leverage pre-existing knowledge and optimize model performance efficiently for various tasks.
MAI4CAREU - Deep Learning - Optimization of Deep Neural Networks
The objective of this unit is to explain the objectives and need for further optimization of deep neural network models. Students will learn various techniques to improve model training and performance. This includes understanding the role of optimization algorithms such as stochastic gradient descent (SGD) and its variants like Adam and RMSprop in efficiently updating model parameters. Regularization techniques like dropout and batch normalization are introduced to prevent overfitting and improve generalization. Additionally, students explore strategies for hyperparameter tuning, such as grid search and random search, to fine-tune model performance. The unit also covers advanced topics like learning rate scheduling and early stopping to further optimize model training. Overall, students gain a comprehensive understanding of the optimization process for deep neural networks, enabling them to build and fine-tune models effectively for various applications.
MAI4CAREU - Deep Learning - Recurrent Deep Neural Networks
This unit introduces recurrent deep neural networks (RNNs) and transformers, architectures designed for processing sequential data with memory and attention mechanisms. The unit first introduces basic RNNs, highlighting their struggle with retaining long-term dependencies, leading to exploration of advanced models like long short-term memory (LSTM) and gated recurrent units (GRU). Attention mechanisms are introduced as pivotal components, allowing models to focus on relevant information within sequences.
Furthermore, the unit introduces transformers, showcasing their innovative architecture that solely relies on attention mechanisms, enabling parallelization and capturing long-range dependencies efficiently. Applications across machine translation, sentiment analysis, and speech recognition demonstrate the versatility of these models in various domains. Through this comprehensive exploration, students gain insight into the evolving landscape of sequential data processing and the transformative potential of attention-based architectures like transformers.
MAI4CAREU - Deep Learning - Attention and Transformers
The first of the advanced units, traces the evolution of recurrent neural networks (RNNs) from basic architectures to the emergence of transformers, underlining the necessity for more efficient and scalable models. As sequences grew longer, these models struggled to capture long-range dependencies effectively. The introduction of attention mechanisms mitigated this challenge by allowing models to focus on relevant information, paving the way for transformers. Transformers revolutionized sequence modelling by solely relying on attention mechanisms, enabling parallelization and efficient handling of long-range dependencies. This narrative underscores the iterative progression towards attention-based architectures driven by the need for scalable models capable of handling diverse sequential data efficiently across domains like machine translation, sentiment analysis, and speech recognition.
MAI4CAREU - Deep Learning - Generative Adversarial Networks
Following up the unit on transformers, the next advanced topic introduces generative modelling, where students will learn the principles of generative adversarial networks (GANs) alongside other influential approaches like deep generative adversarial networks (DGANs), deep belief networks (DBNs), and encoder-decoder architectures. The unit elucidates the foundational principles of GANs, emphasizing the adversarial training process between a generator and discriminator to produce realistic data samples. Additionally, students learn about the advancements introduced by DGANs, which leverage deep architectures to enhance the quality and diversity of generated data samples. DBNs are introduced as probabilistic generative models capable of learning intricate data distributions, while encoder-decoder architectures enable tasks such as image-to-image translation and text-to-image synthesis. Throughout the lecture, the profound impact of these generative models on diverse domains, from computer vision to natural language processing, is showcased, highlighting their transformative role in modern machine learning research and applications.
MAI4CAREU - Deep Learning – Deep Reinforcement Learning
The second-to-the-last unit focuses on deep reinforcement learning (DRL), where students are introduced to the principles and theory behind training agents to make sequential decisions in dynamic environments. The lecture covers the foundational concepts of reinforcement learning (RL), highlighting the interaction between an agent, its environment, and rewards. Students learn how deep learning techniques are integrated with RL algorithms to handle high-dimensional input spaces and complex decision-making tasks. Applications of DRL in domains such as robotics, game playing, and autonomous systems are explored, showcasing its potential for solving real-world problems. However, the lecture also addresses the limitations and challenges of DRL, including sample inefficiency, stability issues, and the need for exploration strategies. Despite these challenges, the lecture underscores the transformative impact of DRL on various fields, from healthcare to entertainment, and its potential to revolutionize the way machines learn and interact with the world. Through this comprehensive exploratory unit, students will gain a deep understanding of the principles, applications, challenges, and impact of DRL in modern artificial intelligence research and practice.
MAI4CAREU - Deep Learning - Emerging and Advanced Topics in Deep Learning
The last, but diverse and enlightening unit, introduces emerging topics in deep learning, teaching how emerging trends and neural network models and architectures reshape a diverse range of domains. The unit first introduces region proposal networks (RPNs) and single-shot detectors (SSDs), streamlining object detection with efficient region proposals and classifications. Next, Graph neural networks (GNNs) are discussed for their prowess in modelling complex data relationships, promising breakthroughs in areas like social networks and drug discovery.
Lastly, the lecture explores the trend of migrating deep neural networks to resource-constrained edge devices, employing techniques like model efficiency, pruning, and quantization. The unit concludes the learning path, by embracing these innovations and showing how deep learning is poised to revolutionize modern societies, offering solutions to pressing challenges across healthcare, transportation, and beyond, and paving the way for a smarter, more connected world. Through this exploration, students complete the learning path into the profound impact of deep learning on the future landscape of technology and society.