Intermediate Deep Learning

Deep Learning

Introduction

These are deep learning papers that are fundamental to research. The intermediate deep learning course includes topics:

  • Deep learning models of the 2010s.
  • Famous deep learning techniques.
  • Basic learning methods.


Contents

# Topic Keyword Paper Source    
1 Fundamentals of Deep Learning Fundamentals, NN, CNN, RNN        
2 Model Learning Technique Optimizer, GD, SDG, Momentum, RMSProp        
    Adagrad Adaptive Subgradient Methods for Online Learning and Stochastic Optimization      
    Adam Adam: A Method for Stochastic Optimization      
    Schedular        
3 Model Design Techniques Activation Functions, Sigmoid, Tanh, ReLU        
    GELU Gaussian Error Linear Units (GELUs)      
    Swish Searching for Activation Functions      
    Batch Normalization Batch Normalization: Accelerating Deep Network Training      
    Layer Normalization Layer Normalization      
    Dropout Dropout: A Simple Way to Prevent Neural Networks from Overfitting      
4 RNN Based Models LSTM Long Short-Term Memory      
    GRU Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation      
    Seq2Seq Sequence to Sequence Learning with Neural Networks      
5 Transformer Attention Neural Machine Translation by Jointly Learning to Align and Translate      
    Transformer Attention Is All You Need      
6 Text Models word2vec Efficient Estimation of Word Representations in Vector Space      
    BERT BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding      
    GPT Improving Language Understanding by Generative Pre-Training      
7 CNN Based Models LeNet Gradient-Based Learning Applied to Document Recognition      
    AlexNet ImageNet Classification with Deep Convolutional Neural Networks      
    ResNet Deep Residual Learning for Image Recognition      
    Inception Going Deeper with Convolutions      
8 Image Detection Models R-CNN Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation      
    Fast R-CNN Fast R-CNN      
    Faster R-CNN Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks      
    YOLO You Only Look Once: Unified, Real-Time Object Detection      
9 Other Learning Methods Contrastive Learning A Simple Framework for Contrastive Learning of Visual Representations      
    Few-Shot Learning Matching Networks for One Shot Learning      
    Knowledge Distillation Distilling the Knowledge in a Neural Network      
10 Generative Models VAE Auto-Encoding Variational Bayes      
    GAN Generative Adversarial Networks      
    Diffusion Denoising Diffusion Probabilistic Models      
11 Graph Based Models GNN The Graph Neural Network      
    GCN Semi-Supervised Classification with Graph Convolutional Networks      
    GAT Graph Attention Networks      
12 Fundamental of Reinforcement Learning Q-Learning        
    SARSA        
    DQN Playing Atari with Deep Reinforcement Learning      
    PG Policy Gradient Methods for Reinforcement Learning with Function Approximation