Convolutional Neural Networks and Modern Architectures

Author

Your Name

Published

January 28, 2025

Introduction

This document covers the fundamental concepts and architectures of Convolutional Neural Networks (CNNs), starting with their general structure and progressing to advanced models like VGGNet and ResNet. The lecture begins with an overview of CNNs, focusing on their architecture and the role of convolutional layers, max pooling layers, and fully connected layers. It then delves into a detailed analysis of AlexNet, one of the pioneering CNN architectures that significantly outperformed traditional machine learning algorithms with an error rate of 18.2%. The discussion extends to transfer learning, a crucial technique for leveraging pre-trained models to address data scarcity issues. Finally, the lecture explores modern CNN architectures, including VGGNet and ResNet, highlighting their design principles, advantages, and practical applications. Key concepts such as one-by-one convolutional filters and the Inception module are also introduced to provide a comprehensive understanding of contemporary CNN advancements.

Convolutional Neural Networks (CNNs)

General Architecture

A typical CNN architecture can be broadly divided into two main components: feature extraction and classification.

Input Layer

The input layer of a CNN typically receives an image as input.

Feature Extraction

This part consists of alternating convolutional and max pooling layers designed to extract relevant features from the input image. This sequence of layers progressively reduces the spatial dimensions while retaining essential information.

Convolutional Layer: Applies filters to the input to create feature maps. These filters are designed to detect specific patterns or features in the input image.
Max Pooling Layer: Reduces the spatial dimensions of the feature maps, retaining only the most important information. This downsampling process helps to reduce computational complexity and provides a degree of translation invariance.

Classification

After feature extraction, the resulting feature maps are flattened into a vector and fed into fully connected layers for classification.

Flatten: Reshapes the feature maps into a vector. This step prepares the data for input into the fully connected layers.
Fully Connected Layers: Standard neural network layers that perform classification. Each neuron in a fully connected layer is connected to every neuron in the previous layer.
Softmax Activation: Often used in the final layer for multi-class classification to produce probabilities for each class. The softmax function ensures that the output probabilities sum to 1, providing a probability distribution over the possible classes.

Convolutional Layers

Filters/Kernels

Filters (or kernels) are small matrices that are convolved with the input image to extract features. Each filter is designed to detect a specific pattern, such as edges, corners, or textures.

Feature Maps

Feature maps are the output of applying a filter to an input. Each filter produces a distinct feature map, representing the presence and location of the specific feature detected by that filter.

Max Pooling Layers

Downsampling

Max pooling reduces the spatial dimensions of the feature maps by selecting the maximum value within a defined window (e.g., 2x2). This process helps to retain the most salient features while reducing computational complexity and providing a degree of translation invariance.

Fully Connected Layers

Fully connected layers are standard neural network layers where each neuron is connected to every neuron in the previous layer. These layers are typically used for classification after the feature extraction phase. They learn complex combinations of features extracted by the convolutional layers to make predictions.

Softmax Activation

Multi-class Classification

The softmax function is often used in the final layer of a CNN for multi-class classification. It converts the output of the previous layer into a probability distribution over multiple classes, ensuring that the sum of probabilities across all classes equals 1. This allows the network to output a probability for each possible class, indicating the likelihood that the input image belongs to each class.

AlexNet

Architecture Overview

AlexNet was one of the first CNN architectures to significantly outperform traditional machine learning algorithms. Presented by Alex Krizhevsky, it achieved an error rate of 18.2% on the ImageNet dataset. Its architecture consists of five convolutional layers followed by three fully connected layers:

Convolutional Layers and Filter Sizes

Layer 1: 11x11 filters with ReLU activation and max pooling.
Layer 2: 5x5 filters with ReLU activation and max pooling.
Layers 3-5: 3x3 filters with ReLU activation and max pooling.

ReLU Activation Function

ReLU (Rectified Linear Unit) is used as the activation function after each convolutional layer. It introduces non-linearity and helps to mitigate the vanishing gradient problem by outputting the input directly if it is positive, otherwise outputting zero.

Max Pooling Layers

Max pooling layers are used to reduce the spatial dimensions of the feature maps, retaining the most important information and reducing computational load.

Fully Connected (Dense) Layers

AlexNet includes three fully connected layers after the convolutional and max pooling layers. These layers learn to interpret the features extracted by the convolutional layers and perform the final classification.

Dropout Regularization

Dropout is a regularization technique used to prevent overfitting by randomly setting a fraction of the neurons to zero during each training iteration. This forces the network to learn more robust features that are not dependent on specific neurons.

Experimental Analysis

Impact of Removing Dense Layers on Performance

Removing the last dense layer resulted in a 1.1% performance drop. Removing both of the last two dense layers significantly increased the error rate, demonstrating their importance in the classification process. However, this also resulted in a large reduction in the number of parameters.

Impact of Removing Convolutional Layers on Performance

Removing two convolutional layers increased the error rate substantially, despite a smaller reduction in the number of parameters compared to removing dense layers. This highlights the importance of the feature extraction performed by the convolutional layers.

Impact of Removing Dropout on Performance

Removing dropout and parts of the convolutional layers led to a significant increase in the error rate, almost failing the classification, indicating dropout’s importance in preventing overfitting and the importance of the depth of the network.

Visualization of Learned Filters

Low-Level Feature Extraction (Edges, Lines)

The first layer of filters in AlexNet learns basic features such as horizontal, vertical, and diagonal lines. These filters are similar to Gabor filters used in traditional computer vision.

Mid-Level Feature Extraction (Corners, Shapes)

The second layer learns more complex shapes like corners and circles, combining the basic features from the first layer.

High-Level Feature Extraction (Object Parts, Objects)

Higher layers learn to recognize complex patterns and object parts, such as dog faces or car parts. These filters are more abstract and represent higher-level concepts.

Relationship to Gabor Filters

Similarities in Learned Filters

Some filters learned by AlexNet resemble Gabor filters, which are hand-designed filters used in traditional computer vision for edge detection. This suggests that the network independently learns filters that have been found useful in traditional approaches, demonstrating the effectiveness of the learning process.

Transfer Learning

Concept and Motivation

Leveraging Pre-trained Models

Transfer learning involves using a model trained on a large dataset for a different but related task. This approach leverages the knowledge gained from the initial training to improve performance on the new task, especially when the new dataset is small. The underlying principle is that features learned from a large dataset can be generalizable and useful for other related tasks.

Addressing Limited Data

Transfer learning is particularly useful when the dataset for the new task is small, as it helps to mitigate the challenges of training a deep network from scratch with limited data. Deep networks typically require large amounts of data to train effectively; transfer learning allows us to leverage pre-existing knowledge to overcome this limitation. For example, a company might have very few images of a specific product they want to recognize.

Strategies

There are two main strategies for transfer learning: fine-tuning a pre-trained model and using a CNN as a feature extractor.

Fine-tuning Pre-trained Models

Fine-tuning involves taking a pre-trained model and further training it on the new dataset. This can involve updating the weights of all layers or only a subset of layers. The initial layers, which have learned general features, can be frozen, while the later layers, which learn task-specific features, are fine-tuned on the new data.

Using CNNs as Feature Extractors

In this approach, the pre-trained CNN is used to extract features from the new dataset, and these features are then fed into a separate classifier (e.g., SVM or random forest). The pre-trained CNN acts as a fixed feature extractor, and only the new classifier is trained on the new data. This is useful when the new dataset is very small and fine-tuning might lead to overfitting.

Adapting Pre-trained Models

Freezing Initial Layers

When the new dataset is very small, it is common to freeze the weights of the initial layers of the pre-trained model and only train the final layers. This is because the initial layers have learned general features that are likely to be relevant to the new task, while the final layers need to be adapted to the specific characteristics of the new data.

Training New Layers

For larger datasets, more layers can be unfrozen and trained on the new data, allowing the model to adapt more specifically to the new task. The more data available, the more layers can be fine-tuned without risking overfitting.

Data Set Size Considerations

Impact on Fine-tuning Strategy

The size of the new dataset influences the fine-tuning strategy:

Very Small Dataset: Freeze most layers (or all convolutional layers) and only train the final layer(s) or use the pre-trained model as a feature extractor and train a separate classifier.
Medium Dataset: Freeze fewer layers and train more layers, allowing for some adaptation to the new task while still leveraging the knowledge from the pre-trained model.
Large Dataset: Potentially train the entire network from scratch or fine-tune all layers, as there is enough data to adapt the model to the new task without overfitting.

The choice between fine-tuning and using the model as a feature extractor also depends on the similarity between the original task and the new task. If the tasks are very similar, fine-tuning is often more effective. If the tasks are quite different, using the model as a feature extractor might be a better choice.

Modern CNN Architectures

VGGNet

Architecture and Design Principles

VGGNet, developed by the Visual Geometry Group (VGG) at the University of Oxford, is characterized by its simplicity and depth. It uses small 3x3 convolutional filters throughout the network and stacks multiple convolutional layers before each max pooling operation, increasing the depth of the network compared to AlexNet. The architecture achieved a significantly reduced error rate on the ImageNet dataset compared to previous models.

Use of Small 3x3 Convolutional Filters

VGGNet exclusively uses 3x3 convolutional filters, which simplifies the architecture and reduces the number of parameters compared to using larger filters. This design choice was based on the observation that multiple small filters can have the same receptive field as a single large filter, but with increased non-linearity.

Receptive Field Analysis with Stacked 3x3 Filters

Two stacked 3x3 filters have the same receptive field as a single 5x5 filter, and three stacked 3x3 filters have the same receptive field as a single 7x7 filter. This means that the network can learn more complex features with smaller filters by stacking them.

Example: Consider a 5x5 image. Applying two 3x3 filters sequentially results in a single output value, similar to applying a single 5x5 filter.
- First 3x3 filter application reduces the 5x5 image to a 3x3 feature map.
- Second 3x3 filter application reduces the 3x3 feature map to a single value.
Similarly, applying three 3x3 filters to a 7x7 image results in a single output value, equivalent to a single 7x7 filter.

Parameter Efficiency Compared to Larger Filters

Using stacked 3x3 filters is more parameter-efficient than using larger filters. For example, three 3x3 filters have $3 \times (3 \times 3) = 27$ parameters, while a single 7x7 filter has $7 \times 7 = 49$ parameters. This reduction in parameters helps to reduce the risk of overfitting.

Increased Non-linearity with Multiple Layers

Stacked 3x3 filters introduce more non-linearity compared to a single large filter, as each 3x3 filter is followed by a non-linear activation function (e.g., ReLU). This increased non-linearity allows the network to learn more complex functions and improve its ability to discriminate between different classes.

Training Details (Batch Size, Optimizer, Learning Rate, Regularization)

Batch Size: 256 samples per batch.
Optimizer: Stochastic Gradient Descent (SGD) with momentum.
Learning Rate: Initial learning rate of $10^{-1}$, decreased by a factor of 10 during training when the validation accuracy stopped improving.
Regularization: L2 regularization and dropout (after the first two fully connected layers) are used to prevent overfitting.

ResNet (Residual Networks)

Skip Connections (Shortcut Connections) and their Purpose

ResNet introduces skip connections, also known as shortcut connections, which allow the gradient to be directly backpropagated to earlier layers, mitigating the vanishing gradient problem that can occur in very deep networks. These connections allow the network to learn residual functions, making it easier to train very deep networks.

Addressing the Vanishing Gradient Problem in Deep Networks

Skip connections help to address the vanishing gradient problem by providing an alternative path for the gradient to flow, making it easier to train very deep networks. This is because the gradient can be directly backpropagated through the skip connection, bypassing the convolutional layers that might otherwise reduce its magnitude.

Residual Blocks and their Structure

A residual block consists of multiple convolutional layers with a skip connection that adds the input of the block to its output. The output $H(x)$ of a residual block can be expressed as $H(x) = F(x) + x$, where $F(x)$ is the residual function learned by the convolutional layers and $x$ is the input to the block.

Residual Block Structure

Input: $x$
Convolutional Layers: Learn a residual function $F(x)$
Skip Connection: Adds the input $x$ to the output of the convolutional layers
Output: $H(x) = F(x) + x$

Dimensionality Matching in Skip Connections

If the input and output dimensions of a residual block differ, a linear transformation (e.g., a 1x1 convolution) is applied to the input to match the output dimensions. This is typically done when the number of channels changes between the input and output of the block. The linear transformation $W_s$ is applied to the input $x$ to match the dimensions of $F(x)$, resulting in $H(x) = F(x) + W_s x$. This matrix is learned during the training.

ResNet as an Ensemble of Smaller Networks

ResNet can be viewed as an ensemble of shallower networks, where each residual block acts as a smaller network, and their outputs are combined to produce the final result. This is because the skip connections allow the network to learn residual functions that are added to the input, effectively creating multiple paths through the network.

The output $y$ of a ResNet with four residual blocks can be expressed as: \[\begin{aligned} H_1 &= x + F_1(x) \\ H_2 &= H_1 + F_2(H_1) = x + F_1(x) + F_2(x + F_1(x)) \\ H_3 &= H_2 + F_3(H_2) = x + F_1(x) + F_2(x + F_1(x)) + F_3(x + F_1(x) + F_2(x + F_1(x))) \\ y = H_4 &= H_3 + F_4(H_3) = x + \sum_{i=1}^{4} F_i(\text{previous } H) \end{aligned}\] This shows that the output is a combination of the input $x$ and the outputs of four smaller networks $F_i$.

One-by-One Convolutional Filters

Concept and Operation

Pointwise Convolutions

One-by-one convolutional filters perform pointwise convolutions, where each filter is a 1x1 matrix applied to each spatial location of the input tensor. They operate across the depth of the input tensor, performing a weighted sum of the values in each channel at each spatial location.

Use in Channel Manipulation

Reducing the Number of Channels (Dimensionality Reduction)

One-by-one convolutions can reduce the number of channels in a tensor by applying a set of 1x1 filters equal to the desired number of output channels. For example, if an input tensor has 32 channels and we want to reduce it to 3 channels, we can apply three 1x1x32 filters. Each filter will produce a single output value at each spatial location, resulting in an output tensor with 3 channels.

Suppose we have an input tensor of size $6 \times 6 \times 32$. Applying a single $1 \times 1 \times 32$ filter results in an output tensor of size $6 \times 6 \times 1$. The filter performs a weighted sum of the 32 input channels at each spatial location, producing a single output value.

If we apply three such filters, we obtain an output tensor of size $6 \times 6 \times 3$. Each filter produces one channel of the output tensor.

Increasing the Number of Channels (Dimensionality Expansion)

Similarly, one-by-one convolutions can increase the number of channels by applying a set of 1x1 filters greater than the number of input channels. For example, if an input tensor has 3 channels and we want to increase it to 10 channels, we can apply ten 1x1x3 filters. Each filter will produce a single output value at each spatial location, resulting in an output tensor with 10 channels. This can be useful for introducing non-linearity or for preparing the tensor for subsequent convolutional layers that require a different number of input channels.

Other Uses

Computational Efficiency: 1x1 convolutions can be used to reduce the computational cost of subsequent convolutional layers by reducing the number of channels before applying more expensive filters (e.g., 3x3, 5x5).
Non-linearity: Applying a non-linear activation function after a 1x1 convolution introduces non-linearity without changing the spatial dimensions or the number of channels.

Inception Module

Motivation and Design Principles

Addressing the Choice of Filter Size

The Inception module, introduced by Google, addresses the challenge of choosing the appropriate filter size by using multiple filter sizes in parallel within a single module. This allows the network to learn features at different scales simultaneously, making it more robust to variations in object size and location within the input image.

Architecture

Parallel Convolutional Paths with Different Filter Sizes

The Inception module includes parallel convolutional paths with different filter sizes (e.g., 1x1, 3x3, 5x5) applied to the same input. Each path extracts features at a different scale, and the results are concatenated to produce the output of the module.

Max Pooling within the Module

In addition to convolutional paths, the Inception module often includes a max pooling path to capture features at different scales and provide some degree of translation invariance.

Concatenation of Feature Maps from Different Paths

The feature maps from the different paths are concatenated along the depth dimension to produce the output of the Inception module. This creates a richer representation of the input, combining features learned at different scales.

Computational Efficiency

Use of 1x1 Convolutions for Dimensionality Reduction

One-by-one convolutions are used within the Inception module to reduce the number of channels before applying more computationally expensive filters (e.g., 3x3, 5x5). This reduces the computational cost while maintaining performance. By reducing the number of channels, the 1x1 convolutions reduce the number of parameters in the subsequent convolutional layers, making the network more efficient. For example, if the input tensor has 256 channels, a 1x1 convolution can be used to reduce it to 64 channels before applying a 3x3 convolution. This significantly reduces the number of computations required. Without this dimensionality reduction, applying a 5x5 filter directly to an input with a large number of channels would result in a very large number of multiplications (around 120 million). Using a 1x1 convolution to reduce the number of channels first can reduce this number significantly (to around 10-20 million).

Inception as a Building Block

Replication of Inception Modules

The Inception module is used as a building block in deeper networks, where multiple Inception modules are stacked to create a deep architecture. This allows the network to learn features at multiple scales at each level of the hierarchy, creating a very powerful and flexible architecture.

Auxiliary Classifiers

Intermediate Supervision

Inception networks often include auxiliary classifiers, which are additional classifiers connected to intermediate layers of the network. These classifiers provide intermediate supervision during training, helping to stabilize training and improve performance. They are connected to the output of intermediate Inception modules and are used to inject additional gradient information during backpropagation. This helps to prevent the vanishing gradient problem and encourages the intermediate layers to learn more discriminative features.

The final classification is based on the output of the last Inception module, but the auxiliary classifiers provide additional guidance during training.

Inception with ResNet

Combining Inception and Skip Connections

The Inception architecture can be combined with ResNet’s skip connections to create even deeper and more powerful networks. This combination leverages the strengths of both architectures to achieve state-of-the-art performance. The skip connections help to mitigate the vanishing gradient problem, while the Inception modules allow the network to learn features at multiple scales.

The resulting architecture, known as Inception-ResNet, is very deep and can achieve very high accuracy on challenging image classification tasks. It consists of multiple Inception modules with skip connections, allowing for very deep networks. It can be seen as an ensemble of networks, similar to ResNet, but with Inception modules as the building blocks.

Conclusion

This lecture provided a comprehensive overview of Convolutional Neural Networks (CNNs), from their basic architecture to advanced models like VGGNet and ResNet. Key concepts such as convolutional layers, max pooling, and fully connected layers were explained, along with the pioneering architecture of AlexNet, which achieved an error rate of 18.2% on the ImageNet dataset. The importance of transfer learning was highlighted, demonstrating how pre-trained models can be adapted for new tasks, especially when data is limited. Modern CNN architectures like VGGNet and ResNet were discussed in detail, emphasizing their design principles and advantages, such as the use of stacked 3x3 convolutional filters in VGGNet and skip connections in ResNet. Additionally, the concept of one-by-one convolutional filters and the Inception module were introduced, showcasing their roles in contemporary CNN advancements, particularly in manipulating channel dimensions and improving computational efficiency.

The lecture underscored the significance of understanding these concepts for practical applications in image classification and computer vision. We saw how different architectures, such as AlexNet, VGGNet, and ResNet, have evolved to address the challenges of training deep networks and improving performance. We also learned about techniques like transfer learning and the use of one-by-one convolutions that can help us to build more efficient and effective models. Future topics may include exploring transformer architectures, which are currently state-of-the-art in many natural language processing tasks and are beginning to be applied to computer vision as well, and further advancements in deep learning.

--- title: "Convolutional Neural Networks and Modern Architectures" author: "Your Name" date: "2025-01-28" format: html: toc: true # Table of Contents toc-depth: 2 code-tools: true theme: cosmo # Or "journal" for Distill-like minimalism --- # Introduction This document covers the fundamental concepts and architectures of Convolutional Neural Networks (CNNs), starting with their general structure and progressing to advanced models like VGGNet and ResNet. The lecture begins with an overview of CNNs, focusing on their architecture and the role of convolutional layers, max pooling layers, and fully connected layers. It then delves into a detailed analysis of AlexNet, one of the pioneering CNN architectures that significantly outperformed traditional machine learning algorithms with an error rate of 18.2%. The discussion extends to transfer learning, a crucial technique for leveraging pre-trained models to address data scarcity issues. Finally, the lecture explores modern CNN architectures, including VGGNet and ResNet, highlighting their design principles, advantages, and practical applications. Key concepts such as one-by-one convolutional filters and the Inception module are also introduced to provide a comprehensive understanding of contemporary CNN advancements. # Convolutional Neural Networks (CNNs) ## General Architecture A typical CNN architecture can be broadly divided into two main components: feature extraction and classification. ### Input Layer The input layer of a CNN typically receives an image as input. ### Feature Extraction This part consists of alternating convolutional and max pooling layers designed to extract relevant features from the input image. This sequence of layers progressively reduces the spatial dimensions while retaining essential information. - **Convolutional Layer**: Applies filters to the input to create feature maps. These filters are designed to detect specific patterns or features in the input image. - **Max Pooling Layer**: Reduces the spatial dimensions of the feature maps, retaining only the most important information. This downsampling process helps to reduce computational complexity and provides a degree of translation invariance. ### Classification After feature extraction, the resulting feature maps are flattened into a vector and fed into fully connected layers for classification. - **Flatten**: Reshapes the feature maps into a vector. This step prepares the data for input into the fully connected layers. - **Fully Connected Layers**: Standard neural network layers that perform classification. Each neuron in a fully connected layer is connected to every neuron in the previous layer. - **Softmax Activation**: Often used in the final layer for multi-class classification to produce probabilities for each class. The softmax function ensures that the output probabilities sum to 1, providing a probability distribution over the possible classes. ## Convolutional Layers ### Filters/Kernels Filters (or kernels) are small matrices that are convolved with the input image to extract features. Each filter is designed to detect a specific pattern, such as edges, corners, or textures. ### Feature Maps Feature maps are the output of applying a filter to an input. Each filter produces a distinct feature map, representing the presence and location of the specific feature detected by that filter. ## Max Pooling Layers ### Downsampling Max pooling reduces the spatial dimensions of the feature maps by selecting the maximum value within a defined window (e.g., 2x2). This process helps to retain the most salient features while reducing computational complexity and providing a degree of translation invariance. ## Fully Connected Layers Fully connected layers are standard neural network layers where each neuron is connected to every neuron in the previous layer. These layers are typically used for classification after the feature extraction phase. They learn complex combinations of features extracted by the convolutional layers to make predictions. ## Softmax Activation ### Multi-class Classification The softmax function is often used in the final layer of a CNN for multi-class classification. It converts the output of the previous layer into a probability distribution over multiple classes, ensuring that the sum of probabilities across all classes equals 1. This allows the network to output a probability for each possible class, indicating the likelihood that the input image belongs to each class. # AlexNet ## Architecture Overview AlexNet was one of the first CNN architectures to significantly outperform traditional machine learning algorithms. Presented by Alex Krizhevsky, it achieved an error rate of 18.2% on the ImageNet dataset. Its architecture consists of five convolutional layers followed by three fully connected layers: ### Convolutional Layers and Filter Sizes - **Layer 1**: 11x11 filters with ReLU activation and max pooling. - **Layer 2**: 5x5 filters with ReLU activation and max pooling. - **Layers 3-5**: 3x3 filters with ReLU activation and max pooling. ### ReLU Activation Function ReLU (Rectified Linear Unit) is used as the activation function after each convolutional layer. It introduces non-linearity and helps to mitigate the vanishing gradient problem by outputting the input directly if it is positive, otherwise outputting zero. ### Max Pooling Layers Max pooling layers are used to reduce the spatial dimensions of the feature maps, retaining the most important information and reducing computational load. ### Fully Connected (Dense) Layers AlexNet includes three fully connected layers after the convolutional and max pooling layers. These layers learn to interpret the features extracted by the convolutional layers and perform the final classification. ### Dropout Regularization Dropout is a regularization technique used to prevent overfitting by randomly setting a fraction of the neurons to zero during each training iteration. This forces the network to learn more robust features that are not dependent on specific neurons. ## Experimental Analysis ### Impact of Removing Dense Layers on Performance Removing the last dense layer resulted in a 1.1% performance drop. Removing both of the last two dense layers significantly increased the error rate, demonstrating their importance in the classification process. However, this also resulted in a large reduction in the number of parameters. ### Impact of Removing Convolutional Layers on Performance Removing two convolutional layers increased the error rate substantially, despite a smaller reduction in the number of parameters compared to removing dense layers. This highlights the importance of the feature extraction performed by the convolutional layers. ### Impact of Removing Dropout on Performance Removing dropout and parts of the convolutional layers led to a significant increase in the error rate, almost failing the classification, indicating dropout's importance in preventing overfitting and the importance of the depth of the network. ## Visualization of Learned Filters ### Low-Level Feature Extraction (Edges, Lines) The first layer of filters in AlexNet learns basic features such as horizontal, vertical, and diagonal lines. These filters are similar to Gabor filters used in traditional computer vision. ### Mid-Level Feature Extraction (Corners, Shapes) The second layer learns more complex shapes like corners and circles, combining the basic features from the first layer. ### High-Level Feature Extraction (Object Parts, Objects) Higher layers learn to recognize complex patterns and object parts, such as dog faces or car parts. These filters are more abstract and represent higher-level concepts. ## Relationship to Gabor Filters ### Similarities in Learned Filters Some filters learned by AlexNet resemble Gabor filters, which are hand-designed filters used in traditional computer vision for edge detection. This suggests that the network independently learns filters that have been found useful in traditional approaches, demonstrating the effectiveness of the learning process. # Transfer Learning ## Concept and Motivation ### Leveraging Pre-trained Models ::: tcolorbox Transfer learning involves using a model trained on a large dataset for a different but related task. This approach leverages the knowledge gained from the initial training to improve performance on the new task, especially when the new dataset is small. The underlying principle is that features learned from a large dataset can be generalizable and useful for other related tasks. ::: ### Addressing Limited Data Transfer learning is particularly useful when the dataset for the new task is small, as it helps to mitigate the challenges of training a deep network from scratch with limited data. Deep networks typically require large amounts of data to train effectively; transfer learning allows us to leverage pre-existing knowledge to overcome this limitation. For example, a company might have very few images of a specific product they want to recognize. ## Strategies There are two main strategies for transfer learning: fine-tuning a pre-trained model and using a CNN as a feature extractor. ### Fine-tuning Pre-trained Models ::: tcolorbox Fine-tuning involves taking a pre-trained model and further training it on the new dataset. This can involve updating the weights of all layers or only a subset of layers. The initial layers, which have learned general features, can be frozen, while the later layers, which learn task-specific features, are fine-tuned on the new data. ::: ### Using CNNs as Feature Extractors ::: tcolorbox In this approach, the pre-trained CNN is used to extract features from the new dataset, and these features are then fed into a separate classifier (e.g., SVM or random forest). The pre-trained CNN acts as a fixed feature extractor, and only the new classifier is trained on the new data. This is useful when the new dataset is very small and fine-tuning might lead to overfitting. ::: ## Adapting Pre-trained Models ### Freezing Initial Layers When the new dataset is very small, it is common to freeze the weights of the initial layers of the pre-trained model and only train the final layers. This is because the initial layers have learned general features that are likely to be relevant to the new task, while the final layers need to be adapted to the specific characteristics of the new data. ### Training New Layers For larger datasets, more layers can be unfrozen and trained on the new data, allowing the model to adapt more specifically to the new task. The more data available, the more layers can be fine-tuned without risking overfitting. ## Data Set Size Considerations ### Impact on Fine-tuning Strategy The size of the new dataset influences the fine-tuning strategy: - **Very Small Dataset**: Freeze most layers (or all convolutional layers) and only train the final layer(s) or use the pre-trained model as a feature extractor and train a separate classifier. - **Medium Dataset**: Freeze fewer layers and train more layers, allowing for some adaptation to the new task while still leveraging the knowledge from the pre-trained model. - **Large Dataset**: Potentially train the entire network from scratch or fine-tune all layers, as there is enough data to adapt the model to the new task without overfitting. The choice between fine-tuning and using the model as a feature extractor also depends on the similarity between the original task and the new task. If the tasks are very similar, fine-tuning is often more effective. If the tasks are quite different, using the model as a feature extractor might be a better choice. # Modern CNN Architectures ## VGGNet ### Architecture and Design Principles VGGNet, developed by the Visual Geometry Group (VGG) at the University of Oxford, is characterized by its simplicity and depth. It uses small 3x3 convolutional filters throughout the network and stacks multiple convolutional layers before each max pooling operation, increasing the depth of the network compared to AlexNet. The architecture achieved a significantly reduced error rate on the ImageNet dataset compared to previous models. ### Use of Small 3x3 Convolutional Filters VGGNet exclusively uses 3x3 convolutional filters, which simplifies the architecture and reduces the number of parameters compared to using larger filters. This design choice was based on the observation that multiple small filters can have the same receptive field as a single large filter, but with increased non-linearity. ### Receptive Field Analysis with Stacked 3x3 Filters ::: tcolorbox Two stacked 3x3 filters have the same receptive field as a single 5x5 filter, and three stacked 3x3 filters have the same receptive field as a single 7x7 filter. This means that the network can learn more complex features with smaller filters by stacking them. ::: - **Example**: Consider a 5x5 image. Applying two 3x3 filters sequentially results in a single output value, similar to applying a single 5x5 filter. - First 3x3 filter application reduces the 5x5 image to a 3x3 feature map. - Second 3x3 filter application reduces the 3x3 feature map to a single value. Similarly, applying three 3x3 filters to a 7x7 image results in a single output value, equivalent to a single 7x7 filter. ::: center ::: ### Parameter Efficiency Compared to Larger Filters Using stacked 3x3 filters is more parameter-efficient than using larger filters. For example, three 3x3 filters have $3 \times (3 \times 3) = 27$ parameters, while a single 7x7 filter has $7 \times 7 = 49$ parameters. This reduction in parameters helps to reduce the risk of overfitting. ### Increased Non-linearity with Multiple Layers Stacked 3x3 filters introduce more non-linearity compared to a single large filter, as each 3x3 filter is followed by a non-linear activation function (e.g., ReLU). This increased non-linearity allows the network to learn more complex functions and improve its ability to discriminate between different classes. ### Training Details (Batch Size, Optimizer, Learning Rate, Regularization) - **Batch Size**: 256 samples per batch. - **Optimizer**: Stochastic Gradient Descent (SGD) with momentum. - **Learning Rate**: Initial learning rate of $10^{-1}$, decreased by a factor of 10 during training when the validation accuracy stopped improving. - **Regularization**: L2 regularization and dropout (after the first two fully connected layers) are used to prevent overfitting. ## ResNet (Residual Networks) ### Skip Connections (Shortcut Connections) and their Purpose ::: tcolorbox ResNet introduces skip connections, also known as shortcut connections, which allow the gradient to be directly backpropagated to earlier layers, mitigating the vanishing gradient problem that can occur in very deep networks. These connections allow the network to learn residual functions, making it easier to train very deep networks. ::: ### Addressing the Vanishing Gradient Problem in Deep Networks Skip connections help to address the vanishing gradient problem by providing an alternative path for the gradient to flow, making it easier to train very deep networks. This is because the gradient can be directly backpropagated through the skip connection, bypassing the convolutional layers that might otherwise reduce its magnitude. ### Residual Blocks and their Structure ::: tcolorbox A residual block consists of multiple convolutional layers with a skip connection that adds the input of the block to its output. The output $H(x)$ of a residual block can be expressed as $H(x) = F(x) + x$, where $F(x)$ is the residual function learned by the convolutional layers and $x$ is the input to the block. ::: ::: mdframed **Residual Block Structure** - Input: $x$ - Convolutional Layers: Learn a residual function $F(x)$ - Skip Connection: Adds the input $x$ to the output of the convolutional layers - Output: $H(x) = F(x) + x$ ::: ### Dimensionality Matching in Skip Connections If the input and output dimensions of a residual block differ, a linear transformation (e.g., a 1x1 convolution) is applied to the input to match the output dimensions. This is typically done when the number of channels changes between the input and output of the block. The linear transformation $W_s$ is applied to the input $x$ to match the dimensions of $F(x)$, resulting in $H(x) = F(x) + W_s x$. This matrix is learned during the training. ### ResNet as an Ensemble of Smaller Networks ::: tcolorbox ResNet can be viewed as an ensemble of shallower networks, where each residual block acts as a smaller network, and their outputs are combined to produce the final result. This is because the skip connections allow the network to learn residual functions that are added to the input, effectively creating multiple paths through the network. ::: The output $y$ of a ResNet with four residual blocks can be expressed as: $$\begin{aligned} H_1 &= x + F_1(x) \\ H_2 &= H_1 + F_2(H_1) = x + F_1(x) + F_2(x + F_1(x)) \\ H_3 &= H_2 + F_3(H_2) = x + F_1(x) + F_2(x + F_1(x)) + F_3(x + F_1(x) + F_2(x + F_1(x))) \\ y = H_4 &= H_3 + F_4(H_3) = x + \sum_{i=1}^{4} F_i(\text{previous } H) \end{aligned}$$ This shows that the output is a combination of the input $x$ and the outputs of four smaller networks $F_i$. # One-by-One Convolutional Filters ## Concept and Operation ### Pointwise Convolutions ::: tcolorbox One-by-one convolutional filters perform pointwise convolutions, where each filter is a 1x1 matrix applied to each spatial location of the input tensor. They operate across the depth of the input tensor, performing a weighted sum of the values in each channel at each spatial location. ::: ## Use in Channel Manipulation ### Reducing the Number of Channels (Dimensionality Reduction) One-by-one convolutions can reduce the number of channels in a tensor by applying a set of 1x1 filters equal to the desired number of output channels. For example, if an input tensor has 32 channels and we want to reduce it to 3 channels, we can apply three 1x1x32 filters. Each filter will produce a single output value at each spatial location, resulting in an output tensor with 3 channels. Suppose we have an input tensor of size $6 \times 6 \times 32$. Applying a single $1 \times 1 \times 32$ filter results in an output tensor of size $6 \times 6 \times 1$. The filter performs a weighted sum of the 32 input channels at each spatial location, producing a single output value. If we apply three such filters, we obtain an output tensor of size $6 \times 6 \times 3$. Each filter produces one channel of the output tensor. ### Increasing the Number of Channels (Dimensionality Expansion) Similarly, one-by-one convolutions can increase the number of channels by applying a set of 1x1 filters greater than the number of input channels. For example, if an input tensor has 3 channels and we want to increase it to 10 channels, we can apply ten 1x1x3 filters. Each filter will produce a single output value at each spatial location, resulting in an output tensor with 10 channels. This can be useful for introducing non-linearity or for preparing the tensor for subsequent convolutional layers that require a different number of input channels. ::: center ::: ## Other Uses - **Computational Efficiency**: 1x1 convolutions can be used to reduce the computational cost of subsequent convolutional layers by reducing the number of channels before applying more expensive filters (e.g., 3x3, 5x5). - **Non-linearity**: Applying a non-linear activation function after a 1x1 convolution introduces non-linearity without changing the spatial dimensions or the number of channels. # Inception Module ## Motivation and Design Principles ### Addressing the Choice of Filter Size ::: tcolorbox The Inception module, introduced by Google, addresses the challenge of choosing the appropriate filter size by using multiple filter sizes in parallel within a single module. This allows the network to learn features at different scales simultaneously, making it more robust to variations in object size and location within the input image. ::: ## Architecture ### Parallel Convolutional Paths with Different Filter Sizes The Inception module includes parallel convolutional paths with different filter sizes (e.g., 1x1, 3x3, 5x5) applied to the same input. Each path extracts features at a different scale, and the results are concatenated to produce the output of the module. ### Max Pooling within the Module In addition to convolutional paths, the Inception module often includes a max pooling path to capture features at different scales and provide some degree of translation invariance. ### Concatenation of Feature Maps from Different Paths The feature maps from the different paths are concatenated along the depth dimension to produce the output of the Inception module. This creates a richer representation of the input, combining features learned at different scales. ::: center ::: ## Computational Efficiency ### Use of 1x1 Convolutions for Dimensionality Reduction One-by-one convolutions are used within the Inception module to reduce the number of channels before applying more computationally expensive filters (e.g., 3x3, 5x5). This reduces the computational cost while maintaining performance. By reducing the number of channels, the 1x1 convolutions reduce the number of parameters in the subsequent convolutional layers, making the network more efficient. For example, if the input tensor has 256 channels, a 1x1 convolution can be used to reduce it to 64 channels before applying a 3x3 convolution. This significantly reduces the number of computations required. Without this dimensionality reduction, applying a 5x5 filter directly to an input with a large number of channels would result in a very large number of multiplications (around 120 million). Using a 1x1 convolution to reduce the number of channels first can reduce this number significantly (to around 10-20 million). ## Inception as a Building Block ### Replication of Inception Modules The Inception module is used as a building block in deeper networks, where multiple Inception modules are stacked to create a deep architecture. This allows the network to learn features at multiple scales at each level of the hierarchy, creating a very powerful and flexible architecture. ## Auxiliary Classifiers ### Intermediate Supervision ::: tcolorbox Inception networks often include auxiliary classifiers, which are additional classifiers connected to intermediate layers of the network. These classifiers provide intermediate supervision during training, helping to stabilize training and improve performance. They are connected to the output of intermediate Inception modules and are used to inject additional gradient information during backpropagation. This helps to prevent the vanishing gradient problem and encourages the intermediate layers to learn more discriminative features. ::: The final classification is based on the output of the last Inception module, but the auxiliary classifiers provide additional guidance during training. ## Inception with ResNet ### Combining Inception and Skip Connections ::: tcolorbox The Inception architecture can be combined with ResNet's skip connections to create even deeper and more powerful networks. This combination leverages the strengths of both architectures to achieve state-of-the-art performance. The skip connections help to mitigate the vanishing gradient problem, while the Inception modules allow the network to learn features at multiple scales. ::: The resulting architecture, known as Inception-ResNet, is very deep and can achieve very high accuracy on challenging image classification tasks. It consists of multiple Inception modules with skip connections, allowing for very deep networks. It can be seen as an ensemble of networks, similar to ResNet, but with Inception modules as the building blocks. # Conclusion This lecture provided a comprehensive overview of Convolutional Neural Networks (CNNs), from their basic architecture to advanced models like VGGNet and ResNet. Key concepts such as convolutional layers, max pooling, and fully connected layers were explained, along with the pioneering architecture of AlexNet, which achieved an error rate of 18.2% on the ImageNet dataset. The importance of transfer learning was highlighted, demonstrating how pre-trained models can be adapted for new tasks, especially when data is limited. Modern CNN architectures like VGGNet and ResNet were discussed in detail, emphasizing their design principles and advantages, such as the use of stacked 3x3 convolutional filters in VGGNet and skip connections in ResNet. Additionally, the concept of one-by-one convolutional filters and the Inception module were introduced, showcasing their roles in contemporary CNN advancements, particularly in manipulating channel dimensions and improving computational efficiency. The lecture underscored the significance of understanding these concepts for practical applications in image classification and computer vision. We saw how different architectures, such as AlexNet, VGGNet, and ResNet, have evolved to address the challenges of training deep networks and improving performance. We also learned about techniques like transfer learning and the use of one-by-one convolutions that can help us to build more efficient and effective models. Future topics may include exploring transformer architectures, which are currently state-of-the-art in many natural language processing tasks and are beginning to be applied to computer vision as well, and further advancements in deep learning.