Introduction to PyTorch for Deep Learning

Author

Your Name

Published

January 28, 2025

Introduction

This document outlines the first lab session of a deep learning course. The primary focus of this session is to introduce PyTorch and its application in building neural networks, starting from basic principles and progressing towards practical implementation. The session is structured around using Google Colab as the coding environment, leveraging its accessibility and free GPU resources.

Session Objectives

The main objectives of this lab session are to:

Familiarize students with the Google Colab environment for deep learning.
Introduce the fundamental concepts of Python and essential libraries such as NumPy and Pandas for data manipulation.
Demonstrate the transition from NumPy to PyTorch for implementing neural networks.
Explain and utilize PyTorch’s automatic differentiation capabilities (autograd).
Guide students through defining neural network models and implementing training loops in PyTorch.
Provide hands-on experience with implementing custom PyTorch modules.
Introduce a practical exercise involving a dynamic neural network architecture to solidify understanding and encourage experimentation.

Session Structure

The session is designed to be interactive, combining guided examples with hands-on exercises. The structure is as follows:

Environment Setup: Introduction to Google Colab, accessing resources, and disabling AI autocompletion features to enhance learning.
Material Distribution: Accessing shared notebooks covering Python basics, NumPy, Pandas/Matplotlib, and the main PyTorch tutorial.
Introductory Notebooks (Optional): Brief overview of Python, NumPy, and Pandas/Matplotlib notebooks for students needing a refresher or introduction to these tools. These notebooks cover fundamental operations and data handling techniques.
Main PyTorch Notebook:
- NumPy to PyTorch Transition: Implementing a simple neural network first in NumPy to illustrate manual gradient computation, then transitioning to PyTorch to demonstrate the advantages of automatic differentiation.
- PyTorch Fundamentals: Introduction to PyTorch tensors, autograd, defining models using nn.Sequential, loss functions, and optimizers.
- Training Loop Implementation: Step-by-step guide to implementing a basic training loop in PyTorch.
- Custom Modules: Creating custom PyTorch modules to build more complex and flexible neural networks.
Exercise: Dynamic Neural Network: A practical exercise involving the implementation of a dynamic neural network architecture. This exercise is designed to challenge students to apply the learned concepts and explore network control.
Q&A and Support: Throughout the session, instructors will be available to answer questions and provide support, both in person and online via chat. A dedicated channel will be available for follow-up questions after the session.

Google Colab and Computational Resources

Google Colab is the chosen platform for this lab due to its ease of access and availability of computational resources.

Access and Setup

Students are required to have a Google account to access Google Colab. The notebooks can be opened directly in Colab via the provided links or by uploading them to Google Drive and opening from there.

Disabling AI Autocompletion

It is recommended to disable the AI-powered autocompletion features (like Gemini) in Google Colab to encourage active learning and prevent reliance on automated code generation during this introductory phase.

Runtime Environment

Google Colab offers various runtime environments, including CPU, GPU, and TPU. For the initial exercises in this lab, using the CPU runtime is sufficient. Instructions on how to change the runtime type to utilize GPUs or TPUs for more computationally intensive tasks will be provided if needed. Note that while GPUs offer significant speedups for deep learning tasks, their usage on Colab may be subject to usage limits. For this session, the exercises are designed to run efficiently on CPU.

Accessing Course Materials

All necessary materials, including Python, NumPy, Pandas, and PyTorch notebooks, are provided through a shared Google Drive folder. The link to this folder has been shared in the chat and via a QR code for easy access. Students are advised to create a copy of these notebooks in their personal Google Drive to ensure they can save their progress and modifications. The original shared folder is read-only to prevent accidental modifications to the source materials.

Notebook Structure

The notebooks are structured to guide students progressively through the concepts, starting with basic implementations and gradually introducing more advanced PyTorch features. Each notebook contains explanations, code examples, and exercises to reinforce learning.

Check-in and Support

A designated time (3:30) is set for a check-in during the lab session to assess student progress and address any immediate questions or difficulties. Furthermore, a dedicated communication channel will be established on Teams to provide ongoing support and answer questions that arise after the lab session. Students are encouraged to utilize this channel to seek assistance and engage in discussions related to the lab materials and exercises.

Initial Setup and Environment

Audience Background

Python Proficiency

At the beginning of the session, a quick poll was conducted to assess the audience’s familiarity with Python. The results indicated that all participants, including remote students, had prior experience with Python coding.

PyTorch Experience

Inquiring about prior PyTorch usage, it was noted that only a few attendees had used PyTorch before. This highlighted the need to start from the basics, catering to those new to the framework.

Deep Learning Model Experience

A significant number of participants indicated they had never coded a deep learning model previously. This confirmed the suitability of starting the lab session from scratch, as planned, to accommodate beginners in deep learning.

Google Colab Setup

Accessing Google Colab

Students were instructed to use Google Colab for all coding activities during the lab. Google Colab was chosen for its accessibility, ease of use, and provision of free computational resources, including GPUs, which can be utilized for more demanding tasks later in the course, although not necessary for the initial exercises. Accessing Colab requires a Google account (Gmail). Users can navigate to Google Colab by searching "Google Colab" on Google or by directly clicking on any of the provided notebook files, which will prompt them to open it with their Google account.

Disabling AI Autocompletion (Gemini)

It was recommended to disable the AI-powered autocompletion feature, specifically mentioning Gemini, which is enabled by default in Google Colab. This suggestion was made to encourage students to actively write and understand the code themselves, rather than relying on AI to generate it, which is crucial for the learning process. Instructions on how to disable this feature within Colab were to be provided if needed.

Accessing and Copying Course Notebooks

Course materials, including several introductory notebooks and the main lab notebook, were made available through a shared Google Drive folder. The link to this folder was shared in the session chat and via a QR code displayed on screen. Students were explicitly instructed to create a copy of the notebooks in their personal Google Drive before starting to work on them. This step is essential because the shared folder is read-only, and any edits made directly to the files in the shared folder would be lost. Creating a copy ensures that students can save their work and progress. To create a copy, students should open a notebook and select "File" $\rightarrow$ "Save a copy in Drive." The copied notebook will be saved in a folder named "Colab Notebooks" within their Google Drive.

Connecting to a Virtual Machine

Google Colab provides access to virtual machines for executing code. By default, Colab connects to a virtual machine with CPU resources. For more computationally intensive tasks, Colab also offers options to connect to virtual machines equipped with GPUs or TPUs. To connect, a "Connect" button is available in the Colab interface. Clicking this button connects the notebook to a virtual machine. While the default CPU instance is sufficient for the initial exercises and is virtually free with no time limits, GPU and TPU resources have usage limits. For the exercises in this lab, running on CPU is adequate, and students were informed that they would not need to switch to a GPU for these tasks. It was also mentioned that changing the runtime type (e.g., from CPU to GPU) will reset the environment, requiring variables to be reinitialized and code cells to be re-executed.

Course Materials Overview

Accessing the Shared Folder

The course materials are organized within a Google Drive folder. Access to this folder is provided through a link shared in the Microsoft Teams chat and via a QR code displayed during the session. This folder contains all the notebooks required for the lab session, excluding solution notebooks to encourage independent problem-solving.

Introductory Notebooks

To ensure all students have the necessary foundational knowledge, three introductory notebooks are provided:

Python Basics Notebook

This notebook is designed for individuals with limited or no prior experience with Python. It covers fundamental Python programming concepts, including basic operations, data structures like lists and sets, and control flow.

NumPy Notebook

This notebook introduces NumPy, a fundamental library in Python for numerical computations. It focuses on array and matrix manipulation, which are essential for implementing mathematical operations in deep learning.

Pandas and Matplotlib Notebook

This notebook covers Pandas and Matplotlib. Pandas is introduced as a library for data manipulation and analysis, particularly for creating and working with data frames, which are useful for tabular data. Matplotlib is presented as a plotting library for data visualization. The notebook also demonstrates how Pandas can be used for basic plotting, offering a convenient way to visualize data directly from data frames.

Main PyTorch Notebook

The central resource for this lab session is a dedicated PyTorch notebook. This notebook is structured to guide students through the process of implementing a simple feed-forward neural network. It begins with a NumPy implementation to illustrate the underlying mathematical operations and the complexity of manual gradient calculation. It then transitions to PyTorch, demonstrating how to implement the same network using PyTorch tensors and leveraging PyTorch’s autograd functionality for automatic differentiation. The notebook includes examples of using nn.Sequential to define neural network models, defining loss functions, and setting up optimizers.

Exploring the PyTorch Notebook

This section details the contents and structure of the primary PyTorch notebook used in the lab session. The notebook is designed to guide students from basic NumPy implementations to advanced PyTorch functionalities for building and training neural networks.

From NumPy to PyTorch: Manual to Automatic Differentiation

NumPy Implementation: Gradients from Scratch

The notebook initiates with a practical example of implementing a neural network using NumPy. This segment is crucial for understanding the underlying mathematical operations involved in neural networks, particularly the manual computation of gradients.

The NumPy implementation deliberately showcases the complexity and intricacies of manual gradient computation. This approach serves a pedagogical purpose: by experiencing the cumbersome nature of calculating gradients by hand using the chain rule, students gain a deeper appreciation for the automatic differentiation capabilities offered by frameworks like PyTorch. The notebook meticulously steps through the process, even verifying the correctness of gradient calculations using pen and paper to ensure accuracy and understanding. This part is intentionally challenging to highlight the benefits of using PyTorch for automatic gradient handling.

Transition to PyTorch Tensors: Embracing Automatic Gradients

Following the NumPy example, the notebook transitions to PyTorch, reimplementing the same neural network using PyTorch tensors. This transition is designed to be seamless, demonstrating the similarities in syntax and operations between NumPy and PyTorch, while introducing the fundamental concept of PyTorch tensors. The code structure remains largely consistent, allowing students to easily compare and contrast the NumPy and PyTorch implementations. The key difference highlighted is the preparation for automatic differentiation, which will be fully leveraged in subsequent sections.

PyTorch Fundamentals: Autograd and Model Building

Introduction to Autograd: Automatic Differentiation Engine

This subsection introduces PyTorch’s automatic differentiation feature, autograd. It explains how PyTorch dynamically builds a computation graph as operations are performed on tensors. This graph tracks all operations, enabling automatic computation of gradients.

Autograd is presented as a game-changing feature that eliminates the need for manual gradient calculations. The explanation emphasizes that when using PyTorch tensors, all operations are recorded in a dynamic computation graph. To compute gradients, one simply needs to call loss.backward() on the loss tensor. PyTorch then traverses the computation graph in reverse, applying the chain rule to automatically compute gradients for all tensors that were involved in the loss calculation and have requires_grad=True. This section underscores the immense simplification and efficiency that autograd brings to neural network development, contrasting sharply with the manual approach demonstrated in NumPy.

Sequential Model Building with `nn.Sequential`

The notebook introduces torch.nn.Sequential as a high-level API for constructing neural networks. nn.Sequential simplifies model definition by allowing users to define a network as a linear sequence of layers.

Defining Loss Functions and Optimizers

This part of the notebook covers how to define loss functions and optimizers in PyTorch. It explains that loss functions are used to quantify the error between the network’s predictions and the actual targets, guiding the learning process. Optimizers, on the other hand, are algorithms that adjust the network’s parameters to minimize the loss function. PyTorch provides a variety of pre-built loss functions (e.g., nn.MSELoss, nn.CrossEntropyLoss) and optimizers (e.g., optim.SGD, optim.Adam) within the torch.nn and torch.optim modules, respectively.

Implementing the Training Loop with Autograd

A complete training loop is presented, integrating autograd, loss functions, and optimizers. The training loop encompasses the following essential steps:

Forward Pass: Input data is passed through the model to obtain predictions.
Loss Computation: The loss function compares the predictions with the actual targets to calculate the loss.
Backward Pass (loss.backward()): autograd computes gradients of the loss with respect to all model parameters.
Optimizer Step (optimizer.step()): The optimizer updates the model parameters using the computed gradients to minimize the loss.
Zero Gradients (optimizer.zero_grad()): Before each backward pass, it’s crucial to zero out the gradients from the previous iteration to prevent accumulation.

This section provides a practical, runnable code example demonstrating how to put these components together to train a simple neural network, highlighting the ease and efficiency of training in PyTorch compared to manual gradient methods.

Custom PyTorch Modules: Building Flexible Networks

Creating Custom Neural Network Modules

The notebook transitions to creating custom neural network modules by subclassing torch.nn.Module. This approach offers greater flexibility and control over network architecture compared to nn.Sequential. It’s explained that custom modules are defined as classes that inherit from nn.Module and must implement two key methods:

__init__(self, ...): The constructor, where layers and sub-modules are defined and initialized.
forward(self, x): Defines the forward pass of the module, specifying how input tensors x are processed through the layers to produce an output tensor.

Implementing the `forward` Pass in Custom Modules

The forward method is emphasized as the core of a custom module. It dictates the exact computations performed by the module. Within the forward method, one specifies how the input tensor is passed through the layers defined in __init__ and what operations are applied. The notebook provides a simple example of creating a custom module, demonstrating the basic structure and usage. It shows how to instantiate the custom module and use it in a similar way to nn.Sequential models, highlighting the modularity and reusability of PyTorch modules.

Exercise: Implementing a Dynamic Neural Network

Challenge: A Dynamically Modulated Network Architecture

The final section of the notebook presents an exercise that challenges students to implement a more complex, dynamic neural network. This network features a unique architecture where a middle linear layer’s presence and repetition are dynamically controlled during the forward pass.

Dynamic Layer Control: Repetition and Dropout Analogy

The exercise requires implementing a neural network with the following structure: Input Layer $\rightarrow$ Activation Function $\rightarrow$ [Middle Linear Layer (repeated 0 to 3 times, dynamically chosen)] $\rightarrow$ Output Linear Layer.

The core challenge lies in the "Middle Linear Layer." This layer is designed to be conditionally included in the network’s forward pass, and if included, it can be repeated 0, 1, 2, or 3 times. The number of repetitions is intended to be dynamically determined (though in the exercise, it might be pre-set or randomly chosen for simplicity). The exercise is described as a way to explore network control and is analogized to a very aggressive form of dropout, where entire layers are dropped out or repeated, rather than individual neurons. The goal is to understand how to build networks with conditional execution paths and to experiment with unconventional architectures. Students are tasked with implementing this network and observing its behavior. The solution to this exercise will be discussed in the next lab session. Students are explicitly advised against using such an architecture in real-world applications, as it is primarily designed for educational purposes to explore dynamic network behavior rather than to achieve state-of-the-art performance.

Concluding Remarks and Next Steps

This section summarizes the concluding discussions of the lab session and outlines the next steps for students.

Q&A and Clarifications

Following the hands-on coding session, a brief question and answer period was held. One notable question from a remote participant concerned the dimensions labeled in a network diagram presented in the notebook. The clarification provided was that dimensions marked as ‘N’ in the diagram referred to the batch size, while ‘H’ indicated feature dimensions or the number of hidden units in a layer. This clarification was made to ensure students correctly interpreted the tensor dimensions within the context of batch processing and network architecture.

Homework and Continued Learning

Exercise Completion

Students were assigned the dynamic neural network exercise from the PyTorch notebook as homework. They are strongly encouraged to attempt to implement this network independently to solidify their understanding of custom modules and dynamic network control. It was mentioned that the solution to this exercise would be reviewed and discussed at the beginning of the next lab session.

Review of Introductory Materials

To ensure a strong foundation, students were also advised to review the introductory notebooks on Python, NumPy, and Pandas/Matplotlib. This review is particularly recommended for those who feel less confident with these fundamental tools, as proficiency in these areas is crucial for more advanced deep learning topics. Completing all notebooks, including the introductory ones and the main PyTorch notebook, was encouraged to gain a comprehensive understanding of the material covered in this session.

Communication and Support Channel

Teams Channel for Questions

To facilitate continued learning and support outside of the lab session, a dedicated channel will be created on Microsoft Teams. Students are invited to use this channel to ask any questions they may have regarding the lab materials, exercises, or related topics. Instructors and teaching assistants will monitor this channel to provide timely responses and assistance.

Encouragement for Active Engagement

Students were encouraged to actively use this communication channel and to not hesitate to ask questions, emphasizing that seeking clarification is a crucial part of the learning process.

Preview of Future Sessions

Next Steps in the Course

A brief preview of future lab sessions was provided. It was indicated that subsequent sessions are planned to build upon the foundational knowledge established in this lab. Future topics may include more complex neural network architectures, potentially focusing on Convolutional Neural Networks (CNNs) and other advanced deep learning techniques.

Importance of Foundational Knowledge

The importance of mastering the basic concepts covered in this introductory lab was reiterated. It was emphasized that a solid understanding of these fundamentals is essential for successfully tackling more advanced topics in deep learning.

Closing Remarks

The instructors thanked the students for their participation in the lab session, both in-person and remote attendees. They expressed hope that the session was beneficial and reiterated the availability of support through the Teams channel. Students were wished a good weekend and informed about the next lesson scheduled for the following week.

Conclusion

This introductory lab session successfully provided a foundational understanding of PyTorch for deep learning. By starting with a NumPy implementation of a neural network and transitioning to PyTorch, students gained a clear appreciation for the benefits of using a dedicated deep learning framework. Key concepts covered included:

Transitioning from NumPy to PyTorch tensors.
Leveraging PyTorch’s autograd for automatic differentiation, significantly simplifying gradient computations.
Building neural network models using nn.Sequential and custom nn.Module classes.
Defining loss functions and optimizers.
Implementing a basic training loop.

The hands-on exercise involving a dynamic neural network architecture was designed to challenge students to apply these concepts in a creative and non-standard setting, emphasizing the principles of network control and flexibility rather than practical applicability.

Looking ahead, future lab sessions will build upon this foundation, exploring more advanced and complex deep learning techniques. While the specific topics are still to be determined, there is an anticipation of delving into areas such as convolutional neural networks and other sophisticated architectures.

It is crucial for students to solidify their understanding of the material presented in this lab. Therefore, it is strongly recommended to:

Complete the assigned exercise on the dynamic neural network to gain practical experience.
Review all provided notebooks, including the introductory notebooks on Python, NumPy, and Pandas/Matplotlib, to ensure a comprehensive grasp of the fundamental tools and libraries.
Utilize the dedicated Teams channel to ask questions and seek clarification on any aspect of the lab materials or exercises. Active participation and inquiry are encouraged to maximize learning outcomes.

The next session will commence with a review of the dynamic network exercise solution, followed by the introduction of more advanced laboratory activities, designed to progressively increase in complexity and depth. Building a strong foundation in these initial concepts is paramount for success in the subsequent, more challenging topics.

Thank you for your active participation in this lab session. We encourage you to engage with the provided materials and exercises to prepare for the upcoming sessions. We look forward to seeing you next week, ready to explore more advanced aspects of deep learning.

--- title: "Introduction to PyTorch for Deep Learning" author: "Your Name" date: "2025-01-28" format: html: toc: true # Table of Contents toc-depth: 2 code-tools: true theme: cosmo # Or "journal" for Distill-like minimalism --- # Introduction This document outlines the first lab session of a deep learning course. The primary focus of this session is to introduce PyTorch and its application in building neural networks, starting from basic principles and progressing towards practical implementation. The session is structured around using Google Colab as the coding environment, leveraging its accessibility and free GPU resources. ## Session Objectives The main objectives of this lab session are to: - Familiarize students with the Google Colab environment for deep learning. - Introduce the fundamental concepts of Python and essential libraries such as NumPy and Pandas for data manipulation. - Demonstrate the transition from NumPy to PyTorch for implementing neural networks. - Explain and utilize PyTorch's automatic differentiation capabilities (`autograd`). - Guide students through defining neural network models and implementing training loops in PyTorch. - Provide hands-on experience with implementing custom PyTorch modules. - Introduce a practical exercise involving a dynamic neural network architecture to solidify understanding and encourage experimentation. ## Session Structure The session is designed to be interactive, combining guided examples with hands-on exercises. The structure is as follows: 1. **Environment Setup:** Introduction to Google Colab, accessing resources, and disabling AI autocompletion features to enhance learning. 2. **Material Distribution:** Accessing shared notebooks covering Python basics, NumPy, Pandas/Matplotlib, and the main PyTorch tutorial. 3. **Introductory Notebooks (Optional):** Brief overview of Python, NumPy, and Pandas/Matplotlib notebooks for students needing a refresher or introduction to these tools. These notebooks cover fundamental operations and data handling techniques. 4. **Main PyTorch Notebook:** - **NumPy to PyTorch Transition:** Implementing a simple neural network first in NumPy to illustrate manual gradient computation, then transitioning to PyTorch to demonstrate the advantages of automatic differentiation. - **PyTorch Fundamentals:** Introduction to PyTorch tensors, `autograd`, defining models using `nn.Sequential`, loss functions, and optimizers. - **Training Loop Implementation:** Step-by-step guide to implementing a basic training loop in PyTorch. - **Custom Modules:** Creating custom PyTorch modules to build more complex and flexible neural networks. 5. **Exercise: Dynamic Neural Network:** A practical exercise involving the implementation of a dynamic neural network architecture. This exercise is designed to challenge students to apply the learned concepts and explore network control. 6. **Q&A and Support:** Throughout the session, instructors will be available to answer questions and provide support, both in person and online via chat. A dedicated channel will be available for follow-up questions after the session. ## Google Colab and Computational Resources Google Colab is the chosen platform for this lab due to its ease of access and availability of computational resources. ### Access and Setup Students are required to have a Google account to access Google Colab. The notebooks can be opened directly in Colab via the provided links or by uploading them to Google Drive and opening from there. ### Disabling AI Autocompletion It is recommended to disable the AI-powered autocompletion features (like Gemini) in Google Colab to encourage active learning and prevent reliance on automated code generation during this introductory phase. ### Runtime Environment Google Colab offers various runtime environments, including CPU, GPU, and TPU. For the initial exercises in this lab, using the CPU runtime is sufficient. Instructions on how to change the runtime type to utilize GPUs or TPUs for more computationally intensive tasks will be provided if needed. Note that while GPUs offer significant speedups for deep learning tasks, their usage on Colab may be subject to usage limits. For this session, the exercises are designed to run efficiently on CPU. ### Accessing Course Materials All necessary materials, including Python, NumPy, Pandas, and PyTorch notebooks, are provided through a shared Google Drive folder. The link to this folder has been shared in the chat and via a QR code for easy access. Students are advised to create a copy of these notebooks in their personal Google Drive to ensure they can save their progress and modifications. The original shared folder is read-only to prevent accidental modifications to the source materials. ### Notebook Structure The notebooks are structured to guide students progressively through the concepts, starting with basic implementations and gradually introducing more advanced PyTorch features. Each notebook contains explanations, code examples, and exercises to reinforce learning. ## Check-in and Support A designated time (3:30) is set for a check-in during the lab session to assess student progress and address any immediate questions or difficulties. Furthermore, a dedicated communication channel will be established on Teams to provide ongoing support and answer questions that arise after the lab session. Students are encouraged to utilize this channel to seek assistance and engage in discussions related to the lab materials and exercises. # Initial Setup and Environment ## Audience Background ### Python Proficiency At the beginning of the session, a quick poll was conducted to assess the audience's familiarity with Python. The results indicated that all participants, including remote students, had prior experience with Python coding. ### PyTorch Experience Inquiring about prior PyTorch usage, it was noted that only a few attendees had used PyTorch before. This highlighted the need to start from the basics, catering to those new to the framework. ### Deep Learning Model Experience A significant number of participants indicated they had never coded a deep learning model previously. This confirmed the suitability of starting the lab session from scratch, as planned, to accommodate beginners in deep learning. ## Google Colab Setup ### Accessing Google Colab Students were instructed to use Google Colab for all coding activities during the lab. Google Colab was chosen for its accessibility, ease of use, and provision of free computational resources, including GPUs, which can be utilized for more demanding tasks later in the course, although not necessary for the initial exercises. Accessing Colab requires a Google account (Gmail). Users can navigate to Google Colab by searching \"Google Colab\" on Google or by directly clicking on any of the provided notebook files, which will prompt them to open it with their Google account. ### Disabling AI Autocompletion (Gemini) It was recommended to disable the AI-powered autocompletion feature, specifically mentioning Gemini, which is enabled by default in Google Colab. This suggestion was made to encourage students to actively write and understand the code themselves, rather than relying on AI to generate it, which is crucial for the learning process. Instructions on how to disable this feature within Colab were to be provided if needed. ### Accessing and Copying Course Notebooks Course materials, including several introductory notebooks and the main lab notebook, were made available through a shared Google Drive folder. The link to this folder was shared in the session chat and via a QR code displayed on screen. Students were explicitly instructed to create a copy of the notebooks in their personal Google Drive before starting to work on them. This step is essential because the shared folder is read-only, and any edits made directly to the files in the shared folder would be lost. Creating a copy ensures that students can save their work and progress. To create a copy, students should open a notebook and select \"File\" $\rightarrow$ \"Save a copy in Drive.\" The copied notebook will be saved in a folder named \"Colab Notebooks\" within their Google Drive. ### Connecting to a Virtual Machine Google Colab provides access to virtual machines for executing code. By default, Colab connects to a virtual machine with CPU resources. For more computationally intensive tasks, Colab also offers options to connect to virtual machines equipped with GPUs or TPUs. To connect, a \"Connect\" button is available in the Colab interface. Clicking this button connects the notebook to a virtual machine. While the default CPU instance is sufficient for the initial exercises and is virtually free with no time limits, GPU and TPU resources have usage limits. For the exercises in this lab, running on CPU is adequate, and students were informed that they would not need to switch to a GPU for these tasks. It was also mentioned that changing the runtime type (e.g., from CPU to GPU) will reset the environment, requiring variables to be reinitialized and code cells to be re-executed. ## Course Materials Overview ### Accessing the Shared Folder The course materials are organized within a Google Drive folder. Access to this folder is provided through a link shared in the Microsoft Teams chat and via a QR code displayed during the session. This folder contains all the notebooks required for the lab session, excluding solution notebooks to encourage independent problem-solving. ### Introductory Notebooks To ensure all students have the necessary foundational knowledge, three introductory notebooks are provided: #### Python Basics Notebook This notebook is designed for individuals with limited or no prior experience with Python. It covers fundamental Python programming concepts, including basic operations, data structures like lists and sets, and control flow. #### NumPy Notebook This notebook introduces NumPy, a fundamental library in Python for numerical computations. It focuses on array and matrix manipulation, which are essential for implementing mathematical operations in deep learning. #### Pandas and Matplotlib Notebook This notebook covers Pandas and Matplotlib. Pandas is introduced as a library for data manipulation and analysis, particularly for creating and working with data frames, which are useful for tabular data. Matplotlib is presented as a plotting library for data visualization. The notebook also demonstrates how Pandas can be used for basic plotting, offering a convenient way to visualize data directly from data frames. ### Main PyTorch Notebook The central resource for this lab session is a dedicated PyTorch notebook. This notebook is structured to guide students through the process of implementing a simple feed-forward neural network. It begins with a NumPy implementation to illustrate the underlying mathematical operations and the complexity of manual gradient calculation. It then transitions to PyTorch, demonstrating how to implement the same network using PyTorch tensors and leveraging PyTorch's autograd functionality for automatic differentiation. The notebook includes examples of using `nn.Sequential` to define neural network models, defining loss functions, and setting up optimizers. # Exploring the PyTorch Notebook This section details the contents and structure of the primary PyTorch notebook used in the lab session. The notebook is designed to guide students from basic NumPy implementations to advanced PyTorch functionalities for building and training neural networks. ## From NumPy to PyTorch: Manual to Automatic Differentiation ### NumPy Implementation: Gradients from Scratch The notebook initiates with a practical example of implementing a neural network using NumPy. This segment is crucial for understanding the underlying mathematical operations involved in neural networks, particularly the manual computation of gradients. ::: tcolorbox The NumPy implementation deliberately showcases the complexity and intricacies of manual gradient computation. This approach serves a pedagogical purpose: by experiencing the cumbersome nature of calculating gradients by hand using the chain rule, students gain a deeper appreciation for the automatic differentiation capabilities offered by frameworks like PyTorch. The notebook meticulously steps through the process, even verifying the correctness of gradient calculations using pen and paper to ensure accuracy and understanding. This part is intentionally challenging to highlight the benefits of using PyTorch for automatic gradient handling. ::: ### Transition to PyTorch Tensors: Embracing Automatic Gradients Following the NumPy example, the notebook transitions to PyTorch, reimplementing the same neural network using PyTorch tensors. This transition is designed to be seamless, demonstrating the similarities in syntax and operations between NumPy and PyTorch, while introducing the fundamental concept of PyTorch tensors. The code structure remains largely consistent, allowing students to easily compare and contrast the NumPy and PyTorch implementations. The key difference highlighted is the preparation for automatic differentiation, which will be fully leveraged in subsequent sections. ## PyTorch Fundamentals: Autograd and Model Building ### Introduction to Autograd: Automatic Differentiation Engine This subsection introduces PyTorch's automatic differentiation feature, `autograd`. It explains how PyTorch dynamically builds a computation graph as operations are performed on tensors. This graph tracks all operations, enabling automatic computation of gradients. ::: tcolorbox `Autograd` is presented as a game-changing feature that eliminates the need for manual gradient calculations. The explanation emphasizes that when using PyTorch tensors, all operations are recorded in a dynamic computation graph. To compute gradients, one simply needs to call `loss.backward()` on the loss tensor. PyTorch then traverses the computation graph in reverse, applying the chain rule to automatically compute gradients for all tensors that were involved in the loss calculation and have `requires_grad=True`. This section underscores the immense simplification and efficiency that `autograd` brings to neural network development, contrasting sharply with the manual approach demonstrated in NumPy. ::: ### Sequential Model Building with `nn.Sequential` The notebook introduces `torch.nn.Sequential` as a high-level API for constructing neural networks. `nn.Sequential` simplifies model definition by allowing users to define a network as a linear sequence of layers. ### Defining Loss Functions and Optimizers This part of the notebook covers how to define loss functions and optimizers in PyTorch. It explains that loss functions are used to quantify the error between the network's predictions and the actual targets, guiding the learning process. Optimizers, on the other hand, are algorithms that adjust the network's parameters to minimize the loss function. PyTorch provides a variety of pre-built loss functions (e.g., `nn.MSELoss`, `nn.CrossEntropyLoss`) and optimizers (e.g., `optim.SGD`, `optim.Adam`) within the `torch.nn` and `torch.optim` modules, respectively. ### Implementing the Training Loop with Autograd A complete training loop is presented, integrating `autograd`, loss functions, and optimizers. The training loop encompasses the following essential steps: 1. **Forward Pass:** Input data is passed through the model to obtain predictions. 2. **Loss Computation:** The loss function compares the predictions with the actual targets to calculate the loss. 3. **Backward Pass (`loss.backward()`):** `autograd` computes gradients of the loss with respect to all model parameters. 4. **Optimizer Step (`optimizer.step()`):** The optimizer updates the model parameters using the computed gradients to minimize the loss. 5. **Zero Gradients (`optimizer.zero_grad()`):** Before each backward pass, it's crucial to zero out the gradients from the previous iteration to prevent accumulation. This section provides a practical, runnable code example demonstrating how to put these components together to train a simple neural network, highlighting the ease and efficiency of training in PyTorch compared to manual gradient methods. ## Custom PyTorch Modules: Building Flexible Networks ### Creating Custom Neural Network Modules The notebook transitions to creating custom neural network modules by subclassing `torch.nn.Module`. This approach offers greater flexibility and control over network architecture compared to `nn.Sequential`. It's explained that custom modules are defined as classes that inherit from `nn.Module` and must implement two key methods: - **`__init__(self, ...)`:** The constructor, where layers and sub-modules are defined and initialized. - **`forward(self, x)`:** Defines the forward pass of the module, specifying how input tensors `x` are processed through the layers to produce an output tensor. ### Implementing the `forward` Pass in Custom Modules The `forward` method is emphasized as the core of a custom module. It dictates the exact computations performed by the module. Within the `forward` method, one specifies how the input tensor is passed through the layers defined in `__init__` and what operations are applied. The notebook provides a simple example of creating a custom module, demonstrating the basic structure and usage. It shows how to instantiate the custom module and use it in a similar way to `nn.Sequential` models, highlighting the modularity and reusability of PyTorch modules. ## Exercise: Implementing a Dynamic Neural Network ### Challenge: A Dynamically Modulated Network Architecture The final section of the notebook presents an exercise that challenges students to implement a more complex, dynamic neural network. This network features a unique architecture where a middle linear layer's presence and repetition are dynamically controlled during the forward pass. ### Dynamic Layer Control: Repetition and Dropout Analogy The exercise requires implementing a neural network with the following structure: Input Layer $\rightarrow$ Activation Function $\rightarrow$ \[Middle Linear Layer (repeated 0 to 3 times, dynamically chosen)\] $\rightarrow$ Output Linear Layer. ::: tcolorbox The core challenge lies in the \"Middle Linear Layer.\" This layer is designed to be conditionally included in the network's forward pass, and if included, it can be repeated 0, 1, 2, or 3 times. The number of repetitions is intended to be dynamically determined (though in the exercise, it might be pre-set or randomly chosen for simplicity). The exercise is described as a way to explore network control and is analogized to a very aggressive form of dropout, where entire layers are dropped out or repeated, rather than individual neurons. The goal is to understand how to build networks with conditional execution paths and to experiment with unconventional architectures. Students are tasked with implementing this network and observing its behavior. The solution to this exercise will be discussed in the next lab session. Students are explicitly advised against using such an architecture in real-world applications, as it is primarily designed for educational purposes to explore dynamic network behavior rather than to achieve state-of-the-art performance. ::: # Concluding Remarks and Next Steps This section summarizes the concluding discussions of the lab session and outlines the next steps for students. ## Q&A and Clarifications Following the hands-on coding session, a brief question and answer period was held. One notable question from a remote participant concerned the dimensions labeled in a network diagram presented in the notebook. The clarification provided was that dimensions marked as 'N' in the diagram referred to the batch size, while 'H' indicated feature dimensions or the number of hidden units in a layer. This clarification was made to ensure students correctly interpreted the tensor dimensions within the context of batch processing and network architecture. ## Homework and Continued Learning ### Exercise Completion Students were assigned the dynamic neural network exercise from the PyTorch notebook as homework. They are strongly encouraged to attempt to implement this network independently to solidify their understanding of custom modules and dynamic network control. It was mentioned that the solution to this exercise would be reviewed and discussed at the beginning of the next lab session. ### Review of Introductory Materials To ensure a strong foundation, students were also advised to review the introductory notebooks on Python, NumPy, and Pandas/Matplotlib. This review is particularly recommended for those who feel less confident with these fundamental tools, as proficiency in these areas is crucial for more advanced deep learning topics. Completing all notebooks, including the introductory ones and the main PyTorch notebook, was encouraged to gain a comprehensive understanding of the material covered in this session. ## Communication and Support Channel ### Teams Channel for Questions To facilitate continued learning and support outside of the lab session, a dedicated channel will be created on Microsoft Teams. Students are invited to use this channel to ask any questions they may have regarding the lab materials, exercises, or related topics. Instructors and teaching assistants will monitor this channel to provide timely responses and assistance. ### Encouragement for Active Engagement Students were encouraged to actively use this communication channel and to not hesitate to ask questions, emphasizing that seeking clarification is a crucial part of the learning process. ## Preview of Future Sessions ### Next Steps in the Course A brief preview of future lab sessions was provided. It was indicated that subsequent sessions are planned to build upon the foundational knowledge established in this lab. Future topics may include more complex neural network architectures, potentially focusing on Convolutional Neural Networks (CNNs) and other advanced deep learning techniques. ### Importance of Foundational Knowledge The importance of mastering the basic concepts covered in this introductory lab was reiterated. It was emphasized that a solid understanding of these fundamentals is essential for successfully tackling more advanced topics in deep learning. ## Closing Remarks The instructors thanked the students for their participation in the lab session, both in-person and remote attendees. They expressed hope that the session was beneficial and reiterated the availability of support through the Teams channel. Students were wished a good weekend and informed about the next lesson scheduled for the following week. # Conclusion This introductory lab session successfully provided a foundational understanding of PyTorch for deep learning. By starting with a NumPy implementation of a neural network and transitioning to PyTorch, students gained a clear appreciation for the benefits of using a dedicated deep learning framework. Key concepts covered included: - Transitioning from NumPy to PyTorch tensors. - Leveraging PyTorch's `autograd` for automatic differentiation, significantly simplifying gradient computations. - Building neural network models using `nn.Sequential` and custom `nn.Module` classes. - Defining loss functions and optimizers. - Implementing a basic training loop. The hands-on exercise involving a dynamic neural network architecture was designed to challenge students to apply these concepts in a creative and non-standard setting, emphasizing the principles of network control and flexibility rather than practical applicability. Looking ahead, future lab sessions will build upon this foundation, exploring more advanced and complex deep learning techniques. While the specific topics are still to be determined, there is an anticipation of delving into areas such as convolutional neural networks and other sophisticated architectures. It is crucial for students to solidify their understanding of the material presented in this lab. Therefore, it is strongly recommended to: - Complete the assigned exercise on the dynamic neural network to gain practical experience. - Review all provided notebooks, including the introductory notebooks on Python, NumPy, and Pandas/Matplotlib, to ensure a comprehensive grasp of the fundamental tools and libraries. - Utilize the dedicated Teams channel to ask questions and seek clarification on any aspect of the lab materials or exercises. Active participation and inquiry are encouraged to maximize learning outcomes. The next session will commence with a review of the dynamic network exercise solution, followed by the introduction of more advanced laboratory activities, designed to progressively increase in complexity and depth. Building a strong foundation in these initial concepts is paramount for success in the subsequent, more challenging topics. Thank you for your active participation in this lab session. We encourage you to engage with the provided materials and exercises to prepare for the upcoming sessions. We look forward to seeing you next week, ready to explore more advanced aspects of deep learning.