Artificial Neural Networks and Deep Learning
The following are last minute news you should be aware of ;-)
04/01/2021: Updated course syllabus (removed batch normalization) 29/12/2020: An extra point for the students who participate to the second phase of the Image Segmentation task! 29/12/2020: The third challenge is out! You have until 31st January 2021 to submit it ... no extensions this time ;-) 29/12/2020: Updated syllabus published 29/12/2020: Past exams published 11/12/2020: Uploaded the material from last week lectures!!! 03/12/2020: Second course challenge is out! Check it here, you have to deliver it by 20th December 2020 19/11/2020: Uploaded slides for today and the next days and past lectures videos 18/11/2020: Updated schedule to reflect Teams merge fron today on 08/11/2020: First homework challenge published here 18/10/2020: Updated links to lectures videos and lab notebooks 14/10/2020: Requested changes of team have been authorized by Presidenza ... stay tuned I will update you on this soon 13/10/2020: IMPORTANT CHANGE !!! -> Tomorrow 14/10/2020 lectures will be issued ONLY ONLINE!!!! 09/10/2020: Uploaded lab notebooks and lab recordings 08/10/2020: Change of schedule on the 15/10/2020 we will have online lecture 08/10/2020: Today's video uploaded and Neural Networks Traning Slides updated in the cross-validation part 06/10/2020: IMPORTANT CHANGE !!! -> Tomorrow 07/10/2020 lectures will be issued ONLY ONLINE!!!! 02/10/2020: Published a guide to install the software which will be used in the labs here 02/10/2020: Published fixed slides about feed forward neural networks 01/10/2020: Pool to request the change of Team is here 23/09/2020: Tomorrow 24/09/2020 we are going to have the online lecture as planned 23/09/2020: Today's lectures published 22/09/2020: Added slides on Perceptron, Hebbian learning and feed forward neural networks 20/09/2020: Added links to the lecture recordings and uploaded slides 14/09/2020: FIX - Team 1 ODD numbers, Team 2 EVEN numbers !!! 14/09/2020: FIX - the hours of the second team were overlapping to the first, now they are correctly one after the other 13/09/2020: The course is about to start ... stay tuned!
Course Aim & Organization
Neural networks are mature, flexible, and powerful non-linear data-driven models that have successfully been applied to solve complex tasks in science and engineering. The advent of the deep learning paradigm, i.e., the use of (neural) network to simultaneously learn an optimal data representation and the corresponding model, has further boosted neural networks and the data-driven paradigm.
Nowadays, deep neural network can outperform traditional hand-crafted algorithms, achieving human performance in solving many complex tasks, such as natural language processing, text modeling, gene expression modeling, and image recognition. The course provides a broad introduction to neural networks (NN), starting from the traditional feedforward (FFNN) and recurrent (RNN) neural networks, till the most successful deep-learning models such as convolutional neural networks (CNN) and long short-term memories (LSTM).
The course major goal is to provide students with the theoretical background and the practical skills to understand and use NN, and at the same time become familiar and with Deep Learning for solving complex engineering problems.
The course is composed of a blending of lectures and exercises by the course teachers and a teaching assistant.
- Matteo Matteucci: the course teacher and this is his webex room
- Giacomo Boracchi: the course co-teacher and this is his webex room
- Francesco Lattari: the course teaching assistant and this is his webex room
Course Program and Syllabus
This goal is pursued in the course by:
- Presenting major theoretical results underpinning NN (e.g., universal approx, vanishing/exploding gradient, etc.)
- Describing the most important algorithms for NN training (e.g., backpropagation, adaptive gradient algorithms, etc.)
- Illustrating the best practices on how to successfully train and use these models (e.g., dropout, data augmentation, etc.)
- Providing an overview of the most successful Deep Learning architectures (e.g., CNNs, sparse and dense autoencoder, LSTMs for sequence to sequence learning, etc.)
- Providing an overview of the most successful applications with particular emphasis on models for solving visual recognition tasks.
We have compiled a detailed syllabus of the course student can use to double check their preparation against before the exam.
- [2020/2021] Course Syllabus: a detailed list of topics covered by the course and which students are expected to know when approaching the exam
Detailed course schedule
A detailed schedule of the course can be found here; topics are just indicative while days and teachers are correct up to some last minute change (I will notify you by email). Please note that not all days we have lectures!!
Note: Lecture timetable interpretation * On Wednesday, in 2.0.2 (EX N.0.2), starts at 15:15, ends at 17:00 * On Wednesday, in 2.1.2 (EX N.1.2), starts at 17:30, ends at 19:15 * On Thursday, in teacher webex room, starts at 16:30, ends at 19:15
New Note: from November 18th teams are merged and lectures are moved online * On Wednesday, in teacher webex room, starts at 16:15, ends at 18:15 * On Thursday, in teacher webex room, starts at 16:30, ends at 19:15
|16/09/2020||Wednesday||15:15 - 17:00||2.0.2||Matteo Matteucci||Course Introduction|
|16/09/2020||Wednesday||17:30 - 19:15||2.1.2|
|17/09/2020||Thursday||16:30 - 19:15||Virtual Room||Matteo Matteucci||Introduction to Deep Learning + Perceptron and Hebbian Learning|
|23/09/2020||Wednesday||15:15 - 17:00||2.0.2||Matteo Matteucci||Hebbian Learning Example and the XOR Problem|
|23/09/2020||Wednesday||17:30 - 19:15||2.1.2|
|24/09/2020||Thursday||16:30 - 19:15||Virtual Room||Matteo Matteucci||Feed forward neural networks and Backpropagation|
|30/09/2020||Wednesday||15:15 - 17:00||2.0.2||Matteo Matteucci||Backpropagation Example|
|30/09/2020||Wednesday||17:30 - 19:15||2.1.2|
|01/10/2020||Thursday||16:30 - 19:15||Virtual Room||Matteo Matteucci||Error Functions Design|
|07/10/2020||Wednesday||15:15 - 17:00||Virtual Room||Francesco Lattari||KERAS NN - Feed forward neural networks (Part 1)(Part 2)|
|07/10/2020||Wednesday||17:30 - 19:15||Virtual Room|
|08/10/2020||Thursday||16:30 - 19:15||Virtual Room||Matteo Matteucci||Overffitting, cross-validation, and Early Stopping|
|14/10/2020||Wednesday||15:15 - 17:00||Virtual Room||Francesco Lattari||KERAS NN - FFNN and Overfitting|
|14/10/2020||Wednesday||17:30 - 19:15||Virtual Room|
|15/10/2020||Thursday||16:30 - 19:15||Virtual Room||Matteo Matteucci||Facing overfitting, network initialization, and other stuff ...|
|21/10/2020||Wednesday||15:15 - 17:00||2.0.2||Giacomo Boracchi||The Image Classification Problem|
|21/10/2020||Wednesday||17:30 - 19:15||2.1.2|
|22/10/2020||Thursday||16:30 - 19:15||Virtual Room||Giacomo Boracchi||Convolutional Neural Networks|
|28/10/2020||Wednesday||15:15 - 17:00||2.0.2||Giacomo Boracchi||CNN Training, Transfer Learning and Visualization. Fully Convolutional CNN (Part1).|
|28/10/2020||Wednesday||17:30 - 19:15||2.1.2|
|29/10/2020||Thursday||16:30 - 19:15||Virtual Room||Giacomo Boracchi||CNN Training, Transfer Learning and Visualization. Fully Convolutional CNN (Part2).|
|04/11/2020||Wednesday||15:15 - 17:00||2.0.2||Francesco Lattari||KERAS NN - Convolutional Neural Networks|
|04/11/2020||Wednesday||17:30 - 19:15||2.1.2|
|05/11/2020||Thursday||16:30 - 19:15||Virtual Room||Giacomo Boracchi||Fully Convolutional CNN, CNN for image segmentation|
|11/11/2020||Wednesday||---||---||---||-- No Lecture (Prove in Itinere) --|
|12/11/2020||Thursday||16:30 - 19:15||Virtual Room||Giacomo Boracchi||CNN for localization and detection|
|18/11/2020||Wednesday||16:15 - 18:15||Virtual Room||Giacomo Boracchi||GANs|
|19/11/2020||Thursday||16:30 - 19:15||Virtual Room||Matteo Matteucci||Recurrent neural networks + LSTM|
|25/11/2020||Wednesday||16:15 - 18:15||Virtual Room||Francesco Lattari||KERAS NN - Autoencoder, classification, segmentation|
|26/11/2020||Thursday||16:30 - 19:15||Virtual Room||Matteo Matteucci||Se2Seq Learning and Word Embedding|
|02/12/2020||Wednesday||16:15 - 18:15||Virtual Room||Matteo Matteucci||Attention Mechanisms|
|03/12/2020||Thursday||16:30 - 19:15||Virtual Room||Matteo Matteucci||The Transformer (and challenge presentation in the first 20 minutes)|
|09/12/2020||Wednesday||16:15 - 18:15||Virtual Room||Francesco Lattari||KERAS NN - Recurrent Neural Networks|
|10/12/2020||Thursday||16:30 - 19:15||Virtual Room||Matteo Matteucci||Questions and Answers|
|16/12/2020||Wednesday||16:15 - 18:15||Virtual Room||---||-- Spare Lecture --|
|17/12/2020||Thursday||16:30 - 19:15||Virtual Room||---||-- Spare Lecture --|
|23/12/2020||Wednesday||16:15 - 18:15||Virtual Room||---||-- Spare Lecture --|
Course evaluation is composed of two parts:
- A written examination covering the whole program graded up to 26/32 ... may be less
- A home project in the form of a "Kaggle style" competition practicing the topics of the course graded up to 6/32 ... may be more
The final score will sum the grade of the written exam and the grade of the home project.
You can find here one example of the exam text to get a flavor of what to expect in the written examination.
Teaching Material (the textbook)
Lectures will be based on material from different sources, teachers will provide their slides to students as soon they are available. As a general reference you can check the following text, but keep in mind that teachers will not follow it strictly
- Deep Learning. Ian Goodfellow, Yoshua Bengio, and Aaron Courville, MIT Press, 2016.
Slides from the lectures by Matteo Matteucci
- [2020/2021] Course Introduction: introductory slides of the course with useful information about the course syllabus, grading, and the course logistics.
- [2020/2021] Machine Learning vs Deep Learning: introduction to machine learning paradigms and definition of deep learning with examples
- [2020/2021] From Perceptrons to Feed Forward Neural Networks: the original Perceptron model, Hebbian learning, feed-forward architecture, backpropagation and gradient descent, error functions and maximum likelihood estimation
- [2020/2021] Neural Networks Training: dealing with overfitting (weight decay, early stopping, dropout), vanishing gradient (ReLU and friends), batch normalization
- [2020/2021] Recurrent Neural Networks: learning with sequences, Recurrent Neural Networks, vanishing gradient, Long Short-Term Memories (LSTM), seq2seq model.
- [2020/2021] Word Embedding: neural autoencoders, language models, word embedding, word2vec, glove.
- [2020/2021] Beyond Sequence 2 Sequence Learning: Neural Turing Machines, attention mechanisms, the Transformer.
Slides from the lectures by Giacomo Boracchi are available in his webpage, for you
- Image Classification: Image classification and related issues, template matching, image classification via nearest neighbors methods, image classification via linear classifiers, image classification via hand-crafted features.
- Convolutional Neural Networks: From hand-crafted features to convolutional neural networks.
- Training Convolutional Neural Networks: How to train CNNs, famous architectures, data augmentation, and the like.
- Convolutional Neural Networks for Image Segmentation: CNN architectures for segmentation and detection.
Slides from the practicals by Francesco Lattari
- [2020/2021] Introduction to Keras: Introduction to Keras and Tensorflow2 (slides + notebook)
- [2020/2021] Facing overfitting in Keras: Techniques to limit overfitting in Keras and Tensorboard use (slides + notebook)
- [2020/2021] Convolutional architectures in Keras: How to build, train, and evaluate convolutional models for image classification in Keras and Tensorflow2 (slides + notebook)
- [2020/2021] Image Segmentation in Keras: How to build, train, and evaluate convolutional models for image segmentation in Keras and Tensorflow2 (slides + notebook)
- [2020/2021] Recurrent architectures in Keras: How to build, train, and evaluate recurrent neural architectures in Keras and Tensorflow2 (slides + notebook)
Past Online Exams
As for the last semester, Politecnico di Milano exams will be held online. We have not yet decided the digital environment we are going to use for the exams, but you can have a flavor of what to expect by looking at past semester exam calls.
Note: this year written exams will be graded 26 points instead of 27. This is because the past year challenges were 5 points while this year they are 6 points worth. This means that the maximum mark is still 26+6=32. You can get an extra point if you participate in the second phase of the Image Segmentation Challenge thus the full mark could reach in principle 33. To get a laude it is required to pass 30.
Lab software setup
For the lab in class we suggest you install TensorFlow 2 on you machine so to be able to follow the coding examples step by step. Here what you should do:
- Install Anaconda according to your distro (Windows/Linux), python 3.7 from 
- From terminal (Anaconda Prompt in Windows):
- conda create -n tf_env python=3.7 tensorflow-gpu
- conda activate tf_env
- pip install --upgrade pip
- pip install jupyter
- pip install pillow
- Test your Tensorflow install
- Run python from terminal (Anaconda Prompt in Windows) with «python»
- import tensorflow
- print(tensorflow.__version__) -> your should get version 2.1.0 or higher
- Test your Jupiter install
- From terminal (Anaconda Prompt in Windows) use the command jupyter notebook -> a Jupiter tab should appear in your browser
- On top right click on «New» and select «Python 3» from the menu -> a Jupiter Notebook should appear in a new tab
- Write code 3b and 3c in cell "In [ ]:" and execute clicking on «Run».
As part of the evaluation (up to 6 marks in the final grade) we are issuing 3 homeworks in the form of "Kaggle style" competitions. They are meant to practice the course topics on simple image recognition tasks.
- Image Classification Homework: the first homework is about image classification with convolutional neural networks. The deadline to submit the results is November the 22nd.
- Image Segmentation Homework: the second homework is about image segmentation. This competition is open also to external participants so it lasts until January, you are not requested to participate to the two Stages, just at the Development one and your delivery is expected by December 20th. There are different datasets and leaderboards you can decide to participate to any of the leaderboard and to use any approach you prefer for segmentation.
- Visual Question answering Homework: the third homework is about visual question answering and it mixes feature extraction from images and text in order to solve an multi domain classification tas. This time we do not want to put pressure on you and the deadline is quite relaxed, you have until the 31st of January to submit your solution.
- Extra point -> Image Segmentation Homework: from 19th January 2021 till 22nd January 2021 there will be the second phase of the image segmentation challenge where new data will be released to train you models and participate to the final leaderboard. One extra point for you if you fine tune your models and participate into this!!!
- Image Classification Homework: the first homework is about image classification with convolutional neural networks. The deadline to submit the results is November the 30th.
- Image Segmentation Homework: the second homework is about image segmentation with convolutional neural networks and the like. The deadline to submit the results is December the 17th.
- Visual Question Answering Homework: the third homework is about visual question answering with convolutional and recurrent neural networks ... plus word2vec. The deadline to submit the results is January the 15th.