Talk: Gradient-based optimization for Deep Learning

This weekend I gave a talk at the Machine Learning Porto Alegre Meetup about optimization methods for Deep Learning. In this material you will find an overview of first-order methods, second-order methods and some approximations of second-order methods as well about the natural gradient descent and approximations to it. I took some long nights to prepare this material, so I hope you like it! You can download the PDF of the slides by clicking on the top-right menu.

– Christian S. Perone

Visualizing sample simplex trajectories in Deep Learning

Softmax is a distribution over choices, it maps a vector into the probability simplex that is defined as \Delta_{n-1}=\{p\in\mathbb{R}^n\; \vert\; 1^\top p = 1 \; \; {\rm and} \;\; p \geq 0 \}, where the sum of all elements of the vector must equal 1. Softmax is used a lot in classification and I thought it would be interesting to visualize (when possible, on lower dimensions) the trajectories of individual samples in that simplex as predicted by the network while the network is being trained.

In the animations below you’ll see the trajectories of the sample individual sample (from the test set) over the simplex of 3 classes (dog, cat, horse) from CIFAR-10 and using a simple shallow CNN both with Adam and SGD. Each frame is generated after 10 optimization steps and the video is from 4 epochs with CIFAR-10 dataset with only the 3 aforementioned classes.

Trajectory of a CNN using Adam with LR of 0.001

Trajectory of a CNN using SGD with LR of 0.001 and momentum

A new professional ethics: Karl Popper and Xenophanes’ epistemology

It is not a secret that I admire the work of Karl Popper, both as a philosopher but also as a very precise historian that tried to dismiss many misunderstandings of the past.

I was reading the book The World of Parmenides, which is a collection of Popper’s essays on the Presocratic Enlightenment, and found a very interesting insight on how the epistemology of Xenophanes led naturally to a professional ethics. This link isn’t widespread nowadays, but it certainly deserves more divulgation as it is a natural consequence of the conjectural knowledge we possess.

Read More

Uncertainty Estimation in Deep Learning (PyData Lisbon / July 2019)

Just sharing some slides I presented at the PyData Lisbon on July 2019 about the talk “Uncertainty Estimation in Deep Learning“:

Cite this article as: Christian S. Perone, "Uncertainty Estimation in Deep Learning (PyData Lisbon / July 2019)," in Terra Incognita, 18/07/2019, https://blog.christianperone.com/2019/07/uncertainty-estimation-in-deep-learning-pydata-lisbon-july-2019/.

PyData Montreal slides for the talk: PyTorch under the hood

These are the slides of the talk I presented on PyData Montreal on Feb 25th. It was a pleasure to meet you all ! Thanks a lot to Maria and Alexander for the invitation !

Cite this article as: Christian S. Perone, "PyData Montreal slides for the talk: PyTorch under the hood," in Terra Incognita, 26/02/2019, https://blog.christianperone.com/2019/02/pydata-montreal-slides-for-the-talk-pytorch-under-the-hood/.