A Spline Perspective on Deep Learning

The Spring 2022 SDS Seminar Series continues on Friday, April 8 from 2:00 p.m. to 3:00 p.m. with Dr. Richard Baraniuk (C. Sidney Burrus Professor of Electrical and Computer Engineering at Rice University). This event is virtual.

Title: A Spline Perspective on Deep Learning

Abstract: We study the geometry of deep learning through the lens of approximation theory via splines. Our core enabling insight is that a large class of deep networks can be written as a composition of continuous piecewise affine (CPA) spline operators, which provides a powerful portal through which to interpret and analyze their inner workings. We explore links to the classical theory of optimal classification via matched filters, the effects of data memorization, vector quantization (VQ), K-means clustering, and subdivision, which open up new geometric avenues to study how deep networks organize signals in a hierarchical and multiscale fashion. We also shed new light on deep network learning by proving that networks with skip connections, such as residual networks (ResNets), have a better-conditioned and less erratic loss surface than that of a classical convolutional network (ConvNet).