Statistical Seminar: Every Monday at 2:00 pm.
Time: 2:00 pm – 3:00 pm
Date: 29th September
Place: 3001

 

Pierre MARION (INRIA Paris) – Large Stepsizes Accelerate Gradient Descent for (Regularized) Logistic Regression

 

 Abstract : 

 

Deep learning practitioners usually use large stepsizes when training neural networks. To understand the impact of large stepsizes on training dynamics, we consider the simplified setting of gradient descent (GD) applied to logistic regression with linearly separable data, where the stepsize is so large that the loss initially oscillates. We study the training dynamics, and show convergence and acceleration compared to using stepsizes that satisfy the descent lemma. I will show some key ideas from the proof and, if time allows, discuss what happens when adding a regularization term.

 

 

 

Organizers:

Anna KORBA (CREST), Karim LOUNICI (CMAP) , Jaouad MOURTADA (CREST)

Sponsors:
CREST-CMAP