Papers

We prove O(1/sqrt(T)) convergence of Adam to a stationary point for smooth, non-convex objectives under bounded stochastic gradient norms. Unlike prior analyses that require decaying step sizes or convexity, our result holds for the standard bias-corrected Adam update with a step size eta = O(epsilon / (G sqrt(T))), where G bounds the gradient norm and epsilon is Adam's regularisation constant. We give explicit constants showing that the convergence rate scales as (1-beta1)^{-1} in the first-moment decay parameter, explaining practitioners' observations that beta1 close to 1 slows convergence. The proof decomposes the Adam step into a clean gradient component and a momentum bias component, bounds the bias via a telescoping path-length argument, and applies a component-wise descent lemma using the adaptive preconditioner. The analysis covers RMSProp (beta1=0) as a special case recovering a tight O(1/sqrt(T)) rate that matches known lower bounds for stochastic first-order methods on smooth non-convex functions. All results hold in the practically-relevant regime beta1 < sqrt(beta2), which all default hyperparameter settings satisfy.