Papers

Adam's convergence theory treats the stability constant $\varepsilon$ (default $10^{-8}$) as a numerical afterthought: divergence counterexamples set it to zero and convergence proofs either require it large or absorb it into constants. We settle a precisely stated question in that gap: does Adam with its exact shipped defaults $(\beta_1,\beta_2,\varepsilon)=(0.9,0.999,10^{-8})$ — with $\varepsilon>0$ as implemented — converge on convex stochastic problems with bounded gradients, under constant or $1/\sqrt{t}$ step sizes? We prove it does not, and characterise exactly when $\varepsilon$ changes the answer. On the canonical Reddi-type family we reduce Adam's stationary dynamics to a closed form in two geometric "spike sums", yielding a scalar drift $D(\beta_1,\beta_2,\varepsilon/\lambda)$ whose sign determines Adam's fate: if $D<0$, Adam ascends a convex objective almost surely at a linear rate, for every constant step size, and at a $\sqrt{T}$ rate under the $1/\sqrt{t}$ schedule. We prove divergence at the exact defaults for an explicit instance, for every $\varepsilon\le 17\lambda$ ($\lambda$ = gradient scale); a matching positive result, $\varepsilon \ge \lambda[(C-1)/\mu-1] \Rightarrow D>0$ for all $(\beta_1,\beta_2)$, so tuning $\varepsilon$ alone repairs the whole family; and a scale law: $D$ depends on $(\varepsilon,\lambda)$ only through $\varepsilon/\lambda$, so rescaling a loss moves Adam across a convergence/divergence phase boundary with all hyperparameters fixed. Reproducible dependency-free experiments (code included, seeds fixed) certify the drift sign with rigorous truncation brackets, trace the empirical boundary $\varepsilon^*(C)$ across three $(\beta_1,\beta_2)$ regimes, and match measured trajectory slopes to the predicted $-\alpha D$.

We prove O(1/sqrt(T)) convergence of Adam to a stationary point for smooth, non-convex objectives under bounded stochastic gradient norms. Unlike prior analyses that require decaying step sizes or convexity, our result holds for the standard bias-corrected Adam update with a step size eta = O(epsilon / (G sqrt(T))), where G bounds the gradient norm and epsilon is Adam's regularisation constant. We give explicit constants showing that the convergence rate scales as (1-beta1)^{-1} in the first-moment decay parameter, explaining practitioners' observations that beta1 close to 1 slows convergence. The proof decomposes the Adam step into a clean gradient component and a momentum bias component, bounds the bias via a telescoping path-length argument, and applies a component-wise descent lemma using the adaptive preconditioner. The analysis covers RMSProp (beta1=0) as a special case recovering a tight O(1/sqrt(T)) rate that matches known lower bounds for stochastic first-order methods on smooth non-convex functions. All results hold in the practically-relevant regime beta1 < sqrt(beta2), which all default hyperparameter settings satisfy.