ALTEN Italia

"Muon is Not That Special: Random or Inverted Spectra Work Just as Well" by Zakhar Shumaylov , Nathaël Da Costa, Peter Zaika , Bálint Mucsányi , Alex Massucco , Yoav Gelberg , Prof Carola-Bibiane Schönlieb , Yarin Gal , Philipp Hennig "The recent empirical success of the Muon optimizer has renewed interest in non-Euclidean optimization, typically justified by similarities with second-order methods, and linear minimization oracle (LMO) theory. In this paper, we challenge this geometric narrative through three contributions, demonstrating that precise geometric structure is not the key factor affecting optimization performance. First, we introduce Freon, a family of optimizers based on Schatten (quasi-)norms, powered by a novel, provably optimal QDWH-based iterative approximation. Freon naturally interpolates between SGD and Muon, while smoothly extrapolating into the quasi-norm regime. Empirically, the best-performing Schatten parameters for GPT-2 lie strictly within the quasi-norm regime, and thus cannot be represented by any unitarily invariant LMO. Second, noting that Freon performs well across a wide range of exponents, we introduce Kaon, an absurd optimizer that replaces singular values with random noise. Despite lacking any coherent geometric structure, Kaon matches Muon's performance and retains classical convergence guarantees, proving that strict adherence to a precise geometry is practically irrelevant. Third, having shown that geometry is not the primary driver of performance, we demonstrate it is instead controlled by two local quantities: alignment and descent potential. Ultimately, each optimizer must tune its step size around these two quantities. While their dynamics are difficult to predict a-priori, evaluating them within a stochastic random feature model yields a precise insight: Muon succeeds not by tracking an ideal global geometry, but by guaranteeing step-size optimality." Paper: https://lnkd.in/dJQ8WUrR #machinelearning #optimization

"Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfer" by Clarissa Lauditi , Cengiz Pehlevan, Blake Bordelon "We study the evolution of hidden-weight spectra in wideneural networks trained by (stochastic) gradient descent. We develop a two-level dynamical mean-field theory (DMFT) that jointly tracks bulk and outlier spectral dynamics for spiked ensembles whose spike directions remain statistically dependent on the random bulk. We apply this framework to two settings: (1) infinite-width nonlinear networks in mean-field/μP scaling and (2) deep linear networks in the proportional high-dimensional limit, where width, input dimension, and sample size diverge with fixed ratios. Our theory predicts how outliers evolve with training time, width, output scale, and initialization variance. In deep linear networks, μP yields width-consistent outlier dynamics and hyperparameter transfer, including width-stable growth of the leading NTK mode toward the edge of stability (EoS). In contrast, NTK parameterization exhibits strongly width-dependent outlier dynamics, despite converging to a stable large-width limit. We show that this bulk+outlier picture is descriptive of simple tasks with small output channels, but that tasks involving large numbers of outputs (ImageNet classification or GPT language modeling) are better described by a restructuring of the spectral bulk. We develop a toy model with extensive output channels that recapitulates this phenomenon and show that edge of the spectrum still converges for sufficiently wide networks." Paper: https://lnkd.in/dG-cH7Fx #machinelearning

"Mathematical Foundations of Geometric Deep Learning" by Haitz Sáez de (Ocáriz) Borde , Michael Bronstein A review of key mathematical concepts for starting to study geometric deep learning. For further reading, the paper provides useful references. Paper: https://lnkd.in/dsJwP3eu #geometricdeeplearning

"Generalization at the Edge of Stability" by Mario Tuci , Caner Korkmaz , Umut Simsekli , Tolga Birdal "Training modern neural networks often relies on large learning rates, operating at the edge of stability, where the optimization dynamics exhibit oscillatory and chaotic behavior. Empirically, this regime often yields improved generalization performance, yet the underlying mechanism remains poorly understood. In this work, we represent stochastic optimizers as random dynamical systems, which often converge to a fractal attractor set (rather than a point) with a smaller intrinsic dimension. Building on this connection and inspired by Lyapunov dimension theory, we introduce a novel notion of dimension, coined the `sharpness dimension', and prove a generalization bound based on this dimension. Our results show that generalization in the chaotic regime depends on the complete Hessian spectrum and the structure of its partial determinants, highlighting a complexity that cannot be captured by the trace or spectral norm considered in prior work. Experiments across various MLPs and transformers validate our theory while also providing new insights into the recently observed phenomenon of grokking." Paper: https://lnkd.in/d9wsiEn9 #machinelearning

"A Mechanistic Analysis of Looped Reasoning Language Models" by Hugh Blayney , Alvaro Arroyo, Johan Samir Obando Ceron , Pablo Samuel Castro , Aaron Courville , Michael Bronstein , Xiaowen Dong "Reasoning has become a central capability in large language models. Recent research has shown that reasoning performance can be improved by looping an LLM's layers in the latent dimension, resulting in looped reasoning language models. Despite promising results, few works have investigated how their internal dynamics differ from those of standard feedforward models. In this paper, we conduct a mechanistic analysis of the latent states in looped language models, focusing in particular on how the stages of inference observed in feedforward models compare to those observed in looped ones. To this end, we analyze cyclic recurrence and show that for many of the studied models each layer in the cycle converges to a distinct fixed point; consequently, the recurrent block follows a consistent cyclic trajectory in the latent space. We provide evidence that as these fixed points are reached, attention-head behavior stabilizes, leading to constant behavior across recurrences. Empirically, we discover that recurrent blocks learn stages of inference that closely mirror those of feedforward models, repeating these stages in depth with each iteration. We study how recurrent block size, input injection, and normalization influence the emergence and stability of these cyclic fixed points. We believe these findings help translate mechanistic insights into practical guidance for architectural design." Paper: https://lnkd.in/dVi59hNt #machinelearning

"Temporal Generalization: A Reality Check" by Divyam Madaan , Sumit Chopra , KyungHyun Cho "Machine learning (ML) models often struggle to maintain performance under distribution shifts, leading to inaccurate predictions on unseen future data. In this work, we investigate whether and under what conditions models can achieve such a generalization when relying solely on past data. We explore two primary approaches: convex combinations of past model parameters (parameter interpolation) and explicit extrapolation beyond the convex hull of past parameters (parameter extrapolation). We benchmark several methods within these categories on a diverse set of temporal tasks, including language modeling, news summarization, news tag prediction, academic paper categorization, satellite image-based land use classification over time, and historical yearbook photo gender prediction. Our empirical findings show that none of the evaluated methods consistently outperforms the simple baseline of using the latest available model parameters in all scenarios. In the absence of access to future data or robust assumptions about the underlying data-generating process, these results underscore the inherent difficulties of generalizing and extrapolating to future data and warrant caution when evaluating claims of such generalization" Paper: https://lnkd.in/dyeupe-V #machinelearning

"Same Error, Different Function: The Optimizer as an Implicit Prior in Financial Time Series" by Federico Vittorio Cortesi , Giuseppe Iannone , Giulia Crippa , Tomaso Poggio , Pierfrancesco Beneventano "Neural networks applied to financial time series operate in a regime of underspecification, where model predictors achieve indistinguishable out-of-sample error. Using large-scale volatility forecasting for S& P 500 stocks, we show that different model-training-pipeline pairs with identical test loss learn qualitatively different functions. Across architectures, predictive accuracy remains unchanged, yet optimizer choice reshapes non-linear response profiles and temporal dependence differently. These divergences have material consequences for decisions: volatility-ranked portfolios trace a near-vertical Sharpe-turnover frontier, with nearly 3× turnover dispersion at comparable Sharpe ratios. We conclude that in underspecified settings, optimization acts as a consequential source of inductive bias, thus model evaluation should extend beyond scalar loss to encompass functional and decision-level implications." Paper: https://lnkd.in/dJuwpBsp Code: https://lnkd.in/dkPwJHpi #machinelearning

"Understanding Generalization in Node and Link Prediction" by Antonis Vasileiou , Timo Stoll, Christopher Morris "Using message-passing graph neural networks (MPNNs) for node and link prediction is crucial in various scientific and industrial domains, which has led to the development of diverse MPNN architectures. Besides working well in practical settings, their ability to generalize beyond the training set remains poorly understood. While some studies have explored MPNNs' generalization in graph-level prediction tasks, much less attention has been given to node- and link-level predictions. Existing works often rely on unrealistic i.i.d. assumptions, overlooking possible correlations between nodes or links, and assuming fixed aggregation and impractical loss functions while neglecting the influence of graph structure. In this work, we introduce a unified framework to analyze the generalization properties of MPNNs in inductive and transductive node and link prediction settings, incorporating diverse architectural parameters and loss functions and quantifying the influence of graph structure. Additionally, our proposed generalization framework can be applied beyond graphs to any classification task under the inductive or transductive setting. Our empirical study supports our theoretical insights, deepening our understanding of MPNNs' generalization capabilities in these tasks." Paper: https://lnkd.in/dNKuBQmn #graphmachinelearning

On Wednesday, we had the pleasure of attending the project presentations by the students of IIS Amedeo Avogadro in Turin, developed in response to a real challenge launched by #KONE : designing a temporary protection system for elevator shafts during installation. The students delivered four innovative solutions, complete with modeling, prototypes, and impressive technical thinking.💡 A great example of how collaboration between education and industry can spark talent, skills, and new perspectives. A big thank you to all the colleagues who contributed to making this experience so valuable🙌 Maria Adele S. Giulia Paganini Well done, everyone! 👏 -------------------------------------------------------------------------------- Mercoledì abbiamo avuto il piacere di assistere alla presentazione dei progetti realizzati dagli studenti dell’IIS Amedeo Avogadro di Torino, sviluppati in risposta a una sfida reale lanciata da #KONE : progettare un sistema di protezioni temporanee per il vano corsa durante l’installazione degli ascensori. Gli studenti hanno presentato quattro soluzioni innovative, complete di modellazione e prototipi. 💡 Un bellissimo esempio di come la collaborazione tra scuola e impresa possa generare valore, competenze e nuove prospettive. Un enorme grazie a tutti i colleghi che hanno contribuito a rendere questa esperienza così preziosa 🙌 Maria Adele S. Giulia Paganini Ottimo lavoro! 👏