ECE 5290/7290 & ORIE 5290: Distributed Optimization for Machine Learning and AI
Instructor: Tianyi Chen | tianyi.chen@cornell.edu | Semester: Fall 2025
Class time: MoWe 1:25 PM – 2:40 PM | Location: Cornell Tech, Bloomberg Center 161
Overview
This is a graduate-level course about theory, algorithms, and applications of distributed optimization and machine learning. The course covers the basics of distributed optimization and learning algorithms, and their performance analyses when they are used to solve large-scale distributed problems arising in AI, machine learning, signal processing, communication networks, and power systems.
- Prerequisites: Linear Algebra, Calculus, Probability, and Basic ML and optimization background
- Format: Lectures + Homework + Exam (only a little) + Project
Public resources: The lecture slides and assignments will be posted online as the course progresses. We are happy for anyone to learn from these resources, but we cannot grade the work of any unenrolled students
Cornell students: Students should ask all course-related questions on Ed Discussion, submit homework on Gradescope, and find all the announcements on Canvas.
Instructor, TAs, and Graders
Tianyi Chen
Instructor
Email
Yuheng Wang
Teaching Assistant
Email
Anisha Kechaphna
Grader
Email
Zhaoxian Wu
Student Volunteer
Email
Modules
Course surveys: To help tailor the course and improve your learning experience, please complete:
- Background Survey (first class) — let us understand your preparation and interests.
- Anonymous Feedback Form — share your thoughts or suggestions during the semester.
Week 1 Aug 25 – Aug 27 2 sessions
-
Mon, Aug 25Introduction & MotivationOverview of course goals, applications of distributed optimization in modern AI and ML systems, and motivating examples from large-scale model training and federated learning.
-
Wed, Aug 27ML fundamentals for optimization (ERM, supervised learning)Review of empirical risk minimization (ERM), supervised learning frameworks, convexity basics, and gradient-based learning methods.
Week 2 Sep 01 – Sep 03 2 sessions
-
Mon, Sep 01No class - Labor day
-
Wed, Sep 03ML Fundamentals Continued: Optimization ViewpointFrom linear and nonlinear regression to general loss functions; connecting optimization to machine learning tasks and understanding challenges in non-convex learning.
Week 3 Sep 08 – Sep 10 2 sessions
-
Mon, Sep 08Optimization Basics I: Gradient Descent on Quadratic ProblemsDerivation and intuition of gradient descent, convergence on quadratic objectives, and the geometry of step sizes and conditioning.
-
Wed, Sep 10Optimization Basics II: Convergence and ComplexityConvergence analysis for smooth and strongly convex functions; understanding sublinear and linear rates of convergence for gradient descent.
Week 4 Sep 15 – Sep 17 2 sessions
-
Mon, Sep 15Beyond Convexity: Non-Convex Landscapes and SmoothnessExtending gradient-based methods to non-convex settings; smoothness assumptions, saddle points, and convergence guarantees.
-
Wed, Sep 17Gradient Methods for Constrained OptimizationProjected gradient methods, constraint handling in distributed settings, and convergence rates under constraints.
Week 5 Sep 22 – Sep 24 2 sessions
-
Mon, Sep 22Stochastic Optimization: SGD, Minibatching, and ConvergenceFundamentals of stochastic gradient descent, convergence properties under noise, and trade-offs between batch size and computation.
-
Wed, Sep 24Variance Reduction and MomentumModern SGD enhancements — variance-reduced methods (SVRG) and momentum techniques for accelerating convergence.
Week 6 Sep 29 – Oct 01 2 sessions
-
Mon, Sep 29Variance Reduction and Momentum in PracticeFinite-sum optimization, adaptive methods (Adam, AdaGrad), and a deeper look at the variance–bias trade-off in stochastic learning.
-
Wed, Oct 01Consensus and Spectrum of GraphsFundamentals of consensus algorithms, properties of doubly-stochastic matrices, and spectral connectivity measures in networks.
Week 7 Oct 06 – Oct 08 2 sessions
-
Mon, Oct 06Gossip and Random WalksRandomized gossip algorithms, asynchronous communication, and relationships between random walks and averaging in networks.
-
Wed, Oct 08Data and Model Parallelism in Distributed TrainingData and model parallelism in distributed training; local SGD, synchronization intervals, and communication–computation balancing.
Week 8 Oct 13 – Oct 15 2 sessions
-
Mon, Oct 13No class - Fall break
-
Wed, Oct 15Communication-Efficient Distributed MethodsQuantization and local updates; techniques to reduce bandwidth while preserving convergence in distributed learning.
Week 9 Oct 20 – Oct 22 2 sessions
-
Mon, Oct 20Communication-Efficient Distributed MethodsSparsification, and worker selections; techniques to reduce bandwidth while preserving convergence in distributed learning.
-
Wed, Oct 22Decentralized Algorithms: Consensus GD and Its ConvergenceGradient tracking algorithms, convergence under directed and time-varying graphs.
Week 10 Oct 27 – Oct 29 2 sessions
-
Mon, Oct 27No class - Asynchronous Office Q&A on Ed Discussion and Emails
-
Wed, Oct 29In-person Exam
Week 11 Nov 03 – Nov 05 2 sessions
-
Mon, Nov 03Robust distributed optimization: adversaries, attacks and defensesRobustness against adversarial clients, Byzantine attacks, and noisy updates; algorithmic defenses and aggregation strategies.
-
Wed, Nov 05The curse of data heterogeneity in distributed learningPersonalization techniques and handling heterogeneous data distributions in federated settings.
Week 12 Nov 10 – Nov 12 2 sessions
-
Mon, Nov 10Transformers - Architecture, Parameters, and MemoriesTransformer architectures, parameter scaling laws, and memory components relevant to efficient and distributed training.
-
Wed, Nov 12Memory footprint of GPT and mixed precision trainingMemory bottlenecks in GPT training, covering parameters, activations, optimizer states, and how mixed-precision methods enable efficient distributed optimization.
Week 13 Nov 17 – Nov 19 2 sessions
-
Mon, Nov 17Analogc computing for energy-efficient AI: Part IPrinciples of analog computing for neural network inference, focusing on in-memory computation, and hardware–algorithm co-design.
-
Wed, Nov 19Analogc computing for energy-efficient AI: Part IIAnalog computing techniques for neural network training, highlighting gradient computation and hardware–algorithm co-design under device nonidealities.
Week 14 Nov 24 – Nov 26 2 sessions
-
Mon, Nov 24Pre-training and fine-tuning LLMsOptimization methods for pre-training and fine-tuning LLMs, including data scaling, parameter-efficient adaptation, and distributed training tradeoffs.
-
Wed, Nov 26No class - Thanksgiving
Week 15 Dec 01 – Dec 03 2 sessions
-
Mon, Dec 01Project presentation - Part IECE/ORIE 5290 student projects: educational presentations highlighting key papers, methods, and open challenges from the recent literature in distributed optimization.
-
Wed, Dec 03Project presentation - Part IIECE/ORIE 5290/ECE 7290 Students project presentations: showcasing research topics, key findings, and proposed extensions in distributed optimization.
Week 16 Dec 08 – Dec 08 1 sessions
-
Mon, Dec 08Project presentation - Part IIIECE 7290 Students project presentations: showcasing research topics, key findings, and proposed extensions in distributed optimization.
Assignments & Project
Synergy with other ECE Courses
Due to space limitations, the current list highlights selected ECE courses offered at Cornell Tech that are open to Master’s students and have strong thematic connections with ECE 5290.
- ECE 5242 — Intelligent Autonomous Systems: Shares strong links through optimization and control for multi-agent and autonomous systems, where distributed algorithms enable real-time decision-making and coordination.
- ECE 5260 — Graph-Based Data Science for Networked Systems: Aligns with distributed and graph-structured optimization, where consensus algorithms form a unifying framework for networked learning and inference.
- ECE 5414 — Applied Machine Learning: ECE 5290 complements this course by emphasizing the optimization algorithms that drive scalable learning, convergence analysis, and the theoretical underpinnings of model training.
- ECE 5415 — Digital Signal Processing and Learning: Connects signal processing and optimization perspectives, showing how filtering, transforms, and iterative methods relate to modern ML and neural network training.
- ECE 5545 — Machine Learning Hardware and Systems: Extends concepts to practical implementation, examining how distributed optimization interacts with computing hardware and systems for efficient ML deployment.
Suggested Readings
These papers offer background and provide deeper insights into distributed optimization and its applications in machine learning.
Advances and Open Problems in Federated Learning
P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, et al.; arXiv preprint, 2021
The definitive survey of federated learning—covering foundations, system challenges, privacy, personalization, and open research directions. Essential background for advanced topics in this course.
DeepSpeed: System Optimizations Enable Training Beyond 100 Billion Parameters
J. Rasley, S. Rajbhandari, O. Ruwase, Y. He; Proceedings of the International Conference for High Performance Computing (SC), 2020
System-level innovations enabling efficient large-scale model training—essential reading for project work.
Local SGD Converges Fast and Communicates Little
S. U. Stich; ICLR, 2019
Formal analysis of local SGD showing near-linear speedup with limited communication.
LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning
T. Chen, G. Giannakis, T. Sun, W. Yin; NeurIPS, 2018
Introduces LAG, which adaptively skips redundant gradient updates to reduce communication cost without harming convergence.
Communication Compression for Decentralized Training
H. Tang, X. Lian, M. Yan, C. Zhang, J. Liu; NeurIPS, 2018
Shows how gradient compression techniques accelerate decentralized training while maintaining convergence.
Achieving geometric convergence for distributed optimization over time-varying graphs
A. Nedić, A. Olshevsky, W. Shi; SIAM Journal on Optimization, 2017
Pioneering paper introducing gradient tracking, enabling linear convergence in time-varying graphs.
Communication-Efficient Learning of Deep Networks from Decentralized Data
H. B. McMahan, E. Moore, D. Ramage, S. Hampson, B. A. Arcas; AISTATS, 2017
Introduces FedAvg, the foundational algorithm for federated learning.
On the Convergence of Decentralized Gradient Descent
K. Yuan, Q. Ling, W. Yin; SIAM Journal on Optimization, vol. 26, no. 3, pp. 1835–1854, 2016
A seminal theoretical study of decentralized gradient descent (DGD), providing a convergence rate analysis for both diminishing and fixed step sizes over static networks.
EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimization
W. Shi, Q. Ling, K. Yuan, G. Wu, W. Yin; SIAM Journal on Optimization, 2015
A breakthrough decentralized algorithm achieving exact convergence to the global optimum using local updates.
Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers
S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein; Foundations and Trends in Machine Learning, 2011
The classic monograph on ADMM—still the most cited reference for distributed convex optimization.
Resources
- Optimization Methods for Large-Scale Machine Learning by Leon Bottou & Frank Curtis & Jorge Nocedal
- First-Order Methods in Optimization by Amir Beck
- Large-Scale Convex Optimization: Algorithms & Analyses via Monotone Operators by Ernest Ryu & Wotao Yin
Acknowledgement
- Parts of the course material were adapted from STAT9910-303: Large-Scale Optimization for Data Science by Yuxin Chen (Wharton Statistics and Data Science, UPenn).
- Parts of the course material were drawn from Optimization for Deep Learning and Foundation Models by Kun Yuan (Center for Machine Learning Research, PKU).
Course Policies
Academic integrity, late submission, and collaboration policies will follow Cornell Tech standards.