In today’s data-driven landscape, personalization, efficiency, and distributed learning have become paramount for tackling complex machine-learning tasks. This talk presents a unified framework that includes three different training techniques: Federated Mixture-of-Experts (FedJETs), Mixture of Prompts (MoPs), and Independent Subnet Training (IST), to address these challenges. We will discuss FedJETs that leverage the diversity of clients to train specialized experts (different small ML models) on different subsets of classes and a gating function to route inputs to the most relevant expert(s). We will discuss MoPs (Mixture of Prompts) that apply to parts of a larger model (like an LLM), identify relevant skills embedded in different groups of prompts, and dynamically weigh experts based on the target task, using a gating functionality. And, finally, to address the challenges of distributed machine learning (ML), where data and models are partitioned across multiple machines, Independent Subnet Training (IST) is introduced. IST decomposes the original network into narrow subnetworks, which are trained locally before exchanging parameters to produce new subnets. The common theme across these works is the idea of “Training a larger model via training smaller versions of it in a distributed fashion”. This talk presents a unified framework that combines FedJETs, MoPs, and IST, enabling personalized and efficient learning across distributed environments.

Bio: Anastasios (Tasos) Kyrillidis is a Noah Harding Assistant Professor at the Computer Science department at Rice University, and an MSR visiting research collaborator. Prior to that, he was a Goldstine PostDoctoral Fellow at IBM T. J. Watson Research Center (NY) and a Simons Foundation PostDoc member at the University of Texas at Austin. He finished his PhD at the CS Department of EPFL (Switzerland). He is a recipient of an NSF CAREER Award, a 2022 Amazon Research Award, a 2023 Microsoft Research Award, a Rice Engineering award for outstanding teaching and research, an IBM Goldstine Fellowship award, and an Alexander S. Onassis Public Benefit Foundation award. His research interests include (but are not limited to): Optimization for machine learning, convex and non-convex algorithms and analysis, large-scale and efficient optimization, with a special focus on machine learning, AI, and quantum algorithms.. 

See all upcoming talks at https://www.anl.gov/mcs/lans-seminars