We review optimization methods that underpin machine-learning approaches. Our goal is to provide an overview of the connections between optimization and machine learning. We discuss the impact of nonsmooth optimization methods, splitting methods, gradient and Newton-type methods. We also examine the impact of different model formulations such as robust optimization, mixed-integer optimization, and complementarity constraints. Our goal is to highlight both the limitations and promises of optimization methodologies and formulations.