Stochastic optimization plays a vital role in machine learning where the full batch of data is either unavailable or too large to process in practice. This course introduces the theoretical foundations and algorithmic development in this area. The topics will start form the basic convex optimization theories as well as numerical methods, and we then focus on the stochastic approximation for stochastic optimization and its various accelerations in many statistical and machine learning models, supplemented with the most recent progress from research literature. Basic theoretic understanding of these stochastic optimization algorithms will also be explained. After this class, the students with some preliminaries of classic optimizations and probability theories are expected to transit into the new optimization world in the machine learning, in which significant progresses have been made during the past decades.