We are living in the age of AI and Machine learning and optimization is known as the heart of artificial intelligence algorithms. Researchers are developing new and complex machine learning and deep learning models to solve real word problems. In each algorithm minimization of the cost function is important which tells the mapping of input and output of the model. Thus, the cost function minimization is the basic function of optimization algorithms. However, optimization perspectives and optimization algorithm qualities are two distinct aspects. However, computationally inexpensive optimization algorithms are always in demand of machine learning algorithms. In this chapter we are presenting important optimization algorithms and their applications.