Throughout the year's few work has been carry out for vision-based Apple disease framework. Mainly apple disease recognition includes two issues: one is infection identification and another is disease classification. Because of advancement of vision-based innovation we got better framework for this issue. The datasets are mainly grouped into four categories i.e. normal, rot, blotch, scab, the last three being the three major kind of defects found in apples. The aim is to distinguish these defected apples from the normal ones. In this chapter, we propose an Alex net and VGG-16 based deep learning model for classification of disease in all categories of apple. The performance of Alex-Net model 95.56 percentage where as VGG-16 produce 94 percentage accuracy rate. In both model highest classification accuracy has been produce for the rot disease apple category.