Introduction: Oral cancer is one of the most dangerous cancers which occurs in the oral cavity. Overuse of tobacco and smoking cigarettes are the primary risk factors for developing oral cancer. Oral cancer diagnosis at an early stage can save the lives of many people with proper treatment. Objective: The proposed work aims at early detection of potentially malignant oral lesions by the development of an automated disease diagnosis system. Building a large dataset of well-annotated oral lesions is a primary key component. A novel strategy to build automatic oral cancerous image classification software is provided in this paper. Methods: In the present work, machine learning models and deep neural networks are used to build an automated diagnosis system. By using the initial data which was gathered in this study, Naive Bayes, KNN, SVM, ANN, and CNN classification models are constructed for the automated detection and classification of oral malignancies. A new CNN network is designed which consists of 43 deep layers, whose network structure is inspired by the standard VGG-16 network. Results: Performance analysis of different machine learning models and deep learning models has been provided. Results demonstrate that the deep learning model has the potential to tackle this challenging task of early detection of oral cancerous lesions. Conclusion: It is observed from experiments that different classifiers can perform well in identifying oral cancerous lesions. Particularly, the deep learning CNN model shows high accuracy in differentiating normal and cancerous images.