MultimodalDeepLearning.PDFVIP

Multimodal Deep Learning for Cervical Dysplasia Diagnosis Tao Xu1?, Han Zhang2?, Xiaolei Huang1, Shaoting Zhang3, and Dimitris N. Metaxas2 1 Computer Science and Engineering Department, Lehigh University, Bethlehem, PA, USA 2 Department of Computer Science, Rutgers University, Piscataway, NJ, USA 3 Department of Computer Science, UNC Charlotte, Charlotte, NC, USA Abstract. To improve the diagnostic accuracy of cervical dysplasia, it is im- portant to fuse multimodal information collected during a patient’s screening visit. However, current multimodal frameworks suffer from low sensitivity at high speci?city levels, due to their limitations in learning correlations among highly heterogeneous modalities. In this paper, we design a deep learning framework for cervical dysplasia diagnosis by leveraging multimodal information. We ?rst employ the convolutional neural network (CNN) to convert the low-level image data into a feature vector fusible with other non-image modalities. We then jointly learn the non-linear correlations among all modalities in a deep neural network. Our multimodal framework is an end-to-end deep network which can learn better complementary features from the image and non-image modalities. It automat- ically gives the ?nal diagnosis for cervical dysplasia with 87.83% sensitivity at 90% speci?city on a large dataset, which signi?cantly outperforms methods using any single source of information alone and previous multimodal frameworks. 1 Introduction Cervical cancer ranks as the second most common type of cancer in women aged 15 to 44 years worldwide [13]. Screening can help prevent cervical cancer by detecting cer- vical intraepithelial neoplasia (CIN), which is the potentially precancerous change and abnormal growth of squamous cells on the surface of the cervix. According to the World Health Organization (WHO) [13], CIN