Centralized Multi Modal Deep Learning for Breast Cancer Diagnosis: A Physics Aware Approach
Downloads
Breast cancer is still a major cause of cancer death in women around the world, so it needs a precise diagnosis that often goes beyond one imaging method. While deep learning has shown great promise in automated diagnosis, current research often suffers from the separation of imaging modalities. This study offers a comprehensive, physics-aware deep learning framework that combines Ultrasound, MRI, and Mammography. We present custom preprocessing pipelines specific to each modality to tackle the different physical degradation models associated with each type of imaging. For MRI, we adopt a 2D slice-based methodology involving Key Slice Extraction to find the most informative tumor cross-section followed by N4 Bias Field Correction and Otsu’s thresholding for intensity normalization. This allows the effective use of 2D Convolutional Neural Networks without incurring the computational cost associated with 3D processing. Using transfer learning from ResNet50 (Ultrasound/Mammography) and DenseNet121 (MRI), our centralized models reached state-of-the-art accuracies of 92.50%, 90.63%, and 92.00%, proving that a centralized multi-modal approach can be effective in enhancing diagnostic precision.
Alom, M. R., et al. (2025). An explainable AI driven deep neural network for accurate breast cancer detection from histopathological and ultrasound images. Scientific Reports, 15, 9771.
Abdullah, M., & Nuzla, F. (2025). Deep learning for multi modal medical imaging fusion: Enhancing diagnostic accuracy in complex disease detection. International Journal of Engineering Technology Research and Management.
Khan, S. (2021). Deep learning in digital breast tomosynthesis. PMC.
Li, T., et al. (2025). Deep learning in multi modal breast cancer data fusion: a literature review. Quantitative Imaging in Medicine and Surgery, 15(11), 11578 11610.
Yala, A., et al. (2019). A Deep Learning Model to Predict Breast Cancer Risk from Mammograms. Radiology, 292(1), 60 66.
Chen, Y., et al. (2025). AI in Breast Cancer Imaging: An Update and Future Trends. Seminars in Nuclear Medicine.
Gao, Y., et al. (2024). An explainable longitudinal multi modal fusion model for predicting treatment response in breast cancer. Nature Communications, 15, 5345.
Ahmadi, M., et al. (2025). Physics informed machine learning for advancing computational medical imaging. Artificial Intelligence Review.
Xiang, Y., et al. (2025). Hyperspectral Image Restoration and Super resolution with Physics Aware Deep Learning for Biomedical Applications. arXiv:2503.02908.
Hirsch, L., et al. (2025). High Performance Open Source AI for Breast Cancer Detection in MRI. Radiology: Artificial Intelligence.
Saldanha, O. L., et al. (2025). Swarm learning with weak supervision enables automatic breast cancer detection in MRI. Communications Medicine, 5, 72.
Alotaibi, M., et al. (2023). Breast cancer classification based on convolutional neural network and image fusion approaches using ultrasound images. Heliyon, 9(10), e20440.
Baccouche, A., et al. (2022). An integrated framework for breast mass classification and segmentation using deep learning. Scientific Reports, 12, 15632.
Houssein, E. H., et al. (2022). An optimized deep learning architecture for breast cancer diagnosis. Psychological Medicine.
Puttegowda, K., et al. (2025). Enhanced machine learning models for accurate breast cancer detection. Results in Engineering.
Lee, R. S., et al. (2017). A curated mammography data set for use in computer aided detection and diagnosis research. Scientific Data, 4, 170177.
Guo, D., et al. (2024). A multimodal breast cancer diagnosis method based on Knowledge Augmented Deep Learning. Biomedical Signal Processing and Control, 90.
Wu, J., et al. (2023). Multimodal microscopic imaging with deep learning for highly effective diagnosis of breast cancer. Optics and Lasers in Engineering, 168.
Sharma, G. N., et al. (2010). Various types and management of breast cancer: an overview. Journal of Advanced Pharmaceutical Technology & Research, 1(2), 109.
Arya, N., & Saha, S. (2021). Multi modal advanced deep learning architectures for breast cancer survival prediction. Knowledge Based Systems, 221.
Park, S., et al. (2012). Characteristics and outcomes according to molecular subtypes of breast cancer as classified by a panel of four biomarkers using immunohistochemistry. Breast, 21(1), 50 57.
Sun, D., et al. (2019). A Multimodal Deep Neural Network for Human Breast Cancer Prognosis Prediction by Integrating Multi Dimensional Data. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 16(3), 841 850.
Arya, N., & Saha, S. (2022). Multi modal Classification for Human Breast Cancer Prognosis Prediction: Proposal of Deep Learning Based Stacked Ensemble Model. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 19(2), 1032 1041.
Yamakawa, Y., et al. (1991). A tentative tumor node metastasis classification of thymoma. Cancer, 68(9), 1984 1987.
Verma, M., et al. (2023). Multimodal Spatiotemporal Deep Learning Framework to Predict Response of Breast Cancer to Neoadjuvant Systemic Therapy. Diagnostics, 13(1).
Li, H., et al. (2024). MRI and RNA seq fusion for prediction of pathological response to neoadjuvant chemotherapy in breast cancer. Displays, 83.
Chen, H., et al. (2019). Attention Based Multi Modal Deep Neural Network with Multimodality Data for Breast Cancer Prognosis Model. BioMed Research International, 2019.
Tong, L., et al. (2020). Deep learning based feature level integration of multi omics data for breast cancer patients survival analysis. BMC Medical Informatics and Decision Making, 20, 225.
Guo, W., et al. (2021). Multimodal Affinity Fusion Network for Predicting the Survival of Breast Cancer Patients. Frontiers in Genetics, 12.
Yan, R., et al. (2021). A multi modal deep learning model to predict breast cancer recurrence and metastasis risk by integrating pathological, clinical and gene expression data. Briefings in Bioinformatics, 22(5).
