An Ensemble Vision Transformer Approach for Automated Detection of Hepatocellular Carcinoma

Hepatocellular carcinoma (HCC) is the most common type of primary liver cancer in adults and the third leading cause of cancer-related mortality worldwide. Early-stage detection and diagnosis are crucial for reducing patient mortality. The conventional clinical standard relies on histopathological image examination, which requires substantial manual effort, extended processing time, and is prone to observer-dependent variation. Advanced machine learning (ML) and deep learning (DL) approaches have been widely applied to support early cancer detection and diagnosis. However, the scarcity of large annotated medical image datasets and reliance on single models may degrade performance, limiting the development of robust automated systems.To address these challenges, this study proposes a robust classification framework to distinguish HCC from healthy liver histopathological images. The proposed pipeline integrates data preprocessing and augmentation, deep feature extraction using ViT-B/16, LASSO-based feature selection, and a calibrated weighted soft voting ensemble classifier combining Logistic Regression (LR), Support Vector Machine (SVM), CatBoost, and XGBoost. The ViT-B/16 model effectively captures both local and global contextual dependencies through multi-head self-attention mechanisms, while LASSO performs simultaneous feature selection and regularization through cross-validation penalization. The final ensemble aggregates the decision boundaries of multiple classifiers and achieves an accuracy of 96.62%, AUC of 99.56%, precision of 96.62%, sensitivity of 96.57%, specificity of 96.67%, F1-score of 96.62%, and MCC of 0.9324, outperforming existing state-of-the-art methods. These findings demonstrate the effectiveness of integrating transformer-based feature extraction, interpretable feature selection, and ensemble classification for improving clinical decision support in liver cancer diagnosis.