Comparative Analysis of Mask-R CNN and YOLOv8 Models for Automated Detection and Classification of Malaria Parasite in Microscopy Images

Abstract

Accurate and efficient detection of malaria parasites in stained blood smear images remains a critical challenge, particularly in resource-limited settings where expert microscopists may be unavailable. This study compares two deep learning instance segmentation models, YOLOv8 and Mask R-CNN, for automated detection, segmentation, and life-stage classification of malaria parasites in publicly available Giemsa-stained microscopy images. A total of 1,328 annotated images were used to fine-tune YOLOv8n and Mask R-CNN (ResNet-50-FPN backbone). YOLOv8 achieved higher detection performance with bounding-box mAP50 of 0.648, mask mAP50 of 0.624, mean accuracy of 96.7%, and F1-score of 0.71, compared to Mask R-CNN’s mAP50 of 0.511, accuracy of 93.2%, and F1-score of 0.48. Bootstrap resampling (1,000 iterations) confirmed the statistical reliability of performance differences with 95% confidence intervals. YOLOv8 also achieved faster inference (9 ms per image) than Mask R-CNN (93 ms), highlighting its potential for real-time screening. Despite data imbalance among parasite stages, both models produced meaningful segmentation masks enabling quantitative morphological analysis. These results demonstrate that lightweight, statistically validated deep learning architectures can deliver reliable, scalable, and interpretable tools for automated malaria detection and quantification, promoting AI integration into diagnostic microscopy workflows.