Real-Time Prevention of Shoulder Surfing Attacks Through Object Detection
Downloads
: Shoulder surfing is a social engineering attack in which an observer covertly views a victim’s device screen to steal sensitive information. Existing countermeasures, such as privacy films and screen-obfuscation software, operate passively and fail to definitively block observers at straight-on angles. This study proposes a real-time shoulder surfing prevention application for Android that integrates Google ML Kit Face Detection with the CameraX API and foreground service. The system detects multiple faces in the front camera field of view and triggers an automatic screen lock when more than one face satisfying four filtering conditions is detected: valid facial landmark structure, horizontal rotation angle below 36°, dynamic eye-open probability threshold adapted to ambient lighting, and a minimum bounding box area ratio for proximity validation. Testing on two devices with different hardware specifications across variations in observation distance (0.5–2 m) and lighting (bright and dim) confirmed the system's functional reliability. On the high-end device, the system achieved up to 17.21 FPS with a minimum response time of 74 ms, though this higher processing intensity resulted in greater battery consumption. Evaluation revealed that detection speed drops at extreme distances due to the loss of face tracking efficiency near the minimum detection threshold, and in low-light conditions due to hardware-level shutter speed constraints. Despite these hardware-dependent limitations, functional correctness was maintained on the mid-range device with response times consistently below 300 ms. Supplementary testing on disguised observer scenarios, including the use of a physical mask, sunglasses, and a face mask, further confirmed that the system successfully detected partially occluded faces as active threats, demonstrating robustness against deliberate concealment attempts. The results confirm that the system provides a successful, automated, on-device shoulder surfing prevention mechanism without requiring internet connectivity, cloud processing, or custom model training.
L. Bošnjak and B. Brumen, “Shoulder surfing experiments: A systematic literature review,” Comput. Secur., vol. 99, Dec. 2020, doi: https://doi.org/10.1016/j.cose.2020.102023.
H. Farzand, S. Macdonald, K. Marky, and M. Khamis, “‘What you think is private is no longer’ -- Investigating the Aftermath of Shoulder Surfing on Smartphones in Everyday Life through the Eyes of the Victims,” Nov. 2024, doi: https://doi.org/10.48550/arXiv.2411.18265.
C. Bassinet, D. Michael, R. Yoann, and W. Clemens, “Mobile phone screen protector glass: A TL investigation of the intrinsic background signal,” Frontiers. Public Health, vol. 10, pp. 1–9, Sep. 2022, doi: https://doi.org/10.3389/fpubh.2022.969330.
B. J. Tang and K. G. Shin, Eye-Shield: Real-Time Protection of Mobile Device Screen Information from Shoulder Surfing. Anaheim, CA: USENIX Association, 2023. Accessed: Nov. 22, 2025. [Online]. Available: https://www.usenix.org/conference/usenixsecurity23/presentation/tang
M. Bâce, A. Saad, M. Khamis, S. Schneegass, and A. Bulling, “PrivacyScout: Assessing Vulnerability to Shoulder Surfing on Mobile Devices,” Proceedings on Privacy Enhancing Technologies, vol. 2022, no. 3, pp. 650–669, Mar. 2022, doi: https://doi.org/10.56553/popets-2022-0090.
H. Farzand, “Understanding Shoulder Surfing & Informing the Design of Protection Mechanisms,” Thesis (PhD), University of Glasgow, Glasgow, 2025. doi: https://doi.org/10.5525/gla.thesis.85064.
F. Fauzi Abdullah and S. Agustin, “Penerapan Biometric Face Recognition Menggunakan Metode Convolutional Neural Network pada Aplikasai Berbasis Android,” INDEXIA : Informatic and Computational Intelligent Journal, vol. 6, no. 1, pp. 1–11, May 2024, doi: https://doi.org/10.30587/indexia.v6i1.4958.
B. Hartanto, B. Wiryawan Yudanto, and Kustanto, “Implementasi Google ML Kit Untuk Liveness Detection Dalam Sistem Face Recognition: Analisis Kinerja dan Keamanan Pada Aplikasi Mobile,” Biner : Jurnal Ilmiah Informatika dan Komputer, vol. 4, no. 1, pp. 32–38, Jan. 2025, doi: https://doi.org/10.32699/biner.v4i1.8870.
E. M. Safitri, Z. Ameilindra, and R. Yulianti, “Analisis Teknik Social Engineering Sebagai Ancaman Dalam Keamanan Sistem Informasi: Studi Literatur,” JIFTI-Jurnal Ilmiah Teknologi Informasi dan Robotika, vol. 2, no. 2, Dec. 2020, doi: https://doi.org/10.33005/jifti.v2i2.26.
M. Anastasiah and H. Pandia, “Analisis Perilaku Pengguna Mobile Banking Terhadap Keamanan Informasi Menggunakan Metode Human Aspects of Information Security Questionnaire (HAIS-Q),” Journal Of Social Science Research, vol. 4, no. 2, 2024, doi: https://doi.org/10.31004/innovative.v4i2.9684.
M. Jamali, P. Davidsson, R. Khoshkangini, M. G. Ljungqvist, and R. C. Mihailescu, “Context in object detection: a systematic literature review,” Artif. Intell. Rev., vol. 58, no. 6, Jun. 2025, doi: 10.1007/s10462-025-11186-x.
P. Tsirtsakis, G. Zacharis, G. S. Maraslidis, and G. F. Fragulis, “Deep learning for object recognition: A comprehensive review of models and algorithms,” International Journal of Cognitive Computing in Engineering, vol. 6, pp. 298–312, Dec. 2025, doi: 10.1016/j.ijcce.2025.01.004.
G. Chinnappa, R. Rajasekaran, S. Sekar, S. Kumar Ashokkumar, S. Ravikumar, and L. Shanmugam, “ATM Shoulder Surfing Detection Using Viola-Jones Algorithm,” TIJER, vol. 11, no. 7, Jul. 2024, Accessed: Nov. 23, 2025. [Online]. Available: https://tijer.org/tijer/papers/TIJERC001306.pdf
firebase.google.com, “Face Detection | ML Kit for Firebase.” Accessed: Dec. 08, 2025. [Online]. Available: https://firebase.google.com/docs/ml-kit/detect-faces
F. Abdillah Ahmad and N. Pratiwi, “Implementation of Face Recognition, Attendance Detection, and Geolocation using TensorFlow Lite and Google ML Kit in a Mobile Attendance Application,” Sistemasi: Jurnal Sistem Informasi, vol. 14, no. 1, pp. 172–186, 2025, [Online]. Available: http://sistemasi.ftik.unisi.ac.id
developers.google.com, “Face detection | ML Kit | Google for Developers.” Accessed: Dec. 05, 2025. [Online]. Available: https://developers.google.com/ml-kit/vision/face-detection
O. Nachmani, T. Saun, M. Huynh, C. R. Forrest, and M. McRae, “‘Facekit’-Toward an Automated Facial Analysis App Using a Machine Learning–Derived Facial Recognition Algorithm,” Plastic Surgery, vol. 31, no. 4, pp. 321–329, Nov. 2023, doi: 10.1177/22925503211073843.
I. Saputra, T. Desyani, and A. Syaripudin, “Implementasi Login Aplikasi dengan Fitur Autentikasi Pengguna Menggunakan Flutter dan ML Kit Face Recognation,” JORAPI : Journal of Research and Publication Innovation, vol. 3, no. 1, pp. 106–115, Jan. 2025, Accessed: Dec. 06, 2025. [Online]. Available: https://jurnal.portalpublikasi.id/index.php/JORAPI/article/view/1358
B. B. Wibowo and E. B. Setiawan, “Implementasi Face Recognition dan Geolocation Pada Sistem Presensi Karyawan Berbasis Mobile APPS,” KOMPUTA : Jurnal Ilmiah Komputer dan Informatika, vol. 13, no. 1, Apr. 2024.
M. Corbett, B. David-John, J. Shang, and J. I. Bo, “ShouldAR: Detecting Shoulder Surfing Attacks Using Multimodal Eye Tracking and Augmented Reality,” Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., vol. 8, no. 3, Sep. 2024, doi: 10.1145/3678573.
