Data Integrity and Consistency Challenges in Distributed Database Systems
Downloads
With the advent of distributed computing, maintaining data consistency and integrity between geographically remote and logically divided systems has become one of the most significant challenges for system designers and database architects. Distributed databases, though they offer higher availability, scalability, and local autonomy, create problems in maintaining synchronized and consistent states of data. This work covers the Database Integrity (DDBI) project—a pioneering endeavor sponsored by Rome Laboratory and conducted at SRI International—where innovative methods such as rule-based enforcement, active trigger algorithms, and intelligent object versioning were presented to address integrity in distributed object-oriented databases. Apart from the results of the DDBI project, this paper addresses root consistency and transaction models like the Two-Phase Commit (2PC) protocol and examines current replication methods in large-scale data environments. A theoretical classification of consistency models is presented to aid in selecting the most appropriate frameworks for different system needs. Moreover, the study explores the implications of Intelligent Management Objects (IMOs) to integrity and discusses consistency-availability trade-offs. Drawn statistics from comparative studies and simulations confirm the effectiveness of the proposed methods. Finally, this paper provides an organized overview of the techniques reshaping the future of distributed database systems and the need for adaptive, robust, and intelligent design paradigms.
(eds), J.H.D.a.L.R., Database Integrity—Challenges and Solutions. Institutional Knowledge at Singapore Management University, 2020.
Aldoseri, A., K.N. Al-Khalifa, and A.M. Hamouda, Re-Thinking Data Strategy and Integration for Artificial Intelligence: Concepts, Opportunities, and Challenges. Applied Sciences, 2023. 13(12).
Bhattarai, B.P., et al., Big data analytics in smart grids: state‐of‐the‐art, challenges, opportunities, and future directions. IET Smart Grid, 2019. 2(2): p. 141-154.
Chanson, M., et al., "Blockchain for the IoT: Privacy-Preserving Protection of Sensor Data". Journal of the Association for Information Systems, 2019: p. 1272-1307.
Diène, B., et al., Data management techniques for Internet of Things. Mechanical Systems and Signal Processing, 2020. 138.
Dong, S., et al., RocksDB: Evolution of Development Priorities in a Key-value Store Serving Large-scale Applications. ACM Transactions on Storage, 2021. 17(4): p. 1-32.
Gadde, H., Leveraging AI for Scalable Query Processing in Big Data Environments. International Journal of Advanced Engineering Technologies and Innovations, 2023. 1(2).
Gadde, H., Optimizing Transactional Integrity with AI in Distributed Database Systems. International Journal of Advanced Engineering Technologies and Innovations, 2024. 1(2).
Habib, G., et al., Blockchain Technology: Benefits, Challenges, Applications, and Integration of Blockchain Technology with Cloud Computing. Future Internet, 2022. 14(11).
Hang, L. and D.H. Kim, Design and Implementation of an Integrated IoT Blockchain Platform for Sensing Data Integrity. Sensors (Basel), 2019. 19(10).
Hang, L., I. Ullah, and D.-H. Kim, A secure fish farm platform based on blockchain for agriculture data integrity. Computers and Electronics in Agriculture, 2020. 170.
Haoran YUAN, X.C., Jianfeng WANG, Jiaming YUAN, Hongyang YAN, and Willy SUSILO, Blockchain-based public auditing and secure deduplication with fair arbitration. Institutional Knowledge at Singapore Management University, 2020.
Hassan, M.U., M.H. Rehmani, and J. Chen, Privacy preservation in blockchain based IoT systems: Integration issues, prospects, challenges, and future research directions. Future Generation Computer Systems, 2019. 97: p. 512-529.
Iqbal, N., et al., A Novel Blockchain-Based Integrity and Reliable Veterinary Clinic Information Management System Using Predictive Analytics for Provisioning of Quality Health Services. IEEE Access, 2021. 9: p. 8069-8098.
Ismail, L., H. Materwala, and S. Zeadally, Lightweight Blockchain for Healthcare. IEEE Access, 2019. 7: p. 149935-149951.
Kishore Reddy Gade JP Morgan Chase, U., Data Quality in the Age of Cloud Migration: Challenges and Best Practices. MZ Journals, 2020.
Kolb, J., et al., Core Concepts, Challenges, and Future Directions in Blockchain. ACM Computing Surveys, 2020. 53(1): p. 1-39.
Kumaran, R., ETL Techniques for Structured and Unstructured Data. International Research Journal of Engineering and Technology (IRJET), 2021. 8(12).
Laigner, R., et al., Data Management in Microservices: State of the Practice, Challenges, and Research Directions. International Journal of Advanced Engineering Technologies and
Innovations, 2021.
Li, W., et al., Secure Data Integrity Check Based on Verified Public Key Encryption With Equality Test for Multi-Cloud Storage. IEEE Transactions on Dependable and Secure Computing, 2024. 21(6): p. 5359-5373.
Liu, R. and Z. Shan, Is Blockchain a Silver Bullet for Supply Chain Management? Technical Challenges and Research Opportunities. Decision Sciences Institute, 2019.
M. R. JAMALI++, A.G.M., M. R. MAREE, Security issues in data at rest in a non-relational Document Database.
SINDHUNIVERSITYRESEARCHJOURNAL(SCIENCE SERIES), 2020.
MACIEJ BESTA, et al., Demystifying Graph Databases: Analysis and Taxonomy of Data Organization, System Designs, and Graph Queries. International Journal of Advanced Engineering Technologies and Innovations, 2023.
Maurice Bailleu, D.G., and Vasilis Gavrielatos, Avocado: A Secure In-Memory Distributed Storage System. USENIX Annual Technical Conference, 2021.
Mohiuddin, A.B. and N. Ahmed, Block chain-Based Solutions for Improved Cloud Data Integrity and Security. BULLET: Jurnal Multidisiplin Ilmu, 2022.
Muvva, S., Blockchain Technology in Data Engineering: Enhancing Data Integrity and Traceability in Modern Data Pipelines. International Journal of Leading Research Publication (IJLRP), 2023.
Paik, H.-Y., et al., Analysis of Data Management in Blockchain-Based Systems: From Architecture to Governance. IEEE Access, 2019. 7: p. 186091-186107.
Pamadi, V.N., O. Goel, and L. Kumar, Effective Strategies for Building Parallel and Distributed Systems. International Journal of Novel Research in Engineering & Pharmaceutical Sciences, 2020. 5(1).
PengCheng Wei1, D.W., Yu Zhao1, Sumarga Kumar Sah Tyagi3, Neeraj Kumar4, Blockchain data-based cloud data integrity protection mechanism. Institutional Knowledge at Singapore Management University, 2019.
Rawat, D.P., MOSAIC OF IDEAS. Multidisciplinary reflection. International Journal of Advanced Engineering Technologies and Innovations, 2024.
Ruan, P., et al., BLOCKCHAINS VS. DISTRIBUTED DATABASES: DICHOTOMY AND FUSION. International Journal of Leading Research Publication (IJLRP), 2021.
Sandhu, A.K., Big data with cloud computing: Discussions and challenges. Big Data Mining and Analytics, 2022. 5(1): p. 32-40.
Singh, S., A.S.M.S. Hosen, and B. Yoon, Blockchain Security Attacks, Challenges, and Solutions for the Future Distributed IoT Network. IEEE Access, 2021. 9: p. 13938-13959.
Solanke, A., Blockchain Implementation for Enterprise Data Integrity: Distributed Ledger Strategies for Supply Chain Transparency and Trustworthy Data Verification. International Journal of Advanced Engineering Technologies and Innovations, 2024.
Sundararajan, A., et al., Survey on synchrophasor data quality and cybersecurity challenges, and evaluation of their interdependencies. Journal of Modern Power Systems and Clean Energy, 2018. 7(3): p. 449-467.
Tao, X., et al., Distributed common data environment using blockchain and Interplanetary File System for secure BIM-based collaborative design. Automation in Construction, 2021. 130.
Thokala, V.S., A Comparative Study of Data Integrity and Redundancy in Distributed Databases for Web Applications. INTERNATIONAL JOURNAL OF RESEARCH AND ANALYTICAL REVIEWS, 2021. 8(4).
Viriyasitavat, W. and D. Hoonsopon, Blockchain characteristics and consensus in modern business processes. Journal of Industrial Information Integration, 2019. 13: p. 32-39.
Wang, H. and J. Zhang, Blockchain Based Data Integrity Verification for Large-Scale IoT Data. IEEE Access, 2019. 7: p. 164996-165006.
Wang, X., et al., Big data management challenges in health research-a literature review. Brief Bioinform, 2019. 20(1): p. 156-167.
Williamson, S.M. and V. Prybutok, Balancing Privacy and Progress: A Review of Privacy Challenges, Systemic Oversight, and Patient Perceptions in AI-Driven Healthcare. Applied Sciences, 2024. 14(2).
Wu, H., et al., VQL: Efficient and Verifiable Cloud Query Services for Blockchain Systems. IEEE Transactions on Parallel and Distributed Systems, 2022. 33(6): p. 1393-1406.
Xu, L.D., Y. Lu, and L. Li, Embedding Blockchain Technology Into IoT for Security: A Survey. IEEE Internet of Things Journal, 2021. 8(13): p. 10452-10473.
Yesin, V., et al., Ensuring Data Integrity in Databases with the Universal Basis of Relations. Applied Sciences, 2021. 11(18).
Yue, D., et al., Blockchain-based verification framework for data integrity in edge-cloud storage. Journal of Parallel and Distributed Computing, 2020. 146: p. 1-14.
Zhang, X. and X. Chen, Data Security Sharing and Storage Based on a Consortium Blockchain in a Vehicular Ad-hoc Network. IEEE Access, 2019. 7: p. 58241-58254.
Zheng, R., et al., bcBIM: A Blockchain‐Based Big Data Model for BIM Modification Audit and Provenance in Mobile Cloud. Mathematical Problems in Engineering, 2019. 2019(1).
Zhu, H., et al., A Secure and Efficient Data Integrity Verification Scheme for Cloud-IoT Based on Short Signature. IEEE Access, 2019. 7: p. 90036-90044.
