Duplicate File Detection System Using MD5/SHA Hashing Algorithm
Duplicate File Detection System Using MD5/SHA Hashing Algorithm
1Tulasi Krishna Kumar1, K. Swathi 2
1 Associate Professor & Training & Placement Officer, 2 MCA Final Semester
1Department of Computer Applications
1 Sanketika Vidya Parishad Engineering College, Visakhapatnam, India
1tulasikrishnakumar@gmail.com, 2kaddalaswathi10@gmail.com
ABSTRACT
A Duplicate File Detection System is an intelligent application designed to identify and manage redundant files stored across a file system or cloud storage environment. Duplicate files consume unnecessary disk space, slow down system performance, and create confusion in file management. The system addresses this problem by using cryptographic hashing techniques — specifically MD5 (Message Digest Algorithm 5) and SHA (Secure Hash Algorithm) — to generate unique digital fingerprints for every file, regardless of file type. These fingerprints are compared to detect exact duplicate files efficiently and accurately. The system supports all file types including documents, images, videos, audio files, executables, and compressed archives. The process begins with file traversal, where the system scans directories and subdirectories recursively to collect file paths. Each file is read in binary mode and passed through MD5 and SHA hashing algorithms to produce a fixed-length hash digest. Files that produce identical hash values are flagged as duplicates. A detailed report is then generated listing all duplicate groups, their file paths, sizes, and the potential disk space that can be reclaimed by removing them. The system is evaluated based on accuracy, speed, collision resistance, and storage efficiency. Performance metrics demonstrate that the SHA-256 algorithm provides stronger collision resistance while MD5 offers faster computation, making both suitable for different use cases. The final system is deployed as a desktop or web-based application providing a user-friendly interface for scanning, reviewing, and deleting duplicate files safely..
Keywords: Duplicate File Detection, MD5 Hashing, SHA-256, File System Scanning, Cryptographic Hash Functions, File Fingerprinting, Storage Optimization, Python, File Management, Data Reduplications, Binary Comparison, Hash Digest, Recursive Directory Traversal, File Integrity, Web Application.