Finding Duplicate Files

[Nim]

Is there an easier way to find duplicate files?

import java.io.*;
public class Test {
	public static void main(String[] args) throws Throwable {
		System.out.print(duplicate(new File("C:\\Users\\Me\\Desktop\\4-01 Living the Dream.avi"),
			new File("C:\\Users\\Me\\Desktop\\4-01 Living the Dream - Copy.avi")));
	}
	public static boolean duplicate(File f1, File f2) throws Throwable {
		FileInputStream fis1 = new FileInputStream(f1),
			fis2 = new FileInputStream(f2);
		byte[] ba1 = new byte[fis1.available()],
			ba2 = new byte[fis2.available()];
		
		if (ba1.length == ba2.length) {
			fis1.read(ba1);
			fis2.read(ba2);
			for (int i = 0; i < ba1.length; i++) {
				if (ba1[i] != ba2[i]) return false;
			}
			return true;
		}
		return false;
		
	}
}

Is this seriously the quickest way? I notice some redundancy when you read the FileInputStream into the byte array, and then check each byte. Is there a way to read the file byte-by-byte and check it on-the-fly? It would replace the byte array, to just one byte variable.

[postmortem]

I doubt your method will work for large files, that cannot fit the memory.

Doing MD5 hash sounds much more reasonable. Even if there's no pre-existing Java solution, you can find pseudo code.

[Nim]

I added the paragraph below my code the minute you posted, in case you didn't catch that. I'm willing to bet that what I've described is in the API. I tried using FileInputStream.read(), but it takes way too long.

[ProgramThis]

Well, if you are worried about reading in large files, you should probably use the buffered input stream read in (and compare) a line at a time. This way you won't run into the problem of trying to read EVERYTHING into memory all at once.

[Nim]

Yeah I did that with FileInputStream.read() and it took a lot longer. I mean, i exited the command prompt because I was tired of waiting.

[Xeel]

This solves your problem: http://www.twmacinta.com/myjava/fast_md5.php

It becomes as easy as this:

byte[] hash = MD5.getHash(new File(filename));

On my cra**y PC in the office (1 CPU core) took me about 2 and half min to calculate hash of a 4.4Gb linux iso image.

[robertsgav]

If you are looking for the solution to the duplicate files or image. You can use following

utility (http://www.duplicatefilesdeleter.com/). its a good solution to the duplicate files.

[keang]

Please don't resurrect old threads. I doubt if the OP is still interested in solving the problem after 2 years.

[kara89r]

I Know About A Free Duplicate File Finder Just Get It On :- http://www.ashisoft.com/

[Shaimali]

i think http://DuplicateFilesDeleter.com will help you better, i tried it and they did a great a job.