That's an interesting point - I wonder if it does make a difference.
The reason I suggested it maybe an issue is as follows:

Assume you generate 2 series of random numbers which happen to contain the same numbers but in a different order eg 7, 98, 68, 27 and 27, 98, 7, 68. If each number happens to be in a different HashSet bucket then both series will be returned as identical sequences by the iterator eg 27, 98, 68, 7.

I've tested this to prove it on my system and these numbers do get reordered to the same sequence.