Filling an array with random lines from a file

[Ryan1590]

Hi

I currently have a program that reads the first 10 lines from a text file and adds them to a vector array, however how would you instead choose lets say 100 random lines from the text file and place those into the vector array.

const char newLine = 10;
	ifstream file("rockyou.txt");
	vector<string> arr;
	for (int i = 0; i < 10; i++) {
		string temp;
		file >> temp;
		arr.push_back(temp);
	}
	for (int i = 0; i < 10; i++) {
		cout << arr[i] << endl;
	}  
}

Thanks for any help you can provide.

[laserlight]

Do you know how many lines there are in the text file? If you do, then you can number the lines from 0 to n-1, where n is the number of lines, and then generate 100 unique random integers within this range. After that, it becomes a matter of reading line by line and copying the corresponding lines to the vector while discarding the rest.

If you don't know how many lines there are, then one way is to do a two-pass solution, i.e., first you read the text file line by line to count the number of lines, then you proceed as described above.

[wolle]

If you don't know how many lines there are, then one way is to do a two-pass solution

Instead of line numbers, character positions may be used instead. A certain number of random character positions within the file are generated and kept in sorted order.

A random character position most likely will be somewhere within a line so that line is skipped and the next line is used (and if there is no next line because eof was reached the first line of the file is used instead). This can all be accomplished using the standard functions of std::istream such as seekg, tellg and getline.

This should be a fast solution since essentially only the randomly selected lines are read and there's just one pass. It may be somewhat more involved to ensure that lines are unique (if that's wanted).

[2kaud]

If the size of the file is such that it all will fit into memory, then an alternative is to read the whole file into a vector, randomly shuffle the vector and then use the top n elements as required. Consider

#include <fstream>
#include <vector>
#include <string>
#include <iostream>
#include <algorithm>
#include <random>
using namespace std;

int main()
{
	const string fnam = "rockyou.txt";
	const size_t lines = 20;

	ifstream ifs(fnam);

	if (!ifs.is_open()) {
		cout << "Cannot open file " << fnam << endl;
		return 1;
	}

	vector<string> fildata;

	for (string l; getline(ifs, l); fildata.push_back(l));
	shuffle(fildata.begin(), fildata.end(), mt19937((random_device())()));

	vector<string> data(fildata.begin(), fildata.begin() + min(lines, fildata.size()));

	for (const auto& v : data)
		cout << v << endl;
}

The vector fildata contains the whole of the file, which is then randomly shuffled using the Mersenne Twister 19937 random generator. The first 'lines' elements are then copied to the vector data for use, if needed - otherwise the shuffled vector fildata could be used.

[Ryan1590]

Hi

Thanks for the reply the file currently contains over 30 million passwords, so I was aiming to randomly select around a hundred of these each time the program starts.

[2kaud]

Hi

Thanks for the reply the file currently contains over 30 million passwords, so I was aiming to randomly select around a hundred of these each time the program starts.

For what purpose? I hope these passwords are not stored 'plaintext' - but are encrypted!

[Ryan1590]

Hi

Its for a program I have been building, currently its a brute force password cracker that will attempt to crack any password that the user enters, however, I want to expand this so the cracker will attempt to find passwords within that 100 sample stored in the array which was taken from the rock you text file, based on parameters the user will change.

[2kaud]

Hi

Its for a program I have been building, currently its a brute force password cracker that will attempt to crack any password that the user enters, however, I want to expand this so the cracker will attempt to find passwords within that 100 sample stored in the array which was taken from the rock you text file, based on parameters the user will change.

As that is not within the ethics of this forum, I'm closing this thread.