access large text file with index

[dinesh999lama]

How can i read large text file containing numbers with its index without first reading the hole datas? Is their any other method where i can directly save the datas to memory and than directly access it with its index like an array? If no how can i atleast read the datas very fast, the size of datas can be 300 mb to 1 gb. so reading the hole file fist is not suitable for performance. For example to write the data:

 rosbag::Bag bagr("test.bag", rosbag::bagmode::Write);
    for(int i=0;i<xw.size();i++)  {
       xo.data = xw.at(i);  bagr.write("numbers", ros::Time::now(), xo); 
       xo.data = yw.at(i);  bagr.write("numbers", ros::Time::now(), xo);
       xo.data = zw.at(i);  bagr.write("numbers", ros::Time::now(), xo); 
    }

And to read the same:

    rosbag::Bag bagr("test.bag");
    rosbag::View viewr(bagr, rosbag::TopicQuery("numbers"));
    BOOST_FOREACH(rosbag::MessageInstance const m, viewr)
    {
               std_msgs::Float32::ConstPtr i = m.instantiate<std_msgs::Float32>();
               iv = iv+1; 
               if(iv%3==0) {
                  x1.push_back(i->data);
               }
               if(iv%3==1) {
                 y1.push_back(i->data); 
               }
               if(iv%3==2) {
                 z1.push_back(i->data); 
               }
    }
    bagr.close();

[VictorN]

If you do not need the whole amount of file data then you could read only a part of it.
Use fseek (or its boost equivalent) to repositions the file pointer to the place/index you need to read and then read a part of data beginning from this position.

[2kaud]

There are basically 2 ways that data is stored in a file. One is that data is read/written in blocks with the size of each block fixed. The other is not. In this case there is usually (not not always) some sort of delimiter(s) used between the data (ie for a text file a LF etc). If non-block layout files are used then it is not possible to directly index its contents without first scanning the file to determine the position of the relevant terminators and storing these in a container (eg a vector) so that particular data can then be accessed directly. This assumes that once the index container has been created then no changes are made to the file which affects its layout.

If a block layout file is used (each record is the same size), then it is easily possible to seek to the position in the file required and read the appropriate block (eg using fseek() mentioned by Victor in post #2 with fread() or c++ .seekg() and .read()).

From the code provided, it looks like the data being written consists of 3 numbers - so this can be read/written as a fixed sized block file.

Consider

#include <iostream>
#include <fstream>
#include <string>
using namespace std;

struct Block {
	int x = 0;
	int y = 0;
	int z = 0;

	Block() {}

	Block(int x1, int y1, int z1) : x(x1), y(y1), z(z1) {}
};

ostream& operator<<(ostream& os, const Block& bk)
{
	return os << bk.x << " " << bk.y << " " << bk.z;
}

class myRec {
public:
	myRec(const string& fm)
	{
		fs.open(fm, fstream::in | fstream::out | fstream::binary);
		if (!fs.is_open()) {
			fs.open(fm, ofstream::out | ofstream::binary);
			if (fs.is_open()) {
				fs.close();
				fs.open(fm, fstream::in | fstream::out | fstream::binary);
			}
		}
	}

	bool is_good() const
	{
		return fs.good();
	}

	bool setData(int rec, const Block& bk)
	{
		fs.seekp(rec * sizeof(Block));
		return !!fs.write((char*)&bk, sizeof(Block));
	}

	bool getData(int rec, Block& bk)
	{
		fs.seekg(rec * sizeof(Block));
		return !!fs.read((char*)&bk, sizeof(Block));
	}

private:
	fstream fs;
};

int main()
{
	myRec f("block1.bin");

	if (!f.is_good()) {
		cout << "Cannot open file" << endl;
		return 1;
	}

	f.setData(0, Block(1, 2, 3));
	f.setData(1, Block(3, 4, 5));
	f.setData(2, Block(6, 7, 8));

	Block bk;

	f.getData(2, bk);
	cout << bk << endl;
}

which enables the file to be treated somewhat like an array for read/write access.

[dinesh999lama]

thank u providing me also the code, but it seems like it works with c++11 and since ros uses c++03 i think i have to modify it little bit which i will do next. So thank u, i'm really happy that i'm getting all important solutions in this forum.

[2kaud]

but it seems like it works with c++11 and since ros uses c++03 i think i have to modify it little bit

What compiler is being used?

The only change for the example should be in the Block struct. The rest of the code should be fine for c++03. Try

struct Block {
	int x;
	int y;
	int z;

	Block() : x(0), y(0), z(0) {}

	Block(int x1, int y1, int z1) : x(x1), y(y1), z(z1) {}
};

[dinesh999lama]

ros is using cxx comiler.

[2kaud]

ros is using cxx comiler.

You should be able to use the option -std=c++11 to enable c++11 support in the compiler.