C++ load data

[dtustudy68]

Dear All,

I need to load data (24 X 28 X 40) from 24 X 28 files into a datastructure so that my program can use
them to do operations.

Each file has 40 data items, which are floating numbers.

I hope that the data structure can be indexed by item name not only by integer.

For example, for 24 states in USA, each state's power consumption is POWC in year Y.

I hope that the data structure indexing is like this

datastructure[newYork][year][powerconsumption]

My question is :

(1) how to load the data from 24 x 28 files by C++?

It is boring to type file's name one by one in program.

Is there an alternative to load the data by indexing the file names ?

for example,

for (filename = name1; filename < filenum ; filename++)
{
load data from filename;
store loaded data in the datastructure.
}

(2) how to index the item of the datastructure by C++ ?


Any help is appreciated.

Jack

July 11 2010

[D_Drmmr]

I hope that the data structure can be indexed by item name not only by integer.

For example, for 24 states in USA, each state's power consumption is POWC in year Y.

I hope that the data structure indexing is like this

datastructure[newYork][year][powerconsumption]

You can use enums to bind an identifier to an integer value. For the year that won't work, however, unless you prefix the year with a letter. Instead, you could just subtract the first year (stored in some variable).
Alternatively, you can use maps for this, but that will make your program a lot slower.

(1) how to load the data from 24 x 28 files by C++?

It is boring to type file's name one by one in program.

Is there an alternative to load the data by indexing the file names ?

You can use a stringstream to generate the file name.

for (int i = 1; i <= filenum; ++i)
{
    std::ostringstream oss;
    oss << "file_" << i;
    std::ifstream ifs(oss.str());
    // read from ifs
}

(2) how to index the item of the datastructure by C++ ?

I don't understand your question.

[dtustudy68]

thanks

I am working on a multi-computer Open MPI cluster system.

If I put some data files in /home/mypath/folder, is it possible that all non-head nodes can access the files in the folder ?

I need to load some data to some nodes, if all nodes can access the data, I do not need to load them to each node one by one.

If multiple nodes access the same file to get data, is there conflict ?

For example,

fopen(myFile) by node 1, at the same time fopen(myFile) by node 2.

Is it allowed to do that on MPI cluster without conflict ?

Any help is appreciated.

Jinxu Ding

July 12 2010

[Philip Nicoletti]

How are the calculations distributed among the nodes ?
Normally, each node would work on a sub-section of the
data, but the amount of data is so small that I don't know
how much improvement you would get using MPI unless
there are a lot of calculations that need to be done.

non-head nodes can access the files if they can see the folder
(for example, it is NFS mounted). I imagine the conflicts would
be the same as running separate codes on a single processor.

IO on MPI systems is often done in one of the following ways:

1) the head node does all the IO and sends/receives the data
from the appropriate nodes.

2) Each node does IO on its local system.

3) a combination of above where you have N reader/writer nodes
(where N is considerably less than the total number of nodes).

4) MPI IO

[dtustudy68]

thanks