Fast way to convert string array to double array...

[ryu]

Hi GURU,

What is the fastest way to convert string array to double array? Currently, I am doing this:

...
...
...
double[] arrDouble = new double[arrString.Length];
for(int i=0; i<arrString.Length; i++)
{
   arrDouble[i] = double.Parse(arrString[i]);
}
...
...
...

This is very slow for big array (about 3 millions).

Any idea?

Thanks GURU for any help.

Cheers

[torrud]

Maybe the Convert class works a little bit faster.

...
...
...
double[] arrDouble = new double[arrString.Length];
for(int i=0; i<arrString.Length; i++)
{
   arrDouble[i] = Convert.ToDouble(arrString[i]);
}
...
...
...

[ryu]

Hi Torrud,

Thanks for replying.

I just tested the code, it doesn't give me any improvement

Here is the speed comparison to convert object.
http://blogs.msdn.com/bclteam/archiv...11/371436.aspx

Please note:

- The link I sent is the comparison to convert object to double.
- You will see double.Parse is very slow because it actually has to convert the object to string.
- It is not the case here because my array is already a string.

Any other idea? Thanks again for any help.

Cheers

[zips]

Here is the speed comparison to convert object.

The information there is obviously at least three years stale.

Doing a series of simple tests with v2005, my results were that Double.TryParse was a few percent faster than any of the others. Since it is also the safest, it seems like the obvious choice.

This is very slow for big array (about 3 millions).

Of course! That's a fact-of-life with floats.

[MadHatter]

uh, so why are you converting the value of the string to a double? .net strings are 16 bit values, so ushort would be a better choice there, so I'm just curious why you'd use such a large data type for that?

[ryu]

Hi Zips & MadHatter,

Thanks for your kind reply.

I have a large data (about 3GB per file). This file is just a simple Comma Delimited file that is generated by our legacy system (quite an old system). Now, I need to read all of this data and do some calculation on it.

The first 3 columns are strings and the rest of the columns are doubles.
The number of columns can be vary between 10000-1000000 depending the total number of attributes they have.

The problem I have is reading these files is too slow. In normal operation, I need to read about 5-10 of this files. And yes, I am using 64bit machine.

If there is nothing I can do, that is ok. I just thought you guys might have a better algorithm to use

Any suggestions?

Cheers

[MadHatter]

parsing numbers out of a CSV, and obtaining an array of doubles based on each character's ASCII or Unicode character value is two (oh so) totally different things...

if you're trying to parse in a CSV file, then you DO NOT iterate over the file contents char by char..., you split it on commas, and parse in the different segments...

what you need is something like this: http://www.codeproject.com/KB/database/CsvReader.aspx

[cjard]

The problem I have is reading these files is too slow

And how do you know that the bottleneck is in the double parsing?

Any suggestions?

Well, youre reading 3 gigs of data. Did you think it was gonna be fast? I have a process that reads and uploads an 80 meg customer file into an oracle database, and using bulked transfer methods, the fastest I can get it to complete is about 4 minutes - I think thats pretty good for 800,000 records considering all the validation, and overheads..

How "slow" is your process and how much "slower" is it than it needs to be?

I suggest you do some more tests before absolutely declaring that the double parsing is the slowest bit

[ryu]

Hi cjard,

I am not sure how does it done in Oracle, but I would imagine that there is a lot of overhead to insert these records in the database. But my problem isn't about inserting the data to the database. I basically wanted to know if there is anything I can do to improve loading speed.

Here is my original test result.

Reading without double.parse takes about 1 minute and 30 seconds.
Reading with double.parse takes about 8 minutes and 50 seconds.

I suspect it is really fast without double.parse is because C# does the optimisation and ignoring the for...loop because it doesn't been used anywhere.

So, in order to get a real value, what I did is to put Console.Write() to print out each of the value out. So, I modified the codes to:

...
...
...
double[] arrDouble = new double[arrString.Length];
for(int i=0; i<arrString.Length; i++)
{
   arrDouble[i] = double.Parse(arrString[i]);   //I will comment this line when testing without double.Parse
   Console.Write(arrString[i]);
}
...
...
...

But Unfortunately, running the 3GB is extremely slow. So, I modified my input file to smaller file (just about 150MBytes --> 60 fields + 645,000 lines.)

Here is the new result:

Reading without double.parse takes about 7 minutes and 55 seconds
Reading with double.parse takes about 9 minutes and 56 seconds

As you can see, even with the small file (150 Mbytes), it takes 2 minutes longer. You may argue that it is a small number and I totally agree with you. All I wanted to know if there is a faster way to parse the double so I can increase the loading speed. If not, that is ok because I can live with it.

Cheers

Btw, please don't be surprise with the number. Console.Write() is really slow. If you gonna ask me how I know Console.Write is slow, well, please google it or simply create a new Thread in CodeGuru and I am sure a lot of people will response to that.

[dglienna]

You need to SPLIT() the data based on the delimiter, into array or classes, whichever you prefer. Not character based.

Read a chunk, read up to the next record start, then parse the chunk.

[ryu]

Hi Dglienna,

I believe that is what I did.

...
...
...
string [] arrString = st.Split(new char[] {','});
double[] arrDouble = new double[arrString.Length];
for(int i=0; i<arrString.Length; i++)
{
   arrDouble[i] = double.Parse(arrString[i]);
}
...
...
...

Is that ok?

Cheers