Multithread writing data

[mce]

Hi,

I have many threads which update simple "real time" data, format as in a database. for example, i have 10 threads, updating data at every 20ms to 30ms. The data is just a string or double value of a particular field.

ID Value
1 10.5

I would like to know whether it is more efficient to put the data in a database or just have a multi-threaded dictionary storing all the data. I don need to have a persistence storage. But i believe if using a SQL database could be more efficient, as database support parallel update by multiple threads to a single table??

[rliq]

9 out of 10 times I believe the answer to this would be Dictionary<,>. (Memory access faster than Disk access)

A database may be faster, if there are millions of items, the database is on a separate machine, the client machine has very little memory (so the Dictionary<,> is swapping to disk anyway) and your network is very fast.

SQL definitely supports multiple 'parallel' updates to the same table.

Having said all of that, the proof is in the pudding, so it's probably best to test it in your environment.

[mce]

Hi rliq,

Thanks for your response.

What if i run the database in memory, or i use a datatable in memory? I have concern about Dictionary, i.e if i have 10 threads, and using lock with the dictionary, will that be too much, since 10 threads is updating the dictionary every 20-30ms, on different items inside the dictionary, and possibly another thread enumerating the dictionary every 10ms to get the latest data and send it over the network.

[rliq]

Can you explain the process a little more... You have 10 threads 'updating'. Is this adding, removing, updating? Is 1 removing and 9 adding or 9 removing and 1 adding etc...?

[mce]

Hi rliq,

Currently it will be 10 threads updating (no removing or deleting), one thread enumerating the dictionary every 10ms to get the latest data updates.

[rliq]

I just wrote this as an experiment...

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Threading;

namespace CodeGuruTest
{
    class Program
    {
        const int NUM_THREADS = 10;

        static Random _random = new Random();

        static int _key = 0;
        static Dictionary<int, string> _data = new Dictionary<int, string>();

        static BackgroundWorker[] _adders = new BackgroundWorker[NUM_THREADS];

        static BackgroundWorker _enumerator;

        static void Main(string[] args)
        {
            for (int i = 0; i < NUM_THREADS; i++)
            {
                _adders[i] = new BackgroundWorker();
                _adders[i].DoWork += new DoWorkEventHandler(AdderThread);
                _adders[i].RunWorkerAsync();
            }

            _enumerator = new BackgroundWorker();
            _enumerator.WorkerReportsProgress = true;
            _enumerator.DoWork += new DoWorkEventHandler(EnumeratorThread);
            _enumerator.ProgressChanged += new ProgressChangedEventHandler(EnumeratorThreadProgressChanged);
            _enumerator.RunWorkerAsync();

            Thread.Sleep(1000 * 60);
        }

        static void AdderThread(object sender, DoWorkEventArgs e)
        {
            while (true)
            {
                Thread.Sleep(_random.Next(10000));
                _data.Add(_key++, "Hello");
            }
        }

        static void EnumeratorThread(object sender, DoWorkEventArgs e)
        {
            BackgroundWorker worker = sender as BackgroundWorker;

            while (true)
            {
                Thread.Sleep(_random.Next(5000));
                worker.ReportProgress(_data.Count);
            }
        }

        static void EnumeratorThreadProgressChanged(object sender, ProgressChangedEventArgs e)
        {
            Console.WriteLine(e.ProgressPercentage);
        }
    }
}

Does this simulate what you are doing with the dictionary?

[mce]

Hi rliq,

Thanks for that.
I am too using the asynchronous threads to do the updates and enumerating.
But on the data update part and enumerating part,

a closer simulation is to update the value of the data in the dictionary(assuming the dictionary, contains roughly 5000 set of data is already created, with an ID as unique key.) The enumerator will loop through every set of this data and repackage them.

So i believe the Dictionary must be multithreaded (your dictionary shown above is not multithreaded), and i am thinking of using csharp object lock to achieve this, but i have concerned over too frequent locking by threads.

[rliq]

That is correct, I just read the Dictionary<> documentation on MSDN. Guess I was just lucky that I did not create two duplicate keys...

Ultimately though, only testing will determine which way is best....

[rliq]

I guess you may already have seen the link, showing an implementation of the "Thread Safe" Dictionary class.