ContextSwitchDeadLock please help

[techtalk7]

I built a TickImporter class to import very large csv files of price data into a trading software for charting.

The function that does all the actual importing is TickImporter.ImportToDataStore().
When calling ImportToDataStore() from a basic Console Application it performs very well and does what it should.

However, I realized it would be very time consuming to import the data by hand because the data comes in many files that need to be organized, merged into one (TickImporter overwrites so importing them separately is not an option), then actually import the file which will take a few hours inside ImportToDataStore() looping through the millions of lines, and then do this again for each of over 20 symbols. As you can imagine this will takes days with a program and possibly weeks manually!

So I decided to build a WPF app to make this easier for other members on the forum that may want to import these same files (which are publicly available). All they have to do is drag all the raw downloaded files into the app and it does the dirty work, organizing, merging, and importing.

However, I noticed it performs significantly slower than its Console Application counterpart (only 5kb/sec as compared to something like 200 kb/sec on the Console App!)

Not only that but after ~350kb I get ContextSwitchDeadLock:

The CLR has been unable to transition from COM context 0x9670e8 to COM context 0x967258 for 60 seconds. The thread that owns the destination context/apartment is most likely either doing a non pumping wait or processing a very long running operation without pumping Windows messages. This situation generally has a negative performance impact and may even lead to the application becoming non responsive or memory usage accumulating continually over time. To avoid this problem, all single threaded apartment (STA) threads should use pumping wait primitives (such as CoWaitForMultipleHandles) and routinely pump messages during long running operations.


Like I said TickImporter.ImportDataToStore() is the main computation intensive area, I get the deadlock while program is inside it.

Im not knowledgeable in threading yet so if you will explain how to put ImportDataToStore() on a different thread please provide code.

And what is with this performance??? Its extremely slow even at the very beginning looks something like 7kb/sec and once it gets over 250kb it drops down to like 3kb/sec and crashes 350ish,. Theres nothing wrong with ImportDataToStore(), its lightning fast in a console app it gets to 350kb before I can blink..


Thank you

[techtalk7]

So I opened my .NET tome to try to find the easiest what to execute it on a separate thread and decided to use a delegate's asynchronous invocation method BeginInvoke().

Because ImportToDataStore() returns void I just fired BeginInvoke() for each large file.

I no longer get ContextSwitchDeadLock but now I just have 20 files loading at 5kb/sec... why is it so dam slow???

[DataMiser]

Hard to say why it is slow when you have not shown any code.

[techtalk7]

Hard to say why it is slow when you have not shown any code.

sorry about that

 
public partial class MainWindow : Window
    {
        public List<String> fileCollection = new List<String>();
        public delegate void ImportHelper(string s, Symbol sym);


        public MainWindow()
        {
            InitializeComponent();
        }

        private void importButton_Click(object sender, RoutedEventArgs e)
        {
            List<string> Symbols = new List<string>();
            List<string> tickFiles = new List<string>();
            string filesDirectory = string.Empty;
            
            foreach (string sourcePath in fileCollection)  //Group files into symbol folders
            {
                statusText.Text = "Organizing files";

                string symbol = System.IO.Path.GetFileNameWithoutExtension(sourcePath).Substring(0, 6); //Parse symbol from file path: "AUDJPY"
                if(!Symbols.Contains(symbol))
                   Symbols.Add(symbol);
                string fileName = System.IO.Path.GetFileName(sourcePath);           //Get file name "AUDJPY-2009-12.csv"
                filesDirectory = System.IO.Path.GetDirectoryName(sourcePath);       //Get directory where files were dragged from

                if (!Directory.Exists(string.Format(@"{0}\{1}", filesDirectory , symbol)))      //Create directory filesDirectory\AUDJPY if doesn't exist
                   Directory.CreateDirectory(string.Format(@"{0}\{1}", filesDirectory, symbol));

                string targetPath = string.Format(@"{0}\{1}\{2}", filesDirectory, symbol, fileName);  
                File.Copy(sourcePath, targetPath, false);                                      //Copy all files to their respective symbol folders
            }

            foreach (string symbol in Symbols)  //Merge files using command line
            {
                statusText.Text = string.Format("Merging {0} files", symbol);

                string command = string.Format("copy *.csv {0}.csv", symbol);
                string workingDirectory = string.Format(@"{0}\{1}", filesDirectory, symbol);
                ExecuteCommand(command, workingDirectory);
                
                string tickFile = string.Format(@"{0}\{1}.csv", workingDirectory, symbol);  //Save merged file path for each symbol
                if(!tickFiles.Contains(tickFile))
                    tickFiles.Add(tickFile);
            }

            foreach(string inputFile in tickFiles) //Import into RightEdge
            {
                string symbol = System.IO.Path.GetFileNameWithoutExtension(inputFile).Insert(3, "/");  //Change AUDJPY to AUD/JPY
                Symbol inputSymbol = new Symbol(symbol);
                inputSymbol.AssetClass = AssetClass.*****;
                statusText.Text = string.Format("Importing {0} into RightEdge", inputSymbol.ToString());

                TickImporter importer = new TickImporter();
                ImportHelper helper = new ImportHelper(importer.ImportToDataStore);
                
                helper.BeginInvoke(inputFile, inputSymbol, null, null);
            }

            statusText.Text = "Done!";
            
        } //End importButton_Click

        static void ExecuteCommand(string command, string workingPath)
        {
            int ExitCode;
            ProcessStartInfo ProcessInfo;
            Process process;

            ProcessInfo = new ProcessStartInfo("cmd.exe", "/c " + command);
            ProcessInfo.CreateNoWindow = true;
            ProcessInfo.UseShellExecute = false;
            // *** Redirect the output ***
            ProcessInfo.RedirectStandardError = true;
            ProcessInfo.RedirectStandardOutput = true;
            ProcessInfo.WorkingDirectory = workingPath;

            process = Process.Start(ProcessInfo);
            process.WaitForExit();

            // *** Read the streams ***
            string output = process.StandardOutput.ReadToEnd();
            string error = process.StandardError.ReadToEnd();

            ExitCode = process.ExitCode;

            Console.WriteLine("output>>" + (String.IsNullOrEmpty(output) ? "(none)" : output));
            Console.WriteLine("error>>" + (String.IsNullOrEmpty(error) ? "(none)" : error));
            Console.WriteLine("ExitCode: " + ExitCode.ToString(), "ExecuteCommand");
            process.Close();
        }
      
        private void listBox_Drop(object sender, DragEventArgs e)
        {
            if (e.Data is DataObject && ((DataObject)e.Data).ContainsFileDropList())
            {
                BitmapImage bitmap = new BitmapImage();
                bitmap.BeginInit();
                bitmap.UriSource = new Uri(@"\Images\excel_file.gif", UriKind.Relative);
                bitmap.EndInit();
                

                int i = 0;
                foreach (string filePath in ((DataObject)e.Data).GetFileDropList())
                {
                    Image image = new Image();
                    image.Margin = new Thickness(0, 0, 20, 10);
                    image.Source = bitmap;

                    App currentApp = (App)Application.Current;
                    fileCollection.Add(filePath);
                    listBox.Items.Add(new StackPanel());

                    string fileName = System.IO.Path.GetFileNameWithoutExtension(filePath);
                    TextBlock fileBlock = new TextBlock();
                    fileBlock.Text = fileName;
                    fileBlock.FontFamily = new FontFamily("Comic Sans");
                    fileBlock.FontSize = 18;
                    fileBlock.FontWeight = FontWeights.Bold;
                    fileBlock.Foreground = new SolidColorBrush(Colors.DarkGreen);

                    StackPanel stackPanel = (StackPanel)listBox.Items[i++];
                    stackPanel.Orientation = Orientation.Horizontal;
                    stackPanel.Children.Add(image);
                    stackPanel.Children.Add(fileBlock);
                }
            }
            
            
        }

     
        
    }

[darwen]

You've not shown any code which is the slow part - you mentioned TickImporter.ImportDataToStore can you post that ?

If this is loading 20 files from disc at the same time then this'll be causing the disc to thrash about which is probably why you're getting the performance hit.

Try with just 2 at a time.

Darwen.

[techtalk7]

You've not shown any code which is the slow part - you mentioned TickImporter.ImportDataToStore can you post that ?

If this is loading 20 files from disc at the same time then this'll be causing the disc to thrash about which is probably why you're getting the performance hit.

Try with just 2 at a time.

Darwen.

Ok I got this resolved, sorry for the trouble. For debugging purposes I set MAX_BATCH_SIZE = 100; which says to write 100 lines to memory then save it to the file. I had forgotten to switch this back to 10,000 after i was done debugging so the program paused every 100 lines to save to the data store, causing the 5kb speed.

I still got ContextSwitchDeadLock at like 50,000 kb into the process even when I called ImportDataStore() asynchronously but since I now know thats not what was affecting the performance I simply turned it off in Debug > Exceptions > Managed Debugging Assistants

Thank you for indirectly helping me to find the problem