Processing huge datatable and saving to sybase(sql anywhere) database table.

Hi ,

I Am very much new to .net and as well as sybase (SQL Anywhere).
I have a .net application in which am supposed to read almost 30 huge text files onvert them to datatable and insert into sybase SQL Anywhere database table.There are a set of business rules which are needed to be applied on some of these text files while processing.
Right now am processing each and every row and validating against the business rules and inserting them which is taking very long time and effecting the performance.

What would be the better way of dealing with this scenario. As performance has become a huge constraint.


It will be great if anyone of you suggest me a better solution so that its improves the performance.

Thanks in advance.
Satish

Satish,

There might be no particular shortcut to what you're doing but what I can tell you as the author of a couple of rules engines is that you should place your engine in the position to process the records and modify them as a memory-based component. In other words, try to avoid applying your business rules as stored-procedures or other code that must be repeatedly loaded from slower storage into memory.

If your business rules must be applied record-by-record as it appears here there may be no shortcut. The system I maintain has to process thousands of discrete time "punches" when constructing employee time cards and the process can be extremely time consuming. That's just the nature of the beast.

Overall I suggest that you get your process working correctly first, then optimize it at each level.

I'm afraid that there's not much more of a detailed answer I could give you at this level. Your process may be fairly complex and it takes time to work these details out. Suggest, however, that you do as much design up front as you can before you code it up because a refit after implementation can take years.

-max