Need Advise: How to store procedural steps.

[e0117214]

I hope someone can give me a bit of help.
This is one of the most complicated projects I have ever been given and I need some advise on how to store procedural steps -- for each customer.

Our company receives over a period of a month about 2000 data files provided by the customers and the project is to validate each file before sending them on for further processing. Each of the files sent by the customer has a similar format, but they are not exactly the same.

There are two types of files, Fixed Length and Variable length files. The file extension tells you what files type you have.
However, in the fixed length, there are 6 different lengths. It is the size of each field that makes them different. Some of the validations are simple. Date has format MM/DD/YYYY and that can be a regular expression. Other validations are not so simple, like Field 4 for customer "ABC Company" is the total amount issued to their client. But for "XYZ Industries", it's Field 5 which is the Check Amount issued to the client. Each company can have a different field to check or a different value to be validated.

Here's my question: Each customer is going to require their own procedural steps or set of validation constants, how do I store this?

What I have come up with is this:
>The validation detail files would all be xml files for each of the customer's reports.
>An SQL Server table will be the master table telling the program which xml file to grab for that incomming file.
>Since they all the customers basically have the same format, each xml file would have all 50 fields laid out.
>Only the xml nodes with text values would be those that are validated, either by regular expressions or actual values.

So far not too bad? Now add the Variable length files to this....

Certain fields within the file are unlimited or nearly unlimited in size and are delimited with a "|" between each value. I may have to check each of the values in that field to see if it is a proper street address format or something like that. And that's only in the first record.

Ouch! my head hurts just thinking about it.
Any help that anyone can give will be deeply appreciated.
Frank

[kungpaoshizi]

Why not just skip all the extra stuff and make a base parsing program to feed a data file into arrays/lists, and then write smaller executables to read each file?
You could put a marker in the filename via the parser, then route each file to the appropriate .exe.
This would also make it easy to implement "multi-threading" without dealing with threading in one application.... /shrug

Otherwise if you really want something dynamic, just parse and then write to a txt different C# procedural steps needed for that specific data file. Then when the txt is ready to be compiled, call the command line compiler on the txt, and then run that unique procedurally correct exe against the data file (or parsed data file).

Hope this helps in some way

[TechGnome]

Here's how I've done it: a SQL table to act as a staging area. Custom import processes for the different files. Each of the importers dumps the data into the SQL Table. Once all of the data is imported, run a data scrub routine... sometimes a simple Stored Proc will work, sometimes it means writing code. Once the data in the table has been scrubed and validated... then another process takes the valid records and moves them to the final location(s) in the database.

-tg