Large database organization

[sephiroth2m]

Hi gurus,
If I have a large database, like a dictionary with hundreds of thousand words, and I have 2 ways of organizing data, one is making my own data structure & algorithm, other is using a SQL database like Access. I just wonder which way is better for searching, could anyone give me a suggestion?

[Marc G]

Depends on a lot of things.
The most important question is: are you going to distribute your app? If so, using your own file format will be easier to let it work on all systems i guess.
Are you going to do some cross referencing between database tables?

If you are just storing 1 big list of words, then i would do the following:
Store the data in alphabetical order in a plain text file.
Finding a specific word can be done very quickly by using a binary search method because the file is sorted.
If you need to search a lot of words, it might be even faster to load the entire database into memory and do the binary search in memory. Because you are only storing words, memory usage is not that big: suppose you have 1 million words and the average word is 10 characters big, it only occupies 10 MB in memory which is not a lot.

[sephiroth2m]

Yes, I just wonder wich way is better (faster) for my app, a binary-search algorithm like yours or the database's algorithm? If the two are equal, I guess it's better to use the database's algorithm because there's no need to reinvent the wheel, right? (suppose all platforms my app runs on support that database interface)

[TheCPUWizard]

way is better (faster) for my app

These two items are completely un-related in most cases. Assume one method returns the result in 1mS, the other takes 10mS. The user will only be looking up individual words interactively. The first method is clearly 10x faster, but the user will never notice. If the second method is easier to write, more robust, more maintainable, then it becomes the better method by far.

The most important question is: are you going to distribute your app?

I agree, if your app is going to require the installation of MSDE or MDAC so that you can lookup words, that is a burden that should not be placed (probably). On the other hand if your application already requires a specific database for other purposes, then storing the dictionary in a file of that databases format would probably be the best choice.

[sephiroth2m]

Thanks friends.

[jolley]

it is true that it depend thething you want to treat .if you want a lot5 of delete or insert,then you will be advised to choose the latter,or you may choose the former ,but i advise you to choose the latter ,because the database is the king of coping with a large number of data.

[Joe Nellis]

Is your data actually a text dictionary?

I just wonder which way is better for searching, could anyone give me a suggestion?

A text dictionary can be represented as a directed acyclic word graph. They offer very fast search and good compression.

[jolley]

thanks lot and which dictionarydoes you suggest most,buddy?