Borland C++-Builder: How to speed up adding of entries in TList-View

[AlionSolutions]

Hi Gurus,
this time it is a VCL-component of Borland that drives me crazy:

I have a TListView for displaying the contents of a directory. When I tested my application and used directories with a small or normal amount of entries everything looked fine. then I tried a directory with over 20.000 entries inside. it worked awful. the application took a loooong time to add all entries to the TListView... a very looong time (on a 1,24Ghz Machine).
I optimized the adding-loop as much as possible (three lines of code are not much to optimize )
but nothing went faster. I set AllocBy to the count of all entries that are added. I set Count to the same value... no effect.
Ok, next I want to try the adding loop in a single thread (must do this anyway for having a progress-bar running), but I don't think that this improves the performance of the adding-task.
I used a profiler and located the bottle-neck in the windows listbox.

Has anybody an idea how to speed up the adding of entries to a TListBox ? The windows-explorer is pretty fast in displaying directories with much contents. so it must be possible in any way to speed this task up.

Thanx in advance

Juergen

[serg_dz]

May be it's not a good answer, but You can write your own ListBox component using Win API. I think this component will be the fastest one.

__________________
Sincerely,
Serg Dzysyak
AlarIT programmer
http://www.AlarIT.com

[Paul McKenzie]

Originally posted by AlionSolutions
I used a profiler and located the bottle-neck in the windows listbox.

Has anybody an idea how to speed up the adding of entries to a TListBox ? The windows-explorer is pretty fast in displaying directories with much contents. so it must be possible in any way to speed this task up.

Thanx in advance

Juergen

If you used Spy or some other utility, you will find that the Explorer Window is not a ListBox, but a SysTreeView32 and a SysListView32 custom control. Therefore assming that you can get a ListBox to go at the same speed as the Explorer window is not a valid assumption.

There is a message that you can send to a listbox to initialize the memory used to store items. The LB_INITSTORAGE message is used to do this. I don't know what TListBox does, what it encapsulates, or how to use it, but for a true Windows Listbox control, this message can be sent to the listbox window to reserve the memory for the items. I'm assuming that the slow performance is that for each item you're adding, a reallocation of memory is done. If the memory is reserved up front, the allocations don't take place, effectively speeding up the loading of items.

Regards,

Paul McKenzie

[AnthonyMai]

I guess instead of complaining about how slow it is to add 20,000 entries to your listbox, you need to look at the problem of your UI design?

How useful is a listbox that lists 20,000 entries at a time? It is as useless as a phonebook that prints 20,000 lines per page!!! Actually it is worth than that. How many lines of text can your listbox display at a time? 20 may be. So to search and find a specific entry, the user prpbably has to click the mouse on the up or down arrow some 1000 times, or he has to drag the side scroll bar using a super fine precision of no more than a few microm movement at a time. Who can do that?

Throw away your design. Then think about some way how you can present 20,000 entries to the user, in a more meaningful and accessible way. Maybe you can first categorize the entries into a dozen different groups, the user further selects a subgroup from each group, and finally in a sub group you only need to display no more than 100 entries at a time.

[AlionSolutions]

Dear friends,
thanx for your tips. first I want to tell, Anthony that my UI-design is great
As I told, I have to code an explorer-lookalike. can you explain me how to categorize 20.000 files in a directory ??? does win-explorer this ??? NO, for this reason i MUST do it...
Ok, but I found some smart solutions, and, dear Anthony I archived a speed as in win-explorer on a 233Mhz PII. What impossible ??
1. Set the number of listentries to the value of expected entries. For example, if I read a directory that has 20.000 entries, I set the Count / Capacity-Values to 20.000 before adding entries.
2. Call BeginUpdate before adding and EndUpdate after adding on the List (this speeded things up very much).
3 Put the adding-task in a single thread. This speeded up adding on factor 5 !!!

Ok, I did some things on the way adding the entries too by using meaningful memory alignment etc. but those actions did speed up the task just a little bit. Now on a 1,24 Ghz machine you cannot see anything of the adding of 20.000 entries, ok my slow harddisk makes it a little bit slow... remember win-explorer not some weird listview to watch db-entries. In this case, dear Anthony, I would have surely selected another design than just reading all entries into the listview. but sometimes you must even do this: Imagine you have to develop a db-application for a db with thousands of entries in a single table. don't tell me that this is bad db-design, because if my customer wants to have an application for his existing db you cannot go to your customer and tell him "hey your db-design is sh.., go ahead and change it, or I cannot build you app".

Greets

Juergen

[PaulWendt]

Hey just another suggestion I thought of: in windows apps, you
can often disable the listview, then add the items, and then
enable the listview. This often can help when you're dealing with
WIN32 listviews. If borland doesn't wrap the WIN32 listview
control, then this may or may not help you

--Paul

[AlionSolutions]

Hi Paul,
yeah that is what I meant with calling BeginUpdate() and EndUpdate.
It's a nice thing, because it had the most powerful effect of all my attempts to speed it up.

Anyway thanx again

Juergen

[PaulWendt]

Alright, Alion. In WIN32 and MFC, BeginUpdate and EndUpdate is
when the drawing is done. I suppose disabling the control just
makes it not respond to drawing messages so perhaps it's the
same result.

--Paul

[AlionSolutions]

ok paul,
disabling is also a possible way.

many ways to rome

Juergen

[AnthonyMai]

Ok, but I found some smart solutions, and, dear Anthony I archived a speed as in win-explorer on a 233Mhz PII. What impossible ??

Did I even say that it is impossible to speed up listview add? I didn't say that. There are always ways to speed things up.

But what's the point of having a listview that contains 20,000 entries at a time. It's useless from the usability point of view, regardless what is the backend database do. You simply shouldn't have a 20,000 listview. If you do the user can't scroll it.

It's NOT a programming issue. It is a human-machine interface issue. Human can only handle these much data at a time. You have to cope with that. If you are start to add 20,000 entries to your listbox, why not 2 million? Why can't the IRS show all social security number of every one on one huge computer screen, in one listview?

[AlionSolutions]

ok, from that point of view you are right. But, as I told, I have to develop an WinExplorer-lookalike. The Explorer displays all entries of a directory, and I don't want to add another proggie, that leads users wrong because of establishing a new style of gui under windows. I don't like the fat win-style, but I have to work in the environment that suis my monetary needs too. so it is not my decision to place thousands of entries in a list.