Solving long link times

[coderdave]

At our company our link times have grown to twenty minutes for a debug build. As we are getting near the end of the project and we have less and less time this is not acceptable and we need to find a solution.

About our project layout:
- One solution
- links to 37 of our internal static libraries
- each static library has on average 20ish C++ files.
- native c++ code only

The problem is certainly linking with debug symbols. If we link without debug symbols then we can link in a few seconds but when we add debug symbols back to the linker option we get to 20 minutes.

If there is no solution to speed up the time I'd like to try and re-architecture our code base to avoid problematic code. For example, my guess is that certain templated classes that get used a lot through out are the culprit. My reasoning for this is that half of our code base does not use templates and linking to this is very fast, and the other half does use it and it links very slowly.

Does anyone know a way to determine what makes it link slowly? I need to find exactly which are the biggest classes to refactor. I need to be sure that this would help link times before I invest a week into the refactor since time is crucial at this point in the project.

[sunny_sz]

No any code here, we may not understand what exactly you want.

[GCDEF]

Have you turned on incremental linking?

[coderdave]

Yes we do have incremental linking on

[JVene]

You've not mentioned, but it may help a little to know the version of the compiler/linker you're using.

Have you checked with, say, task manager during the link to note if CPU is 100% bound and that you're not paging?

Building 32bit targets?

Native only?

Is code analysis turned on?

Have you tried deleting the project's ncb file lately (especially on VS2005)?

[coderdave]

- Code analysis off
- native only
- 32 bit targets
- not cpu bound and not paging
- i disabled ncb files
- visual studio C++ 2003 compiler/linker

[googler]

You can enable the /verbose option of the linker to see in detail what it's doing. That might help you a little bit.

Templates require the linker to do a lot of COMDAT-folding, a relatively expensive operation. You can turn off COMDAT-folding, but this could result in a huge code bloat.

I don't see how that relates to the debug symbol issue though. In fact, I don't see why debug symbols should slow it down that much unless they're causing the production of huge output files.

[MrViggy]

Are you linking across a network? In other words, is all your code on a network drive? I found that when all my code is on a network drive, my CPU is only at 70-80% usage. However, when all my code is on the local hard drive, the CPU is pegged at 100% (and compile times speed up).

Viggy

[JVene]

The "not CPU bound' reply seems key to me.

It suggests the machine is busy waiting on the disks.

If it's not paging, then the linker must be sifting through gobs and gobs of library debug info (assuming debug is the culprit based on your observations).

Is the compilation all local? Is it possible that you're drawing file info from a network disk?

Could there be some gain from a defrag?

Maybe even another drive you install to devote to the libraries/link tasks?

I doubt there's enough room to even think of a RAM drive for this, but that depends on the size of your libraries.

Can you give just a few more numbers? What's the sum of the sizes of the temp files created when you compile a debug build?

...do you really need a full debug build?

...is the debug information in a program database? What's the debug format....


I admit I'm stabbing a bit here - just trying to see if we hit upon something. It sounds like you've found a straw that breaks the camel's back.

[coderdave]

"Is the compilation all local? Is it possible that you're drawing file info from a network disk?"

All local.

"Could there be some gain from a defrag?"
No, this is a studio wide issue and I've defragged before.

"Maybe even another drive you install to devote to the libraries/link tasks?"
That may be a solution - I'd have to experiment but its not the solution I'm really looking for.

"I doubt there's enough room to even think of a RAM drive for this, but that depends on the size of your libraries."
Not sure what you mean but I've got a gig free of RAM when its linking.

"Can you give just a few more numbers? What's the sum of the sizes of the temp files created when you compile a debug build?"
- All obj files, pdb files, incremental files, and libs together are 316mb.
- pdb files for each library is around 1mb with a range of 500kb to 2.1mb
- libs range from 500kb to 11mb
- final pdb is 24mb

"...do you really need a full debug build?"
Unfortunately yes

"...is the debug information in a program database? What's the debug format...."
The debug information is stored in the .pdb format

With /verbose on, the linking goes slower per library. It goes very fast in the beginning but as you get futher towards the end each library slows to a crawl. It has no correlation with the order that the libraries link.

[mmesser]

This might be a few months too late to help. We also have a large solution - about 4 times the size of yours. We sometimes get mysterious problems like this.

I sometimes run out of some resource on my computer. I don't know what. To avoid this, I have to be careful not to have too many applications or tools running.

This can cause my computer to lock up or cause the development environment to have an internal error. If this happens in the middle of a compile, it can cause a variety of odd symptoms that persist.

Deleting the ncb file and doing a full rebuild sometimes helps.

Sometimes I need to delete the suo file. It tells which files are loaded, what breakpoints you have set, etc. Problems here can slow down editing and debugging.

I have found that linking slows down if I edit source files while building.