advice on build speed

[superbonzo]

I'm aware of the programming techniques used to minimize building time; here, I'm interested in the hardware/software tools used to optimize the build process, such as dedicated build servers or specific compilers (btw, I'm thinking at debug builds only...) and similar.

googling around I noticed there's a plethora of possibilities out there, so I decided to ask here what are the most used solutions in this field.

just to make things precise, I'm speaking of relatively small compilation times (1-10 minutes) happening with template code. Of course, I can modify the code and/or my coding habits in order to reduce them but I'd like to know if there are other solutions capable of shrinking them ~1 order of magnitude.

Thank you

[ninja9578]

First thing to check is how many threads is your compiler running? For some dumb reason, MinGW will only compile using a single thread, if you're using that, try another compiler. For daily builds, you could turn down the optimization level, some optimizations take a very long time to perform.

[JVene]

Without any specifics on your targets, it's likely that any recommendation for a particular compiler toolset would come with the common complications of moving code among compilers, depending on it's portability (which is a lesson in and as of itself).

You may find various "make" tools that can drive your own compiler in multiple threads/processes. It could simply a matter of tossing off compilation of the various CPP files in parallel instead of in a series - and in the old days (the 80's) we'd arrange for this manually, sometimes by creating our own build tools to drive compilation (an unpopular concept now, but it was the genesis of several new build tools emerging from that time).

Some compilers support incremental builds, but when templates are involved that can negate the feature.

Another technique from the old days, virtually lost in present practice, is the separation of template function bodies from the template declarations in order to place that code in separate compilation units. At one time, this was a necessity due to RAM constraints, somewhere in the middle 90's when 64Mbytes of RAM was considered a lot.

The concept is helpful in the situation you seem to describe - a build/debugging/development phase involving template heavy code. This doesn't work with existing libraries of template code not constructed to work this way, but if your own template code is increasing build times when you're only making small changes to code, this technique could help.

It's called explicit template instantiation. Search here for a few posts (some of my own) on the concept. You can, basically, segregate the function body of templates into their own CPP files. This means the compiler need not generate code for all your compilation units. Thus, if you change non-template code, the compiler won't have to process the template functions at all. On the other hand, even if you do change template code, it only needs to generate template function code within the compilation units where they are explicitly instantiated, limiting the time the compiler takes to process the compilation.

This works because the linker finds what it needs through name mangling, just like any other member function - the function bodies of the appropriate templates are found at link time just like any other compilation, even though most people have been taught that such function code must be generated "in-line" during every compilation unit that calls those functions (this was never strictly true).

In the days of the Pentium-90, compilation times could be reduced from several hours to less than one hour, depending on the nature of a project and it's use of templates. The impact on a 10 to 20 minute compilation may not be dramatic, but then what you're probably hoping for is to shave some time off of each of those "debug/test/change/compile/repeat" cycles commonly part putting the pieces together.

[superbonzo]

thanks for response JVene,

>>Without any specifics on your targets, it's likely that any recommendation for a particular compiler toolset would come with the common complications of moving code among compilers, depending on it's portability (which is a lesson in and as of itself).

well, I write only standard conformant code ( or I hope so ) and if I need to use "dirty" C++ features I try isolating them as much as I can, so compiler-portability shouldn't be a problem ( OS/hardware portability is another story, I code only for general purpose hardware in ms windows; anyway, OS specific features are always properly incapsulated (again, I hope so) ).

>>It's called explicit template instantiation...

I know what explicit template instantiation is and how it's used.

I forgot mentioning that the template code I referred to are a set of metaprocedures (is this the right term? ... I'd like more something like "higher order programming" ) based on boost mpl and fusion libraries. Thus, explicit template instantiation is not a viable solution...

That sead, my question is not specific to what I'm doing right now. I'm just courious to know if there exist dedicated hardware and/or software capable of increasing debug build speed...

First thing to check is how many threads is your compiler running? For some dumb reason, MinGW will only compile using a single thread, if you're using that, try another compiler. For daily builds, you could turn down the optimization level, some optimizations take a very long time to perform.

thanks for your response too, but I use VC++2008 that should already use multiple threads during compilation

[JVene]

... I use VC++2008 that should already use multiple threads during compilation

Since this is in the non-visual C++ forum, I simply assumed you were in another compiler (I see, you're question is non-VC++ oriented, so it's not like you're posting in the wrong place).

It's true that VC++ spins off to multiple threads, but either the team builder or some other addon supports "farm" oriented compilation in VC environments.

[superbonzo]

something like distcc ? ( I've never used it, simply found googling )

the problem is: how easy are these distributed build systems to configure/mantain/use ? what are the realistic performance gains ? will I have to change my coding habits to use these tools or are they completely automated ?

I ask because it would cost me time and money, so I'd like to hear some real world experiences...

[Peter_APIIT]

I also encounter a project that a build time quite slow.

What are the approach can reduce build time ?

I wonder can ABI or bridge pattern reduce build time.

What is you all opinion ?

Thanks.

[JVene]

Google for distributed build, a number of 3rd party tools will come up.

To my knowledge, which is limited on this specific technique, the link is still performed in a single machine. Most distributed build solutions work by scheduling compilation units in client machines, then linking them in a final step on one machine.

Many of these tools strive to work without implication on how you develop or define a project. In theory there need not be any alteration.

It's difficult to offer much of an analysis of the impact of various design patterns on build time, but it seems obvious that the more classes created, the more functions created and the more templates used, the longer compilation times will be.

In my own experience, the greatest impacts on build time are either template usage or include layout (interdependent between includes). Templates are mixed, because in many cases impact is negligible because the template code is simply replacing what would have been "normal" code anyway (if used well, not indiscriminately).

When designing, the separation of implementation from interface code helps isolate what must be compiled when changes are made. This means that there is limited effect when a complete rebuild is called for, but in typical debug/change/compile/debug cycles, there are significant time savings from well separated units of logic. Most good design patterns help improve this effect, so if you have a design pattern that tends to create interdependencies (none come to mind at the moment), there could be minor implications on what must be recompiled when small changes are required. Template code usually ends up being rather imposing in this regard, which is one reason separating function bodies into separate compilation units can be of some help (and can be constructed so as to be an optional feature of the template code).

Also, it may be worth reviewing the condition of the build machine occasionally. Even minor improvements accumulate into real time savings, achieved by knowing if more RAM would be helpful, disk defrags should be run, moving the compiler to the inside location of a separate, dedicated drive, location of the paging file, using a 64 bit OS, even the size of the allocation blocks, can all contribute in one small way or another. Saving a minute per build, times the number of debug/build/test/debug cycles, can return 30 minutes or an hour of productive work in a day.

[superbonzo]

>> in typical debug/change/compile/debug cycles, there are significant time savings from well separated units of logic...

agreed. But, regarding my situation and as stated in my post, the most slowdowns happen in "debugging" meta template procedures where non trivial computations are done by the compiler as arbitrarely specified by the code (maybe this is what you mean by an "indiscriminated" template use ).

>> Also, it may be worth reviewing the condition of the build machine occasionally.

In fact, when testing build times on different computer at my disposal I noticed an high dependence on CPU power.



so, I'll take a look at distcc for the moment and see what happens .... bye!

[Amleto]

My work has used a free trial of incredibuild. It certainly worked wonders for compiling - it was lightning. I should have been though, as we had several quad-core machines available. However, linking time seemed a lot worse than without incredibuild.

[ninja9578]

The best advice I can give is to go through your code and make sure that all the header files that you include are necessary. About a week ago I did that (I'd been writing my software application for about 10 months so it was subject to code rot) I found lots of headers that I no longer needed. Less headers for me meant less compiler dependancies and less recompiling of code each time.