Passing STL arguments through shared libaries

[sszd]

We have a small group on our program redesigning an API that is to be used by 3rd party developers to create a shared library that our system will use. The current API interface is extremely old, very cumbersome to use, and very C-like. The goal is to create a more C++-like interface, make it much easier to use, and add some additional functionality. The library delivered by the 3rd party developers adhering to the new API will be a shared object (.so file) used by our system on a Redhat Linux platform.

An individual on our team recently pointed out that the API should not use any STL type arguments, pointing out that “it would be in violation of the API (Application Binary I/F) spec that compilers must follow”. He further argued that the only way this would work would be if the library and executables are compiled and linked exactly the same so all template implementations are the same across the linked modules. Since we do not have control over what compiler versions will be used, or what compiler flags will be used by the 3rd party developers, it cannot be guaranteed that the library will work.

I’ve since done some light reading on the internet and have not come to a definitive conclusion as to whether these statements are accurate. It seems that there is much discrepancy (or confusion) as to whether or not the API is designed in a Windows environment (which uses DLL’s) or a Linux environment (which uses SO’s).

So, is it true that defining a shared object API that passes STL arguments to and from its interface is not a smart design and may cause problems? Or, is this just a problem for MS and how their DLL’s are laid out?

Any insight would be greatly appreciated. Thanks.

[Codeplug]

It is accurate - for all platforms.
You also should not pass CRT objects either - like FILE since the .so may be using a completely different CRT - you don't know.
For the same reasons, you don't want malloc() memory in one modules and free() it in another.

>> ... and very C-like
You don't even know what compiler was used to create the .so - so only a standard ABI should be used when crossing module boundaries.

gg

[sszd]

Thanks for the quick response. So, we're talking about basic POD types? If I understand you correctly, the only safe API design would be to use only POD types for all input and output to a shared object.

[2kaud]

There is, of course, nothing to stop you implementing your 'shared library' as header files in the same way that the c++ standard library is implemented and include them in the programs that need to use them. As the header files will be compiled along with your code then the API interface can be as desired.

[Codeplug]

Yeah, POD's.
You could even expose your own structures, as long as the user's dev environment builds them with the same padding and alignment as yours.

gg

[sszd]

Yeah, POD's.
You could even expose your own structures, as long as the user's dev environment builds them with the same padding and alignment as yours.

gg

Great! Thanks for your help.

[sszd]

There is, of course, nothing to stop you implementing your 'shared library' as header files in the same way that the c++ standard library is implemented and include them in the programs that need to use them. As the header files will be compiled along with your code then the API interface can be as desired.

Ok, so what you're saying is to define a standard set of header files providing the necessary API. Then, provide these headers to the 3rd party developers for usage in their implementation of the library. These developers will then provide a binary shared object (.so file). Finally, regardless of what platform, comp‎iler version, or compiler flags they used, we will then be able to link our code with the shared library using our own platform, compiler version, and compiler flags as long as we use the same header files where necessary.

If this works, that's great, however, what about runtime dynamic linking vice compile time static linking? Is there a way
to solve that problem?

[Codeplug]

>> As the header files will be compiled along with your code then the API interface can be as desired.
My interpretation is your API will still be C-like, but you could still provide classes that wrap that API, which could use std:: facilities etc. Clients would use their own C++ compiler and libraries to compile the code - but nothing fancy crosses the API boundary.

GDI+ does this on Windows for example. (C++ code wrapping a C api that is)

gg

[2kaud]

Ok, so what you're saying is to define a standard set of header files providing the necessary API. Then, provide these headers to the 3rd party developers for usage in their implementation of the library. These developers will then provide a binary shared object (.so file). Finally, regardless of what platform, comp‎iler version, or compiler flags they used, we will then be able to link our code with the shared library using our own platform, compiler version, and compiler flags as long as we use the same header files where necessary.

If this works, that's great, however, what about runtime dynamic linking vice compile time static linking? Is there a way
to solve that problem?

What I'm suggesting for consideration is that you could implement the whole of the required shared library in the header files - just like the c++ Standard Library. No creation/linking of .so files etc. All the code for the library is in the header files. That's the way the c++ standard library is implemented. If you do

#include <string>

then all the required code for the string class is included with the program code and gets compiled along with the program code. No .so (or .dll) file, no linking, no issues with using non-POD etc.

[sszd]

What I'm suggesting for consideration is that you could implement the whole of the required shared library in the header files - just like the c++ Standard Library. No creation/linking of .so files etc. All the code for the library is in the header files. That's the way the c++ standard library is implemented.

Thanks for the clarification. Unfortunately that would mean the library providers would have to expose possibly proprietary code. Some may be willing to do so, but others I know are not.

[sszd]

My interpretation is your API will still be C-like, but you could still provide classes that wrap that API, which could use std:: facilities etc. Clients would use their own C++ compiler and libraries to compile the code - but nothing fancy crosses the API boundary.

This is something we are certainly considering and will probably end up with.

[tiliavirga]

There's a module system in pipeline. I think it's planned for C++ 17,

http://www.open-std.org/jtc1/sc22/wg...2015/n4465.pdf

[superbonzo]

An individual on our team recently pointed out that the API should not use any STL type arguments, [...] as to whether these statements are accurate.

strictly speaking, IMO it's not accurate, because there's nothing wrong in <using> STL type arguments in the API, what's wrong is to <pass> them through module boundaries ( the former does not imply the latter ).

Indeed, there's also a third option along with what codeplug already suggested ( 1) use a C API, 2) use a C++ wrapper over a C API ) that is, you can write/use an existing interface solely devoted to the marshalling of the c++ types ( the server/client never directly "sees" the objects on the other side, that are continuously transferred by using some intermidiate, binary portable representation ).

albeit technically equivalent to codeplug suggestion ( you always have a C api under the hood ), the advantage is that you don't need to write and mantain a C version of <your> library ( whether user exposed or not ) as you only focus on the marshalling mechanism. Moreover, you can use existing quality implementations ( from boost asio to google protocol buffers etc... ) and you'll end up with a more general solution ( it would work across network boundaries as well with almost no effort ).

the disadvantages are the dipendency on such third party libraries, possible performance issues if overused, etc... so it's neither the holy grail nor the easier of solutions, of course.

There's a module system in pipeline

AFAIK modules are not designed for ABI portability ( the OP wants a DLL/so )

[tiliavirga]

AFAIK modules are not designed for ABI portability ( the OP wants a DLL/so )

That's right. But modules still is a portability (componentization for the purpose of sharing) option that most likely will be in the standard pretty soon.

And there are ongoing discussions and even a proposal for a true ABI for C++,

https://isocpp.org/blog/2014/05/n4028

[superbonzo]

that's interesting, but it still wouldn't solve the OP problem, that is ODR failures accross binary boundaries. For that we'd need something like standardized binary c++ modules ...