What is the best way to exit from a dll function when an error is encountered?

[OReubens]

basically:
if you make both yourself: do whatever you want that works. Anything else is "guidelines".

If you make the dll, and users/clients of this dll still need to make their programs:
do whatever you want that is as compatible as possible with all the compilers you want to support, then document how users are required to respond to your way of handling errors.

If you make an addon dll that needs to work in a preexisting program:
do whatever method the preexisting program prescribes.

...