How did you create the crypto.dll? cryptopp version 5.6.1.0 is available from Wei Dai's home page as a set of source files to be compiled to produce a .dll. Using their standard build produces cryptopp.dll and cryptopp.lib files. If you have (or can make) the .lib file why are you bothering using LoadLibrary etc when you could just link with the .lib file and include the .h file with your program? This would be my preferred way of doing it when these files are available.

However, as the source of the dll is available then if you build a debug version then you can use the debugger to debug into the dll when dllmain(..) is called to find the problem.

Code:

int code = (decryptAddress)((char *) pswd, (char *) out, (char*)key);

This is not the usual way that a C++ (or C) programmer calls a function. Why are there parentheses around the name of the function?
Because decryptAddress is variable containing a pointer to a function and not a function name. It could have been written

Code:
int code = (*decryptAddress)((char *) pswd, (char *) out, (char*)key);
but the '*' is now optional.