are not part of the EXE, but rather only utilized by the EXE
Sure, this is the entire concept of dynamic linking. It will work as long as all of the library (DLL) files are available on the client computer. For .NET applications, this typically means they have the same .NET framework installed as was your compile target (plus any external libraries you utilized).

However, I doubt that copying a file into a location on a drive is going to be simpler than an install. I haven't actually made an installer before, but this one looks pretty good: http://nsis.sourceforge.net/Main_Page

The situation will be similar under linux.