UnauthorizedAccessException when accessing Virtual COM port

[MontgomeryBurns]

Hi,
My application opens a virtual COM port to access an USB device. (I'm using CDC ACM Port Driver by Thesycon GmbH).
My app opens the port and closes it correctly, if I disconnect the USB cable from the device before calling SerialPort.Close(), then the UnauthorizedAccessException is thrown. Notice that it's not the Close() that throws the exception. It's a separate thread (which is not created from my code) that throws it.
This is what the IDE's output window shows:

'WindowsFormsApplication3.vshost.exe' (Managed): Loaded 'C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll'
'WindowsFormsApplication3.vshost.exe' (Managed): Loaded 'C:\WINDOWS\assembly\GAC_MSIL\Microsoft.VisualStudio.HostingProcess.Utilities\9.0.0.0__b03f5f7f11d50a3a\Microsoft.VisualStudio.HostingProcess.Utilities.dll'
'WindowsFormsApplication3.vshost.exe' (Managed): Loaded 'C:\WINDOWS\assembly\GAC_MSIL\System.Windows.Forms\2.0.0.0__b77a5c561934e089\System.Windows.Forms.dll'
'WindowsFormsApplication3.vshost.exe' (Managed): Loaded 'C:\WINDOWS\assembly\GAC_MSIL\System\2.0.0.0__b77a5c561934e089\System.dll'
'WindowsFormsApplication3.vshost.exe' (Managed): Loaded 'C:\WINDOWS\assembly\GAC_MSIL\System.Drawing\2.0.0.0__b03f5f7f11d50a3a\System.Drawing.dll'
'WindowsFormsApplication3.vshost.exe' (Managed): Loaded 'C:\WINDOWS\assembly\GAC_MSIL\Microsoft.VisualStudio.HostingProcess.Utilities.Sync\9.0.0.0__b03f5f7f11d50a3a\Microsoft.VisualStudio.HostingProcess.Utilities.Sync.dll'
'WindowsFormsApplication3.vshost.exe' (Managed): Loaded 'C:\WINDOWS\assembly\GAC_MSIL\Microsoft.VisualStudio.Debugger.Runtime\9.0.0.0__b03f5f7f11d50a3a\Microsoft.VisualStudio.Debugger.Runtime.dll'
'WindowsFormsApplication3.vshost.exe' (Managed): Loaded 'C:\Temp\WindowsFormsApplication3\WindowsFormsApplication3\bin\Debug\WindowsFormsApplication3.vshost.exe'
'WindowsFormsApplication3.vshost.exe' (Managed): Loaded 'C:\WINDOWS\assembly\GAC_MSIL\System.Core\3.5.0.0__b77a5c561934e089\System.Core.dll'
'WindowsFormsApplication3.vshost.exe' (Managed): Loaded 'C:\WINDOWS\assembly\GAC_MSIL\System.Xml.Linq\3.5.0.0__b77a5c561934e089\System.Xml.Linq.dll'
'WindowsFormsApplication3.vshost.exe' (Managed): Loaded 'C:\WINDOWS\assembly\GAC_MSIL\System.Data.DataSetExtensions\3.5.0.0__b77a5c561934e089\System.Data.DataSetExtensions.dll'
'WindowsFormsApplication3.vshost.exe' (Managed): Loaded 'C:\WINDOWS\assembly\GAC_32\System.Data\2.0.0.0__b77a5c561934e089\System.Data.dll'
'WindowsFormsApplication3.vshost.exe' (Managed): Loaded 'C:\WINDOWS\assembly\GAC_MSIL\System.Deployment\2.0.0.0__b03f5f7f11d50a3a\System.Deployment.dll'
'WindowsFormsApplication3.vshost.exe' (Managed): Loaded 'C:\WINDOWS\assembly\GAC_MSIL\System.Xml\2.0.0.0__b77a5c561934e089\System.Xml.dll'
The thread 0xfb4 has exited with code 0 (0x0).
The thread 0xfbc has exited with code 0 (0x0).
'WindowsFormsApplication3.vshost.exe' (Managed): Loaded 'C:\Temp\WindowsFormsApplication3\WindowsFormsApplication3\bin\Debug\WindowsFormsApplication3.exe', Symbols loaded.
>>>>> System.IO.Ports.SerialPort.Open() returned without any error
The thread 0xeec has exited with code 0 (0x0).
>>>>> System.IO.Ports.SerialPort.Close() returned without any error
The thread 0x814 has exited with code 0 (0x0).
The thread 0x888 has exited with code 0 (0x0).
A first chance exception of type 'System.UnauthorizedAccessException' occurred in System.dll
An unhandled exception of type 'System.UnauthorizedAccessException' occurred in System.dll

Additional information: Access to the port is denied.

The program '[0xF50] WindowsFormsApplication3.vshost.exe: Managed' has exited with code -532459699 (0xe0434f4d).

Is there a way to prevent this exception to be thrown?