Your server side code looks rather odd.

Here are some snippets from my server code.

In my server start routine I have

Code:
SocketHandler = New TcpListener(hIPAddress, hPort)  ' Set the IP AND Port Number
SocketHandler.Start()
I have a timer running which checks to see if there are any pending connections every 100ms.

Code:
If SocketHandler.Pending() Then
            CurThreadStart = New ThreadStart(AddressOf ConnectionBegin)     ' Set info for the new thread
            CurThread = New Thread(CurThreadStart)                          ' Initialize the new thread variable
            CurThread.SetApartmentState(ApartmentState.STA)
            CurThread.Start()                                               ' Start the thread
            SyncLock CurThread                                              ' Lock the thread for update
                VAActiveThreads += 1                                        ' increment active threads
                cInfo.ActiveClients += 1
                CurThread.Name = "ClientCon" + VAActiveThreads.ToString        ' Name the thread
                Console.WriteLine(CurThread.Name & " Thread Started")
            End SyncLock
End If
Then in my ConnectionBegin which is launched by the code above in a new thread I have.

Code:
Dim ClientSocket As Socket = SocketHandler.AcceptSocket
Followed by a loop that will read from the port until there is a idle timeout condition.