Handling connection attempts in asynchronous sockets

Hi everybody,
while working with my first Socket-relateg project I came across with the following problem.
I have two apps: client and server. The Server receives commands from a third-part app and pushes commands to selected client app. Client app should process received command and send back a reply. I want my server app to work asynchronously, so that after receiving a command (accepting new connection and adding it to a client list) it should immediately start waiting for another incomming connections.
Everything looks fine untill the number of incoming connections reaches about 20 per second (or rather the period of time between connection calls is short enough). In this case the third-paty app keeps getting the following error:
"Only one usage of each socket address (protocol/network address/port) is normally permitted <IP_Address>:<port>"

I wanted to use the Socket.Poll method but all of my attempts have failed. I don't really know how to make the server to wait untill listener socket is free and can accept incoming connections.

The following is part of my Server source code:

private bool StartListening()
{
try
{
serverSocket = new Socket(AddressFamily.InterNetwork,
SocketType.Stream,
ProtocolType.Tcp);

IPEndPoint ipEndPoint = new IPEndPoint(IPAddress.Any, this.portServer);

serverSocket.Bind(ipEndPoint);
serverSocket.Listen(4);
serverSocket.BeginAccept(new AsyncCallback(OnConnectRequest), serverSocket);
}
catch (Exception ex)
{
[...]
return false;
}
return true;
}

public void OnConnectRequest(IAsyncResult ar)
{
Socket listener = (Socket)ar.AsyncState;
Socket socketCommunication = null;
lock (listener)
{

bool bNewClientCreated = true;
try
{
socketCommunication = listener.EndAccept(ar);
// NewConnection - adds clientSocket to list of clients,
// sets up OnDataReceived callback which receives and processes
// received command
NewConnection(socketCommunication);
}
catch (SocketException sockExc)
{
bNewClientCreated = false;
[...]
}
catch (Exception exc)
{
bNewClientCreated = false;
[...]
}

if (bNewClientCreated)
{
listener.BeginAccept(new AsyncCallback(OnConnectRequest), listener);
}
else
{
// closes client socket and removes it from the list
CloseClientSocket(socketCommunication);
}
}
}

I will appreciate any help as I ran out of ideas.
Regards
Rob