High performance and UDP

I'm building UDP server and it's the first time I build something that should have high performance using UDP.

How does it work if I have one socket using BeginReceiveFrom from IPAddress.Any and other sockets using BeginReceiveFrom from specific remote end points. Will the packets go to the specific socket, or could the IPADress.Any BeginReceiveFrom get them instead?

Or should I just use one socket for all operations? Does that scale well? For instance, what happens if 500 packets should be send through the same socket to different remote end points at virtually the same time (using BeginSendTo)?

Or third =) Use one socket for all incoming packets and one socket per remote end point for send operations?