The ones that are listening to the port will receive the broadcast message. That's the definition of it being broadcast.

To solve your problem, you have to have the ones listening to the port, reply back to the PC that sent the broadcast. That PC can get the IP address of the ones that replied from the reply packet. The "From" address is in the packet header.