This all started out because Silverlight has a Rect class which is ALMOST identical to the normal Rectangle class, however its IntersectsWith() method just seems to waste memory
I'm gonna say this is exactly where it started going wrong for you. What made you think that the silverlight Rect class 'wastes' memory when you use IntersectsWith? I assume you're referring to Rect.Intersect (Rect) by the way. It's a struct so there's going to be no actual memory allocations going on there...

Anyway, you were going to have to use custom classes/structs regardless of performance. If you want your library to be able to run on GDI, XNA, Silverlight and whatever else, then you have to make absolutely sure that your library does not use *any* type defined in those toolkits. This immediately rules out using any of the pre-existing Point/Shape classes. You should then write a translation layer for each toolkit which will take your Point/Polygon/Rect types and mangle them into the correct form for the current toolkit. Sure, it will involve duplicating some stuff, but don't worry about performance until you profile and see that's where the problem is. You can always add caching so that you don't have to do the translation *every* time.