Well, comments are always nice :P

Code:
PORTB ^= ~(1 << pos);  //PORTB XOR  the 2's compliment of 2 ^ pos
PORTB = ~PORTB;  //2's compliment of PORTB


If you want something more readable
Code:
#define TIMES_TWO_TO_THE_POWER_OF\
   <<
#define BINARY_NEGATION(x)\
   ~(x)
#define SELF_XOR(var, x)\
   var ~= x

SELF_XOR(PORTB, BINARY_NEGATION(1 TIMES_TWO_TO_THE_POWER_OF pos);
PORTB = BINARY_NEGATION(PORTB);
I personally find the latter harder to understand, but I've been using the binary operators for quiet some time