I beleive lot of such knowledge comes from experience. One does this thing a lot and then sees patterns in these data that a normal eye wouldn't be able to see. Also, comparison of files could prove useful. You could have multiple sets of files ( encoded and decoded ) and then try to see common patterns etc. Another possibility I can think of is, some of these guys would've worked on those formats at one time themselves.. ( read, programmers in anti-virus being ex-virus writers themselves )