Then it's all kinds of hokey to conclude that there's no sex-based differences in mathematical ability because boy's scores and girl's scores even out.

But in general, I think the question of whether there are sex-based differences gets too much attention. We know that sex-based differences in cognition, if they exist, are small compared to the variance within the sexes. We have evidence that cultural factors have a big effect. Those two facts give us all the information we need for most practical purposes.