You make sapiosexuality seem less about being attracted to their minds and more about being attracted to the concept of intelligence itself.
Also from how I've heard you express this fetish, that furthers that it's less about the brain and more about the idea of 'smarts'.
By that logic you could say that about men who express liking boobs or women with them, as being attracted to the boobs and not the woman. By that logic, a man sniffing a woman's bra is him being more into the idea of "boobs"
People just like to find any excuse they can to tear down something they don't understand
I understand that not everyone, in fact most, who identity as sapiosexual, do not exclusively feel attracted to intelligence but rather it is only a factor they have.
On the other hand, I myself soley feel this, and without it there isn't really anything that could describe me and I can even be mistaken as asexual.
Which is why for me, this is an important description of my sexuality, as it is the only aspect of my sexuality I know of