I'm in the same boat.
If I were to be categorized as racist it's be a very old world racism, where skin color holds no importance but instead spirit and will mean everything. In such a case the spirit of a culture is what is superior, not the skin color that is predominant in a ruling culture.
It is about culture, yes.
I'd argue a lot of people within "superior" cultures actually lack the spirit that made them great and instead we've relied on an intellectual elite to drive us forward. Discriminating based on that sounds less like old world racism and more like snobbism.
I've always been a snob, in a sense. I don't disrespect people outside of what i consider the 'elite', but i expect them to be less impressive and they usually are. It is not their fault, but i always have a problem choosing between disrespecting them and being patronizing. Its like it has to be one, but cant be both, because I can see why they are that way, but if i tell them, it will be rude, so i have to go around it and its like i am talking to a child(and i may even like them, i just dont know what to do lol)... With my 'elite' friends, i am always 'disrespectful'(more like bluntly rude), because they can take it and not care, but instead they give me the same treatment.
Either way tho, it doesn't change the result, where a culture is subjectively superior then another. Whatever the reason why.