I do see how it can be damaging for women's sports, and from that perspective, I sympathize. I understand you don't think men are superior as a whole, but only in terms of many sports.
However, I think the problem is more that the community around sports activities has been built the way it is, rather than something else. If trans women had always been allowed in women's sports and it was already the "status quo" then I think it would not be a problem.
It's similar to how someone might make the argument that black and white people should have their own categories, as black/white people have different genomes and, let's be fair, many sports are dominated by black people. It's of course a rather racist argument to make, but there are established, biological differences between white, black, and asian men, such as bone and muscle density. That is, the average bone density is shifted towards higher values for black men vs white men, just as it is shifted towards higher values for men vs women and chimpanzees vs humans.
Thus, by allowing black people to compete in "white sports" we destroyed the community around white sports. Yet, now that it is the status quo, nobody questions it, despite the same biological arguments that you're making being applicable here. In fact, the exact same arguments that you're using have been used by those boycotting black people entering white sports in the past.
At any rate, I'm merely playing the devil's advocate here, I think any argument about separating sports for black and white people is discriminatory.
However, I want to be logically consistent. Otherwise, I am forced to admit that I am a hypocrite. And I don't want to be one. I understand that there must be a flaw somewhere in my logic, as I am ultimately forced to admit that chimpanzees should be admitted to MMA, but I do not see it. Thus, until I find the flaw, I will believe that chimpanzees should be allowed to participate in human sports.