If you are approaching me to tell me I don't belong somewhere, it's because you know you have the power and don't like that I am transgressing it by coming into a space that you have deemed your own.
While these people undoubtedly view me as female and profess to be protecting women, the irony isn't lost on me that I now feel safer in the men's loos than the women's, when by their logic sharing loos with men is unsafe for women.
Individuals need to challenge their assumptions that the person who doesn't 'look like a woman' is the threat in these situations because often it's the other way around.