You know, as a white, male, cis, heterosexual, I've been in women's restrooms while women were there at least twice in my life; and I've had a woman come into a men's restroom while I was in there at least once. In all three of those cases, I was presenting as male (because that's what I do). And in all three of those cases, it was essentially a non-issue.
#1: A friend tricked me into going into the wrong bathroom in a pizza parlor. The doors didn't have labels on them. I was peeing in a guy-style urinal when a woman walked in and said something like, "Gah! This is the women's room!" At which point I said replied, "I'm going to kill my friend." And she walked into her stall, and I finished and washed my hands and left.
#2: My mom had polio when she was very young, around six or seven. She spent most of her life walking with crutches or braces. We were at a renfaire, and she'd gone into the bathroom, into the handicapped stall (which was almost a room unto itself) and after doing her business discovered that the floor was wet. My father is the one who usually helps her up, but with the potential for a slick floor (which is a massive danger if you have no muscular control below your waist) he needed help. So I walked in, past a bunch of women (who, again, were either using stalls or waiting to use stalls, and in any case not the purpose of my visit nor any part of my interest) helped Mom get back upright and over to the wheelchair she used for for places like renfaires, and left again. I don't think anybody gave us a second glance.
#3: College, men's dorm, about three in the morning. I'd been in the shower, as one occasionally is. I heard someone come in the door, then go back out a few moments later. Being finished, I turned off the shower, wrapped a towel around myself, and stepped out the main area with the stalls. I was standing there when a woman came in. (So, for context, this was well after curfew, when most people were asleep, and her boyfriend had probably been the one scouting the room a moment earlier. It was sheer bad luck that I'd finished my shower right then.) We exchanged brief, embarrassed smiles and nodded to each other. Then she went into one of the stalls, and I went out (past the boyfriend, IIRC) and back to my dorm room.
So... Who goes in which bathroom? Who gives a poot? As long as they're there "on business" as it were, it doesn't seem to matter. And if their business is more nefarious, well, it doesn't matter a bit whether they're in the restroom that matches their perceived identity or not.
Can we please, PLEASE quit pretending that that this discussion has anything to do with protecting anyone from anything?