People can also choose to avoid establishments with single sex bathrooms, single sex schools, single sex dorms, or single sex floors within single sex dorms if those things make them uncomfortable, so don't see how their rights are being violated without the CLT ordinance either.
You're saying that everyone should be comfortable with folks of any sex/gender in an enclosed stall bathroom with them. So if that is the case, why can't trans people be comfortable with folks of any sex/gender in an enclosed stall bathroom with them, as exists under the status quo? They are the ones saying they are not comfortable so need a change ... not everyone else. Yet you are telling everyone else that they need to be comfortable with it but not the trans people.