Yeah, Varys' only allegiance is to "the realm", which he says about 7992981 times during the series. He did not support the Mad King's kids until he determined one of them (at least at the time) had a good heart and could potentially unite the 7 kingdoms in peace, get rid of slaves, etc. Not to mention, she logistically had the superior armies/dragons.
When he tried to help assassinate her, he just thought she was another crazy Targ that married a crazy horse lord to bring blood and chaos to the realm. Plus he hates magic and dragons = magic.