That's generally the way I remember the Gilmore situation.
USC left because Paul Dietzell, then the AD, thought the school could make more of a splash, and more money, as an independent. He probably harbored some desire that they eventually wind up in the SEC, but the reason given was the desire to be independent. There was some speculation at the time that he also wanted relaxed admission standards for football and the ACC wouldn't agree. Frank McGuire, the basketball coach, opposed the move but Dietzell wanted USC to be a football school.