This is exactly the crushing problem faced by Wake Forest football.
In a good year for a "big game" we have 22-25,000 Wake fans and 8-12,000 opposing fans (maybe more if the opponent is UNC or NCSU). In a bad year or for a "minor" opponent those numbers drop considerably.
Most of us have been to games at VaTech, NCSU, ECU, Clemson, S. Carolina etc. with 60-75,000 home fans and about 10,000 opposing fans even for games that are not "big". Students go to the game and stand for most of it. The crowd is loud, partly because there IS a crowd.
Yes, I understand about the size of the schools and fan bases involved, but this is what we recruit against; it must be terribly difficult to convince an 18 year-old with stars in his eyes to come to Wake under the best of circumstances.
*This should in no way be read as an excuse for this year or last as I do believe the results have been worse than the talent level.*