Subsidized, yes, obviously, but I'm still not sure why you think a group being tax exempt affects the tax rates for the rest of us. Most tax-exempt groups exist to provide services to their communities. They aren't there sucking up the community's resources and burdening society. I'm not quite clear on how you figure that they owe something to the government.
Now, some of the big corporate run "non-profits" and "charities" that we are starting to see have questionable practices is a separate issue, but using a local church in your town as an example lets think about this. Members of the church contribute portions of their post-tax income to pay for the building, utilities, priest's salary, etc. Then they volunteer their time and often more of their money aiding the community, whether through food drives, soup kitchens, shelters, welfare-to-work programs, youth programs, etc... The reason that federally recognized non-profits are often also exempt from state and local property taxes and the like are because they effectively reduce some burden on the government to provide those kinds of services to the community. Thus, if your politicians aren't into wasting money on some agenda, they shouldn't have to raise your taxes a dime because a new church popped up.
I understand that, but my point was that we wouldn't have to rely so much on those loans if schools had to compete for real money with competitive and reasonable prices. The government pumping cash into the system (whether through subsidized interest rates, or direct grants) has done nothing to help with this. Schools have no incentive to control their spending, and thus their tuition rates, because the government will just print more money, lower rates even when the country is broke to encourage banks to offer more loans (even to people who may never be able to pay them back... hmmm where have I seen this before) and call it a day.
It's not so simple as to just say that taking away government funding would make college harder to afford. I am of the opinion that the cost is high because of government meddling in the first place.