It's a little disconcerting to walk into a gun range where you know everyone in the building is armed - especially when they're carrying highly visible firearms like long guns. After a while you get used to it, and then you actually kinda like knowing that nobody can start anything. I've occupied a firing range position next to a person with a sawed off shotgun who was outfitted like a gang member (who was practicing). I can honestly tell you I was more comfortable being in that situation than I would have been in any other situation with that person while I knew he was armed.
The scenario where no-one has a gun is not possible, but I can definitely say that I would be less comfortable with my ability to protect my family in that scenario. Because if multiple muscle heads show up (especially with various clubs or stabbing weapons) there's not much I can do in retaliation.
There aren't too many people in my life that I can think that would make me less comfortable knowing that they own guns. I know a lot of gun owners, and generally I would be more comfortable if that list grew rather than shrank. So for example, my neighbor across the street, and my daughter's school teacher, and my co-workers and wife's co-workers, etc. etc.