It's a kinda/kinda not sitch.
In essence, any invisible virus snot that doesn't land in your nose or pie holes won't infect you... directly. Invisible virus snot on your skin won't do jack, with two exceptions: if you then huff it in off your top lip in your 12 inhalations a minute, or if you touch the bit of your face before you wash it and subsequently stick your hand somewhere gooey (ears, eyes, poop chute, front bum). I'm actually not entirely sure that something so fine and light would attach to the skin - the interactions between the boundary layer of air and the surrounding atmosphere may make it less likely - but it definitely would if you had a beard. Also eyebrows and lashes, as anyone with hayfever will attest.
Those bits of you the virus can get into unmasked are surprisingly small - if you breathe through your nose normally, we're talking about maybe 2sqcm, and a low pressure region below it, which you could definitely snort up.
With the mask in place, there's a 200sqcm surface designed to do nothing but get invisible virus snot stuck to it, like a giant snot net. If it's also something like a plasterer's dust mask, rather than something a bit more discerning, your nose will be directly in contact with a piece of material which is both covered in invisible virus snot and not able to prevent it from passing through.
Best case, you end up with something literally covered with invisible virus snot. Even ignoring the people who will touch it while wearing it to adjust it, because they haven't had PPE mask training and aren't used to it and it's not actually fitted for them (there's a cool thing with either isoamyl acetate or saccharin for proper mask fitment), you've got people who will touch it, ungloved, while removing it - if not the actual mask itself, then the ear loops which may also trap the snot and are not a natural place to touch.
Overall, if you can manage to take off a disposable mask, while gloved*, and chuck it in the bin without touching it (pedal bin is good; we also have a new robo-bin which opens when you trigger two touchless sensors), along with the gloves which you remove without touching outside to skin, and then wash your face, hair, and hands, then a fitted mask of appropriate grade will be better than not having one in preventing you from exposure to the virus.
*People are all about the mouth and nose. Eyes rarely make the list, and hands don't even get a mention usually.
So we'd rather have the virus on your face, in your beard, after you arrive back home, than on a mask? Because even if you wash your hands first, and then rub your face, if the virus was going to be on a mask, then it's on your cheek instead, and you just washed your hands but not your cheek. So one rub is all it takes. This is not worse (as best I can tell) than removing your mask with your hand and then rubbing your face without washing. In fact it's probably easier to remember to wash your hands after removing a mask than it is to wash your face when you get home (without washing virus into your eyes).
I'm also not advocating that people toss their masks out. I'd advocate that people don't wear it for 24 hours (or more), let virus die in the fibers, and then re-use.
I agree that gloves would be the way to go, but short of that, washing hands after getting home and removing the mask and eyewear seems second best.
Washing eyewear would probably be a good move to remove any virus on the outside of it, which might live for days. But then, that's true of anyone who wears glasses in public right now. Which is still better than
not wearing glasses in public right now.
Edit:
So I guess let me put it this way. What I intend to do on my next grocery shopping trip (which I have avoided for weeks now) is to put on safety glasses (mine have like a little foam seal under them), put on a mask, and wear cloth gloves (rather than latex).
I'll consider my hands contaminated the entire time. When I get home, I'll use those gloves to carefully remove the mask, and let it sit for at least 24 hours, and I'll use the gloves to remove eyewear and I might consider wiping that with alcohol if I need to go anywhere for the next 5 days. I'll put the gloves, mask, and eyewear in the garage to sit for at least 24 hours.
Then I'll go inside and wash my hands before touching anything. (I skipped the part where I clean cold grocery stuff before putting it in the fridge and leave pantry stuff in the garage for a few days).
Sure, I might have virus on my head or neck, or in my clothes, but I figure this is quite a bit improved over me just taking my open eyes into the grocery store and breathing through my nose.