No one said no downtime. I'd argue though that spending $300,000,000 walking on grass and hitting a little white ball is not really an efficient use of government funds and there are many more cost efficient ways a POTUS could be spending downtime.
Pretty sure you have to leave that up to the guy who is deciding how to unwind.
Wouldn't you? And how do you know no one is helped by the prayers?
Let's investigate this. You have a guy saying something (praying) while everyone else is standing around thinking about what he's saying.
If there is a God, that's one person praying and others listening.
If there are multiple Gods, that's one person praying to a false god, and others listening.
If there is no God, that's one person wishing and others listening to the wish.
Either way, it's a single individual's thoughts being broadcast. Some of the people in the audience will agree, some will not. God gets to receive thoughts from some prayers and some people thinking "that's not what I think". If there are multiple Gods, it just incurs wrath.
Now let's imagine that there were a moment of silence to collect thoughts.
If there is a God, that's almost everyone praying slightly different prayers but each of which is the content of their hearts. Nobody is disagreeing because they're all just composing their own thoughts.
If there are multiple Gods then maybe one or two of the people in the audience is actually praying to one of them. This may incur slightly less wrath than when one person chooses the wrong deity to pray to.
If there are no Gods then it's just everyone making themselves happy.
So when we have one guy leading the prayer, none of those scenarios (if there is a god, if there are multiple gods, or if there are no gods) are better for anyone than if you just have a moment of silence to let all the crazy people have all their crazy thoughts in private. That is, nobody is better served except the guy getting paid.