It's worth considering that unless we're actually being observed as a species, the chances are that no (or very few, at least) alien civilisation actually knows we exist yet.
Unless they're able to somehow detect our presence by our impact on the atmosphere, no civilisation within a hundred light years would know we exist, because our radio signals have only been broadcasting for just over a hundred years. Since these travel at the speed of light they haven't yet reached stars and planets more than a hundred light years away.
Obviously, a hundred light years is still a relatively wide space in which there could potentially be life, but it also limits the time in which aliens would have had a chance to get to Earth with certainty that we existed here.
As for the Roswell stuff, that's even less time - radio waves would only have been reaching 40-50 light years from us. And the "Nazi aliens" even less time, 30-odd light years.
Obviously, that all relies on them detecting radio waves rather than being somehow able to detect our presence through another means, but as far as detecting things in space goes, radio waves are a pretty good method. And thinking about it, if our atmosphere has changed chemically from human impact, it'll only have been doing so since the industrial revolution, so you're only talking another hundred or so years of detection by light signatures.