Besides humans in the Galaxy, there was someone else. There are traces—usually burned ones. That’s why there is an armed special detachment that urgently flies out every time there’s a report of possible contact. If needed, they’ll burn all traces of the outsiders as well—right down to themselves.