There are actually an infinite number of possible things that could have happened which we can't rule out. Let me preface what I'm going to say with I don't mind what anyone believes. If it makes you happy to believe one thing over another thing, then I think that's great.

With that out of the way, when we're having these kinds of discussions, it's not really productive to bring up an infinite number of possibilities that we can't rule out. Which one should we choose to support? Without some criteria, we will just talk about it forever -- which might be fun, but it's not productive.

This is where science comes in. I'm going to say something a bit controversial. Science doesn't really talk about absolute truth. We can't know what that is -- because there are an infinite number of possibilities that we can't rule out. Instead it talks about things that are "productive" (my term).

In a very hand-wavy way, science is about making observations and building a model that explains those observation. There are a couple of other important things. The model should also make predictions about thing we haven't observed that we can test. We try it and if it also fulfills those predictions, we thing it's a "productive" model (again, my term). Why is it productive? Because it allows us to reason about things and make conclusions that are likely to work out for us. However, we shouldn't mistake that for the absolute truth. There really is a possibility that things appear one way to us, but are very different in ways that we can't see. The key is that from the perspective of science that's fine -- as long as we can keep making predictions and they keep working out for use, we can use our model.

But what if we have 2 models that are both equivalent? For example, let's say they both explain the observations we see and they make predictions that work out. Which should we pick? By convention, we pick the simplest one. In truth, the simplest one might not be correct, but it's always easier to work with a simpler model... because it's simpler ;-)

So while there may truly be advanced civilizations out there -- we can't rule it out, there is no test we can currently do that would show that advanced civilizations exist. Our models without advanced civilizations (except our own) explain everything we can currently observe and everything we can currently test. So we choose the model with no other advanced civilizations.

You really don't have to believe that, but when discussing with others we have to set some rules. If we keep in mind other advanced civilizations, it's just as reasonable to keep in mind an infinite number of other possibilities -- some of which would be truly bizarre. In order to keep the conversation on track, we have to choose somehow and the scientific method is a good, proven way to narrow the discussion down to something that is productive. However, if you find people who want to talk about the kinds of things you want to talk about, I think there is nothing at all wrong with that. You should do what makes you happy.

I hope that makes things a bit clearer. I often meet people who make comments like you did and I think it's really mystifying to those people why science oriented people act this way.