>For example, one of the most convincing things we could observe an AI do would be to make a novel observation about life, derived from current circumstances, that strikes us as deep and insightful, perhaps because it was hovering at the edge of our own awareness, or because it immediately activates mental patterns that we did not activate but are also meaningfully connected to the situation at hand for us.

I think you've come much closer than I to getting a definition of consciousness, but I still don't find it satisfactory. You could have a computer mimic a human without actually having anything resembling an inner "human experience". For example, a giant look up table with a preprogrammed response to any possible input. We have created extremely simple versions of this that are almost effective enough to pass the turing test. Just by gather a large number of human responses to chatbots. Though you could say the process that creates the giant look up table is intelligent, the table itself obviously is not.

There are AIs like AIXI which could be vastly intelligent without reasonably being considered conscious. They don't actually "think" so much as brute-force every possible solution to a given problem, or every possible explanation for a series of inputs, etc. Given enough computing power this would actually be effective. You could then ask it to mimic a conscious being and it could do so easily (given either a definition of consciousness or a set of observations about other conscious beings.) Maybe doing that requires the intelligence to actually simulate a conscious process somewhere along the line, so I don't know if that counts. It may be able to do so through mere, without having to actually run a full simulation. Providing a "deep insight" about life is nowhere near as complicated as a full human consciousness.

Maybe there is no such thing as consciousness. In the sense that there is no simple way to define it that is fully consistent with everything we want it to mean. That there are always going to be arbitrary seeming exceptions and gray areas. But that has deep and disturbing implications for morality. Maybe it's not relevant to our day to day life, but if we ever want to program a friendly artificial intelligence, we are screwed.

>I think it's silly or meaningless to say we possibly -- or probably! -- live in an "artificial" universe.

Well assuming that there are more simulated beings in all of existence, whatever that means, than there are non-simulated ones, it is very likely we are simulations. Though it's impossible to know what exists beyond our own universe. What is the distribution of universes? Do 50% of universes allow for infinite computing power, for example? What programming language do they use, which would determine how many bits a given universe would take to specify in that language, and therefore how often it gets simulated? How many universes even have anything close to intelligent beings?

An infinitely powerful computer would allow you to simulate every possible universe all at once, instantly (you would also be able to simulate every possible branch of non-deterministic universes, continuous variables, time travelling, and other things that people claim makes the universe "uncomputable.) And you could instantly do it an infinite number of times. If it were possible for such a thing to exist in a universe, that universe would contain every other possible universe. Including copies of itself and others universes with infinite computing power, which would contain more universes, etc.

(Very interesting short story that explains the implications of this http://qntm.org/responsibility)

So what is the top level universe? Why does something exist rather than nothing? Perhaps everything that can exist does exist. Maybe every computable program has been (is being?) run, including one that specifies our universe. That's the simplest possible explanation that explains the existence of the universe. Though I can not wrap my head around why something exists rather than nothing. None of this has any practical implications for our daily life, and the answer is probably unknowable, but it bothers me.

Sorry for going off on such a tangent. I hope this is comprehensible, as I know philosophical writings rarely are.