Consciousness and intelligence: You're right that proving something is "conscious" does seem difficult or impossible. Making something that is merely "intelligent," on the other hand, may be easy, depending on how you define the word. I think that's because consciousness is something we experience in ourselves, looking inwards, marveling at what it means that we are a mind living in a body, while intelligence is something we ascribe to other things in varying degrees if they exhibit certain behaviors. It follows that for us to consider something else to be conscious, we have to have that same feeling as when we experience our own consciousness, but via identification with something else (a robot or AI). We have to be like, "Wow, consciousness," and then be like, "Oh, that's not me, that's him!" It's the same way for intelligence, but it's a much broader term because there are many kinds of intelligence. Animals exhibit a lot of intelligence, as do some computer systems. In both cases (consciousness and intelligence), I think making systems that we recognize as more intelligent and more conscious involves understanding the brain and replicating its various functionality (since it's brains that will be making the call; it's brains that encode the distinction; the rest of the universe doesn't care). As we learn more about consciousness, we may be able to identify it as a particular kind of intelligence. For example, it may basically be the small part of our brain that directs our attention and awareness from moment to moment and decides what computational tasks to take on, while the rest of the brain is basically a large set of elaborate coprocessors specialized for different types of computation.

Simulated universes: I still think this is all meaningless metaphor. It's not easy to articulate why, but I'll try.

Some lines of thought just combine ideas in deep ways, but don't actually tell you anything new. For example, Zeno's paradox -- to get from point A to point B, you have to first go halfway, then halfway again, and so on forever -- doesn't actually teach us that nothing can ever go from point A to point B. It just demonstrates a glitch in the metaphors we use to reason about time and space. Actually, it's a glitch in the metaphor of traversing space as achieving a goal. Normally, if reaching a particular goal involves an infinite to-do list of subgoals, we wouldn't expect to ever get to the end and actually achieve the goal. However, if the goal is traversing some infinitely subdividable interval of space, we are provided with a mechanism to generate an infinite list of subgoals that are presumably all involved in achieving the larger goal. It's all just mental gymnastics and the normal tools of intuition breaking down.

To take a closer example, consider this line of thinking: "If you pick a random marriage proposal, chances are it is an imagined one rather than an actual one, because for every actual marriage proposal there are on average two or more imagined ones that never came to pass." This strange kind of thinking also mashes together diverse intuitive concepts. There's the idea that every class of thing (like "marriage proposal") has an extension set, an ensemble of instances. There's the fiction that an "event" is a discrete thing that we could identify and count if we had to, like counting the number of thoughts I've had today ("a thought" is a singular noun after all). There's the concept from probability theory of picking a member at random from a set (with the presumption that this is a well-defined act). There's the idea that an imagined X is still an X, because we still call it one. Unicorns are still unicorns even if they don't happen to exist.

The most intellectual, abstract part of the brain has a tendency to focus on what could possibly be rather than what is. We can get so wrapped up in generalities that we forget that this is the universe -- what we see around us, not what we imagine or are capable of conceiving of. Statements like, "Perhaps everything that can exist does exist" sound to me like a projection of our own models of reality out onto reality. Similarly, it's fascinating that we can conceptualize and talk about infinite computing power, but that doesn't mean there is any reality to it.