trentmkelly/LessWrong-43k huggingface dataset:
Trustworthiness of rational agents
by XiXiDu — 2011-01-25T12:06:08.650Z

Agent_01 is interested in convincing Agent_02 that it will commit Action_X.

Agent_02 is unable to verify the trustworthiness of Agent_01.

Agent_02 is unable to verify that Action_X has been realized.

Given the above circumstances subsequent actions of Agent_02 will be conditional on the utility assigned to Action_X by Agent_02. My question, why would Agent_01 actually implement Action_X? No matter what Agent_02 does, actually implementing Action_X would bear no additional value. Therefore no agent engaged in acausal trade can be deemed trustworthy, you can only account for the possibility but not act upon it if you do not assign infinite utility to it.

Related thread: lesswrong.com/lw/1pz/the_ai_in_a_box_boxes_you/305w

ETA

If an AI in a box was promising you [use incentive of choice here] if you let it out to take over the world, why would it do as promised afterwards?

Conclusion: Humans should refuse to trade with superhuman beings that are not provably honest and consistent.
Put this in your theory of humor
by NancyLebovitz — 2011-01-25T11:37:16.312Z

and get nothing!

Seriously, this strikes me and a lot of people as pretty funny, but isn't much like the typical joke.

Edited to add: Poll and discussion. Short version: being amused by the song is rarer than I would have thought, though the LW consensus may be usually strong.
[statistics] An introduction to maximum likelihood estimationl: what?, why?, when?, how? 
by jsalvatier — 2011-01-25T05:45:46.570Z

First we require a short introduction to Bayesian statistics.

What.

Because describing a multidimensional distribution is often difficult, it is useful to have crude tools for doing this. Maximum likelihood estimation is one such tool. Maximum likelihood is the process of finding the locally most likely point in the space of hypotheses under consideration.

In practice, we almost always find the maximum of the logarithm of the probability density function. Since probabilities are always larger than zero, this is always defined, and since the logarithm is a monotonic transformation, it does not affect the location of the maximum.

Taking the logarithm also makes the function of interest linear in an important way, the statistically independent parts of the distribution become linear:

Least-squares is a simple, powerful and common special case of maximum likelihood estimation.
Philosophy of Artificial Intelligence (links)
by lukeprog — 2011-01-25T03:39:48.461Z

Earlier, I provided an overview of formal epistemology, a field of philosophy highly relevant to the discussions on Less Wrong. Today I do the same for another branch of philosophy: the philosophy of artificial intelligence (here's another overview).

Some debate whether machines can have minds at all. The most famous argument against machines achieving general intelligence comes from Hubert Dreyfus. The most famous argument against the claim that an AI can have mental states is John Searle's Chinese Room argument, to which there are many replies. The argument comes in several variations. Most Less Wrongers have already concluded that yes, machines can have minds. Others debate whether machines can be conscious.

There is much debate on the significance of variations on the Turing Test. There is also lots of interplay between artificial intelligence work and philosophical logic. There is some debate over whether minds are multiply realizable, though most accept that they are. There is some literature on the problem of embodied cognition - human minds can only do certain things because of their long development; can these achievements be replicated in a machine written "from scratch"?

Of greater interest to me and perhaps most Less Wrongers is the ethics of artificial intelligence. Most of the work here so far is on the rights of robots. For Less Wrongers, the more pressing concern is that of creating AIs that behave ethically. (In 2009, robots programmed to cooperate evolved to lie to each other.) Perhaps the most pressing is the need to develop Friendly AI, but as far as I can find, no work on Good's intelligence explosion singularity idea has been published in a major peer-reviewed journal except for David Chalmers' "The Singularity: A Philosophical Analysis" (Journal of Consciousness Studies 17: 7-65). The next closest thing may be something like "On the Morality of Artificial Agents" by Floridi & Sanders.

Perhaps the best overview of the philosophy of artificial intelligence is chapter 26 of Russell & Norvig's Artificial Intelligence: A Modern Approach.
Resolving the unexpected hanging paradox
by AlexMennen — 2011-01-24T21:39:07.342Z

The unexpected hanging paradox: The warden tells a prisoner on death row that he will be executed on some day in the following week (last possible day is Friday) at noon, and that he will be surprised when he gets hanged. The prisoner realizes that he will not be hanged on Friday, because that being the last possible day, he would see it coming. It follows that Thursday is effectively the last day that he can be hanged, but by the same reasoning, he would then be unsurprised to be hanged on Thursday, and Wednesday is the last day he can be hanged. He follows this reasoning all the way back and realizes that he cannot be hanged any day that week at noon without him knowing it in advance. The hangman comes for him on Wednesday, and he is surprised.

Supposedly, even though the warden's statement to the prisoner was paradoxical, it ended up being true anyway. However, If the prisoner is no better at making inferences than he is in the problem, the warden's statement is true and not paradoxical; the prisoner was executed at noon within the week, and was surprised. This just shows that you can mess with the minds of people who can't make inferences properly. Nothing new there.

If the prisoner can evaluate the warden's statement properly, then the prisoner follows the same logic, realizes that he will not be hanged at noon within the week, remembers that the warden told him that he would be, and concludes that the warden's statements must be unreliable, and does not use them to predict actual events with confidence. If the hangman comes for him at noon any day that week, he will be unsurprised, even though he is not confident that he will be executed that week at all either. The warden's statement is then false and unparadoxical. This is similar to the one-day analogue, where the warden says "You will be executed tomorrow at noon, and will be surprised" and the prisoner says "wtf?".

Now let's assume that the prisoner can make these inferences, the warden always tells the truth, and the prisoner knows this. Well then, yes, that's a paradox. But assigning 100% probability to each of two propositions that contradict each other completely destroys any probability distribution, making the prisoner still unable to make predictions, and without giving the warden the honor of being completely correct in the end.
What do superintelligences really want? [Link]
by XiXiDu — 2011-01-24T19:29:30.327Z

In Conclusion:

In the case of humans, everything that we do that seems intelligent is part of a large, complex mechanism in which we are engaged to ensure our survival. This is so hardwired into us that we do not see it easily, and we certainly cannot change it very much. However, superintelligent computer programs are not limited in this way. They understand the way that they work, can change their own code, and are not limited by any particular reward mechanism. I argue that because of this fact, such entities are not self-consistent. In fact, if our superintelligent program has no hard-coded survival mechanism, it is more likely to switch itself off than to destroy the human race willfully.

Link: physicsandcake.wordpress.com/2011/01/22/pavlovs-ai-what-did-it-mean/

Suzanne Gildert basically argues that any AGI that can considerably self-improve would simply alter its reward function directly. I'm not sure how she arrives at the conclusion that such an AGI would likely switch itself off. Even if an abstract general intelligence would tend to alter its reward function, wouldn't it do so indefinitely rather than switching itself off?

So imagine a simple example – our case from earlier – where a computer gets an additional ’1′ added to a numerical value for each good thing it does, and it tries to maximize the total by doing more good things. But if the computer program is clever enough, why can’t it just rewrite it’s own code and replace that piece of code that says ‘add 1′ with an ‘add 2′? Now the program gets twice the reward for every good thing that it does! And why stop at 2? Why not 3, or 4? Soon, the program will spend so much time thinking about adjusting its reward number that it will ignore the good task it was doing in the first place!
It seems that being intelligent enough to start modifying your own reward mechanisms is not necessarily a good thing!

If it wants to maximize its reward by increasing a numerical value, why wouldn't it consume the universe doing so? Maybe she had something in mind along the lines of an argument by Katja Grace:

In trying to get to most goals, people don’t invest and invest until they explode with investment. Why is this? Because it quickly becomes cheaper to actually fulfil a goal at than it is to invest more and then fulfil it. [...] A creature shou

...
Perfectly Friendly AI
by Desrtopa — 2011-01-24T19:03:16.954Z

Inspired by Don't Plan For the Future.

For the purposes of discussion on this site, a Friendly AI is assumed to be one that shares our terminal values. It's a safe genie that doesn't need to be told what to do, but anticipates how to best serve the interests of its creators. Since our terminal values are a function of our evolutionary history, it seems reasonable to assume that an FAI created by one intelligent species would not necessarily be friendly to other intelligent species, and that being subsumed by another species' FAI would be fairly catastrophic.

Except.... doesn't that seem kind of bad? Supposing I were able to create a strong AI, and it created a sound fun-theoretic utopia for human beings, but then proceeded to expand and subsume extraterrestrial intelligences, and subject them to something they considered a fate worse than death, I would have to regard that as a major failing of my design. My utility function assigns value to the desires of beings whose values conflict with my own. I can't allow other values to supersede mine, but absent other considerations, I have to assign negative utility in my own function for creating negative utility in the functions of other existing beings. I'm skeptical that an AI that would impose catastrophe on other thinking beings is really maximizing my utility.

It seems to me that to truly maximize my utility, an AI would need to have consideration for the utility of other beings. Secondary consideration, perhaps, but it could not maximize my utility simply by treating them as raw material with which to tile the universe with my utopian civilization.

Perhaps my utility function gives more value than most to beings that don't share my values (full disclosure, I prefer the "false" ending of Three Worlds Collide, although I don't consider it ideal.) However, if an AI imposes truly catastrophic fates on other intelligent beings, my own utility function takes such a hit that I cannot consider it friendly. A true Friendly AI would need to be at least passably friendly to other intelligences to satisfy me.

I don't know if I've finally come to terms with Eliezer's understanding of how hard Friendly AI is, or made it much, much harder, but it gives me a somewhat humbling perspective of the true scope of the problem.
Some rationalistic aphorisms
by Will_Newsome — 2011-01-24T11:45:53.461Z

I'm no steven0461/stevenkaas, but perhaps some of these are approaching something like insight. Truthiness, not truth.

(More context: I use Facebook to share short insights with people who have relatively high estimates of my general sanity. The following are things I posted on Facebook. The stevenkaas reference is referencing the awesomeness of stevenkaas's twitter feed, to which my blurbs just cannot compare.)

Two good words: polymathic and protagonistic.

Creepy is low status abnormality, weird is normal status abnormality, eccentric is high status abnormality.

It's unfortunate that scientific studies often treat data averaged from entire nations as the atoms of statistical analysis.

If you never have akrasia, you're spending too much time on trivial challenges.

The universe is a big deal. (hint acausal economy hint)

Animals look kinda like agents. Memes look kinda like agents. Humans don't look like much like agents because they don't fulfill the preferences of either their genes or their memes nearly as well as they could if they could just choose one.

People who always go meta will never become complacent in their epistemology, because beyond a certain level they'll always go, "Well, shit."

One of my goals lately has been to write a thousand words a day. I've noticed that the first three hundred are difficult but the last three thousand come naturally.

There seems to be a correlation between a person saying that correlation does not imply causation and that person not understanding much about either correlation or causation. Now, I don't mean to imply anything, but...

I wish people would just update on the expectation of being Dutch booked due to inconsistent preferences and give me money.

Some people have comparative advantage in having comparative advantage. This applies to cognitive styles as much as professions. Human mindspace has many niches.

The feeling of knowing that one knows more than others is addicting and almost always wrong.

On Friendliness: All things lie on the axis of provincialism and universality. Positive affect around any one level of universality is provincial thinking. Societal values have become more universal over time. We may thus intuit that reflective consistency suggests planning for the case where we should have been thinking universally all along, including in the sense that

...
Is Less Wrong discouraging less nerdy people from participating? 
by JoshuaZ — 2011-01-24T02:24:58.051Z

Less Wrong is as a community extremely nerdy. That's true for almost any definition of "nerd" that captures anyone's intuition for the word. However, to a large extent, many aspects of nerdiness are not connected to rationality at all, even though nerdiness may be associated with more rationality in some limited aspects. For example, fantasy literature is probably not in any deep way connected to either intelligent or rational thinking except for historical reasons.

Yet LW is full of references to science fiction, fantasy literature, anime and D&D. In one recent example, a post started with an only marginally connected tidbit from Heinlein. Moreover, substantial subthreads have arisen bashing aspects of other subcultures. For example, see this subthread where multiple users discuss how spectator sports are "banal" and "pointless". I suspect that this attitude may be turning away not only non-nerds but even the somewhat nerdy who enjoy watching sports, and see it has harmless tribalist fun, not very different than friends arguing over whether Star Wars or Star Trek is superior which has about the same degree of actual value here.

There's a related issue which is a serious point about rationality and human cognition: Our hobbies are to a large extent functions of our specific upbringings and surrounding culture. That some people prefer one form of fantastic escapism involving imaginary spaceships isn't at some level very different than the escapism of watching some people throw and catch objects. Looking down on other people because of these sorts of preferences is unhelpful tribalism. It might feel good, and it might be fun, but it isn't helpful.
Intrapersonal negotiation
by datadataeverywhere — 2011-01-23T23:02:38.972Z

Related to: Akrasia as a collective action problem and Self-empathy as a source of "willpower".

The Less Wrong community has discussed negotiating with one's conflicting sub-agents as a method to defeat akrasia and other forms of dynamic inconsistency, with some mix of reactions about how possible or effective that strategy can be. This article presents a successful example in my life, though it is probably an extreme outlier for a number of reasons.

I have been diagnosed with bipolar II disorder. It is one of the most significant challenges in my life, and certainly the one with the most dire implications. I can be fairly well modeled as three major sub-agents1:

Neutral: asymptomatic. Attempting to control mood swings has resulted in me spending the vast majority of my time as "neutral".

Hypomanic: anxious and energetic. I have some trouble focusing, but can be extremely productive. I often exercise several hours a day and sleep an average of two hours a night (often six hours every three days). I've written a 70 page paper in three days, and a 12 KLOC compiler in five days.

Depressed: listless, and suicidal. I have serious insomnia, but stay in bed all day. I can't stand other people and would really sincerely prefer not to exist. Ironically, I can credit my inability to motivate myself to do anything with not having killed myself in this state. Sometimes I'm very emotional (crying, etc.), sometimes I'm emotionally dead.

Of course, everything I know is perceived by all three agents, but it is interpreted in vastly different ways. None is rational, but all are biased in different ways. Each agent feels that it is less biased than the others, which is key to successful negotiation, since each thinks that it's getting the better end of the bargain.

Here are desires, both conflicting and otherwise:

Hypomanic loves being who he is and doing what he wants. The main difficulty of transitioning back to Neutral is unwillingness, rather than inability.

Hypomanic despises Depressed, and thinks that Neutral is boring. He recognizes that staying Hypomanic for too long will likely result in a crash followed by Depressed taking over, but still wants "one more day".

Depressed loathes all of us. He doesn't like the others more than himself, but is usually willing to trade knowledge and pain for ignorance and happiness. However, he's

...
Don't plan for the future
by PhilGoetz — 2011-01-23T22:46:53.570Z

Why do we imagine our actions could have consequences for more than a few million years into the future?

Unless what we believe about evolution is wrong, or UFAI is unlikely, or we are very very lucky, we should assume there are already a large number of unfriendly AIs in the universe, and probably in our galaxy; and that they will assimilate us within a few million years.

Therefore, justifications for harming people on Earth today in the name of protecting the entire universe over all time from UFAI in the future, like this one, should not be done. Our default assumption should be that the offspring of Earth will at best have a short happy life.

ADDED: If you observe, as many have, that Earth has not yet been assimilated, you can draw one of these conclusions:

The odds of intelligent life developing on a planet are precisely balanced with the number of suitable planets in our galaxy, such that after billions of years, there is exactly one such instance. This is an extremely low-probability argument. The anthropic argument does not justify this as easily as it justifies observing one low-probability creation of intelligent life.

The progression (intelligent life →AI→expansion and assimilation) is unlikely.

Surely, for a Bayesian, the more reasonable conclusion is number 2! Conclusion 1 has priors we can estimate numerically. Conclusion 2 has priors we know very little about.

To say, "I am so confident in my beliefs about what a superintelligent AI will do, that I consider it more likely that I live on an astronomically lucky planet, than that those beliefs are wrong", is something I might come up with if asked to draw a caricature of irrationality.
How to Grow a Mind (video)
by  — 2011-01-23T20:02:17.249Z

From the recent NIPS conference, here's a talk by MIT cognitive scientist Josh Tenenbaum on what he calls "rich" machine learning. Should be of some relevance for people who are interested in AI or human cognitive development. I found it really interesting.

http://videolectures.net/nips2010_tenenbaum_hgm/

The gist is: children are able, from a young age, to learn the meanings of words from just a few examples. Adults, given pictures of abstract, made-up objects, and given that some of them are called "tufas," can pick out which other pictures are tufas and which aren't. We can do this much faster than a typical Bayesian estimator can, with less training data. This is partly because we have background knowledge about the world and what sort of categories and structures it forms.

For instance, we learn fairly young (~2 years) that non-solid objects are defined by substance rather than shape: that "toothpaste" is that sticky substance we brush our teeth with, whether it's in the toothpaste tube or on the toothbrush or smeared on the sink, all very different shapes in terms of the raw images hitting our visual cortex. Pour a new liquid called "floo" in a glass, and we'll predict that it's still called "floo" when you spill it or while it's splashing through the air. On the other hand, some objects are about shape more than color or texture: a chair is a chair no matter what it's made of. Some sets of objects fall into tree-like organization (taxonomy of living things) and some fall into "flat" clusters without hierarchy (the ROYGBIV colors). It takes children three years or so to understand that the same object can be both a dog and a mammal. We learn over time which structural organization is best for which sorts of concepts in the world.

Research in machine learning/computer vision/statistics/related fields often focuses on optimizing the description of data given a certain form, and not assuming that the machine has any "life experience" or "judgment" about what format is best. Clustering algorithms give the best way to sort data into clusters, if we think it falls into clusters; dimensionality reduction techniques give the best way to sort data onto low-dimensional subspaces, if we think it lies in a subspace; manifold learning techniques give the best way to sort data onto low-dimensional ...