trentmkelly/LessWrong-43k huggingface dataset:
What's state-of-the-art in AI understanding of theory of mind?
by ioannes — 2019-07-03T23:11:34.426Z

Sparked by Eric Topol, I've been thinking lately about biological complexity, psychology, and AI safety.

A prominent concern in the AI safety community is the problem of instrumental convergence – for almost any terminal goal, agents will converge on instrumental goals are helpful for furthering the terminal goal, e.g. self-preservation.

The story goes something like this:

AGI is given (or arrives at) a terminal goal
AGI learns that self-preservation is important for increasing its chances of achieving its terminal goal
AGI learns enough about the world to realize that humans are a substantial threat to its self-preservation
AGI finds a way to address this threat (e.g. by killing all humans)

It occurred to me that to be really effective at finding & deploying a way to kill all humans, the AGI would probably need to know a lot about human biology (and also markets, bureaucracies, supply chains, etc.).

We humans don't have yet a clean understanding of human biology, and it doesn't seem like an AGI could get to a superhuman understanding of biology without running many more empirical tests (on humans), which would be pretty easy to observe.

Then it occurred to me that maybe the AGI doesn't actually to know a lot about human biology to develop a way to kill all humans. But it seems like it would still need to have a worked-out theory of mind, just to get to the point of understanding that humans are agent-like things that could bear on the AGI's self-preservation.

So now I'm curious about where the state of the art is for this. From my (lay) understanding, it doesn't seem like GPT-2 has anything approximating a theory of mind. Perhaps OpenAI's Dota system or DeepMind's AlphaStar is the state of the art here, theory-of-mind-wise? (To be successful at Dota or Starcraft, you need to understand that there are other things in your environment that are agent-y & will work against you in some circumstances.)

Curious what else is in the literature about this, and also about how important it seems to others.
Blatant lies are the best kind!
by Benquo — 2019-07-03T20:45:56.948Z
Link: http://benjaminrosshoffman.com/blatant-lies-best-kind/

Mala: But then why do people get so indignant about blatant lies?

Noa: You mean, indignant when others call out blatant lies? I see more of that, though they often accuse the person calling out the lie of being unduly harsh.

Mala: Sure, but you can't deny - you've seen yourself - that people actually do get more indignant when they say that, than when they're pointing out a subtle pattern of motivated reasoning. How do you explain that, if "blatant lie" isn't a stronger accusation?

Noa: I think I see the problem. A stronger accusation can mean an accusation of greater wrongdoing, or it can mean a better-founded accusation. Blatant lying is ... well ... blatant! If someone pretends not to see that, that's terrible news about their ability or willingness to help detect deception.

Mala: But then the indignation is misplaced. Suppose Jer is talking with Horaha, trying to persuade him that their mutual acquaintance Narmer is behaving deceptively. Jer indignantly points out a blatant lie Narmer told. The proper target of the extra indignation due to blatancy is Horaha, not Narmer.

Noa: Who said otherwise?

Mala: Come on, you know that people get extra-indignant at the liar about blatant lies, despite your so far unsubstantiated claim that they are the best kind.

Noa: Sure, people make that mistake. People also yell at their friends because a stranger was mean to them earlier in the day or because they stubbed their toe. I never claimed - OK, it's helpful of you to point out that people make this mistake, but still, there is a good reason for indignation here, and understanding its proper target might help us avoid this kind of slippage.

Mala: So, why is blatant lying the best kind? Is it because it's purer somehow?

Olga: Hold on, you two, you've skipped over something.

Noa: What's that?

Olga: Sometimes Horaha and Narmer are the same person - sometimes the liar is the person we're trying to point out the lie to. Let's call the liar in the composite situation Menes.

Jer might be uncertain about Menes's true intentions. Menes might have an unexamined tendency to lie about something, but might - if this is drawn to his attention - reflect on what's going on, and what he's trying to accomplish.

When Menes says something to Jer - for instance, that everybody knows we don't have true freedom of speec...
What was the official story for many top physicists congregating in Los Alamos during the Manhattan Project?
by moses — 2019-07-03T18:05:12.944Z

There is a consensus among most people I know that if top researchers are necessary for the successful development of AGI, then it would be impossible to do in in secrecy, because the world would notice top AI researchers leaving their jobs and congregating at a military facility.

But the US government did pull this off with physicists. How?
Open Thread July 2019
by ryan_b — 2019-07-03T15:07:40.991Z

If it’s worth saying, but not worth its own post, you can put it here.

Also, if you are new to LessWrong and want to introduce yourself, this is the place to do it. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are welcome. If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, and seeing if there are any meetups in your area.

The Open Thread sequence is here.
Self-consciousness wants to make everything about itself
by jessicata — 2019-07-03T01:44:41.204Z
Link: https://unstableontology.com/2019/07/03/self-consciousness-wants-to-make-everything-about-itself/

Here's a pattern that shows up again and again in discourse:

A: This thing that's happening is bad.

B: Are you saying I'm a bad person for participating in this? How mean of you! I'm not a bad person, I've done X, Y, and Z!

It isn't always this explicit; I'll discuss more concrete instances in order to clarify. The important thing to realize is that A is pointing at a concrete problem (and likely one that is concretely affecting them), and B is changing the subject to be about B's own self-consciousness. Self-consciousness wants to make everything about itself; when some topic is being discussed that has implications related to people's self-images, the conversation frequently gets redirected to be about these self-images, rather than the concrete issue. Thus, problems don't get discussed or solved; everything is redirected to being about maintaining people's self-images.

Tone arguments

A tone argument criticizes an argument not for being incorrect, but for having the wrong tone. Common phrases used in tone arguments are: "More people would listen to you if...", "you should try being more polite", etc.

It's clear why tone arguments are epistemically invalid. If someone says X, then X's truth value is independent of their tone, so talking about their tone is changing the subject. (Now, if someone is saying X in a way that breaks epistemic discourse norms, then defending such norms is epistemically sensible; however, tone arguments aren't about epistemic norms, they're about people's feelings).

Tone arguments are about people protecting their self-images when they or a group they are part of (or a person/group they sympathize with) is criticized. When a tone argument is made, the conversation is no longer about the original topic, it's about how talking about the topic in certain ways makes people feel ashamed/guilty. Tone arguments are a key way self-consciousness makes everything about itself.

Tone arguments are practically always in bad faith. They aren't made by people trying to help an idea be transmitted to and internalized by more others. They're made by people who want their self-images to be protected. Protecting one's self-image from the truth, by re-directing attention away from the epistemic object level, is acting in bad faith.

Self-consciousness in social justice

A documented phenomenon in social justice is "white women's tears". Here's a case study (e

...
Opting into Experimental LW Features
by Raemon — 2019-07-03T00:51:19.646Z

We recently added an "beta testing" option for LessWrong. If you go to your account page, you'll see a checkbox for "Opt into experimental features".

Right now, the primary feature under development that isn't yet released to the public is Single Line Comments. This is an attempt to fit more overall comments in some areas.

Recent Discussion

For example, recent discussion now looks like this, where each post loads 4 comments (and highlights them if you haven't read them), but only shows significant amounts of text from the most recent comment:

You can click on a recent discussion item to mark that item as 'read', and make the green line go away.

Posts with 50+ Comments

On posts with 50 or more comments, comments below 10 karma will appear as a single-line comment:

Mousing over a SingleLineComment will show a hovercard with a preview of the comment.

Clicking on a thread will fully expand all of the children of the comment that you clicked on. Doing a search (Control+F, or Command+F) will also expand all comments (so that you won't run into annoying things where you try to search but the comment isn't displaying all the text so you can't find the quote you're looking for.

This is all still under developed. Admins of the site have been using it for the past month or two to see how it works, and tweaking it until it felt usable. It's now at a point where it seemed good to let other LW users try it out and see if it seems like an improvement.

(Before releasing it publicly, we plan to build some sort of safety-valve-checkbox where you can turn it off easily)
How/would you want to consume shortform posts?
by Raemon — 2019-07-02T19:55:56.967Z

The LessWrong team might be putting more work into the shortform concept in the next couple weeks. I wanted to check in about what specific features people might want for that.

Right now, some users set up "shortform feed" posts, which are a hacky way to give yourself a more casual space for off-the-cuff writing. In the past few weeks we've seen an uptick of people creating feeds for themselves.

It seemed useful to make this a bit more official (i.e. there's just a "create shortform post" button you can click instead of navigating to a post and writing a comment).

But there's several open questions about the best way to consume shortform content. Right now most shortform comments appear in Recent Discussion and then quickly disappear, which is very low-discoverability.

Possible considerations include:

Some people like low-discoverability. Sometimes what you want is a low-visibility place where you can write up thoughts without people immediately judging you for it, to help flesh out early stage ideas. For those people, the status quo is sorta fine.
Some people want more visibility. I know I personally would prefer my shortform feed to get maybe 5x the visibility it currently gets – I care less about the visibility and more about the vibe of "low effort casual space."

There's a few possible routes to more visibility:

Treating shortform more as a special case, where new shortform posts get displayed at the top of recent discussion, or potentially each day listing on the Daily/AllPosts page. (This might include entire comments, or might include a quick list of which shortform feeds got updated recently)
More generally improve the visibility of comments on old posts, with shortform just being one type of old post. This has the benefit of making commenting on old posts feel less like shouting into the void.
Creating a special shortform feed view page, and/or create a setting on Recent Discussion where shortform becomes the primary thing you're skimming, rather than a mixture of post-discussion.

There might be entirely different frames or ideas I haven't thought of.

Anyone have preferences on how to consume shortform?

(Also relevant would be if you find shortform actively annoying and would want to make sure it didn't get too much in your way)
Re-introducing Selection vs Control for Optimization (Optimizing and Goodhart Effects - Clarifying Thoughts, Part 1)
by Davidmanheim — 2019-07-02T15:36:51.071Z

This is the first post in a small sequence I'm writing on "Optimizing and Goodhart Effects - Clarifying Thoughts" (I have re-organized to make part 2, "Revisiting What Optimization Means" separate.)

Related to: How does Gradient Descent Interact with Goodhart?, Constructing Goodhart, Selection vs Control, Classifying Specification Problems as variants of Goodhart's Law

Next Posts: Revisiting What Optimization Means with Selection vs. Control, then Applying Overoptimization to Selection vs. Control

Introduction

Goodhart's law comes in a few flavors, as originally pointed out by Scott, and formalized a bit more in our joint paper. When discussing that paper, or afterwards, we struggled with something Abram Demski clarified recently, which is the difference between selection and control. This matters for formalizing what happens, especially when asking about how Goodhart occurs in specific types of optimizers, as Scott asked recently.

Epistemic Status: This is for de-confusing myself, and has been helpful. I'm presenting what I am fairly confident I understand well for the content written so far, but I'm unclear about usefulness for others, or how clear it comes across. I think that there's more to say after this post, and this will have a few more parts if people are interested. (I spent a month getting to this point, and decided to post and get feedback rather than finish a book first.)

In the first half of the post, I'll review Abram's selection/control distinction, and suggest how it relates to actual design. I'll also argue that there is a bit of a continuum between the two cases, and that we should add an addition extreme case to the typology, direct solution. The second section will revisit what optimization means, and try to note a few different things that could happen and go wrong with Goodhart-like overoptimization.

The third section will talk about Goodhart in this context using the new understanding - trying to more fully explain why Goodhart effects in selection and control fundamentally differs. After this, Part 4 will revisit Mesa-optimizers, and .

Thoughts on how selection and control are used in tandem

In this section, I'll discuss the two types of optimizers Abram discussed; selection, and control, and introduce a third, simpler optimizer, direct solution. I'm also going to m...
The Right Way of Formulating a Problem?
by Virgil Kurkjian — 2019-07-02T14:40:09.288Z

David Chapman Writes,

Finding a good formulation for a problem is often most of the work of solving it.

I agree with this intuitively, and I feel like I have seen this principle at work in my own work and in the problems I have tried to solve. However, when I try to convince others of this idea, I struggle to find examples that they can connect with or that they find compelling.

I suspect that programmers find this idea appealing because we routinely work with formal systems, and all of us know the experience of making a minor change in perspective and seeing an impossible problem turn into an easy one. So I'm most interested in examples that have nothing to do with code, examples that a lay audience would be able to grasp.

I would be particularly interested in examples from the history of science or medicine, if anyone can think of some. Scott and Scurvy is the only example I currently know of, and while interesting, does not seem like a perfect fit.

Much appreciated!
Everybody Knows
by Zvi — 2019-07-02T12:20:00.646Z

“Everybody knows that the dice are loaded.

Everybody rolls with their fingers crossed.

Everybody knows the war is over.

Everybody knows the good guys lost.”

– Leonard Cohen, Everybody Knows

“It is known.” – Dothraki saying

It is not known. Everybody doesn’t know.

When someone claims that everyone knows something, either they are short-cutting and specifically mean ‘everyone in this well-defined small group where complex common knowledge of this particular thing is something we have invested in,’ they are very wrong about how the world works, or much more commonly, they are flat out lying.

Saying that everybody knows is almost never a mistake. The statement isn’t sloppy reasoning. It’s a strategy that aims to cut off discussion or objection, to justify fraud and deception, and to establish truth without evidence.

Not Everybody Knows

Let us first establish quickly that everyone doesn’t know. There are many ways to see this.

One way to see this is to point out that when Alice tells Bob that everybody knows X, either Bob is asserting X because people act as if they don’t know X, or Bob does not know X. That’s why Alice is telling Bob in the first place.

A second way is to attempt to explain something in detail as you would to a child.

A cleaner way is to consider some examples of things that a lot of people don’t know. According to the first Google hit, 32 million American adults can’t read, and 50% can’t read a book at the 8th grade level. Various other tests of basic skills from school don’t look much better. Here are some more basic facts many Americans don’t know, including 20% who think the Sun revolves around the Earth. Nigerian prince scams still make over $700,000 per year. Doctors can’t do basic job-relevant probability calculations within an order of magnitude. Just yesterday (as of writing this) I had to explain to a college graduate that Bitcoin was more volatile than the stock market, and Forex was not a responsible retirement savings plan.

What does the claim that ‘everybody knows’ mean?

There are a few different things ‘everybody knows’ is standing in for when someone claims it.

In most of them, the claim that literal actual ‘everybody knows’ is sort of the Bailey, and the thing we’ll describe here is the implicit Motte that ‘everyone knows’ is your real message. Which of course, in turn, not everybody knows. As is often the case, the Bailey is blatantly ...
Collaborative VS adversarial truth seeking
by Elo — 2019-07-01T23:48:09.817Z

There is a question of how the ground rules of discussions should flow. This applies to debates, discussion, challenges of belief, general scientific discourse and more.
There are two cultures in the particular trade-off I want to talk about. Collaborative and adversarial.

I pitch collaborative as, “let’s work together to find the answer (truth)” and I pitch adversarial as, “let’s work against each other to find the answer (truth)”. As fairly neutral descriptions, these are quite balanced. In an entrepreneurial/business setting, adversarial work is called red team.

I made some neat diagrams a while ago in a model of arguments.

As an epistemic practice, it's important to know what practice one is in (what practice one's counterpart is in,) and how this might shape interactions. Internally the stance for each is different. For collaborative, it might look something like, “I need offer my alternative view, between us, we need to seek the truth”. For adversarial, it might look something like, “I need to argue strongly for my (correct) view and if I ever sense that my view is wrong I need to quit”.

Externally (and from a collaborative culture perspective) someone in adversarial culture probably presents as defensive, aggressive, stubborn, yelling and shouting, ontologically arrogant, pulling out every stop possible (every fallacy) to advocate for their opinion. Once they have decided they are lost (or are convinced of the other side's opinion), they probably look like they do a backflip and staunchly argue against their own position. But before they do, they probably stop the argument and run away.

Externally (and from an adversarial culture perspective) someone in collaborative culture probably presents as weak willed, fragile (too politically correct), softly spoken, trying to be manipulative with friendliness, likely to not say what they really think, hard to pin to a belif or position, lacking confidence, and the sort of person who doesn't admit defeat even when they are clearly wrong.

Adversarial culture sees itself as the true arena for ideas, where concepts go head to head and there is only one true and final position. Adversarial sees ourselves as having a duty to call people out if they are wrong, this can get us into trouble and self-invested burdensome quarrels at times where we seem to fight over tiny details that don't mat...
PlayStation Odysseys
by Jacob Falkovich — 2019-07-01T17:41:52.499Z

Cross-posted from Putanumonit.com

Note: heavy spoilers for one epic poem, minor spoilers for several epic video games.

Very related: the Greek myths are about loneliness.

Antihero’s Journey

Merriam-Webster defines “odyssey” as:

n. a long wandering or voyage usually marked by many changes of fortune.

The word brings to mind an adventure, a quest, a hero’s journey that leads them to self-discovery, empowerment, and saving the village. This is the plot structure of the video game Assassin’s Creed: Odyssey. In it, the player controls a mercenary who sails from her home near Ithaca to find her parents and prove her worth to the Greek world. And of course, The Odyssey itself starts with the tale of a young man sailing from Ithaca seeking fame and news of his father.

But that man is not Odysseus. It’s his son, Telemachus, who goes on a quest. Odysseus himself shows up only in book 5 of the epic poem, old and tired. Most of his travels are behind him at that point and told only in flashback. His is not a hero’s journey of transformation and discovery. He is by and large the exact same man who sailed for Troy two decades before, a fact that is remarked upon by the other characters. It is Telemachus who transforms from a shy youth into an assertive prince when he finally fights by his father’s side against Penelope’s suitors.

And yet, the game is not called Assassin’s Creed: Telemacheia, even though the protagonist’s journey mirrors Telemachus much more than his father. The son’s story is relegated to a secondary plot, while the father’s story has proved to be memorable and compelling to audiences for three thousand years.

Why is Odysseus such a compelling hero? When I read the Odyssey as a teenager I remember admiring Odysseus’ heroics. But upon rereading Emily Wilson’s version[1], I noticed that Odysseus isn’t just a static character, he’s kind of a huge dick.

Odysseus is brave, formidable, and intelligent, but also a consummate liar, greedy, violent, and selfish. Some behavior that is repugnant to modern readers, like murder, pillaging, enslaving, and fornicating, was probably considered “all in the game” in Bronze Age Greece. Other sins, like the fact that the survival ratio of men under his command would make even comrade Stalin blush, are criticized by the other characters in the poem itself. And even aside from his indiffere

...