trentmkelly/LessWrong-43k huggingface dataset:
Power as Easily Exploitable Opportunities
by TurnTrout — 2020-08-01T02:14:27.474Z

(Talk given at an event on Sunday 28th of June. TurnTrout is responsible for the talk, David Lambert edited the transcript.

If you're a curated author and interested in giving a 5-min talk, which will then be transcribed and edited, sign up here.)

TurnTrout: Power and power-seeking play a big part in my thinking about AI alignment, and I think it's also an interesting topic more generally.

Why is this a big deal? Why might we want to think about what power is? What is it, exactly, that people are thinking of when they consider someone as powerful?

Well, it seems like a lot of alignment failures are examples of this power-seeking where the agent is trying to become more capable of achieving its goals, whether that is getting out of its box, taking over the world, or even just refusing correction or a shutdown.

If we tie these together, we have something I've called the catastrophic convergence conjecture, which is that if the goals aren't aligned and it causes a catastrophe, it is because of power-seeking.

But I think that when people are first considering alignment, they think to themselves, "What's the big deal? You gave it a weird goal, and it does weird stuff. We’ll just fix that.”

So why exactly do unaligned goal maximizers tend to cause catastrophes? I think it's because of this power seeking. Let me explain.

The way I think about power is as the ability to achieve goals in general.

In the literature, this is like the dispositional power-to notion.

Whereas in the past, people thought "Well, in terms of causality, are the agent's actions necessary and/or sufficient to cause a wider range of outcomes?", here, I think it's best thought of as your average ability to optimize a wide range of different goals. So if you formalize this as your average optimal value in, say, a Markov decision process (MDP), there's a lot of nice properties and you can prove that, at least in certain situations, it links up with instrumental convergence. Power seeking and instrumental convergence are very closely related.

But there's a catch here. We're talking about average optimal value. This can be pretty weird. Let's say you're in the unfortunate situation of having a dozen soldiers about to shoot you. How powerful are you according to average optimal value? Well, average optimal value is still probably quite high.

There's probably an...
Let Your Mind Be Not Fixed
by Gordon Seidoh Worley — 2020-07-31T17:54:43.247Z

When I was a teenager, I knew my mind was not like adults.

Sure, I was like adults in a lot of ways. I believed things. I trusted things. I knew I was right sometimes.

But they didn't change their minds. I did about once a month!

Woe was me. How could I ever be as sure and confident as an adult? I felt inadequate, like a kid going through life dressed up in his father's suit. The closer I got to 18, the more I became worried I was never going to figure it out. I was going to keep bouncing around like a child from one idea to the next, believing whatever good argument I last heard, and never settling into my own confident understanding of life.

Then one day, not too long after legally crossing the threshold into adulthood, I realized the truth.

Those adults were idiots.

Okay, that's a bit strong, but it reflects the magnitude of the realization I had.

I had been trying to pattern match to what I saw adults doing. When I listened to public figures or read history books and biographies and talked to the adults in my life, they all seemed to have their minds made up about a wide variety of topics, and importantly they seemed to keep their minds made up. I, by contrast, kept making and unmaking my mind, and so I thought I was deficient by the standards of adulthood.

What I realized was instead going on was that these adults were stuck. Their minds had become fixed, ideas had been crystalized, and there was no way for them to easily rework the lattice of thought. They could tack stuff on at the edges, sure, but they couldn't tear everything down if it turned out they'd worked themselves into a corner.

This was almost 20 years ago, and I didn't have the advantage of ideas like fixed vs. growth mindset or the correlation of neuroplasticity with age that might have given me some clues about what was going on. Instead I had to figure out for myself that some people lost the ability to significantly rework their understanding of the world when it failed to correlate with reality, and losing this ability wasn't a key feature of being an adult, just something that became more common with age.

So it wasn't that I had to fix my mind to grow up. Instead it was a dangerous trap I was at growing risk of falling into.

Would I?

In some ways, yes, I have, but not for the reasons I was worried about.

It turned out there is more than one way to fix thought.

There's the bad way, where the mind g...
Enforcing Type Distinction
by lionhearted (Sebastian Marshall) — 2020-07-31T11:39:20.026Z

The phrase "Enforcing Type Distinction" was mentioned and fleshed out here 12 years ago, but only appeared exactly once and that exact phrasing didn't catch on.

I think "Enforcing Type Distinction" has value as a concept (to think more clearly), as a phrase (to discuss the thing), and potentially as a community norm (something we could all agree should happen more often, analyze when and where it does not happen in general discourse, and politely encourage each other be more mindful of).

For context, I got there by re-reading Trying to Try:

The first elementary technique of epistemology—it's not deep, but it's cheap—is to distinguish the quotation from the referent. Talking about snow is not the same as talking about "snow". When I use the word "snow", without quotes, I mean to talk about snow; and when I use the word ""snow"", with quotes, I mean to talk about the word "snow".

Which led to The Quotation is not the Referent:

The problem, in my view, stems from the failure to enforce the type distinction between beliefs and things. [...] It is immensely helpful, when one is first learning physics, to learn to keep track of one's units—it may seem like a bother to keep writing down 'cm' and 'kg' and so on, until you notice that (a) your answer seems to be the wrong order of magnitude and (b) it is expressed in seconds per square gram. Similarly, beliefs are different things than planets. If we're talking about human beliefs, at least, then: Beliefs live in brains, planets live in space. Beliefs weigh a few micrograms, planets weigh a lot more. Planets are larger than beliefs... but you get the idea.

After reading those two posts, you get to a familiar concept — words aren't things; a lot of confusion are about words and not things, etc.

Also discussed in Disputing Definitions:

"If a tree falls in the forest, does it make a sound?"

[...]

""Do you think that 'sound' should be defined to require both acoustic vibrations (pressure waves in air) and also auditory experiences (someone to listen to the sound), or should 'sound' be defined as meaning only acoustic vibrations, or only auditory experience?"

Tom Bretton's comment on TQINTR, from 12 years ago, seems like it's worth another look:

The problem is that identity has been treated as if it were absolute, as if when two things are identical in one system, they are i

...
"Go west, young man!" - Preferences in (imperfect) maps
by Stuart_Armstrong — 2020-07-31T07:50:59.520Z

Many people are very nationalistic, putting their country above all others. Such people can be hazy about what "above all others" can mean, outside of a few clear examples - eg winning a total war totally. They're also very hazy on what is meant by "their country" - geography is certainly involved, as is proclaimed or legal nationality, maybe some ethnic groups or a language, or even just giving deference to certain ideals.

Consider the plight of a communist Croatian Yugoslav nationalist during the 1990s...

I'd argue that the situation these nationalists find themselves in - strong views on poorly defined concepts - is the general human state for preferences. Or, to use an appropriate map and territory analogy:

Most people forge their preferences by exploring their local territory, creating a mental map of this, and taking strong preferences over the concepts within their mental map. When the map starts to become imperfect, they will try to extend the concepts to new areas, so that their preferences can also be extended.

Some of the debates about the meaning of words are about this extension-of-preferences process. Scott Alexander recommends that we dissolve concepts such as disease, looking for the relevant categories of 'deserves sympathy' and 'acceptable to treat in a medical way'.

And that dissolving is indeed the correct thing for rationalists to do. But, for most people, including most rationalists, 'sick people deserve sympathy' is a starting moral principle, one we've learnt by example and experience in childhood. When we ask 'do obese people deserve sympathy?' we've trying to extend that moral principle to a situation where our map/model (which includes, say, three categories of people: healthy, mildly sick, very sick) no longer matches up with reality.

Scott's dissolving process requires decomposing 'disease' into more nodes, and then applying moral principles to those individual nodes. In this case, a compelling consequentialist analysis is to look at whether condemnation or praise is effective at changing the condition; ie does fat-shaming people make them less likely to be fat, or others less likely to become fat in the first place? Here the moral principle involved is something like "it's wrong to harm someone (eg through shaming them) if there is no benefit to them or others from doing so".

And that's a compelling moral principle, but it's not th...
Can you gain weirdness points?
by Nicholas / Heather Kross — 2020-07-31T03:41:47.050Z

A useful idea I've been looking into more lately, is "weirdness points". Basically, some good ideas are really unusual. People are often biased against unusual ideas. It's often seen to be easier to fight for one weird thing, than to fight for multiple weird things. Therefore, we ought to prioritize what we fight for, so the most important things get our weirdness points, and tamper down on the weirdness in other areas of our lives/work.

Typical social explanations of weirdness points aren't completely helpful. Power, status, and wealth would seem to bestow weirdness points. But politicians, celebrities, and wealthy people aren't always as free from the weirdness constraint as would be guessed.

Maybe communities and media are fracturing so much that weirdness points are more dependent on your community than your actions. (The social psych idea, "idiosyncrasy credits", is defined in terms of a group's expectations, not those of society-at-large or people-who-are-reachable-but-not-already-on-your-side.)

Weirdness points seem like a valuable (and limited) resource, especially if you are promoting or enacting multiple ideas (A.I. safety and improving rationality and open borders, for example). As with anything valuable to our goals, we ought to figure out if we can get it, and at what cost.

So, the questions for discussion:

What actually determines weirdness points?
Are weirdness points important or useful or accurate, as a predictive model? How constrained are people's actions, really, in promoting weird ideas? In what contexts?
How can one gain weirdness points?
Has anyone at any time "hacked" weirdness points, by successfully promoting multiple weird things / having weird habits / having a weird personality, without eventually running their status/credibility into the ground? (The only person I can think of offhand might be Richard Feynman.)
Free Educational and Research Resources
by Rachel Shu — 2020-07-31T00:24:41.110Z

This is a list of free educational resources that are frequently overlooked or somewhat obscure, but still contain a large trove of information. The selection covers software, lessons, research papers, and reference material. I will skip covering online things that I think most people know, like Wikipedia, Project Gutenberg, or Khan Academy. Where possible I will suggest tax-funded, public domain, or FOSS options. Some of these are easy to use while others, like Zotero, require more of a learning curve. Please reply if you have any suggestions to add to the list; I will add them if they meet those criteria.

Table of Contents

In-Person

Online Education

Multimedia

Books and Audiobooks

Research

Videos

Art

In-Person

Library Card: Everyone should have one, or even multiple; at least have your local branch, and consider finding a way to get a card in a major city, which has more lending options. They're very easy to get, and open up a lot of digital resources, like Libby, RBDigital, and Kanopy, listed below.

Community College Enrollment: Yes, even if you went to an Ivy League. The only reason not to is if you are currently affiliated with a larger college or university. Why? By taking even a single community college course a year you can get longterm access to a lot of student discounts on software and scholarly journals, not to mention museums and such. Not only are the classes cheap or free, but the curriculum quality and even the instructor quality is often comparable to lower-division courses at state universities and private colleges, possibly even better, if the equivalent courses would be taught by a seasoned instructor instead of an overworked grad student. You can fill in gaps in your academic education or satisfy a hobby by taking some sort of evening-class elective. Depending on your community college, maybe you can even qualify for these benefits simply by taking an online course.

Some software that you can get with any valid .edu:

All the Autodesk programs, including Maya and AutoCAD

Windows 10 and Microsoft Office, for most schools in the US and some elsewhere.

FreeStudentSoftware.com has a more complete list.

Don't forget that a lot of people like to help students. If there's something you want and you can't get, try emailing the author or developer, asking if they would make something available to you. If you know anyone who can buy the thing for...
Sunday August 2, 12pm (PDT) — talks by jimrandomh, johnswenthworth, Daniel Filan, Jacobian
by Bird Concept — 2020-07-30T23:55:44.712Z

This Sunday at 12pm (PDT), we're running another session of "lightning talks" by curated LessWrong authors (see here for previous weeks' transcripts).

Each talk will be 3-5 minutes followed by discussion. Afterwards, we'll have a hangout in breakout rooms. The talks will be short and focus on presenting one core idea well, rather than rushing through a lot of content.
We want to give top LessWrong writers an interesting space to discuss their ideas, and have more fruitful collaboration between users. Think of it like a cross between an academic colloquium and some friends chatting by a whiteboard.

If you're a curated author and interested in giving a 5-min talk at a future event, which will then be transcribed and edited, sign up here.

Speakers

jimrandomh

Curated posts: A Significant Portion of COVID-19 Transmission Is Presymptomatic

johnswentworth

Curated posts: Gears-Level Models are Capital Investments, Everyday Lessons from High-Dimensional Optimization, Interfaces as a Scarce Resource, ...

Daniel Filan

Curated post: Insights from 'The Strategy of Conflict'

Jacobian

Curated posts: Winning is for Losers, Seeing the Smoke

Details

When? Sunday August 2, 12pm (PDT)

Where? Zoom.
Dealing with Curiosity-Stoppers
by adamShimi — 2020-07-30T22:05:02.668Z

Introduction

Curiosity is a virtue. It
promotes epistemic honesty, ignites creativity, and improves both competence
and well-being. Multiple posts already discussed
different types of curiosity
the contrast between signalling curiosity and being curious,
the scientific evidence behind curiosity,
why curiosity seemed to leave children,
even the limits of curiosity.

Yet my own issues with curiosity come not with generating it but with
keeping it. Everyday, a myriad of subjects and pieces of content
spark the flame of curiosity in me; also everyday,
recurring thoughts dampen, or sometimes blow out this flame. I thus
need ways to address these thoughts more than techniques to become curious.

What I call curiosity-stoppers -- inspired from but slightly different of this
post
from the Sequences -- brings a lot of negative to my life.
My difficulties to focus on a specific topic of study stems in part from
curiosity-stoppers in the way, which then make almost any other topic
seems more interesting by contrast. When curiosity-stoppers proliferate,
I cannot find anything I'm happy or interested to do, and I feel completely
empty and drained. Even when these thoughts don't overpower me,
they consistently push me to postpone reading, listening, watching or writing content
I'm genuinely curious about and which might improve my life and my research.

I intend this post as an exploration of my own curiosity-stoppers, as well as
my personal counter-measures to each of them. In contrast with
this post,
mine isn't a scientific examination of the question based on an extensive
literature. I merely catalog what I found grappling with my own issues
with curiosity-stoppers. Since I find it difficult to believe I'm a one-of-a-kind
special snowflake, I believe others will share part of my experience. I hope
naming and addressing this issue directly will help them deal with it too.

Definitions

Let's get down to basics and define the main terms.
First is "a discovery": a moment where I discover some new idea about the world
-- every time I understand something new. This includes learning a new mathematical
definition or theorem, understanding how a system works, or figuring out
the meaning behind a text. But also discovering what happens next
in the novel I'm reading, finding the right way to write a specific scene in
a short story, or learning a small trick about how to prove mathemati...
What if memes are common in highly capable minds?
by Daniel Kokotajlo — 2020-07-30T20:45:17.500Z

The meme-theoretic view of humans says: Memes are to humans as sailors are to ships in the age of sail.

If you want to predict where a ship will go, ask: Is it currently crewed by the French or the English? Is it crewed by merchants, pirates, or soldiers? These are the most important questions.

You can also ask e.g. "Does it have a large cargo hold? Is it swift? Does it have many cannon-ports?" But these questions are less predictive of where it will go next. They are useful for explaining how it got the crew it has, but only to a point--while it's true that a ship built with a large cargo hold is more likely to be a merchant for more of its life, it's quite common to encounter a ship with a large cargo hold that is crewed by soldiers, or for a ship built in France to be sailed by the English, etc. The main determinants of how a ship got the crew it currently has are its previous interactions with other crews, e.g. the fights it had, the money that changed hands when it was in port, etc.

The meme-theoretic view says: Similarly, the best way to explain human behavior is by reference to the memes in their head, and the best way to explain how those memes got there is to talk about the history of how those memes evolved inside the head in response to other memes they encountered outside the head. Non-memetic properties of the human (their genes, their nutrition, their age, etc.) matter, but not as much, just like how the internal layout of a ship, its size, its age, etc. matter too, but not as much as the sailors inside it.

Anyhow, the meme-theoretic view is an interesting contrast to the highly-capable-agent view. If we apply the meme-theoretic view to AI, we get the following vague implications:

--Mesa-alignment problems are severe. The paper already talks about how there are different ways a system could be psuedo-aligned, e.g. it could have a stable objective that is a proxy of the real objective, or it could have a completely different objective but be instrumentally motivated to pretend, or it could have a completely different objective but have some irrational tic or false belief that makes it behave the way we want for now. Well, on a meme-theoretic view these sorts of issues are the default, they are the most important things for us to be thinking about.

--There may be no stable objective/goal at all in the system. It may have an object...
Welcome to the SSC Dublin Meetup
by Lucius Bushnaq — 2020-07-30T18:56:36.627Z

We are a small group of about five people. We tend to meet once a month and just spend a few hours talking about SSC related topics. Usually two to three SSC articles are selected in advance as conversation starters.

The meetup dates and times are voted on through in the telegram chat. Just message me and I'll add you.

Everyone is welcome to participate. To quote Scott: "Please don’t feel like you “won’t be welcome” just because you’re new to the blog, demographically different from the average reader, or hate SSC and everything it stands for. You’ll be fine!"
PSA: Tagging is Awesome
by abramdemski — 2020-07-30T17:52:14.047Z

I'd like to supplement the open call for taggers with a few points:

Tagging is a neglected cause area. There is still a huge amount to do, and tagging makes a real difference in making LessWrong content easier to discover, explore, and re-find.
The problem is real. People find LessWrong difficult to read because it is full of deep inferential distances and special jargon. Tags offer an easy way to disambiguate jargon, and reference all the relevant material on a subject.
Tagging isn't just altruistic. Want to promote an idea/topic? Tagging posts and writing a good tag description is a great way to make that topic easier to discover and explore. If writing blobs of text which lots of people later read pumps your ego, tagging is a good way to do that. Write that tag before someone else! But it's also useful -- not just to other people, but also, to yourself. Tagging posts on subjects you love, and upvoting the tag on the most relevant ones, will make it easier for you to reference them later.
You will probably discover things you want to read. Tagging naturally gets you searching for content on LessWrong related to your favorite topics. You are likely to discover than more has been written on these topics than you previously realized.
Tagging is easy. Whenever you think of a tag you want to exist (usually because you're reading a post and decide to tag it with something, only to discover the tag doesn't exist yet), just do a search for that thing on LessWrong and tag all the relevant results! There are other approaches, of course, but if everyone did this, then we'd be in a pretty good position: any new tag created would already be put on most of the relevant posts. (This strategy doesn't work for tags which cover a significant portion of the content on LessWrong, such as the core tags, of course.)

If you're not sure what to tag, take a look at the top posts without tags. You may want to familiarize yourself with the core tags and the concepts portal, so that you're not missing some obvious ones when you tag things.