trentmkelly/LessWrong-43k huggingface dataset:
GPT-X, Paperclip Maximizer? Analyzing AGI and Final Goals
by meanderingmoose — 2020-10-22T14:33:33.105Z

Epistemic status: Low confidence, feel I may be missing something important about how to "put goals into" intelligent systems.

In the artificial intelligence literature, it’s common to see researchers examining the question of “what is the right final goal to give an AGI”. Researchers frequently offer examples of what might happen if we give a superintelligent AGI the wrong final goal; for example, Nick Bostrom zeros in on this question in his book Superintelligence, focusing on a superintelligent AGI with a final goal of maximizing paperclips (it was put into use by a paperclip factory). In his scenario, the AGI interprets its goal literally (as computers tend to do) and turns the entire universe into paperclips. Bostrom shares this story to show that we must be careful with the final goal we put into an AGI system; we’ll want whatever final goal we do put in to be fully representative of human values (and much work has gone into figuring out how to specify them). While Bostrom (and others) are right to call out the slipperiness of language and the importance of exact specification, there’s a deeper question that often seems to be left unaddressed – specifically, whether we’ll be able to give an AGI any final goals at all.

The assumed answer to this question is “yes”, and it’s easy to see why; computers and final goals have gone hand-in-hand since the days of Babbage. We come up with a problem, then structure the computer in a way that solves it, plain and simple. This holds true even for the more complex problems that we’ve recently made progress on, where we don’t know the exact path the computer will take (e.g. training a computer to play Go). For these types of programs, we still formally pose the problem (e.g. minimize Go losses), then let it run an algorithm (e.g. reinforcement learning and gradient descent) which guarantees that the program’s behavior will target the given final goal. These programs are “narrow”, in that they target an exactly specified goal in a limited domain. We have every reason to believe that these “narrow” programs will continue to improve (and they’re already quite powerful, outclassing humans in tasks like Go and object recognition), but they’re in a separate class from generally intelligent programs. To achieve the world-changing impacts that Bostrom and others talk about, we’ll need systems with general intelligence; specifi...
Introduction to Cartesian Frames
by Scott Garrabrant — 2020-10-22T13:00:00.000Z

This is the first post in a sequence on Cartesian frames, a new way of modeling agency that has recently shaped my thinking a lot.

Traditional models of agency have some problems, like:

They treat the "agent" and "environment" as primitives with a simple, stable input-output relation. (See "Embedded Agency.")
They assume a particular way of carving up the world into variables, and don't allow for switching between different carvings or different levels of description.

Cartesian frames are a way to add a first-person perspective (with choices, uncertainty, etc.) on top of a third-person "here is the set of all possible worlds," in such a way that many of these problems either disappear or become easier to address.

The idea of Cartesian frames is that we take as our basic building block a binary function which combines a choice from the agent with a choice from the environment to produce a world history.

We don't think of the agent as having inputs and outputs, and we don't assume that the agent is an object persisting over time. Instead, we only think about a set of possible choices of the agent, a set of possible environments, and a function that encodes what happens when we combine these two.

This basic object is called a Cartesian frame. As with dualistic agents, we are given a way to separate out an “agent” from an “environment." But rather than being a basic feature of the world, this is a “frame” — a particular way of conceptually carving up the world.

We will use the combinatorial properties of a given Cartesian frame to derive versions of inputs, outputs and time. One goal here is that by making these notions derived rather than basic, we can make them more amenable to approximation and thus less dependent on exactly how one draws the Cartesian boundary. Cartesian frames also make it much more natural to think about the world at multiple levels of description, and to model agents as having subagents.

Mathematically, Cartesian frames are exactly Chu spaces. I give them a new name because of my specific interpretation about agency, which also highlights different mathematical questions.

Using Chu spaces, we can express many different relationships between Cartesian frames. For example, given two agents, we could talk about their sum (⊕), which can choose from any of the choices available to either agent, or we could talk about their tensor (⊗), which can accompli...
"Little glimpses of empathy" as the foundation for social emotions
by Steven Byrnes — 2020-10-22T11:02:45.036Z

(Note: I published a much-improved discussion of this topic two years later: See Symbol Grounding & Human Social Instincts. I suggest you read that one instead. By the way, I have started calling it “transient empathetic simulations” instead of “little glimpses of empathy”.)

(Transcript of the little casual talk I gave here.)

I’ve been mulling over the algorithms behind social emotions. How does the brain know when to feel jealous? How does the brain know when to feel guilty? And so on. Knowing the answer would be useful for understanding the human condition, and perhaps also for programming AI's, like if we want to make AI's that are pro-social for the same reason that humans are pro-social—well, at least some humans!—so that the AI’s would wind up with human-like intuitions even in novel circumstances. So again, how does the brain know when to feel jealous? What are the calculations? I’ve looked in the literature, and I haven’t found an answer. The books and papers say things like, “The amygdala helps process emotion”, or whatever. Uhh, OK, what does that mean? The amygdala has 10 million neurons. You can do a lot of calculations with 10 million neurons! So, what are those neurons calculating?

Now, the reason social emotions are especially tricky is that they need to interface with your understanding of the world. And your understanding of the world is some horribly complicated data structure in your brain. Like, my friend Rita just won a trophy and I didn’t, and that makes me jealous. OK. Rita winning the trophy is represented by neuron firing pattern #6,000,000,423. So that’s supposed to trigger the jealousy circuit. How does that work? You can’t just say “The genome wires them up,” because, how does the genome know to do that? How many of my ancestors were friends with Rita when she won a trophy? None! There has to be more to the story than that. And likewise, you can’t just say “A learning algorithm will figure out the connection,” because there’s no ground truth about when you should be jealous. There's ground truth over evolutionary time, but there's no ground truth within a lifetime. So how do you learn it?

I couldn’t find anyone talking about this in a way that made sense to me, so eventually I paused my literature search, in favor of just making things up. A proud tradition here at lesswrong! I’m at the stage of trying to formulate a hypothesis. T...
The date of AI Takeover is not the day the AI takes over
by Daniel Kokotajlo — 2020-10-22T10:41:09.242Z

Instead, it’s the point of no return—the day we AI risk reducers lose the ability to significantly reduce AI risk. This might happen years before classic milestones like “World GWP doubles in four years” and “Superhuman AGI is deployed."

The rest of this post explains, justifies, and expands on this obvious but underappreciated idea. (Toby Ord appreciates it; see quote below). I found myself explaining it repeatedly, so I wrote this post as a reference.

AI timelines often come up in career planning conversations. Insofar as AI timelines are short, career plans which take a long time to pay off are a bad idea, because by the time you reap the benefits of the plans it may already be too late. It may already be too late because AI takeover may already have happened.

But this isn’t quite right, at least not when “AI takeover” is interpreted in the obvious way, as meaning that an AI or group of AIs is firmly in political control of the world, ordering humans about, monopolizing violence, etc. Even if AIs don’t yet have that sort of political control, it may already be too late. Here are three examples: [UPDATE: More fleshed-out examples can be found in this new post.]

Superhuman agent AGI is still in its box but nobody knows how to align it and other actors are going to make their own version soon, and there isn’t enough time to convince them of the risks. They will make and deploy agent AGI, it will be unaligned, and we have no way to oppose it except with our own unaligned AGI. Even if it takes years to actually conquer the world, it’s already game over.

Various weak and narrow AIs are embedded in the economy and beginning to drive a slow takeoff; capabilities are improving much faster than safety/alignment techniques and due to all the money being made there’s too much political opposition to slowing down capability growth or keeping AIs out of positions of power. We wish we had done more safety/alignment research earlier, or built a political movement earlier when opposition was lower.

Persuasion tools have destroyed collective epistemology in the relevant places. AI isn’t very capable yet, except in the narrow domain of persuasion, but everything has become so politicized and tribal that we have no hope of getting AI projects or governments to take AI risk seriously. Their attention is dominated by the topics and ideas of powerful ideological factions tha

...
Learning is (Asymptotically) Computationally Inefficient, Choose Your Exponents Wisely
by Shmi — 2020-10-22T05:30:18.648Z

Epistemic status: A speculation based on personal observations, but looks coherent to me. (Sorry, no links or literature search)

Summary: Studying a subject gets progressively harder as you learn more and more, and the effort required is conjectured to be exponential or worse, which means that sooner or later you hit the limit of what you can learn in a reasonable amount of time. If that limit is not where your skills/knowledge are yet useful, you have basically wasted your time compared to learning something else instead. Consider evaluating your aptitudes and the usability threshold before investing the proverbial 10,000 hours into something specific.

I have noticed this over and over, in my own life as well as watching and teaching others: It is usually not too hard to learn the very basics of something, be it math, music or basketball, and for awhile the effort you put in brings tangible rewards in terms of deeper understanding, better skills and more fluency. Different people, however, need a different amount of effort in different areas per amount of material learned (and retained), right off the bat. (This part seems pretty uncontroversial, unless one is ideologically against admitting that not all people are born all around equal, and that just because they aren't, it doesn't mean that it implies differences in moral weight or something.) This initial part of learning is basically linear: the gain is roughly proportional to the effort you put in. That initial "honeymoon" phase tends to peter out eventually.

Sooner or later, one notices that to make the same apparent amount of progress you have to put more effort than you used to. Maybe it's solving an equation that makes you sweat more, maybe you struggle to play a particularly challenging passage, maybe a certain dribbling drill leaves you frustrated. You can still get through it, but the process not nearly as much fun or easy as it used to feel. Maybe getting over the hump lets you make progress easier for awhile. Maybe drawing on your skills in other areas helps you get over the hump, if you are lucky. But still, the more you learn, the harder the learning process becomes, if not monotonically so, than on average.

My actual claim is that learning becomes not just harder linearly, or quadratically (e.g. when you have to spend, say, an extra hour on learning the same amount of new material compare...
Babble challenge: 50 ways of solving a problem in your life
by Bird Concept — 2020-10-22T04:49:42.661Z

Back again. Let’s

become stronger

.

This week’s challenge:

Years ago you found yourself hurled into existence, facing a vast universe with a mind capable of the Art of Rationality, reading a LessWrong post at this very moment.

Yet in your life there is a particular problem. I don’t know what it is. Maybe your chair is uncomfortable; you’re not getting as high scores as you want at the Math Olympiad; or you’ve got insomnia.

Whatever it is, pick one specific problem in your life.

Find a way to solve it.

You have 1 hour to come up with 50 ways.

(But no need to implement the solutions within 1 hour!)

Looking back

Here are the champions who made it to 50 last week, with stars indicating their streak:

★★★ Slider, gjm, Harmless, jacobjacob, Tetraspace Grouping

★★ athom, johnswentworth, ryan_b, Ericf, Bucky, Mark Xu, CptDrMoreno, Yonge

★ TurnTrout, Tighe, knite

Why measure streaks?

Last week Bucky commented:

I don't like measuring things by streaks - if you want to do a list I think doing it by total number of challenges completed is better. Streaks are a less accurate indication of effort put in or potential gains achieved and have more potential to create unhealthy incentives.

But I disagree. I replied:

One of the goals of the challenge is building a culture of practice. I think consistency is an incredibly important part of that. That's how you get compound returns. A portfolio that grows 7% every year will grow ~30x over fifty years. But a portfolio that grows that much only every other year will only grow about ~5x. (Even though the first one only put in "twice as much effort".)

Moving forwards

I’m now entering week 4 out of the 7-week babble streak I committed to. If you want more regularity in practicing your creativity, feel free to post a comment committing to also going all the way to 7.

This week we’re trying something new: applied babble. I haven’t tried it before, so am very curious to see what will happen. Feel free to add a note to your comment about how useful you found the exercise, and whether you thought about good things you hadn’t considered before.

Rules

50 answers or nothing. Shoot for 1 hour.

Any answer must contain 50 ideas to count. That’s the babble challenge.

However, the 1 hour limit is a stretch goal. It’s fine if it takes longer to get to 50.

Post your answers inside of spoiler

...
Virtue ethics and why the rationalist community might care about it.
by David Gross — 2020-10-22T03:53:28.988Z

This post introduces “virtue ethics” and explains why you might find it worth pursuing.

Virtue ethics is meant to be of practical help

The two main forks of the rationalist endeavor are clear thinking and effective action. Virtue ethics addresses both of these: Clear thinking is the result of honing the “intellectual virtues” and effective action typically involves the skillful use of various other virtues.

Ethics, said Aristotle, “differs from the other [philosophical subjects] in not being a subject of merely intellectual interest — I mean we are not concerned to know what goodness essentially is, but how we are to become good, for this alone gives the study its practical value.”[1]

Where virtue ethics fits alongside its alternatives

Since Aristotle’s time, however, ethical philosophy took a turn for the “meta” — arguing interminably about fascinating things like the nature of good and evil and the plausibility of free will — and losing much of its practical focus along the way. The two main families of ethical approaches that developed from all of that wrangling are usually called deontological and consequentialist. Deontologists hope that we can discern — whether by logical necessity or via revelation — some articulable rules of behavior by which everyone ought to govern their lives. Consequentialists urge us instead to carefully anticipate the likely results of our possible choices and to make decisions such that those results are optimal by some metric or other.

(One branch of the consequentialist family tree leads to the effective altruism movement in which the rationalist community has done much fruitful work, and which adds a welcome dose of the practical to what had become a very theoretical discipline.)

These approaches to ethics are not rigidly exclusive. They tend to inform each other, and as a practical matter most of us tend to mix and match from them rather than dogmatically sticking to one or another. Virtue ethics also can be a complement to either a deontological or consequentialist ethics. If you’re a deontologist, you need wisdom and insight to discover the rules, and self-control to resist the temptations to break them — and those are virtues that virtue ethics can help you with. If you’re a consequentialist you need discernment to understand the situation you are in, imagination to come up with the full range of alternative courses of action,...
Rationalist Town Hall: Pandemic Edition
by Ben Pace — 2020-10-21T23:54:03.528Z

On Sunday 1st November at 12 pm PT, Divia Eden, Anna Salamon, Zvi Mowshowitz, Oliver Habryka and I (Ben Pace) are hosting a Town Hall at this Zoom link, for rationalists to share models of the ongoing pandemic and how we’re responding to it.

Topic of the Town Hall

We’re 8 months into a global pandemic, and there’s much to discuss, such as:

What are the likely trajectories for the pandemic to take over the next year?
What personal risk-avoiding measures should people be taking?
How is the pandemic affecting the rationality community?
What have rationalists done in response to the pandemic?

We will give some prepared thoughts, but we’re interested in hearing from many others in and around the rationality community about their thoughts, updates and plans.

It’s also the US election 2 days after the event. Several people have discussed tail-risk outcomes post-election, so it seems like a natural time to bring that up, get some clarity over the concerns, discuss whether they’re worth preparing for and what any such preparation would look like, so there’ll be some discussion of this towards the end of the event.

Who should attend? And how do I take part?

This is an event for the LessWrong and rationality community and those around it. You are welcome to attend even if you do not think of yourself as 'a rationalist'. For example, if you nonetheless care about the community, or think you may find the discussion personally valuable.

The majority of content during the Town Hall will be determined by the discussion here. If you have something you’d like to contribute to the Town Hall, please write it as an answer to this post. It could be something you’d like to let everyone know about, a question you’d like to see answered by the panel, or an issue you’d like to raise to the community.

A week from today, on Wednesday 28th October at 13:00PM (PT), the hosts will pick the speakers and topics for the event, informed by the Answers on this post. For a number of thoughtful and/or highly-upvoted submissions, we will invite the authors to speak at the event and take questions.

Most people will not get a chance to speak during the Zoom call. If you would like to talk further after that the Town Hall, the LessWrong Walled Garden will be open until 4pm (PT) for people to continue discussion, and many of us will be there to talk with.

Please RSVP with this Google Form, and/or sign up to the Fa...
Malicious link!
by bmeacham — 2020-10-21T23:24:24.539Z

Not sure how to report this, so posting as a question. On the page titled "All Debates Are Bravery Debates" there is a link to http-colon-slash-slash-raikoth.net/libertarian.html. The link text is "an essay arguing against libertarianism." On Windows, that link goes to a page that asks me to download something. I immediately backed out of there. On Linux it goes to a page titled "raikoth.net" with a bunch of links on it. I have not tried it on Mac. I strongly urge you to remove that link, as it appears to be malicious.
What's the difference between GAI and a government?
by DirectedEvolution — 2020-10-21T23:04:01.319Z

I have zero technical AI alignment knowledge. But this question has kept recurring to me for like a year now so I thought I'd ask.

A lot of the arguments for the danger of GAI revolve around the notion that an agent that is smarter than a human is un-boxable, self-creating and self-enhancing, and not necessarily aligned with human interests.

That pattern-matches very well onto "governments," "corporations," and other forms of collective agencies. They have access to collective intelligence far beyond what's accessible to an individual. That intelligence brings them power beyond the ability of even the most clever individual to avoid long-term. Their goals aren't necessarily aligned with human values. They use their intelligence and power to enhance their own intelligence and power. They're not always successful, but they are often able to learn from their mistakes. If one agency destroys itself, another takes its place.

How much bearing does this have on technical AI alignment work? Can AI alignment work translate into solutions for the problems we presently have in aligning these agencies to human values? Do the restraints that have so far prevented governments/corporations from paperclipping the world map onto any proposed strategies for AI alignment?