trentmkelly/LessWrong-43k huggingface dataset:
The ignorance of normative realism bot
by Joe Carlsmith — 2022-01-18T05:26:07.676Z

(Cross-posted from Hands and Cities)

And you want to travel with her, and you want to travel blind
And then you know that she will trust you
For you’ve touched her perfect body with your mind

— Leonard Cohen, “Suzanne”

Non-naturalist normative realism claims that there are objective normative facts that are irreducibly “over and above” facts about the natural world (see here for more details). This post lays out what I see as the most important objection to such realism: namely, that it leaves us without the right type of epistemic access to the normative facts it posits.

To illustrate the point, I discuss various robots in unfortunate epistemic situations. Non-naturalist realism, I claim, makes ours analogous.

Thanks to Ketan Ramakrishnan and Katja Grace for discussion. My views on this topic have been especially influenced by the work of Sharon Street.

I. What’s in the envelope?

In a previous post, I discussed a robot whose value system is built to assume the existence of a certain envelope, inside of which is written an algorithm for scoring the “goodness” of a given world. The robot’s objective is to maximize the goodness of its world, as scored by the envelope. Call such a robot an “envelope bot.”

In some situations, this robot is totally screwed. Here’s one. Suppose that Rosie the envelope bot wakes up in a simulation created by Eccentric Ed. Ed’s avatar appears before Rosie and tells her:

“Outside the simulation, back in my office, I have the envelope on my desk. No one has ever opened the envelope, including me, and no one knows what process determined its contents.

Separately, though, I’ve given you a bunch of “evaluative attitudes” – e.g., intuitions about what’s in the envelope, desires to do different things, experiences of different things as “calling to you” or “to be done,” and so on. I chose the content of these by calling up Angela Merkel and asking her what kind of stuff reminds her of apple pie. Turns out Angela has a weird association between apple pie and helium, so you ended up with a set of evaluative attitudes that tend to favor maximizing the amount of simulated helium, albeit in somewhat incoherent ways.

You’ve got about 100 simulated years to live. After that, I’ll bring you out of the simulation, open up the envelope, and tell you how much you improved the world by its lights.”

My first claim is that Rosie, here, is in a bad w...
How to Build New Countries à la 1729?
by jdcampolargo — 2022-01-17T22:53:46.448Z

Originally published here.

How do you build a country? Is that even something you build? We don’t know but we may have to, and this essay will explain how to achieve that.

We live in a world where it might be faster and cheaper to start a new country than for the FDA to approve a drug. Yes, it may take us less than twelve years and less than $1 billion

1

.

1729 is laying down the foundations for what they may look like.

Balaji

calls it, “The Network State,” which he envisions will be a country you can start from your computer so nations are built from the internet rather than disrupted by it.

“A network state is a social network with a clear leader, an integrated cryptocurrency, a definite purpose, a sense of national consciousness, and a plan to crowdfund territory.”

I want to explain the fundamental concepts to build the Network State but before we do that, let’s talk about how to start a country, where we’ll be located, and our plan to get recognized by the United Nations.

Building the Network State

Building the network state is how we start building a crypto-civilization. A new world where anyone can decide to start or join a country. It will be similar to creating a company where you can create one, or join one as an employee. This new world will be a place to advance free speech, free market, equality of opportunity, privacy, and prosperity.

This will be a place to combine collectivism and individualism to pursue a higher purpose decided by all. Purposes could be setting up a colony in the Moon, only-flying cars cities, figuring out how to live forever, and any other ambitious goal.

“Ok, I get it but how do you actually start a country?”

7 Ways to Start a Country

There are six known ways to start a country. With the additional one, being the one 1729 is pursuing.

(For more details on the ways to start a country,

here

)

Here, we introduce cloud countries. The idea is simple. Cloud-first, land last.

We start with what you cannot buy, community. We recruit a group of people who want to build a new country. This will be similar to ethnic diasporas, which are spread all over the world and connected by communications channels and the motherland.

For instance, the Venezuelan diaspora is everywhere but we’re connected because we’re Venezuelans and we can decide to be connected through organizations like Code for Venezuela (technolo...
The unfalsifiable belief in (doomsday) superintelligence scenarios?
by Hickey — 2022-01-17T21:02:21.445Z

Having read Bostrom's superintelligence book and a couple of adjacent papers by him and Yudkowsky. There is no clear line of argument from beginning to end but rather a disjunctive list of possibilities that all lead to similar extinction events. This leads the entire theory to not be falsifiable, cut off one road toward superintelligence or a related extinction outcome and a new one will pop up. Whatever amount of evidence mounted against it will never be enough.

Is there an existing response to this problem that someone could point me to? Either the unfalsifiability not being a problem or some key detail in the argument that I have missed.
Scalar reward is not enough for aligned AGI
by Peter Vamplew — 2022-01-17T21:02:16.106Z

This post was authored by Peter Vamplew and Cameron Foale (Federation University), and Richard Dazeley (Deakin University)

Introduction

Recently some of the most well-known researchers in reinforcement learning Silver, Singh, Precup and Sutton published a paper entitled Reward is Enough, which proposes the reward-is-enough hypothesis: “Intelligence, and its associated abilities, can be understood as subserving the maximisation of reward by an agent acting in its environment". Essentially, they argue that the overarching goal of maximising reward is sufficient to explain all aspects of natural and artificial intelligences.

Of specific interest to this forum is the contention that suitably powerful methods based on maximisation of a scalar reward (as in conventional reinforcement learning) provide a suitable pathway for the creation of artificial general intelligence (AGI). We are concerned that the promotion of such an approach by these influential researchers increases the risk of development of AGI which is not aligned with human interests, and this led us to work with a team of collaborators on a recent pre-print Scalar Reward is Not Enough which argues against the assumption made by the reward-is-enough hypothesis that scalar rewards are sufficient to underpin intelligence.

The aim of this post is to provide an overview of our arguments as they relate to the creation of aligned AGI. In this post we will focus on reinforcement learning methods, both because that is the main approach mentioned by Silver et al, and also because it is our own area of expertise. However the arguments apply to any form of AI based on maximisation of a numeric measure of reward or utility.

Does aligned AGI require multiple objectives?

In discussing the development of intelligence, Silver et al argue that complex, general intelligence may arise from the combination of complex environments and simple reward signals, and provide the following illustrative example:

“For example, consider a signal that provides +1 reward to the agent each time a round-shaped pebble is collected. In order to maximise this reward signal effectively, an agent may need to classify pebbles, to manipulate pebbles, to navigate to pebble beaches, to store pebbles, to understand waves and tides and their effect on pebble distribution, to persuade people to help collect pebbles, to use tools and vehicles

...
Is there a good way to read deep into LW comment histories on mobile?
by Maxwell Peterson — 2022-01-17T19:02:31.140Z

I sometimes like going through and reading a user’s comments, for those who make particularly good ones. But my mobile browser (Safari) loves refreshing pages when I haven’t been to the page for an hour. When this happens, I have to re-scroll down, and my current expedition requires minutes of scroll-click Load More-scroll to get back to where I was. I think Chrome on mobile does this too. Does anyone have any ideas on how I can do this better? (I really prefer reading on my phone).
How I'm thinking about GPT-N
by delton137 — 2022-01-17T17:11:49.447Z

There has been a lot of hand-wringing about accelerating AI progress within the AI safety community since OpenAI's publication of their GPT-3 and Scaling Laws papers. OpenAI's clear explication of scaling provides a justification for researchers to invest more in compute and provides a clear path forward for improving AI capabilities. Many in the AI safety community have rightly worried that this will lead to an arms race dynamic and faster timelines to AGI.

At the same time there's also an argument that the resources being directed towards scaling transformers may have counter-factually been put towards other approaches (like reverse engineering the neocortex) that are more likely to lead to existentially dangerous AI. My own personal credence on transformers slowing the time to AGI is low, maybe 20%, but I think it's important to weigh in.

There is also a growing concern within the AI safety community that simply scaling up GPT-3 by adding more data, weights, and training compute could lead to something existentially dangerous once a few other relatively simple components are added.

I have not seen the idea that scaling transformers will lead to existentially dangerous AI (after combining with a few other simple bits) defended in detail anywhere but it seems very much an idea "in the water" based on the few discussions with AI safety researchers I have been privy too. It has been alluded to various places online also:

Connor Leahy has said that a sufficiently large transformer model could serve as a powerful world model for an otherwise dumb and simple reinforcement learning agent, allowing it to rapidly learn how to do dangerous things in the world. For the record, I think this general argument is a super important point and something we should worry about, even though in this post I'll mainly be presenting reasons for skepticism.

Gwern is perhaps the most well-known promoter of scaling being something we should worry about. He says "The scaling hypothesis regards the blessings of scale as the secret of AGI: intelligence is ‘just’ simple neural units & learning algorithms applied to diverse experiences at a (currently) unreachable scale."

Observe the title of Alignment Newsletter #156: "The scaling hypothesis: a plan for building AGI". Note: I'm not sure what Rohin Shah's views are exactly, but from what I read they are pretty nuanced.

Zac Hatfield-Dodd

...
Truthful LMs as a warm-up for aligned AGI
by Jacob_Hilton — 2022-01-17T16:49:24.270Z

This post is heavily informed by prior work, most notably that of Owain Evans, Owen Cotton-Barratt and others (

Truthful AI

), Beth Barnes (

Risks from AI persuasion

), Paul Christiano (unpublished) and Dario Amodei (unpublished), but was written by me and is not necessarily endorsed by those people. I am also very grateful to Paul Christiano, Leo Gao, Beth Barnes, William Saunders, Owain Evans, Owen Cotton-Barratt, Holly Mandel and Daniel Ziegler for invaluable feedback.

In this post I propose to work on building competitive, truthful language models or truthful LMs for short. These are AI systems that are:

Useful for a wide range of language-based tasks
Competitive with the best contemporaneous systems at those tasks
Truthful in the sense of rarely stating

negligent falsehoods

in deployment

Such systems will likely be fine-tuned from large language models such as GPT-3, hence the name.

WebGPT

is an early attempt in this direction. The purpose of this post is to explain some of the motivation for building WebGPT, and to seek feedback on this direction.

Truthful LMs are intended as a warm-up for aligned AGI. This term is used in a specific way in this post to refer to an empirical ML research direction with the following properties:

Practical. The goal of the direction is plausibly achievable over the timescale of a few years.
Valuable. The direction naturally leads to research projects that look helpful for AGI alignment.
Mirrors aligned AGI. The goal is structurally similar to aligned AGI on a wide variety of axes.

The remainder of the post discusses:

The motivation for warm-ups (

more

)
Why truthful LMs serve as a good warm-up (

more

)
The motivation for focusing on negligent falsehoods specifically (

more

)
A medium-term vision for truthful LMs (

more

)
How working on truthful LMs compares to similar alternatives (

more

)
Common objections to working on truthful LMs (

more

)

Warm-ups for aligned AGI

There are currently a number of different empirical ML research projects aimed at helping with AGI alignment. A common strategy for selecting such projects is to select a research goal that naturally leads to helpful progress, such as

summarizing books

or

rarely describing injury in fiction

. Often, work on the project is output-driven, taking a no-holds-barred approach to achieving the selected goal, which has a number of advantages that aren't discussed here. On the other hand, goal selection...
Different way classifiers can be diverse
by Stuart_Armstrong — 2022-01-17T16:30:04.977Z

With thanks to Lee Sharkey and Michael Cohen for the conversations that lead to these ideas.

In a previous post, I talked about how we could train classifiers on the same classification problem - a set of lions vs a set of huskies - but using different approaches to classify.

What we want is something we can informally call a 'basis' - a collection of classifiers that are as independent of each other as possible, but that you can combine to generate any way of dividing those two image sets. For example, we might have a colour classifier (white vs yellow-brown), a terrain classifier (snow vs dirt), a background plant classifier, various classifiers on the animals themselves, and so on. Then, if we've done our job well, when we find any not-too-complex classifier Cn, we can say that it's something like '50 colour, 60% nose shape and −10% plant[1]'.

We shouldn't put too much weight on that analogy, but we do want our classifiers to be independent, each classifier distinct from anything you can construct with the all others.

Here are four ways we might achieve this this.

Randomised initial seeds

An easy way of getting an ensemble of classifiers is to have bunch of neural nets (or other classification methods), initialise them with different initial weights, and train them on the same sets. And/or we could train them on different subsets of the lion and husky sets.

The advantage of this method is that it's simple and easy to do - as long as we can train one classifier, we can train them all. The disadvantage is that we're relying on luck and local minima to do the job for us. In practice, I expect these methods to all converge to "white vs yellow-brown" or similar. Even if there are local minima in the classification, there's no guarantee that we'll find them all, or even any. And there's no guarantee that the local minima are very independent - 99.9% colour and 0.01% nose shape might be a local minima, but it's barely different from a colour classifier.

So theoretically, this isn't sound; in practice, it's easy to implement and play around with, so might lead to interesting insights.

Distinct internal structure

Another approach would be to insist the classifiers internal structures are distinct. For example, we could train two neural net classifiers, C1 with weights →w1 and C2 with →w2. They could be trained to minimise their individual classification losses and regul...
PIBBSS Fellowship: Bounty for Referrals & Deadline Extension
by Anna Gajdova — 2022-01-17T16:23:56.912Z

Crossposted to EA Forum
TL;DR: 500 USD bounty for successful applicants referrals

Background

The PIBBSS Summer Research Fellowship is seeking applicants studying complex and intelligent behavior in natural and social systems, such as ecology, evolutionary biology, cognitive science, neuroscience, sociology, legal theory, political economy, statistical mechanics, social anthropology, linguistics, media studies, and more.

Accepted fellows will conduct AI alignment research under the mentorship of experienced alignment researchers.

You can learn more at

https://www.pibbss.ai/

or by reading the post

Introducing the Principles of Intelligent Behaviour in Biological and Social Systems (PIBBSS) Fellowship

.

The extended application deadline is Jan 23rd.

Bounty

We are offering a bounty of 500 USD for each applicant you refer who we end up accepting to the program.
We are offering a bounty of 100 USD for each applicant you refer, and we end up interviewing (i.e. they get to stage 3 of the application process).

We expect to accept 5-15 fellows and to interview 20-40 applicants.

What does count as a referral?

Roughly, it means that you were either the first one who informed someone about the fellowship or that your encouragement was the main reason they applied.

How can I claim the bounty?

Fill in

this form

and list the people you’ve referred for the fellowship, you can add details of your interactions. You should also encourage the people you referred to mention you in their application (we ask them how they learned about the fellowship). We’ll also try reaching out to people mentioned in applications, even when they don’t claim the bounty.

In case the people you referred succeed, we’ll get in touch with you after the end of the application process in mid-March. The final decision about bounty eligibility is ours.

Bounty can also be claimed retroactively, i.e. when you’ve already referred someone, and they applied.

Questions?

If you have any questions about the bounty or the fellowship, you can ask in the comments or reach out to

anna@pibbss.ai

or

contact@pibbss.ai

.
Polymarket Covid-19 1/17/2022
by Zvi — 2022-01-17T16:10:01.246Z

Polymarket’s prediction markets are back and they are more plentiful than ever, both on Covid-19 and otherwise. Americans are not eligible to trade, but plenty of my readers reside elsewhere. And even if you can’t trade, the information component is still free.

Liquidity in any given market is not yet as high as I’d like, but it is still high enough to very much count as real money, provide insights on real problems and allow the astute reader who is eligible to participate to perhaps pick up some value.

To celebrate having all these new markets, and to take advantage of the information and the intuition pumps, I’ll go over what’s available, and say how far I’d be willing to trade a given contract or set of contracts on the assumption that I had access to the markets plus an adequate bankroll.

I’m going to limit this post to Covid-19 markets, because there’s enough value out there that I don’t want to postpone posting while I go through the other stuff.

One problem with the old Buy/Sell/Hold is that it is a conservative system. If something is trading at 50% and I say I’d buy it to 60%, I’m doing that because even after adjusting for the market’s opinion I still think the probability is substantially above 60%. Sometimes I’m uncertain for some reason and I’m avoiding making big mistakes, but my gut says it is much higher. Other times I’m not uncertain, so I’m looking for a smaller edge. This makes it tricky when the time comes to assign a Briar score or otherwise evaluate predictions mathematically.

My solution to that is that I’ll be including my fair price explicitly, in addition to what I would do, while keeping in mind that they are not always well-considered. Then one can either evaluate market performance (and stop where I say to stop) or go full Green Knight and see if my full estimates are superior.

And of course, note all of these carefully:

Markets tend to improve in accuracy over time.
That means lack of market movement over time is (some) evidence of accuracy.
Others are also reading this, so if you don’t get to it fast, that’s adverse selection for you.
If liquidity has gone up, that’s also evidence of accuracy, more adverse selection.
Many of these markets are time-sensitive, and their fair value changes over time rapidly even if nothing happens.
None of this is investment advice!

Without further ado, let’s see what we’ve got. Keep in mind, ...
Value Notion - Questions to Ask
by aysajan — 2022-01-17T15:35:16.293Z

Continuing my

previous post

on the subject of value notion, in this post I will try to make another small step towards a big goal of value notion. Specifically, I attempt to explore what kind of major questions we can ask about the value of something. The list may not be comprehensive, but I believe those questions are important ones.

Let’s start with an example. If I want to understand the value of an individual, for instance, Elon Musk, I can look at several things. First, what is the main purpose of Elon Musk? We can say his purpose is to make human life multiplanetary or save humanity from AI dominance or simply be a good father. Some purposes are more valuable than others. For instance, in general an individual with good deeds is more valuable than someone who wants to harm others.

Next important question is: the value to whom? Who/what is at the receiving end? Depending on the object, the value might be different. I can say that my value to my family is bigger than my value to a random person on the street. Additionally, this “whom” does not need to be a person (and that’s why “willingness to pay” is not general enough). For instance, the beneficiary of a tree in front of my house could be the soil (I can also claim that the tree is the beneficiary of the soil), the birds that set a nest, or a squirrel playing on that tree.

Third question is: the value at what time? The same thing might have different values at different times. For instance, my value as an adult is bigger than my value as a child. As an adult, my family can count on me, and my employer can count on me on certain things. If I am a child, I usually count on someone else to take care of me.

In general, there are at least three elements that need to be evaluated when we want to answer the value question: (1) main purpose, (2) the receiver, and (3) time. The value of something with the main purpose of X to Y at time Z might be very different from the value of the same thing to Y at time C, which again might be very different from the value of the same thing to B at time Z. But I feel like it is too early (and might be wrong) to claim that the value of something is a function of main purpose, receiver, and time. More on this in later posts.

More examples

Let’s look at some more examples, in which the object is not a human being.

Value of a book. Let’s take an economic book as an e

...