trentmkelly/LessWrong-43k huggingface dataset:
Bay Solstice 2022 Call For Volunteers
by Scott Alexander — 2022-09-04T06:44:19.043Z

It's September, which means going to beaches, holding barbecues, and . . . preparing for this year's Bay Area Winter Solstice. This year's team will include me, Shauna, Nova, and Mingyuan. In the meantime, we're looking for volunteers.

First, we have logistics volunteers. Some of these, especially AV and lighting, will require a commitment to be kind of agenty in figuring out what you need to do, attending dress rehearsals to help test things, etc. The simple tasks will just require you to show up day of. If you want to help with any of the following logistics tasks, please email smkravec@celest.ai:

Someone to monitor the Twitch stream/simulcast and give updates to AV on that.
Someone to work closely with Nova on the video side of AV; filming the event and running the live stream (possibly that's worth a separate role, they'd be responsible for the audio submix to the stream)
Someone to help setup and run lighting
5-10 people who can help with simple tasks on the day of, like ushering, taking tickets, and moving large objects

Second, we have creative tasks. We seem pretty good for people willing to do creative tasks this year, so you don't need to volunteer unless you're really enthusiastic. If you do volunteer, expect to have to audition. If you're interested in any of the following creative tasks, please email scott@slatestarcodex.com:

Giving a speech (either writing your own, or giving a reading selected by the team)
Singing a song (either your own, or performing one selected by the team)

We'll give volunteers more information by email, and we'll try to give everyone more information as we get closer to the event.

Thanks!
The shard theory of human values
by Quintin Pope — 2022-09-04T04:28:11.752Z

TL;DR: We propose a theory of human value formation. According to this theory, the reward system shapes human values in a relatively straightforward manner. Human values are not e.g. an incredibly complicated, genetically hard-coded set of drives, but rather sets of contextually activated heuristics which were shaped by and bootstrapped from crude, genetically hard-coded reward circuitry.

We think that human value formation is extremely important for AI alignment

. We have empirically observed

exactly one

process which reliably produces agents which intrinsically care about certain objects in the real world, which reflect upon their values and change them over time, and which—at least some of the time, with non-negligible probability—care about each other. That process occurs millions of times each day, despite genetic variation, cultural differences, and disparity in life experiences. That process produced you and your values.

Human values look so strange and inexplicable. How could those values be the product of anything except hack after evolutionary

hack

? We think this is not what happened. This post describes the shard theory account of human value formation, split into three sections:

Details our working assumptions about the learning dynamics within the brain,
Conjectures that reinforcement learning grows situational heuristics of increasing complexity, and
Uses shard theory to explain several confusing / “irrational” quirks of human decision-making.

Terminological note: We use “value” to mean a contextual influence on decision-making. Examples:

Wanting to hang out with a friend.
Feeling an internal urge to give money to a homeless person.
Feeling an internal urge to text someone you have a crush on.
That tug you feel when you are hungry and pass by a donut.

To us, this definition seems importantly type-correct and appropriate—see Appendix A.2. The main downside is that the definition is relatively broad—most people wouldn’t list “donuts” among their “values.” To avoid this counterintuitiveness, we would refer to a “donut shard” instead of a “donut value.” (“Shard” and associated terminology are defined in section II.)

I. Neuroscientific assumptions

The shard theory of human values makes three main assumptions. We think each assumption is pretty mainstream and reasonable. (For pointers to relevant literature supp...
Breaking Newcomb's Problem with Non-Halting states
by Slimepriestess — 2022-09-04T04:01:04.350Z

Note: I dug around quite a bit and haven't found anyone who previously discussed this, but if it's been talked about previously somewhere links would be appreciated.
Epistemic Status: Interesting Conjecture, I think my logic is all sound but please check me

I think I discovered a toy decision theory agent that generates non-halting states under certain conditions and is capable of breaking various game theory problems in some novel ways.

Imagine an idealized agent employing what I'm informally calling conditional decision theory. Lets name her Marion. Marion goes into Newcomb's Problem with the following algorithm:

"If I Predict that the Predictor has put money in both boxes, I will take both boxes, because that would result in gaining $1,001,000, however, if I predict that the Predictor has only put money in one box, I will only take box B, since this will result in the accurate prediction that I will take one box, which will mean that both boxes contain money and allow me to take both boxes."
The Predictor wants to fill Box B conditional on its prediction that Marion doesn't also take Box A. Marion wants to take Box A conditional on her prediction that Box B is filled.

If Marion takes both boxes, The Predictor will have predicted that she takes both boxes, causing it to not fill box B. Marion knows that the Predictor will predict this, so she only takes box B. The Predictor knows Marion will only take Box B, therefore box B is filled. Marion knows this and thus takes both boxes. The Predictor knows this so it leaves box B empty. Marion knows box B will be empty and thus only takes Box B. The Predictor knows this so it fills box B. Marion knows this so she takes both boxes.

This is, as far as I can tell, non-halting under the premise that both agents have identical predictive power. Marion can't win using this algorithm, but she can keep the Predictor from winning. If Marion and The Predictor have identical predictive and reasoning capability, then the answer as to "What is the outcome?" is unfalsifiable, the retrocausal inferences recurse to infinity and neither one of them can accurately predict the other.

This only works assuming you're basically playing two Predictors against one another, in the real world the predictive errors stack up with each successive recursion until one of them falls out of equilibrium and fails to accurately predict the othe...
Monthly Shorts 8/22
by Celer — 2022-09-04T02:30:18.748Z

Politics

In a 2017 analysis of 229,000 FOIA requests, those from journalists accounted for just 8 percent. In 2020, there were nearly 800,000 requests made. At some federal agencies, the vast majority of requests are now from commercial operators who resell or use data for profit. Their turf is where a lot of the battle over the erosion of the freedom of information in America has been fought.

Wired explaining why and how a law can change in practice even when the words stay the same. You’ll also learn that the Israeli government was reselling US military supplies to Iran in the 80s, which is surprising even by the standards of the people behind NSO, which supplies spyware to authoritarian governments but not to Estonians and Ukranians fighting them.

Questions I didn’t even realize I needed an answer to: I finally know where the term “fundamentalist” came from (though readers should beware that the common usage has substantially drifted, and many people today who match this definition would not be considered fundamentalists):

If you guessed at random on the New York Regents exam in Algebra this year, you passed (at least by enough to get an appeal, which is apparently automatically granted). An interesting institutional component here is that they weren’t willing to fail people, and they weren’t willing to cancel the test, so they just administered the test, functionally removed the penalty for failure, and hoped nobody would notice. It seems that they mostly succeeded! I wonder how much time and treasure was wasted on this test?

Being a careful person who tries to deliver you only the finest perspectives, I talked with a NY math teacher I know. They confirmed it all.

Matt’s Thoughts In Between is going on permanent sporadic posting, which marks a reasonable time to read it. #223 was just published, and I started reading what I assume was #65 (#66 is the first in my inbox). Consistently interesting and thoughtful, Matt Clifford focused, for those 222 articles published on the more frequent schedule, on emerging developments and entrepeneurs, covering everything from the political consequences of remote work to who actually shares fake news. Recommended.

From a great Buzzfeed investigation, The World Wide Fund for Nature funds, trains, and equips paramilitaries who commit human rights abuses in exchange for their help against poachers. Gains from trade!

I ran into on...
Fully Live Electronic Contra
by jefftk — 2022-09-04T01:30:01.369Z

Contra dance traditionally has live music, which I think is key to why
the music and dancing have been able to stay tightly coupled as
they've both changed over the decades. Over the past ~14y, however,
there have been a few different approaches to pulling elements of
electronic music into contra dance:

2008:
fully pre-recorded music. Took off as "techno contra" (despite not
generally using Techno) after Forrest
organized one and made a well-produced
video at YDW. Later DJ Improper and
others would would prepare sets but remix them live.

2009:
Ed Howe and John Cote brought live looping to
contra dance as Perpetual
e-Motion, building a complex texture out of many layers of
fiddle and a range of effects. Super popular at their
peak in ~2011.

2011:
Julie Vallimont and Brendan Carey-Block formed Double Apex, with
Brendan playing live fiddle over Julie's loops, beats, and samples.
Also very popular. Julie brought this approach to a series of duos
over the years, with Jon Cannon (Delta Wave), Ed Howe (Cosmic
Echo), Andy Reiner (Firecloud), and
Noah VanNorstrand (Buddy System).

(more)

For years
I've wanted to add a "fully live" type
here, where all sounds are initiated in the moment by the musicians.
I now have something in this category I'm reasonably happy with:

Even though there's a lot going on, it's all live. Cecilia is playing fiddle,
bringing in an octaver effect at 0:16. I'm playing piano with my
hands and drums with my feet (note the drum pattern changes at 0:16
and 0:47). I'm also playing bass (starting at 0:16), where the beat
of the drum initiates the notes and the piano left hand chooses which
note to play. And then I'm using a breath controller (measuring how
hard I'm blowing) to pulse a supersaw
that, as with the bass, follows my left hand.

There are still places where I don't quite have the sound I'm
imagining, where listening back
makes me grimace a little, or where I can't yet do as many things at
once as would sound best, but I'm excited about Kingfisher, and this is a
really fun area to explore.

Comment via: facebook
How To Know What the AI Knows - An ELK Distillation
by Fabien Roger — 2022-09-04T00:46:30.541Z

This is what I submitted in May 2022 for the Safety Distillation Contest.

Why we care about knowing what an AI knows

Let’s say you have successfully created ScienceAI, an AI that is really good at writing research papers, and that you ask it to write a research paper about how to cure cancer. The resulting paper gives a recipe for a drug that should be taken by everyone and which should prevent cancer, according to the paper ScienceAI has written. However, even the experts in the relevant fields can’t tell you what that recipe would do.

In this kind of situation, it would really help to know what the AI knows. If you could ask it precise questions about the effects of this recipe like “would the end product of this recipe kill the recipient?” or “would one of the step of the recipe kill everyone on earth?”, and have the AI answer truthfully, it would make using this AI much safer. And at least, it would be a first step toward having AI that can actually improve the world without risking destroying it in the process.

This problem of having the AI tell you what it knows is called Eliciting Latent Knowledge (ELK): you have the AI tell you what it knows, but which doesn’t immediately follows from the input you give it or the output it gives you (you elicit latent knowledge).

What does an AI know?

To better understand the problem and paths to potential solution, it is better to flesh out a little what it means to say that an AI “knows” something. To that extent, I think an analogy can be useful.

Think of an AI as a factory which takes requests, and produces ideas, through a chain of intermediate ideas. For example, take a look at this very little factory:

It represents an AI that computes if a given individual is at risk for obesity given weight, height, and time spent doing exercise. For what we care about (having explanations for the AI decisions and making sure it is safe), we will say that it “knows” that this individual has a high body mass index: it is an idea that is one of the causes for the final prediction.

I have written the labels of the intermediate ideas inside the factory, but in many cases, you don’t have access to them, because you trained the network using gradient descent (you can think of gradient descent as “making a lot of random little changes to the factory and keeping the ones that improve the performances”). Therefore, if you try to deciphe...
Private alignment research sharing and coordination
by porby — 2022-09-04T00:01:22.337Z

Coordinating alignment efforts seems difficult when the work involves information that would ideally remain private. I'm not aware of any existing infrastructure to help with this, nor are the people I've asked so far. My understanding is that there are informal social networks which serve this purpose to some degree (e.g. passing google docs around), but it's difficult to safely scale these up.

I've seen some attempts to help manage potentially risky information- for example, Conjecture's recent policy discussions:

Conjecture: Internal Infohazard Policy - LessWrong

These policies are primarily interested in containing information. That's important, but it sure would be nice to have a system that simultaneously helps with both privacy management and coordination.

What kind of use cases could be targeted?

Motivating hypothetical example 1:

Hypothetical me is an independent researcher focusing on characterizing out of distribution behavior. I stumble across One Weird Trick for rapid grokking and generalization. I've run a variety of benchmarks and, aimed properly, it does remarkably well on HHH-style benchmarks. Unfortunately, the technique is trivially applicable to capability research; I observe extremely rapid training convergence and perplexity beating text-davinci-002 in a 2B model I trained on my workstation.

It seems like I should tell someone about the fact that this exists, on the off chance it interacts positively with some other safety work, but telling the wrong person could shorten timelines by approximately 100%. I'm not deeply embedded in the field and don't have high-trust working relationships with anyone. Now what? Send an e-mail to Eliezer composed mostly of AAAAAA?

Motivating hypothetical example 2:

I'm another independent researcher looking to start a new alignment research project, but it's a topic which lightly touches on capability. The capability parts are already out in the public literature, but they're in an obscure subfield and I suspect mostly unknown by mainstream capabilities research. I worry that drawing attention to it could be a net negative even if my work is not capability focused.

I think the research has potentially significant alignment value. I can't find any other related work in alignment. I'm not sure if this is because other people aren't publishing about it, if I'm not looking in the right spots, or if this act...
AXRP Episode 18 - Concept Extrapolation with Stuart Armstrong
by DanielFilan — 2022-09-03T23:12:01.242Z

YouTube link

Concept extrapolation is the idea of taking concepts an AI has about the world - say, “mass” or “does this picture contain a hot dog” - and extending them sensibly to situations where things are different - like learning that the world works via special relativity, or seeing a picture of a novel sausage-bread combination. For a while, Stuart Armstrong has been thinking about concept extrapolation and how it relates to AI alignment. In this episode, we discuss where his thoughts are at on this topic, what the relationship to AI alignment is, and what the open questions are.

Topics we discuss:

What is concept extrapolation

When is concept extrapolation possible

A toy formalism

Uniqueness of extrapolations

Unity of concept extrapolation methods

Concept extrapolation and corrigibility

Is concept extrapolation possible?

Misunderstandings of Stuart’s approach

Following Stuart’s work

Daniel Filan:
Hello, everybody. In this episode, I’ll be speaking with Stuart Armstrong. Stuart was previously a senior researcher at the Future of Humanity Institute at Oxford, where he worked on AI safety and x-risk, as well as how to spread between galaxies by disassembling the planet Mercury. He’s currently the head boffin at Aligned AI, where he works on concept extrapolation, the subject of our discussion. For links to what we’re discussing, you can check the description of this episode, and you can read the transcript at axrp.net. Well, Stuart, welcome to the show.

Stuart Armstrong:
Thank you.

Daniel Filan:
Cool.

Stuart Armstrong:
Good to be on.

What is concept extrapolation

Daniel Filan:
Yeah, it’s nice to have you. So I guess the thing I want to be talking about today is your work and your thoughts on concept extrapolation and model splintering, which I guess you’ve called it. Can you just tell us: what is concept extrapolation?

Stuart Armstrong:
Model splintering is when the features or the concepts on which you built your goals or your reward functions break down. Traditional examples are in physics when (for instance) the ether disappeared. It didn’t mean when the ether disappeared that all the previous physics that had been based on ether suddenly became completely wrong. You had to extend the old results into a new framework. You had to find a new framework and you had to extend it. So model splintering is when the model falls apart or the f...
An Update on Academia vs. Industry (one year into my faculty job)
by David Scott Krueger (formerly: capybaralet) — 2022-09-03T20:43:37.701Z

I've been an assistant professor (equivalent) for ~1 year now at Cambridge. Shortly after accepting the position, I wrote AI x-risk reduction: why I chose academia over industry.
Since then, I've had a lot of conversations on academia vs. industry with people getting into AI x-safety (e.g. considering applying for PhDs). This post summarizes that experience, and describes a few other updates from my experience in the last 1.5 years.
Summary of recent conversations:

Most people haven't read my previous post and I point them to it.
A common attitude is: "Industry seems better, WTF would you do academia right now?"
A perhaps equally common attitude is: "I am new and have no strong opinions, just trying to figure out what people think."
The main reasons I hear for industry over academia are:

Short timelines
Need to access Foundation models
Academic incentives to work on topics less relevant to x-safety.

I think these are all valid points, but I there are countervailing considerations. My response to these 3 points has generally been something like:

Academia still seems like the best option ATM for rapidly training and credentialing people. Even under fairly short timelines, this seems likely to be more valuable than direct work. Furthermore, it is a mistake to simply focus on efforts on whatever timelines seem most likely; one should also consider tractability and neglectedness of strategies that target different timelines. It seems plausible that we are just screwed on short timelines, and somewhat longer timelines are more tractable. Also, people seem to be making this mistake a lot and thus short timelines seem potentially less neglected.
There are and will be open source foundation models. Organizations like Anthropic seem keen on collaborating and providing access (although we haven't yet pitched them anything concrete). It's not clear that you have that much better access when you are working out one of these orgs (I'd be curious what people at the orgs think about this one!): my impression is that it is still clunky to do a lot of things with a large model when you are at an org, and things like retraining are obviously very expensive; these orgs seem to also favor large group projects, which I assume are directed by leadership; so in practice, there might be less difference between being entry-l

...
Request for Alignment Research Project Recommendations
by Rauno Arike — 2022-09-03T15:29:17.880Z

Recent lists of alignment research projects such as Richard Ngo's list of conceptual research projects and Neel Nanda's list of future research directions for understanding grokking and phase changes have been tremendously useful for me as a beginner in the field. They've given me a much clearer idea about the sorts of projects that would be useful for testing my fit for research while also having some potential to be at least somewhat valuable contributions. I believe many others feel the same.

Reading these lists has made me wonder whether there are other people as well who have ideas they think deserve further exploration but which they don't have time to work on themselves. If you have any such ideas, I'd love to see them, even if they're not that thoroughly thought out! Links to lists from other people would also be much appreciated. I think it would be valuable for many people who wish to start contributing to the field but feel like they haven't had enough opportunities to develop their research taste yet to have a resource containing various tractable project ideas related to multiple different research agendas.