trentmkelly/LessWrong-43k huggingface dataset:
How would you build Dath Ilan on earth?
by Yair Halberstadt — 2022-05-29T07:26:17.322Z

I sometimes get a little bit melancholic when I read about Dath Ilan. Humanity could be so much better off, but due to economic illiteracy, we can't have nice things.

Better to light a small flame than to curse the darkness. So if you had to build a model econoliterate city on earth, how would you go about it?

There's lots of micronations projects out there. What I'm interested in is:

How do you keep the city econoliterate, immediately and a hundred years in the future.
How do you make this practical to achieve - most micronations projects dont go anywhere, and are actively thwarted by governments.

I will not cease from mental fight,
Nor shall my pen sleep in my hand:
Till we have built Dath Ilan,
In this earths green & pleasant land.
Distributed Decisions
by johnswentworth — 2022-05-29T02:43:58.961Z

Consider two prototypical “agents”: a human, and a company.

The human is relatively centralized and monolithic. As a rough approximation, every 100 ms or so observations flow into the brain from the eyes, ears, etc. This raw input data updates the brain’s world-model, and then decisions flow out, e.g. muscle movements. This is exactly the sort of “state-update model” which

Against Time In Agent Models

criticized: observations update one central internal state at each timestep, and all decisions are made based on that central state. It’s not even all that accurate a model for a human, but let’s set that aside for now and contrast it to a more obviously decentralized example.

In a company, knowledge and decisions are distributed. A cashier sees and hears customers in the store, and interacts with them in order to sell things. Meanwhile, a marketing editor tweaks some ad copy. Each mostly makes decisions based on their local information; most of that local information is not propagated to other decision-makers. Observations don’t update a single centralized state which informs all decisions. Instead, different decisions have different input information from different sources.

In

Optimization at a Distance

, I suggested a mental picture of agents kinda like this:

Note that this particular drawing doesn’t have any inputs to the optimizer (i.e. observations) for simplicity, but it’s easy to add inputs. The optimizer need not be strictly causally upstream of the target; it could have interactions back-and-forth with the target.

It’s like a phased array: there’s lots of little actions distributed over space/time, all controlled in such a way that their influence can add up coherently and propagate over a long distance to optimize some far-away target. Optimization at a Distance mainly emphasized the “height” of this picture, i.e. the distance between optimizer and target. This post is instead about the “width”: not only are the actions far from the optimization target, the actions themselves are also distributed in spacetime and potentially far apart from each other.

Contrast: Bayesian Updates

Suppose I want to watch my favorite movie,

10 Things I Hate About You

, in the evening. To make this happen, I do some optimization - I steer myself-in-the-evening and my-immediate-environment-in-the-evening into the relatively small set of states in which I’m watching t...
Distilled - AGI Safety from First Principles
by Harrison G — 2022-05-29T00:57:47.237Z

This is meant to be supplemental to

Richard Ngo’s AGI Safety from First Principles seq

uence. It can serve as an overview/guide to the sequence, and an outline for navigating his core claims. It's also a submission for the AIMS Distillation contest.

Preamble: (you can skip this if you’re just looking to get into the arguments right away)

This post can be read independently of the sequence as well. Just be aware that some details may be left out in this case, and you should refer to Ngo’s sequence to get a more-detailed idea of the arguments if anything seems dissatisfying here.

Most of these ideas are Ngo’s, not mine. I provide some examples and different ways of viewing things, but the core arguments don’t necessarily represent my views (and they might not represent Ngo’s either, though I tried my best to give a fair summary!)

Intended Audience

: Those new to the case for why the development of artificial general intelligence (AGI) might pose an existential threat, or those looking to have the case laid out more clearly “from first principles” (without taking any previous claims for granted).

It’s helpful if you have a vague idea of modern machine learning (what’s a neural net? What is reinforcement learning?), but you can probably get by just from considering some of the examples.

What are Ngo’s Core Claims?

Ngo’s argument stands on 4 key premises, stated below in his words:

We’ll build AIs which are much more intelligent than humans (i.e. superintelligent).
Those AIs will be autonomous agents which pursue large-scale goals.
Those goals will be misaligned with ours; that is, they will aim towards outcomes that aren’t desirable by our standards, and trade off against our goals.
The development of such AIs would lead to them gaining control of humanity’s future.

Essentially, if all 4 premises hold, the conclusion is, “the development of superintelligent AGI will occur and poses an existential threat to humanity” (and provided you’re concerned about existential risks to humanity, this is important).

I think Ngo’s 4 core claims can be viewed as analogous to pillars holding up a structure, the conclusion:

Why is this analogy useful? It leads to a couple nice ways of thinking about the problem:

Zooming in, each pillar contains a “stack of subclaims”, spanning from the bottom of the pillar to the top. Any of these subclaims can be contested, makin

...
What have been the major "triumphs" in the field of AI over the last ten years?
by lc — 2022-05-28T19:49:53.382Z

Contrary to what seems to be the experience of others, when I'm talking to normies about AI safety, the most common dissenting reaction I get isn't that they think AI will be controllable, or safe. Convincing them that computers with human-level intelligence won't have their best interests at heart by default tends to be rather easy.

More often the issue is that AGI seems very far away, and so they don't think AI safety is particularly important. Even when they say that's not their sticking point, alerting them to the existence of tools like GPT3 tends to impart a sense of urgency and "realness" to the problem that makes them take it a bit more seriously. There's this significant qualitative difference in the discussion before and after I show them all of the crazy things OpenAI and DeepMind have built.

I have a general sense that progress has been speeding up, but I'd like to compile a list of relevant highlights. Anybody here willing to help?
Utilitarianism
by C S SRUTHI — 2022-05-28T19:35:33.587Z

Following is a brief of some parts of this paper on Aligning AI with shared human values.

The "why" behind most human actions is a universal seeking of pleasure and aversion to pain, so it seems natural that morality should be focused on "the greatest good for the greatest number of people".

This is why Utilitarianism emerged as a key idea in human values- that we make moral decisions from the position of a benevolent disinterested spectator.

In the paper this is mathematically translated as “maximizing the expectation of the sum of everyone’s utility functions.”

A utility function maps various scenarios to a scalar representing the pleasure associated with them. For eg: Completing a project on time and receiving complements for it is more pleasurable that Missing the project deadline.

This understanding of utility can help AI agents deal with imprecise commands by choosing the alternative with higher utility.

Utility can’t be modeled as a regression task because utilities only hold the ordering under positive affine transformations i.e.

a(u1) + b > a(u2) +b preserves u1>u2 only when a is positive and we can't guarantee that when performing regression.

To remove any biases that may occur because people have different perspectives, we remove the examples where there is substantial disagreement in ranking
Bayesian Persuasion?
by Karthik Tadepalli — 2022-05-28T17:52:14.567Z

I'm an economist and quite new to AI alignment. In reading about the perils of persuasive AI, I was reminded of an influential model in economic theory: the Bayesian persuasion model (Kamenica and Gentzkow, 2011). It's used to model situations in which a decisionmaker wants to learn from a biased expert (e.g. a judge learning from a prosecutor). The punchline of the basic model I linked is that in equilibrium, communication depends on the curvature of the expert's payoff function: if the expert's payoff is concave in the decisionmaker's beliefs, then no communication occurs, whereas if the expert's payoff is convex in the decisionmaker's beliefs, then the expert discloses all their information.
To my non-expert eyes, this approach seems like it could be very useful in modelling the challenge of learning from AI while trading off the risk of persuasive AI. Does it seem promising, and if so, has it been done?
What is Going On With CFAR?
by niplav — 2022-05-28T15:21:51.397Z

Whispers have been going around on the internet. People have been talking, using words like "defunct" or "inactive" (not yet "dead").

The last update to the website was December 2020 (the copyright on the website states "© Copyright 2011-2021 Center for Applied Rationality. All rights reserved."), the last large-scale public communication was end of 2019 (that I know of).

If CFAR is now "defunct", it might be useful for the rest of the world to know about that, because the problem of making humans and groups more rational hasn't disappeared, and some people might want to pick up the challenge (and perhaps talk to people who were involved in it to rescue some of the conclusions and insights).

Additionally, it would be interesting to hear why the endeavour was abandoned in the end, to avoid going on wild goose-chases oneself (or, in the very boring case, to discover that they ran out of funding (though that appears unlikely to me)).

If CFAR isn't "defunct", I can see a few possibilities:

It's working on some super-secret projects, perhaps in conjunction with MIRI (which sounds reasonable enough, but there's still value left on the table with distributing rationality training and raising the civilizational sanity)

They are going about their regular business, but the social network they operate in is large enough that they don't need to advertise on their website (I think this is unlikely, it contradicts most of the evidence in the comments linked above)

So, what is going on?
An inquiry into the thoughts of twenty-five people in India
by KatjaGrace — 2022-05-28T08:30:15.355Z

Sometimes I get excited about running surveys. Here is a Positly one from November 2020 in which I asked the following questions, to participants from India:

What are you looking forward to this week?

What do you think of as the most important thing going on in the world right now?

If you had a spare half hour right now, what would you do with it?

What is something you changed your mind about recently?

What in life is more important than other people realize?

If someone gave you $5 right now, what would you do with it?

Who is someone you think of as a hero?

Are you paying attention to the US election?

What was the biggest news story this year?

I don’t recall any notable constraints other than the India location requirement, but I barely remember doing this.
Functional Analysis Reading Group
by Ulisse Mini — 2022-05-28T02:40:26.712Z

We've been running a Functional Analysis reading group on the EleutherAI Discord (partially motivated by a desire to grok Infrabayes). According to hard earned advice it's useful to pair up when learning difficult things.

We're as of this posting on section 2.3 of "A Friendly Approach To Functional Analysis" (which TurnTrout reviewed and may be found on libgen). I should have posted here sooner, but better late then never. A motivated person can still catch up (we're going at ~2 sections/week, I'm spending ~5hrs/wk, if you spend more you can easily catch up).

TlDR get on EleutherAI's discord and join the Functional Analysis Thread!

PS: I'm likely to start other reading groups in the future, comment if you're interested in one on abstract algebra, calculus on manifolds, logic, probability, since those are what I'm currently interested in.
Understanding Selection Theorems
by adamk — 2022-05-28T01:49:43.062Z

This post is a distillation of a corpus of ideas on Selection Theorems by johnswentworth, mainly this post. It was made for EA UC Berkeley's Distillation Contest.

Introduction

Selection Theorems are tools for answering the question, "What will be the likely features of agents we might encounter in an environment?" Roughly speaking, if an environment selects for agents according to some metric, then Selection Theorems try to give us some guarantees or heuristics about the classes of agents that will be highly fit in the environment.

This selective pressure need not be biological. If an AI researcher scraps some agent designs and keeps others, or a set of market traders can thrive or go bankrupt based on their trading performance, then these might also be situations where Selection Theorems can apply.

The value of strong Selection Theorems is quite alluring: if we can make specific claims about the types of agents that will arise in particular environments, it might become easier to know what aligned systems look like and how to make them. They may also be instrumental in resolving our general confusion about the nature of agency.

The goals of this post will be to provide intuition and examples for the notion of agent type signatures, define and describe Selection Theorems through an example, and discuss what future research might look like. I do not assume any technical background from readers, other than a baseline familiarity with ideas like agency and expected utility.

A Toy Scenario

Suppose I am trying to build a robotic agent which mops and polishes floors. I have two prototype mopping agents, A and B, with different internal architectures:

Both agents are relatively safe and competent at this job, but there are a few key differences:

Agent A normally thinks rooms are cleanest when the floors are mopped and polished, but after polish is applied, it thinks rooms are cleanest without polish. As a result, it quickly becomes stuck applying and removing polish. Agent B shows no such problem. It mops floors, polishes them, and then moves on to the next room.
Agent A cleans any mess it sees. However, if I hold a picture of a different room close to Agent A's camera, it immediately behaves as though it is in the room of the picture, forgetting its previous surroundings. Agent B, meanwhile, is robust to these tricks: if it begins cleaning, and halfway through is shown a pi

...
Infernal Corrigibility, Fiendishly Difficult
by David Udell — 2022-05-27T20:32:50.773Z

Enormous spoilers for mad investor chaos and the woman of asmodeus (planecrash Book 1).

1.

Aspexia Rugatonn, Grand High Priestess of Asmodeus, measures the woman kneeling before her with a careful eye and a half-dozen magics. If Carissa Sevar is an exceptional woman in ways beyond a native talent for wizardry, this is not yet evident. But then, if Sevar was that self-evidently extraordinary, she'd have been fast-tracked more than she was.

There are not many times when Asmodeus intervenes directly in Cheliax; Aspexia prefers not to be ignorant about any of them. She is knowledgeable of history and secrets, though, and so less confused by this intervention than others might be. While other possible readings exist, the degree to which Church and Queen have been ordered not to take the initiative in originating actions impinging on Carissa Sevar are suggestive of circumstances having triggered some divine compact to which Asmodeus is signatory. The divine view of reality and negotiation gives more prominence than mortals do to notions of 'leaving things alone to become as they would otherwise have been'; perhaps because gods have been able to formulate a sensible notion of what that means between themselves, where mortals could not.

An obvious further guess is that this compact's signatories include Irori among their number, and that Asmodeus is contesting with Irori for Carissa Sevar's soul in some ancient challenge governed by rules. Though if Carissa Sevar is wavering between Lawful Neutrality and Lawful Evil, Asmodeus is being unsubtle in His blandishments - the temptations more seem like inducements that would be offered to a soul already standing on Asmodean ground, not a soul wavering between a choice of paths. Overt blandishments for a soul to set proudly aside, while being more covertly tempted by a sense of being treated as important and valuable? Perhaps. Carissa Sevar's eidetically reported reaction seems not particularly expected of a nascent follower of Irori, but that could be a masquerade. Sevar has not been mindread more than she would be otherwise; they are not to be proactive about her correction.

Someone else in Aspexia's position might wonder whether Asmodeus would be pleased, if she disobeyed Asmodeus's orders in order to preemptively insinuate temptations to Sevar, show her how importa

...
New Water Quality x Obesity Dataset Available
by Elizabeth — 2022-05-27T19:50:07.693Z

Tl;dr: I created a dataset of US counties’ water contamination and obesity levels. So far I have failed to find anything really interesting with it, but maybe you will. If you are interested you can download the dataset here. Be warned every spreadsheet program will choke on it; you definitely need to be use statistical programming.

Photocredit: DALL-E and a lot of coaxing

Many of you have read Slime Mold Time Mold’s series on the hypothesis that environmental contaminants are driving weight gain. I haven’t done a deep dive on their work, but their lit review is certainly suggestive.

SMTM did some original analysis by looking at obesity levels by state, but this is pretty hopeless. They’re using average altitude by state as a proxy for water purity for the entire state, and then correlating that with the state’s % resident obesity. Water contamination does seem negatively correlated with its altitude, and its altitude is correlated with an end-user’s altitude, and that end user’s altitude is correlated with their average state altitude… but I think that’s too many steps removed with too much noise at each step. So the aggregation by state is basically meaningless, except for showing us Colorado is weird.

So I dug up a better data set, which had contamination levels for almost every water system in the country, accessible by zip code, and another one that had obesity prevalence by county. I combined these into a single spreadsheet and did some very basic statistical analysis on them to look for correlations.

Some caveats before we start:

The dataset looks reasonable to me, but I haven’t examined it exhaustively and don’t know where the holes are.
Slime Mold Time Mold’s top contender for environmental contagion is lithium. While technically present in the database, litium had five entries so I ignored it. I haven’t investigated but my guess is no one tests for lithium.
It’s rare, but some zip codes have multiple water suppliers, and the spreadsheet treats them as two separate entities that coincidentally have the same obesity prevalence.
I’ve made no attempt to back out basic confounding variables like income or age.
“% obese” is a much worse metric than average BMI, which is itself a much worse metric than % body fat.
None of those metrics would catch if a contaminant makes some people very fat while making others thin ( SMTM thinks paradoxical effects ar

...
Bootstrapping Language Models
by harsimony — 2022-05-27T19:43:45.686Z

Note: I'm pretty uncertain of my conclusions in this post, and want to hear other people's thoughts on this.

It seems possible to bootstrap language models to some degree. What are the safety implications of this?

For example, you could:

Ask a language model to write prompts for short stories

Give the language model these prompts and have it generate short stories

Use these new short stories as training data, thus improving the models ability to write short stories (optionally, you could select for good short stories and train only on those)

Repeat until the model is really good at writing short stories.

Would this lead to arbitrarily large increases in capability?

In the case where we don't select for well-written short stories, I would guess no. I would expect that a technique like this would improve the model's ability to write good short stories to a limited extent at the cost of getting worse on unrelated tasks.

Conceptually, retraining a model on a subset of possible outputs seems like it would bias the model to produce similar output in the future; a form of fine-tuning. I would expect that this would increase the model's performance at related tasks (e.g. write long stories) and reduce its performance on unrelated tasks (making lists?).

At best, the language model produces short stories similar to the original training data. After it trains on the newly generated data, the new short stories should be similar in quality to the output it was producing before (slightly better, because it has been fine-tuned to produce short stories). It seems hard to produce training data with significantly higher quality overall without filtering the output somehow.

In the case where we filter the output to produce high quality training data, it seems one could increase capability. To obtain a large amount of high-quality data, the model needs to produce good short stories frequently enough, and the filter must do a good job of selecting these stories.

The rate at which a model can produce good short stories depends on how much overlap it's output distribution has with the hypothetical output distribution of a model trained on high-quality data. As the model improves, it will produce high-quality data with higher frequency.

The quality of the filter determines the upper-bound of performance. If the filter cannot discriminate high-quality stories from extremely-high-q...
Resources For the Redwood Research Triplebyte Test
by Aiyen — 2022-05-27T19:40:36.841Z

After working for a few years as a web developer, I am now applying to the MLAB2 AI Alignment Boot Camp. I'm very excited about this, and studying for the required coding test, the Triplebyte assessment for Redwood Research. Does anyone have any suggestions for good resources to prepare? This is an excellent opportunity and I do not want to let it slide simply because I didn't know the best way to get ready for it.
Sneaking Suspicion
by benwr — 2022-05-27T19:17:34.214Z

Over time I've learned to pay more attention to a sensation I think of as "Sneaking Suspicion."

Sneaking Suspicion is a "flavor" that some of my thoughts come with: it's a quiet little (metaphorical) voice that whispers "wait, but, are you sure?"

Lots of other voices in my head are much louder than Sneaking Suspicion. Sometimes I don't notice the whisper, even when in retrospect it had been whispering for years. But Sneaking Suspicion has a much, much better track record for honesty and correctness than most of those loud voices.

I notice that the people whose epistemics I trust the most appear to be extremely good at noticing their own analogues of Sneaking Suspicion. I think it's a skill similar to Noticing Confusion. But, at least for me, the term "confusion" doesn't really call to mind the right experience. It's not a "wha-- ???!" kind of feeling (Pikachu hurt itself in its confusion!). It's a lot more timid, and it has to be coaxed gently into speaking.