trentmkelly/LessWrong-43k huggingface dataset:
User-Controlled Algorithmic Feeds
by jefftk — 2022-11-12T15:20:03.110Z

I've been

playing with

Mastodon more, and the main way you see
what other people have posted is by reading your feed. This consists
of everything any of the people you follow has posted or boosted. If
you're always going to read everything that's come out since your last
visit, which is the strategy I use for email, then order doesn't
matter very much. But for social media a strategy of reading
everything means following very few people, spending more time on the
site than you probably want to, or falling behind. What would be
better?

The way mainstream social networks (Facebook, Twitter, etc) handle
this is with prioritization: run an algorithm that considers every
post it could potentially show you and picks the best according to
some inscrutable and highly proprietary ranking, optimized for a
combination of engagement (getting you to post/comment/like),
retention (getting you to keep using the site), monetization (getting
you to view and interact with ads), and other
things. It's common to complain about the algorithm, especially
when it fails to show you things you really would have wanted to see
or when what it's optimizing for is clearly not what's best for you.

Mastodon's approach is very different: its prioritization algorithm is
"newest first". This is the ideal algorithm if what matters most is
breaking news, but since that's not most of what I care about it has
some downsides:

I'd rather see Pat's only post of the week than Sam's 100th.

I'm more interested in the things the people I follow post than
the things they boost.

I want to read everything Alex posts because it's relevant to my
work, consistently interesting, or they're someone I'm close to. But
if this were my bar for following people I'd miss a lot of things by
people who I do still often like reading.

If I only have a short amount of time available, all things
being equal I'd rather a Jo post than a Rowan one, but this is
not a strong preference.

The problem with a traditional social network's algorithmic feed
algorithm isn't that it uses an algorithm but that (a) it's goals are
not your goals and (b) you can't tweak it. Imagine a feed prioritized
according to:

Every post gets assigned a number, and are shown highest to
lowest.

Once a post has been shown to you it's removed from the pool of
eligible posts.

You control the rules that

...
Vanessa Kosoy's PreDCA, distilled
by Martín Soto — 2022-11-12T11:38:12.657Z

This is an expanded version of my answer to application problem 3 for Nate Soares and Vivek Hebbar's SERI MATS stream.

Why this post?

Vanessa's presentation has already been turned into a post by carado (as part of Refine) and a video in Connall and Jack's excellent series (as part of SERI MATS). But I felt like carado's post was still too mathy and disconnected, and the video glossed over some interesting details. So my motivation was writing a summary that explained the proposal in more common Alignment language (especially so as to compare it more easily with other proposals), and without presupposing familiarity with Infra-Bayesianism (providing only the required motivating details).

Summary

Infra-Bayesianism (tentatively) provides a robust framework for talking about agent foundations, including formalizations of how to identify agents and their utility functions. PreDCA is a protocol we can have an artificial agent follow (given its top-level architecture implements Infra-Bayesianism), so that the agent uses these tools to identify who created it ("the user", a human) and infer their utility function. The implementation is strict enough to outright negate some of the usual failure modes (if the theoretical tools work as intended), but the implemented framework is general enough (including belief updating and world modeling) to allow for general intelligence.

What needs to happen for this to work

The most important requirement is that the agency framework provides the correct, common sense answers when computationally applied to real world fuzzy scenarios.
The AGI needs to be fit into a very specific top-level architecture (without drastically losing capabilities).
It needs to be the case that maximizing the utilities of a group of present humans allows for future moral development without lock-in (or that we find this lock-in ethically successful).
We need a good enough model of what constitutes a human (or different patches or theoretical breakthroughs to prevent acausal attacks).
Collaterally, the necessary further technical developments need to converge on applicability.

Key points

A common failure mode is wire-heading: the AGI tampering with the communication channel through which the human specifies preferences/rewards, or the AGI tampering with the human's preferences directly, or similar. PreDCA makes this impossible by eliminating this communication channe...
Poster Session on AI Safety
by Neil Crawford — 2022-11-12T03:50:11.528Z

Context

I co-presented the above poster at the PPE Society Sixth Annual Meeting 2022 (henceforth ‘PPE Meeting’) in New Orleans on the 4th of November. Most of the 380 attendees were academics doing research on areas of philosophy that interact with politics or economics. The poster session, which was held at the end of a day of talks, lasted 1h30. There were around 6 posters being presented. In addition to providing attendees with a preview of the poster prior to the session, I gave them the following description:

In our poster session, we want to give an overview of the ‘alignment problem’ of artificial general intelligence (AGI). This is the problem of how to get AGI to do what we want. So far, it seems surprisingly and worryingly difficult. As things stand, AGI will likely be misaligned, resulting in catastrophic consequences. In addition to arguing why AGI is likely to be misaligned, we will also try to defend the assumption that AGI will be developed this century.

Goals

Practice talking with other academics about AI safety
Increase academics’ exposure to AI safety
Highlight AI safety as a special problem amongst the many problems within machine ethics and ethics of AI more generally
Increase my own understanding of AI safety and, in particular, get to grips with the core arguments for why we should be concerned
Gain a better understanding of how other academics conceive of AI safety, which arguments they perceive as unpersuasive and which they perceive as persuasive
Convince at least a couple of academics to take AI safety seriously and increase the likelihood that they connect their research to AI safety

Results

I believe I by and large achieved these goals. However, I was disappointed by the low turnout at the poster session. Out of (supposedly) 380 attendees, I estimate that around 60 were aware of my presentation topic, 25 were able to at least glance at the poster and 8 came to read the poster or interact with us.

Reactions

Which research question(s) am I working on?
Humans do all these things that I’ve claimed make AGI dangerous by default; are humans therefore just as dangerous?
Children do some of these things and they don’t immediately share our values but through praise and blame we mould them to alignment; can we not do something similar with AGI? In particular, could we train them to optimise for praise minus blame (or some similar

...
Is AI Gain-of-Function research a thing?
by MadHatter — 2022-11-12T02:33:21.164Z

Epistemic status: highly uncertain and fairly confused, seeking out opinions

First let me just post the definition of (biological) gain-of-function research from Wikipedia, so that everyone is on the same page:

Gain-of-function research (GoF research or GoFR) is medical research that genetically alters an organism in a way that may enhance the biological functions of gene products. This may include an altered pathogenesis, transmissibility, or host range, i.e. the types of hosts that a microorganism can infect. This research is intended to reveal targets to better predict emerging infectious diseases and to develop vaccines and therapeutics. For example, influenza B can infect only humans and harbor seals.[1] Introducing a mutation that would allow influenza B to infect rabbits in a controlled laboratory situation would be considered a gain-of-function experiment, as the virus did not previously have that function.[2][3] That type of experiment could then help reveal which parts of the virus's genome correspond to the species that it can infect, enabling the creation of antiviral medicines which block this function.[3]

The question I have is, should we have similar intuitions about certain kinds of AI and/or AI safety research? This is related in a complicated way to AI capabilities research, so let's define some terms carefully:

AI capabilities: the ability of AI systems to do things that we want them to do (which they may also be able to use in ways we do not want them to use) such as knowing whether a picture is of a cat, or knowing how to cure cancer
Gain of function: the ability of AI systems to do things that no sane person wants them to do, such as, I don't know, knowing whether they are deployed in the real world or not

This is more of a spectrum than it is two distinct classes; for instance, which class "recursive self-improvement" belongs to probably depends on how seriously you take AI risk in the first place. And one could imagine a sufficiently edgy or depressed person wanting to program AI's to do all manner of nonsense.

There are then a number of questions:

Should people be doing AI gain-of-function research at all?
If they shouldn't, how will we ever test out proposed alignment solutions? Or are AI safety researchers expected to get everything important right the first time it matters without any experimental evidence to guide them before the first ti

...
Why don't organizations have a CREAMO?
by Shmi — 2022-11-12T02:19:57.258Z

That is, a Chief Risk Evaluation And Mitigation Officer.

A bad handle on risk seems to be pervasive even in good organizations, both for- and non-profit. The unfolding FTX debacle shows that even Rationality-inspired orgs suck at risk assessment as much as any other place. If you go through the list of failed, ailing and failing startups in the ratosphere, they don't seem to do any better than your average ones, in terms of evaluating risk and acting on it. One can imagine that they would be good at handling known unknowns, and it's the black swans that would be a real danger. And yet, orgs get blindsided by entirely predictable events (why did FTX or its donors learn nothing from the Mt. Gox fiasco?). I can see some reasons where accurate risk evaluation is not in the interests of top-level human mesa-optimizers within an organization, because the compensation incentives are so misaligned. I can also see how everyone would hate that person. Or maybe this role is distributed between other executive officers? Certainly some rudimentary risk evaluation is done at every level, from the board down, but it doesn't seem to be the main focus of anyone. Or maybe I am missing something here and this role is not useful in general.
"Rudeness", a useful coordination mechanic
by Raemon — 2022-11-11T22:27:35.023Z

I think the concept of "rudeness" is underappreciated. (Or, if people are appreciating it, they're doing so quietly where I can't find out about it)

I think a lot of coordination-social-tech relies on there being some kind of social karmic balance. A lot of actions aren't expressly illegal, and not even blatantly all-the-time socially sanctioned. But if you do them a bit, it's considered rude, and if you're rude all the time, you get a reputation for being rude, and this comes with some consequences (i.e. not invited to as many parties).

The-concept-of-rudeness gives you a tool to softly shape your culture, and have some kind of standards, without having to be really rigid about it.

[Edited to add]
I'm writing this post because I was writing another coordination/epistemic-norms post, and I found myself wanting to write the sentence "If you do [X thing], it should be considered a bit rude. If you do [X' worse version of X thing], it's more rude." And then I realized this was resting on some underlying assumptions about coordination-culture that might not be obvious to everyone. (i.e. that it's good to have some things be considered "rude")

I come from a game-design background. In many games, there are multiple resources, and there are multiple game-mechanics for spending those resources, or having them interact with each other. You might have life-points, you might have some kind of "money" (which can store value and then be spent in arbitrary quantities), or renewable resources (like grains that grow back every year, and spoil if you leave them too long).

Many good games have rich mechanics that you can fine-tune, to shape the player's experience. A flexible mechanic gives you knobs-to-turn, to make some actions more expensive or cheaper.

The invention of "money" in real life was a useful coordination mechanic.

The invention of "vague social capital that you accumulate doing high status respectable things, and which you can lose by doing low status unrespectable things" predates money by a long time, and is still sometimes useful in ways that money is not.

A feeling-of-rudeness is one particular flavor of what "spending-down social capital" can feel like, from the inside of a social interaction.
[/edit]

Different cultures have different conceptions of "what is rude." Some of that is silly/meaningless, or actively harmful. Some of it is arbitrary...
Internalizing the damage of bad-acting partners creates incentives for due diligence
by tailcalled — 2022-11-11T20:57:41.504Z

Suppose you are cooperating with someone. It seems like there would be good reason to keep an eye on your partner to make sure that your partner does not do very bad things. For example, here are some reasons to keep an eye on your partner:

It would be bad for the victims of your partner if your partner did bad things
It would be bad for your reputation to be associated with people who do bad things
It would be bad for you if your partner gets caught doing bad things and suddenly you are missing a partner you thought you could rely on

But how vigilant should you be about keeping an eye on your partners? And who should you keep an eye on?

Here's one proposal: To the extent that your partner helps you, and it then turned out that their help was funded by bad things, you should try to help the partner's victims as much as your partner helped you. For instance if your partner stole $10000 from a bank and then gave you $1000, you should return this $1000 to the bank.

This seems to me to give you nice certain incentives. For instance, it neatly defines which people you must keep an eye on, and gives you proportional incentives to keep an eye on them based on how entangled they are in your organizations. It also directly links diligence to cooperation, so you have a logical reason to give to your partners for why you want extra checks when they suggest helping you. And it also seems like it would help your reputation in case something genuinely does go wrong.

(I'm not sure if it should be 1:1 exactly. An argument for giving more than 1:1 back is that helping others later is likely worse than not harming them in the first place, and also you might not always catch bad actors so paying back more would counterbalance this incentives-wise. An argument for giving less than 1:1 back is that otherwise you would have much greater likelihood of not being able to cover the cost. Probably any significant self-inflicted cost of bad partners would massively improve the incentives.)

Inspired by discussion on twitter.
Speculation on Current Opportunities for Unusually High Impact in Global Health
by johnswentworth — 2022-11-11T20:47:03.367Z

Epistemic Status: armchair speculation from a non-expert.

Short version: I expect things to get pretty bad in the Sahel region over the next year in particular. The area is an obvious target for global health interventions even in good times, and impact is presumably higher in bad times. A simple baseline intervention: fill a backpack with antibiotics, fly to the region, and travel around distributing the antibiotics.

What’s The “Sahel” Region?

The Sahel is a semi-arid region along the southern edge of the Sahara desert. Think roughly Mali, Niger, Chad and Sudan.

Bad How?

Based on statistics on the Sahel, it’s one of the few remaining regions on Earth where the population is near Malthusian equilibrium. Fertility is high, contraception is rare; about half the population is under age 16. Infant mortality is around 6-8%, and ~a quarter of children are underweight. (Source: CIA World Factbook entries on Mali, Niger, Chad and Sudan.)

Being near Malthusian equilibrium means that, when there’s an economic downturn, a substantial chunk of the population dies.

Die How?

Traditional wisdom says: war, famine, disease. In this case, I’d expect famine to be the main instigator. Empty bellies then induce both violence and weak immune systems. On priors, I’d expect infectious disease to be the main proximate killer.

The Next Year In Particular?

The global economy has been looking rough, between the war in Ukraine shocking oil and food markets, and continuing post-Covid stagflation. Based on pulling a number out of my ass without looking at any statistics, I’d guess deaths from violence, starvation, and disease in the Sahel region will each be up an order of magnitude this year/next year compared to a good year (e.g. the first-quartile best year in the past decade).

That said, the intervention we’ll talk about is probably decently impactful even in a good year.

So What’s To Be Done?

Just off the top of my head, one obvious baseline plan is:

Fill a hiking backpack with antibiotics (buy them somewhere cheap!)
Fly to N'Djamena or take a ferry to Timbuktu
Obtain a motorbike or boat
Travel around giving away antibiotics until you run out
Repeat

Note that you could, of course, substitute something else for "antibiotics" - maybe vitamins or antifungals or water purification tablets or iron supplements or some mix of those is higher marginal value.

There are some possibly-nonobvious considerations her...
Catharsis in Bb
by jefftk — 2022-11-11T17:40:01.262Z

Catharsis is an

Amy
Cann fiddle tune that's exciting, difficult, and overplayed.
There are a

zillion
recordings of it, but here's one from a contra dance:

It's in Gm, dark and brooding. But what if it were major instead? A
long time ago, when I still played guitar, I figured out a set of
chords in Bb:

|: Bbmaj7 / Cm7 / Dm7 / Cm7 F9add11 :|
Bbmaj7 / Cm7 / Dm7 / G9add11 /
Cm7 F9add11 Dm7 G9add11
Cm7 F9add11 Bbmaj7 Cm7F9add11

(notation explanation)

The fiddles still play the same notes, but with the chords you
interpret the tune as being in Bb instead of Gm.

It's super cheesy, and combined with a tune as overexposed as
Catharsis I wouldn't normally play it at a dance, but what about an
open band? When you have a group of musicians who are just together
for the evening, cheesy and overdone is not a downside!

Except, how would you get an open band to play those chords? Say
"F9add11" to a folk guitarist (including me—I had to look up the
names of these chords for this post) and they won't know what to do.
It turns out, however, that these chords can each be pretty well
approximated as a combination of a bass note and a well-known chord.
For example, Bbmaj7 is Bb, D, F, and A. If the bass player or piano
left hand plays Bb, that leaves D, F, and A for higher-pitched rhythm
instruments like guitar. And D, F, and A just makes Dm, which your
guitar players can do. So:

chord
bass
treble

Bbmaj7
Bb
Dm

Cm7
C
Eb

Dm7
D
F

F9add11
F
Cm

G9add11
G
Dm

Which gives us:

|: Dm / Eb / F / Eb Cm :|
Dm / Eb / F / Dm /
Eb Cm F Dm Eb Cm Dm EbCm

It's still not a great fit for an open band: you need fiddlers who are
comfortable with a tune this hard, and while the chords are now closer
to the folk repertoire they're flatter than ideal. But while I haven't
tried this with a real group of musicians, with the right ones I do think it should work!

Comment via: facebook, mastodon
Instrumental convergence is what makes general intelligence possible
by tailcalled — 2022-11-11T16:38:14.390Z

TL;DR: General intelligence is possible because solving real-world problems requires solving common subtasks. Common subtasks are what give us instrumental convergence. Common subtasks are also what make AI useful; you want AIs to pursue instrumentally convergent goals. Capabilities research proceeds by figuring out algorithms for instrumentally convergent cognition. Consequentialism and search are fairly general ways of solving common subtasks.

General intelligence is possible because solving real-world problems requires solving common subtasks

No-free-lunch theorems assert that any cognitive algorithm is equally successful when averaged over all possible tasks. This might sound strange, so here's an intuition pump. Suppose you get a test like

2+2 = _
3*2 = _

and so on. One cognitive algorithm would be to evaluate the arithmetic expression and fill the answer in as the result. This algorithm seems so natural that it's hard to imagine how the no-free-lunch theorem could apply to this; what possible task could ever make arithmetic score poorly on questions like the above?

Easy: While an arithmetic evaluator would score well if you e.g. get 1 point for each expression you evaluate arithmetically, it would score very poorly if you e.g. lose 1 point for each expression you evaluate arithmetically.

This doesn't matter much in the real world because you are much more likely to encounter situations where it's useful to do arithmetic right than you are to encounter situations where it's useful to do arithmetic wrong. No-free-lunch theorems point out that when you average all tasks, useful tasks like "do arithmetic correctly" are perfectly cancelled out by useless tasks like "do arithmetic wrong"; but in reality you don't average over all conceivable tasks.

If there were no correlations between subtasks, there would be no generally useful algorithms. And if every goal required a unique algorithm, general intelligence would not exist in any meaningful sense; the generally-useful cognitions are what constitutes general intelligence.

Common subtasks are what give us instrumental convergence

Instrumental convergence basically reduces to acquiring and maintaining power (when including resources under the definition of power). And this is an instance of common subtasks: lots of strategies require power, so a step in lots of strategies is to accumulate or preserve power. Therefore, j...