trentmkelly/LessWrong-43k huggingface dataset:
Intelligence in Commitment Races
by David Udell — 2022-06-24T14:30:21.525Z

A distillation of my understanding of the commitment races problem.

Greaser Courage

It's 1950 and you're a greaser in a greaser gang. Recreationally, you're driving your heavy American cars (without seatbelts) at each other at speed, and seeing who swerves first. Whoever swerves first is a coward and is "chicken"; their opponent is courageous and the victor. Both drivers swerving is less humiliating than just you swerving. Both not swerving means both drivers die.

The bolts on your steering wheel have been loosened in the chop shop, and so your steering wheel can be removed if you pull it out towards you.

If you remove your steering wheel and throw it prominently out your window, then your opponent will see this and realize that you are now incapable of swerving. They will then swerve, as they prefer humiliation to death. This wins you glory that you live to see.

But if you both simultaneously throw your steering wheels out the window, then neither of you will be able to slow down in time and both of you will die.

Commitment Races

The two above greasers are thus in a situation where one can individually do better by throwing out their steering wheel quickly, but fares worse if both adopt this strategy. Both-drivers-having-their-steering-wheels is a commons that each greaser can take from, but both do poorly if the commons is depleted by over-exhaustion. Greasers with unloosened steering wheels don't share a both-drivers-having-their-steering-wheels commons -- because each greaser can commit ahead of time to not swerving, this commons exists. Because the greasers are both itching to commit first to not swerving, we say that the greasers playing chicken with loosened steering wheels are in a commitment race with each other.

Besides throwing out steering wheels, other kinds of actions allow agents to precommit to act ahead of time, in view of their opponents. If you can alter your source code so that you will definitely pass up a somewhat desirable contract if your ideal terms aren't met, you'll be offered better contracts than agents that can't alter their source code. Alice the human and Bot the human-intelligence AGI are trading with each other. Bot has edit access to his own source code; Alice does not have access to hers. Before beginning any trading negotiations with Alice, Bot is sure to modify his own source code so that he won't accept less than almost all of ...
Linkpost: Robin Hanson - Why Not Wait On AI Risk?
by Yair Halberstadt — 2022-06-24T14:23:50.580Z
Link: https://www.overcomingbias.com/2022/06/why-not-wait-on-ai-risk.html

Raises an interesting critique of AI Risk. I think he is wrong, but would be interested in how others would respond.
"Science Cathedrals"
by Alex Vermillion — 2022-06-24T03:30:12.854Z

My favorite place in the world is Chicago's Museum of Science and Industry. When I was younger, I would constantly beg my parents to take me there. It was pretty far, and it's not exactly enjoyable to drive through Chicago, but I'd ask anyways. It felt to me like the one place in the world I'd been that felt like a monument to trying.

It had things like explanations (along with simulations!) of how lightning worked, or a scale-ish model of the solar system (in a long hallway to the planetarium area), a moonrock, explanations of how cities might be built in the future, and so much more[1]. On the ceiling of the main are were the names of famous scientists and inventors, from Tesla to Foucault to Galileo and way more.

Looking back from where I stand now, it feels like the building was a cathedral to Science, the same as a normal cathedral is a cathedral to Religion.

What other places exist like this? Where can I take friends or a kid one day to marvel at the way the world works? Things like special planetariums or such are probably on track, but there's surely much more.

a whole-ass train, a captured German U-boat, the Apollo 8 command module, and I'm not running out of steam yet. ↩︎
LessWrong Has Agree/Disagree Voting On All New Comment Threads
by Ben Pace — 2022-06-24T00:43:17.136Z

Starting today we're activating two-factor voting on all new comment threads.

Now there are two axes on which you can vote on comments: the standard karma axis remains on the left, and the new axis on the right lets you show much you agree or disagree with the content of a comment.

How the system works

For the pre-existing voting system, the most common interpretation of up/down-voting is "Do I want to see more or less of this content on the site?" As an item gets more/less votes, the item changes in visibility, and the karma-weighting of the author is eventually changed as well.

Agree/disagree is just added on to this system. Here's how it all hooks up.

Agree/disagree voting does not translate into a user's or post's karma — its sole function is to communicate agreement/disagreement. It has no other direct effects on the site or content visibility (i.e. no effect on sorting algorithms).
For both regular voting and the new agree/disagree voting, you have the ability to normal-strength vote and strong-vote. Click once for normal-strength vote. For strong-vote, click-and-hold on desktop or double-tap on mobile. The weight of your strong-vote is approximately proportional to your karma on a log-scale (exact numbers here).

Ben's personal reasons for being excited about this split

Here's a couple of reasons that are alive for me.

I personally feel much more comfortable upvoting good comments that I disagree with or whose truth value I am highly uncertain about, because I don’t feel that my vote will be mistaken as setting the social reality of what is true.
I also feel very comfortable strong-agreeing with things while not up/downvoting on them, so as to indicate which side of an argument seems true to me without my voting being read as “this person gets to keep accruing more and more social status for just repeating a common position at length”.
Similarly to the first bullet, I think that many writers have interesting and valuable ideas but whose truth-value I am quite unsure about or even disagree with. This split allows voters to repeatedly signal that a given writer's comments are of high value, without building a false-consensus that LessWrong has high confidence that the ideas are true. (For example, many people have incompatible but valuable ideas about how AGI development will go, and I want authors to get lots of karma and visibility for excellent contributio

...
Book review: The Passenger by Lisa Lutz
by KatjaGrace — 2022-06-23T23:10:19.626Z

Spoiler warning: spoilers for The Passenger by Lisa Lutz, and In the Cart, by Anton Chekhov.

I took up this book looking for a page-turner. It was, but hours before the end I thought its main contribution to my mental life would be the visceral knowledge that page-turners can be undelicious. It felt cold, and getting into its world felt bad. The protagonist slunk around dark and uncomfortable places, killing people, scheming harshly, perceiving low beams as dangers to the heads of tall men, and that sort of thing. With some amount of fretting about what she was becoming. I wanted to turn the pages, but I also kind of wanted it to end, and for me to read something more squarely enjoyable next time.

Then the end was different, and kept coming back to me in the day, and turned the rest around and gave it new meaning. Like the image of a long dark tunnel becomes something different if you step away and see that it starts in the loveliest of fields. After a long dark tunnel of a book, we turn around and see where the heroine came from, and her teenage innocence and goodness newly fills out the potentially empty person we have been watching. And her past isn’t really discordant—I think I just wrote her off fast. This endless night began in that day, and seeing so makes the full picture both warm and intense, where it had seemed like cheap suspense and violence.

In that abstract description, I suppose it is a bit like ‘In The Cart’ by Chekhov: we watch a dreary life, then at last that life is pulled into a perspective where we see its warm and hopeful start, unreachably distant. And it changes the flavor.

I have not much idea if what I saw was what Lisa Lutz was going for, but if she was, and if other readers are like me (haha) then I think having 85% of the book be unpleasant was probably called for.
20 Critiques of AI Safety That I Found on Twitter
by dkirmani — 2022-06-23T19:23:20.013Z

In no particular order, here's a collection of Twitter screenshots of people attacking AI Safety. A lot of them are poorly reasoned, and some of them are simply ad-hominem. Still, these types of tweets are influential, and are widely circulated among AI capabilities researchers.

1

2

3

4

5

(That one wasn't actually a critique, but it did convey useful information about the state of AI Safety's optics.)

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

Conclusions

I originally intended to end this post with a call to action, but we mustn't propose solutions immediately. In lieu of a specific proposal, I ask you, can the optics of AI safety be improved?
The Limits of Automation
by milkandcigarettes — 2022-06-23T18:03:12.881Z
Link: https://milkandcigarettes.com/metaphor/limits-of-automation

Too much time is wasted because of the assumption that methods already in existence will solve problems for which they were not designed; too many hypotheses and systems of thought in philosophy and elsewhere are based on the bizarre view that we, at this point in history, are in possession of the basic forms of understanding needed to comprehend absolutely anything. — Thomas Nagel, The View from Nowhere.

One way to think about the challenges we’re facing in our day-to-day and professional lives is to classify them based on our knowledge of our knowledge.

Put another, less mind-bending way: do we know how to solve it? And if so, do we know how we’re doing it? Can we explain everything in such exact detail so that even a machine can imitate our process, or are there blanks that have to be filled in by experience? This neatly categorizes any problem to one of four kinds.

The Known Knowns are the things we know how to solve and exactly how we’re solving them. These problems are reducible to their individual constituent parts and can be automated as factories, recipes, and algorithms. The solutions are straightforward, even if sometimes finicky and exacting. They lend themselves to mnemonical, atomic, modularized forms. They are what we hope to arrive at, so that we can give the keys to a well-oiled machine and continue with the more interesting parts of our lives.

The Known Unknowns are the things we know will forever be out of reach. These are the inherent limits, difficulties, and impossibilities of physical, mathematical, and computational reality — the Bekenstein bound, Gödel’s incompleteness theorems, and computational complexity separations are just some of the results that perpetually obstruct our way. We can only accept them and work around them.

The Unknown Knowns, or I know it when I see it, are the things we do know how to solve, but aren’t exactly sure how we’re doing so. These are problems of both high-level subjective and cultivated taste and low-level sensorimotor skills. (Hence the dangerous apple-bite economical polarization in which automation forces the comfortable center into becoming either a cog or a maverick).

There’s taste: picking the right song for the mood, the right word for a sentence, the right color for a painting. And sense: reading facial expressions, navigating in a packed crowd, deciding if a pause is a break for thought or a spa...
Is CIRL a promising agenda?
by Chris_Leong — 2022-06-23T17:12:51.213Z

Richard Ngo writes

Since Stuart Russell's proposed alignment solution in Human Compatible is the most publicly-prominent alignment agenda, I should be more explicit about my belief that it almost entirely fails to address the core problems I expect on realistic pathways to AGI.
Specifying an update rule which converges to a desirable goal is just a reframing of the problem of specifying a desirable goal, with the "uncertainty" part as a red herring. https://arbital.com/p/updated_deference/… In other words, Russell gives a wrong-way reduction.
I originally included CIRL in my curriculum (https://docs.google.com/document/d/1mTm_sT2YQx3mRXQD6J2xD2QJG1c3kHyvX8kQc_IQ0ns/edit?usp=drivesdk…) out of some kind of deferent/catering to academic mainstream instinct. Probably a mistake; my current annoyance about deferential thinking has reminded me to take it out.

Howie writes:

My impression is that ~everyone I know in the alignment community is very pessimistic about SR's agenda. Does it sound right that your view is basically a consensus? (There's prob some selection bias in who I know).

Richard responds:

I think it's fair to say that this is a pretty widespread opinion. Partly it's because Stuart is much more skeptical of deep learning (and even machine learning more generally!) than almost any other alignment researcher, and so he's working in a different paradigm.

Is Richard correct and if so why? (I would also like a clearer explanation why Richard is skeptical of Stuart's agenda. I agree that the reframing doesn't completely solve the problem, but I don't understand why it can't be a useful piece).
[Link] OpenAI: Learning to Play Minecraft with Video PreTraining (VPT)
by Aryeh Englander — 2022-06-23T16:29:19.323Z

https://openai.com/blog/vpt/

We trained a neural network to play Minecraft by Video PreTraining (VPT) on a massive unlabeled video dataset of human Minecraft play, while using only a small amount of labeled contractor data. With fine-tuning, our model can learn to craft diamond tools, a task that usually takes proficient humans over 20 minutes (24,000 actions). Our model uses the native human interface of keypresses and mouse movements, making it quite general, and represents a step towards general computer-using agents.

And from the Conclusion:

VPT paves the path toward allowing agents to learn to act by watching the vast numbers of videos on the internet. Compared to generative video modeling or contrastive methods that would only yield representational priors, VPT offers the exciting possibility of directly learning large scale behavioral priors in more domains than just language. While we only experiment in Minecraft, the game is very open-ended and the native human interface (mouse and keyboard) is very generic, so we believe our results bode well for other similar domains, e.g. computer usage.
Half-baked AI Safety ideas thread
by Aryeh Englander — 2022-06-23T16:11:54.207Z

[Cross-posted from the EA Forum. The EA Forum version of this post is for both half-baked EA ideas and half-baked AI Safety ideas, whereas this version of the post is for half-baked AI Safety ideas specifically.]

I keep having ideas related to AI safety, but I keep not having enough time available to really think through those ideas, let alone try to implement them. Practically, the alternatives for me are to either post something half-baked, or to not post at all. I don't want to spam the group with half-thought-through posts, but I also want to post these ideas, even in their current state, in case some of them do have merit and the post inspires someone to take up those ideas.

Originally I was going to start writing up some of these ideas in my Shortform, but I figured that if I have this dilemma then likely other people do as well. So to encourage others to at least post their half-baked ideas somewhere, I am putting up this post as a place where other people can post their own ideas without worrying about making sure they formulate those ideas to the point where they'd merit their own post.

If you have several ideas, please post them in separate comments so that people can consider each of them individually. Unless of course they're closely related to each other, in which case it might be best to post them together - use your best judgment.

[This post was also inspired by a suggestion from Zvi to create something similar to my AGI Safety FAQ / all-dumb-questions-allowed thread, but for ideas / potentially dumb solutions rather than questions.]
Nonprofit Boards are Weird
by HoldenKarnofsky — 2022-06-23T14:40:11.593Z

Note: anything in this post that you think is me subtweeting your organization is actually about, like, at least 3 organizations. (I'm currently on 4 boards in addition to Open Philanthropy's; I've served on a bunch of other boards in the past; and more than half of my takes on boards are not based on any of this, but rather on my interactions with boards I'm not on via the many grants made by Open Philanthropy.)

Writing about ideal governance reminded me of how weird my experiences with nonprofit boards (as in "board of directors" - the set of people who formally control a nonprofit) have been.

I thought that was a pretty good intro. The rest of this piece will:

Try to articulate what's so weird about nonprofit boards, fundamentally. I think a lot of it is the combination of great power, unclear responsibility, and ~zero accountability; additionally, I haven't been able to find much in the way of clear, widely accepted statements of what makes a good board member.

Give my own thoughts on what makes a good board member: which core duties they should be trying to do really well, the importance of "staying out of the way" on other things, and some potentially helpful practices.

I am experienced with nonprofit boards but not with for-profit boards. I'm guessing that roughly half the things I say below will apply to for-profit boards, and that for-profit boards are roughly half as weird overall (so still quite weird), but I haven't put much effort into disentangling these things; I'm writing about what I've seen.

I can't really give real-life examples here (for reasons I think will be pretty clear) so this is just going to be me opining in the abstract.

Why nonprofit boards are weird

Here's how a nonprofit board works:

There are usually 3-10 people on the board (though sometimes much more). Most of them don't work for the nonprofit (they have other jobs).

They meet every few months. Nonprofit employees (especially the CEO1) do a lot of the agenda-setting for the meeting. Employees present general updates and ask for the board's approval on various things the board needs to approve, such as the budget.

A majority vote of the directors can do anything: fire the CEO, dissolve the nonprofit, add and remove directors, etc. You can think of the board as the "owner" of the nonprofit - formally, it has final say in every decision.

In practi

...
Covid 6/23/22: Under Five Alive
by Zvi — 2022-06-23T14:00:00.958Z

America has now approved the Covid vaccine for children as young as six months old. Should you vaccinate your very young child?

Yes, I believe that you should. I don’t think you absolutely have to do that. Children that young are at minimal risk from Covid. However, vaccination decreases the chance they spread the infection to others, including you, and there is some chance that there are long term effects from Covid that far exceed the risks from the vaccine. Also, even if getting Covid isn’t that dangerous, it is still universally a stress ball and a giant pain in the ass. That’s worth avoiding on its own merits.

Either way, I consider it a small mistake. The kids will be alright.

We might not be, if the Responsible Authorities get their way. A commission to look at the failures of our Covid response are going to call for an overhaul of our public health system… that concentrates power to determine response with the Federal Government, because that’s going to go great. This probably won’t happen, and there’s always some chance the new job involves doing a semi-competent job of it rather than being captured by the same craziness as the FDA and CDC. But I am not holding my breath on that.

They have forgotten what actually happened. Federalism wasn’t the problem. Federalism saved us from a much worse problem, with states able to react in ways their populations demanded and could tolerate, and in the face of an administration bent on a Blinking Spirit approach to the pandemic – that if they didn’t look at it, maybe it would go away. It would similarly have saved us if, as I would have counterfactually expected, a Hillary Clinton administration had tried to impose extreme lockdowns over a long period and red states would have been having even less of it than they did in our timeline.

There are of course alternative scenarios where the correct response was extreme measures that would have suppressed a deadly virus, or that some states would have gone and destroyed their economies if left to their own devices. In extremis, the Feds do need to take control of such matters, and I am sure that in a true full emergency they would do so and be allowed to, but I notice that making it easier for them, making it a default outcome, is not making me feel better.

Another development is there is yet another Long Covid preprint making the rounds, causing the exact same caus...
How do states respond to changes in nuclear risk
by  — 2022-06-23T12:42:15.855Z

Thanks to Matthew Gentzel for introducing this idea to me, all mistakes in this article are my own

Modelling risk responses

There are two ways we can think about reducing nuclear risk. We can reduce the probability that there's a nuclear war and we can reduce the harm of a nuclear war if one does occur. Counter-intuitively, both of these types of risk reduction can increase the overall risk of a nuclear war via changing the behaviour of actors.

A state when deciding what to do is trading off the expected harm from nuclear war and the benefits of achieving a goal. For instance, the Soviet Union desperately wanted to take West Berlin, but decided not to because the risk of it leading to war was too high. When we reduce the risk of nuclear war, either by making the war less destructive or reducing the probability that something like the invasion of Berlin leads to nuclear war, we reduce the cost of invading Berlin. If the only way to get nuclear war was a state deliberately taking actions that have a chance of leading to war, then reducing the risk of nuclear war would always increase the risk of nuclear war because it makes it more likely that it's worthwhile for states to take aggressive actions that risk nuclear war.

There are two reasons why reducing the expected harm of nuclear war could decrease the risk of nuclear war. Firstly, there's the risk of nuclear war that doesn't depend on states taking specific, risky actions. For instance, a large fraction of the risk of nuclear war probably comes from accidental use. Reducing the total yield that nuclear states possess reduces the harm of nuclear war if it does start accidentally, for instance via an early warning system failure. Secondly, the model of states deliberately choosing actions can break down. I think this is pretty common in professionalised militaries. For instance, the Soviet troops stationed in Cuba during the missile crisis decided unilaterally to shoot down an American spy plane in Cuban airspace.

We can generalise this model by modelling states as expected utility maximisers, choosing between a large range of actions, all of which carry some level of nuclear risk. If we reduce the probability of nuclear war by reducing the total probability of nuclear war it’s unlikely that this new level of nuclear risk will be optimal for agents. Unless the state is already minimising nu...