trentmkelly/LessWrong-43k huggingface dataset:
Four levels of understanding decision theory
by Max H — 2023-06-01T20:55:07.974Z

[Update 2023-06-09: See this comment for some caveats / motivations / context about this post.]

There are multiple levels on which an agent can understand and implement a particular decision theory. This post describes a taxonomy of understanding required for robust cooperation in the Prisoner's Dilemma.

The main point I'm trying to convey in this post is that the benefits of implementing and following a particular decision theory don't necessarily come from merely understanding the theory, even if your understanding is deep, precise, and correct.

For example, just because you correctly understand logical decision theories and their implications (e.g. why LDT agents might cooperate with each other in the prisoner's dilemma under certain conditions), that doesn't mean that you yourself are a LDT agent, even if you want to be.

Actually implementing a decision theory in real situations often requires hard cognitive work to correctly model your counterparties, as well as the ability to make your own mind and decision process model-able and legible enough to your counterparties. You must have some way of making your outwardly-professed intentions highly correlated with your actual behavior and decision process. The difficulty of creating a robust and outwardly-visible correlation between your behavior prior to making a decision, and your actual decision, varies by the kind of agent you are, and with the capabilities of the agent you are trying to cooperate with or defect against - for a human cooperating with another human, it might be very difficult. For agents whose decision process source code is easily accessible, it may be relatively easy.

The rest of this post describes different levels of comprehension an agent might have about decision theory when applied to the Prisoner's Dilemma. Achieving higher levels in the taxonomy likely requires the agent to be relatively more capable and coherent than agents that achieve only lower levels. For the final level, the agent must have some degree of control over its own internal thought patterns and decision processes.

Note: the taxonomy in this post is kind of a fake framework, but it might be useful in clearing up some common misconceptions (e.g. ones that lead to worries that this post is trying to address), and explaining why Decision theory does not imply that we get to have nice things.

Level 1: Understanding that good

...
Things I Learned by Spending Five Thousand Hours In Non-EA Charities
by jenn — 2023-06-01T20:48:03.940Z
Link: https://jenn.site/2023/05/things-i-learned-by-spending-five-thousand-hours-in-non-ea-charities/

From late 2020 to last month, I worked at grassroots-level non-profits in operational roles. Over that time, I’ve seen surprisingly effective deployments of strategies that were counter-intuitive to my EA and rationalist sensibilities.

I spent 6 months being the on-shift operations manager at one of the largest food banks in Toronto (~50 staff/volunteers), and 2 years doing logistics work at Samaritans (fake name), a long-lived charity that was so multi-armed that it was basically operating as a supplementary social services department for the city it was in(~200 staff and 200 volunteers). Both orgs were well-run, though both dealt with the traditional non-profit double whammy of being underfunded and understaffed.

Neither place was super open to many EA concepts (explicit cost-benefit analyses, the ITN framework, geographic impartiality, the general sense that talent was the constraining factor instead of money, etc). Samaritans in particular is a spectacular non-profit, despite(?) having basically anti-EA philosophies, such as:

Being very localist; Samaritans was established to help residents of the city it was founded in, and now very specialized in doing that.
Adherence to faith; the philosophy of The Catholic Worker Movement continues to inform the operating choices of Samaritans to this day.
A big streak of techno-pessimism; technology is first and foremost seen as a source of exploitation and alienation, and adopted only with great reluctance when necessary.
Not treating money as fungible. The majority of funding came from grants or donations tied to specific projects or outcomes. (This is a system that the vast majority of nonprofits operate in.)
Once early on I gently pushed them towards applying to some EA grants for some of their more EA-aligned work, and they were immediately turned off by the general vibes of EA upon visiting some of its websites. I think the term “borg-like” was used.

Over this post, I’ll be largely focusing on Samaritans as I’ve worked there longer and in a more central role, and it’s also a more interesting case study due to its stronger anti-EA sentiment.

Things I Learned

Long Term Reputation is Priceless
Non-Profits Shouldn’t Be Islands
Slack is Incredibly Powerful
Hospitality is Pretty Important

For each learning, I have a section for sketches for EA integration – I hesitate to call them anything as strong as recommendations, beca...
self-improvement-executors are not goal-maximizers
by bhauth — 2023-06-01T20:46:16.964Z

Agents are adaptation-executors with adaptations that accomplish goals, not goal-maximizers. Understanding agents as maximizing goals is a simplification used by humans to make them easier to understand. This is as true when the goal is self-improvement as it is with anything else.

Let's think about self-improvement in terms of goal specification. You want to accomplish a goal G, so as you add levels, the task becomes more indirect:

make a system S0 to accomplish G

make S1 to make a S0 that accomplishes G

make S2 to make a S1 that makes a S0 that accomplishes G

Some people think that you can just tell S2 to do G and it will figure things out, but those people are interpreting S2 as a goal-maximizer, and agents are adaptation-executors. You have to specify the adaptations that will cause the actions that create a suitable S1. Reinforcement learning is a method (not the only method!) for specifying adaptations that accomplish a certain goal in the training environment, not a way of giving a system an inherent goal. I do think AI self-improvement is possible - I wrote this. So, I do think in some sense that specification of S2 can be easier than direct specification of S0, but I'm not under any illusions that it's the same problem.

I'm posting this because I saw this post (from yesterday) saying that people should try designing AIs that want to be shut down as a tripwire for dangerous recursive self-improvement, and that idea is based on the above misunderstanding. Also, "shut yourself down" is a particularly poor goal for 2 reasons.

It's binary. Training ML systems works much better when you can indicate partial accomplishment of the goal.

It's ill-defined. If you try to train a system to "shut itself down" what you are training it to do is "take provided action X that also coincidentally causes shutdown". You then need to make things harder if you're testing development of more-capable systems, and if you then take away X but leave other shutdown route Y, it's no more likely to do Y than anything else. Having negative average rewards over time doesn't train it to do Y until it does Y a bunch and gets reinforcement learning on doing Y.
Experimental Fat Loss
by johnlawrenceaspden — 2023-06-01T20:26:18.296Z

With the end of the world nigh, and a public panic about to start, this seems an ideal time to worry about weight loss and the obesity epidemic.

Coincidentally, for the first time in my life, I'm getting fat.

SlimeMoldTimeMold's 'Chemical Hunger' series

https://slimemoldtimemold.com/2021/07/07/a-chemical-hunger-part-i-mysteries/

seemed to draw a lot of interest round these parts, and even if it's not lithium

https://www.lesswrong.com/posts/7iAABhWpcGeP5e6SB/it-s-probably-not-lithium

it does seem to me that the molds raise some most interesting questions.

I find the whole 'seed oil' craziness to be a compellingly interesting argument, although, as Scott Alexander wrote:

https://slatestarcodex.com/2020/03/10/for-then-against-high-saturated-fat-diets/

it does seem to be flat wrong. But I think it's important to be interested in ideas that look like they have to be right but aren't.

I want to draw everyone's attention to the 'Experimental Fat Loss' substack

https://exfatloss.substack.com

Which seems to me the very model of sanity and empiricism, a little like reading the early Proceedings of the Royal Society, were Robert Hooke to have become interesting in losing weight.

In particular his definition of what it would mean for a diet to 'work'

https://exfatloss.substack.com/p/the-definition-of-diet-success

He does seem to have found something that works for him,

https://exfatloss.substack.com/p/losing-43lbs-in-144-days-on-ex150-diet

and I find him sufficiently trustworthy-seeming that I'm going to see if it will work for me, and if it doesn't, use his methods to play around and see if I can find something that does.

But I would welcome the comments of wiser and more sceptical persons on all these things.
Yudkowsky vs Hanson on FOOM: Whose Predictions Were Better?
by 1a3orn — 2023-06-01T19:36:48.351Z

TLDR

Starting in 2008, Robin Hanson and Eliezer Yudkowsky debated the likelihood of FOOM: a rapid and localized increase in some AI's intelligence that occurs because an AI recursively improves itself.

As Yudkowsky summarizes his position:

I think that, at some point in the development of Artificial Intelligence, we are likely to see a fast, local increase in capability—“AI go FOOM.” Just to be clear on the claim, “fast” means on a timescale of weeks or hours rather than years or decades; and “FOOM” means way the hell smarter than anything else around, capable of delivering in short time periods technological advancements that would take humans decades, probably including full-scale molecular nanotechnology. (FOOM, 235)

Over the course of this debate, both Hanson and Yudkowsky made a number of incidental predictions about things which could occur before the advent of artificial superintelligence -- or for which we could at the very least receive strong evidence before artificial superintelligence.

On the object level, my conclusions is that when you examine these predictions, Hanson probably does a little better than Yudkowsky. Although depending on how you weigh different topics, I could see arguments from "they do about the same" to "Hanson does much better."

On one meta level, my conclusion is that Hanson's view --- that we should try to use abstractions that have proven prior predictive power -- looks like a pretty good policy.

On another meta level, my conclusion -- springing to a great degree from how painful seeking clear predictions in 700 pages of words has been -- is that if anyone says "I have a great track record" without pointing to specific predictions that they made, you should probably ignore them, or maybe point out their lack of epistemic virtue if you have the energy to spare for doing that kind of criticism productively.

Intro

There are number of difficulties involved in evaluating some public figure's track record. We want to avoid cherry-picking sets of particularly good or bad predictions. And we want to have some baseline to compare them to.

We can mitigate both of these difficulties -- although not, alas, eliminate them -- by choosing one document to evaluate: "The Hanson-Yudkowsky Foom Debate". (All future page numbers refer to this PDF.) Note that the PDF includes the (1) debate-via-blogposts which took place on OvercomingBias, (2) an...
Progress links and tweets, 2023-06-01
by jasoncrawford — 2023-06-01T19:03:14.459Z
Link: https://rootsofprogress.org/links-and-tweets-2023-06-01

Richard Rhodes, illegal floor plans, and cyborg insects

Announcements/opportunities

Plymouth Street can help top talent get an O-1 or EB-1 visa
Econ Twitter science fiction summer book club, July 12

Works in Progress Issue 11

Olivine weathering: “Fixing global warming is a simple matter of applying energy to an unfavorable material situation.” (Related, sulfur stratospheric injection is cheap and easy for solar radiation management)
Edison as a technical entrepreneur (by @eric_is_weird)
Also: why Britain is NIMBY, how we solved acid rain, and more in the full issue

Other links

Richard Rhodes interview by Dwarkesh Patel
The birth of the electrical grid (by @_brianpotter)
Anton Howes in the Telegraph on a new Great Exhibition (based on the full essay featured in the last digest)
The Manual of Illegal Floor Plans

News

Neuralink receives FDA approval for a human clinical study (via @shivon)
Oklo announces two more sites for nuclear plants (via @caorilne)

Queries

Any private companies that hire metascientists to improve their internal processes?
What is the best history of the climate crisis?
Why did the open hearth process use external fuel, but Bessemer did not?
As an economy grows, do zero-sum activities occupy a larger share?
Before no-fault divorce, how widely did states differ in the “fault”?
Is there a difference between an “institute” and a “center”?

Quotes

Extending the number of operations we can perform without thinking about them
What does rule of law require?

Tweets

Why I am reluctant to share stories about supposed breakthroughs. E.g., did AI really discover a new antibiotic against resistant bacteria? (Domain experts: yes, but this isn’t actually interesting/important)
Allen vs. Mokyr on the limiting factor of innovation (excerpted from my recent post)
I have questions for Bret Devereaux on “Why No Roman Industrial Revolution?”
RL can’t learn long-horizon tasks with sparse rewards; LLMs can (e.g., in Minecraft)
Alarmists, accelerationists, and pragmatists on AI (see also my essay on solutionism)
The “mac” in “tarmac” comes from John L. McAdam, 1756–1836, British surveyor
We already have a word for “degrowth.” It’s called impoverishment (@paulg)
Why does it take three years to restore… a carousel?
Childhood should be serious; your life should be serious
Cyborg insects

Charts

We are not running out of resources
Uncertainty about the future does not imply that AGI will go well
by Lauro Langosco — 2023-06-01T17:38:09.619Z

Subtitle: A partial defense of high-confidence AGI doom predictions.

Introduction

Consider these two kinds of accident scenarios:

In a default-success scenario, accidents are rare. For example, modern aviation is very safe thanks to decades of engineering efforts and a safety culture (e.g. the widespread use of checklists). When something goes wrong, it is often due to multiple independent failures that combine to cause a disaster (e.g. bad weather + communication failures + pilot not following checklist correctly).
In a default-failure scenario, accidents are the norm. For example, when I write a program to do something I haven’t done many times already, it usually fails the first time I try it. It then goes on to fail the second time and the third time as well. Here, failure on the first try is overdetermined―even if I fix the first bug, the second bug is still, independently, enough to cause the program to crash. This is typical in software engineering, and it can take many iterations and tests to move into the default-success regime.

See also: conjuctive vs disjunctive risk scenarios.

Default-success scenarios include most engineering tasks that we have lots of experience with and know how to do well: building bridges, building skyscrapers, etc. Default-failure scenarios, as far as I can tell, come in two kinds: scenarios in which we’re trying to do something for the first time (rocket test launches, prototypes, new technologies) and scenarios in which there is a competent adversary that is trying to break the system, as in computer security.[1]

Predictions on AGI risk

In the following, I use P(doom) to refer to the probability of an AGI takeover and / or human extinction due to the development of AGI.

I often encounter the following argument against predictions of AGI catastrophes:

Alice: We seem to be on track to build an AGI smarter than humans. We don’t know how to solve the technical problem of building an AGI we can control, or the political problem of convincing people to not build AGI. Every plausible scenario I’ve ever thought or heard of leads to AGI takeover. In my estimate, P(doom) is [high number].

Bob: I disagree. It’s overconfident to estimate high P(doom). Humans are usually bad at predicting the future, especially when it comes to novel technologies like AGI. When you account for how uncertain your predictions are, your estimate...
What are the arguments for/against FOOM?
by FinalFormal2 — 2023-06-01T17:23:11.698Z

The most promising AI model afaik right now is LLMs. My understanding is that progress in LLMs is slowing down as the gains we can make from more data and run times dry up, and that opportunity right now lies with smaller, more efficient models and creative applications of the existing technology.

I also can't see how LLMs could FOOM when creating them takes a lot of time and processing power. If an LLM understood themselves well enough to alter individual nodes and expect improvements then that would make sense, but that sounds hard enough that I would expect they're already basically superintelligence?

All that being said, I have not finished Nick Bostrom's Superintelligence, so I'd be very interested to hear what I might be missing here.
Change my mind: Veganism entails trade-offs, and health is one of the axes
by Elizabeth — 2023-06-01T17:10:02.075Z

Introduction

To me, it is obvious that veganism introduces challenges to most people. Solving the challenges is possible for most but not all people, and often requires trade-offs that may or may not be worth it. I’ve seen effective altruist vegan advocates deny outright that trade-offs exist, or more often imply it while making technically true statements. This got to the point that a generation of EAs went vegan without health research, some of whom are already paying health costs for it, and I tentatively believe it’s harming animals as well.

Discussions about the challenges of veganism and ensuing trade-offs tend to go poorly, but I think it’s too important to ignore. I’ve created this post so I can lay out my views as legibly as possible, and invite people to present evidence I’m wrong.

One reason discussions about this tend to go so poorly is that the topic is so deeply emotionally and morally charged. Actually, it’s worse than that: it’s deeply emotionally and morally charged for one side in a conversation, and often a vague irritant to the other. Having your deepest moral convictions treated as an annoyance to others is an awful feeling, maybe worse than them having opposing but strong feelings. So I want to be clear that I respect both the belief that animals can suffer and the work advocates put into reducing that suffering. I don’t prioritize it as highly as you do, but I am glad you are doing (parts of) it.

But it’s entirely possible for it to be simultaneously true that animal suffering is morally relevant, and veganism has trade-offs for most people. You can argue that the trade-offs don’t matter, that no cost would justify the consumption of animals, and I have a section discussing that, but even that wouldn’t mean the trade-offs don’t exist.

This post covers a lot of ground, and is targeted at a fairly small audience. If you already agree with me I expect you can skip most of this, maybe check out the comments if you want the counter-evidence. I have a section addressing potential counter-arguments, and probably most people don’t need to read my response to arguments they didn’t plan on making. Because I expect modular reading, some pieces of information show up in more than one section. Anyone reading the piece end to end has my apologies for that.

However, I expect the best arguments to come from people who have read...
The unspoken but ridiculous assumption of AI doom: the hidden doom assumption
by Christopher King — 2023-06-01T17:01:49.088Z

Epistemic Status: Strawman (but maybe also Steelman?)

Advocates for AI existential safety (a.k.a. "AI doomers"), have long been fear-mongering about AGI. Reasonable people have long pointed out how ridiculous the existence of AGI would be. The doom position has many arguments for how AGI could destroy humanity, but they all rely uncritically on the same unempirical, unproven, unfalsifiable, sci-fi-ish assumption:

The hidden doom assumption: The future is something which exists.

Indeed, most people who have thought about and accepted the hidden doom assumption also accept AI doom. But rarely do they spell this assumption out. Although it is clearly laughable, I will now proceed to demolish the doom position by tearing apart this assumption. It is sad that people can be so scared of a hypothetical concept.

Philosophical naivety

The position that the future exists, known as eternalism is widely known as philosophically naive. Quoting a famous philosopher and historian:

In the vast, beguiling constellation of human intellectual endeavors, few enigmas are as tantalizing as the concept of 'the future'. Yet, I dare assert, dear reader, it is a mere illusion, a phantasm conceived within the human propensity for forward-thrusting ambition. Harken to the sage words of Heraclitus, who postulated that "no man ever steps in the same river twice, for it's not the same river and he's not the same man." Indeed, the very fabric of time, like the ceaseless flux of Heraclitean river, is one of relentless and unrepeatable change. Thus, one might indeed conclude, borrowing the lofty wit of Hegel, that the future is but an "Unhappy Consciousness", a state of desire for an entity that does not yet, and can never truly exist.

One may counter-argue, armed with the deceptively empirical utterances of that notorious rationalist Descartes - "cogito ergo sum" - that the future exists in anticipation. Yet, this line of thought is patently absurd, redolent with the absurdity of Sartre's existential angst or Camus' 'Sisyphus' futilely heaving his boulder. For if we contend that the future exists in potentiality, we step upon the slippery slope of Zeno's paradoxes, perpetually dividing time into smaller and smaller fractions, ad infinitum. Alas, it is the very nature of the future, as the insightful Nietzsche succinctly put, to be "an ocean of infinite tomorrows," each tomorrow forever dancing jus

...