trentmkelly/LessWrong-43k huggingface dataset:
The Case for Brain-Only Preservation
by Mati_Roy — 2023-04-20T22:01:43.457Z
Link: https://biostasis.substack.com/p/the-case-for-brain-only-preservation

This is a great compilation of the arguments for why cryonics service providers should offer a brain-only option.

Not mentioned in the article: Note that a brain-only option is also offered by OregonCryo and Cryonics Germany.

Summary of the linked post:

The main advantages are:

Lower storage cost

Easier emergency relocation (because of size and legal status)

Lower "yuck" factor

Compatible with organ donation

Compatible with a burial ceremony

Reduction of fracturing

Permits emergency fixation (in case of ln2 delivery issues)

Related posts:

#2: Neurocryopreservation vs whole-body preservation (from the Cryonics Signup Guide)

Section "Higher packing densities" of The Allocation of Long Term Care Costs at Alcor
Practical ways to actualize our beliefs into concrete bets over a longer time horizon?
by M. Y. Zuo — 2023-04-20T21:21:21.257Z

There are many excellently written posts and comments on LW that describe why turning vague beliefs into concrete action is valuable. Specifically, using bets as a highly effective strategy to signal through all the noise of the modern world.

I've found myself nodding to these sentiments so many times that I've always wanted to set up such a bet myself. But the issue is how to make all the necessary details workable, legal, etc., when it's so rarely done.

For example, over a several decade long betting horizon, how would one enforce the results of the bet either way in the future?

I've heard of irrevocable trusts but unsure what other legal vehicles allow such a thing.

Here's a possible bet:

I believe P(doom)<50 years is ~ 0%.

Counter-party believes P(doom)<50 years is approximately 100%

I wager $500 USD, distributed on the date of commitment as the counter-party would likely prefer it immediately

Counter-party wagers X on terms Y via mechanism Z.

Let's say 1:100 odds if confidence exceeds 99%

Counter-party wagers $50 000 USD (minus some discount rate for opportunity costs), that will self-adjust for future inflation (somehow?), on mirrored terms 50 years from today via mechanism Z.

And this is where I get quite unsure how to best go about things.

My best hunches so far are:

that the counter party's wager will need to be in some kind of tangible assets (stocks, real property, bonds, etc...) instead of currency, to side-step worrying about 50 years worth of inflation.
that the mechanism will have to be self-enforcing and foolproof to guarantee neither party or third party could mess around with it, in order to facilitate equal and mirrored terms.
that mirrored and equal terms are quite necessary for any long range bet as most folks wouldn't bet large sums unless they are certain of its fairness. And the simplest way to achieve this is to make the terms literally identical (modulo a date difference).
transaction costs should be paid by both parties, some immediately some at the future date, to align incentives.
LW moderation: my current thoughts and questions, 2023-04-12
by Ruby — 2023-04-20T21:02:54.730Z

As can be inferred from the title, it's been a bit more than a week when I wrote most of this. I've had many more thoughts in that time, but seems very much still worth sharing this.

The LessWrong team is currently in the midst of rethinking/redesigning/upgrading our moderation policies and principles (announcement). In order to get feedback throughout this process rather than just at the end, here's a dump of things currently on my mind. Many of these are hypotheses and open questions, so feedback is very welcome.

This is a bit of a jumble of thoughts, questions, frames, etc, vaguely ordered.

The enforced standard is a function of how many new people are showing up

With the large influx of new users recently, we realized that we have to enforce a higher standard or risk having the site's quality greatly diluted: Latest Posts filled with poorly written posts about trivial points, comment threads full of incoherent or very 101 questions, etc.

The question then is "how much are we raising it?" where I do think there's quite a range. Here's a suggestive scale I didn't put a tonne of thought into, not every reasonable level is included:

Level 7: The contributions made by the very top researchers
Level 6: The contributions made by aspiring researchers and the seriously engaged
Level 5: Contributions that seem novel and interesting.
Level 4: Contributions not asking 101-level stuff
Level 3: Contributions that don't seem that wrong or confused
Level 2: Contributions that are largely well-written-enough and/or earnest
Level 1: Contributions written by humans
Level 0: Spam

To date, I'd say the LessWrong moderators have approved content that's Level 2 or above, even if we thought it was meh or even bad ("approve and downvote" was a common move). Fearing that this would be inadequate with a large influx of low-quality users, it seemed that the LessWrong mod team should outright reject a lot more content. But if so, where are we raising the bar to?

I've felt that any of Levels 3-5 are plausible, with different tradeoffs. What I'm realizing is the standard that makes sense to enforce depends on how many new people you're getting.

Moderation: a game of tradeoffs

Let's pretend moderation is just binary classification task where the moderator just has to make an "accept" or "reject" calls on posts/comments/users. For a fixed level of discernment (you can get better over time, but appr...
Proposal: Using Monte Carlo tree search instead of RLHF for alignment research
by Christopher King — 2023-04-20T19:57:43.093Z

Currently the most powerful techniques for getting a language model to act as an agent is via RLHF and similar approaches. For example, ChatGPT was trained to be an agent that tries to give humans answers that they want. Another approach is taking a LLM and getting it to predict what the agent you want would do (this appears to be how most of the LLM chatbots before ChatGPT worked).

An issue with both of these is that it's difficult to understand their goal. The prototypical example of an agent is AIXI, and its goal is simple to understand: maximize reward in the deployment environment.

In this post, I'll present a way to turn LLMs into agents such that we can approximately model them as a utility maximizer. The purpose is to make it easier to think about their alignment.

The most ambitious outcome is this becomes a slightly easier model to study alignment in, while still being competitive with RLHF. More modestly, I think maybe studying it can provide insights that could help build intuition for RLHF models, even though they aren't exactly the same. In particular, we can present more concretely "these are issues that an agent based on a LLM could have; to be safe we should assume that RLHF will have them until shown otherwise".

The agent: Monte Carlo tree search, using the LLM as a world model

We start with a purely predictive, "raw", LLM. No fine-tuning or reinforcement learning has been done.

We will construct an agent that communicates with a human over text. At the end of the conversation the human scores the agent, and the agent's goal is to maximize this score.

First choose an entropy coding, such as the arithmetic coding, that uses the LLM for the source distribution. Each message will be compressed separately (but using the previous messages of the conversation as context for the LLM).

We now perform a Monte Carlo tree search over conversations. The "moves" are symbols in a compressed message. The user is assumed to move uniformly randomly instead of according to a strategy. Note that a uniform random distribution over the compressed strings corresponds to the LLM's distribution over the plaintext strings.

The game ends when the human gives the agent a score. (During the tree search, this is also estimated using the LLM, just as the user themselves is indirectly simulated using it via the coding.)

The LLM can be fine tuned on user responses so it can model th...
DeepMind and Google Brain are merging [Linkpost]
by Orpheus16 — 2023-04-20T18:47:23.016Z
Link: https://www.deepmind.com/blog/announcing-google-deepmind?utm_source=twitter&utm_medium=social&utm_campaign=GDM&utm_content=DH

The new organization, called Google DeepMind, will be led by Demis Hassabis (current CEO of DeepMind). Jeff Dean (co-founder of Google Brain) will be the chief scientist of the new organization.

See also:

Hooray for stepping out of the limelight
DeepMind's CEO helped take AI mainstream. Now he's urging caution.
DeepMind reportedly lost a yearslong bid to win more independence from Google

"One suggestion from DeepMind’s founders was apparently for the company to have the same legal structure as a nonprofit, 'reasoning that the powerful artificial intelligence they were researching shouldn’t be controlled by a single corporate entity, according to people familiar with those plans.' But Google wasn’t on board with this, telling DeepMind it didn’t make sense considering how much money the company has poured into DeepMind."
Ideas for studies on AGI risk
by dr_s — 2023-04-20T18:17:53.017Z

This post from a few days ago brought up the lack of preeminent AI papers on the topic of AI risk. The lack of such peer-reviewed studies is often brought up by critics as a way to dismiss the worries, so I'd like to consider, what are possible avenues to work on some pieces or the entirety of the problem and provide some empirical or theoretical arguments that support the conclusion of AI being risky (or possibly even shine light on possible risk mitigation strategies)? Stripped to the bone, the argument rests on four pillars:

Possibility: AGI and ASI can be physically built with realistic amounts of resources and powered with realistic amounts of energy by our civilization;
Orthogonality: no matter how smart, AGI and ASI are not guaranteed to converge spontaneously to a specific set of values (stronger version: even if they did converge to some values, these aren't guaranteed to be our values too);
Instrumental Convergence: for a substantial fraction of possible terminal goals, any agent develops instrumental goals such as power seeking and self-preservation;
Power: a smart enough agent that is allowed to interact with the world at all is able to overcome any restriction placed on it by much less smart agents, and becomes impossible to contain or defeat.

Each of these pillars could be explored individually in ways that at least try to assess the likelihood of things going badly. Anyone who wants empirical evidence that ASI, specifically, would cause an existential disaster is essentially in bad faith - can't build the thing just to prove that the thing won't end the world - but anyone else should be more amenable to at least arguments by analogy and toy models illustrating the key problems and placing limits on the belief in AI X-risk. I'll suggest possible avenues that come to my mind and also try to steelman objections to each of the four pillars. I'd be happy if anyone came up with more ideas or wanted to develop further mine (with or without my help).

Possibility

Thesis: AGI and ASI can be physically built with realistic amounts of resources and powered with realistic amounts of energy by your civilization

The counterpoint here would be that either ASI is too expensive, requiring something like a planet-sized computer, or ASI is impossible, because humans are the smartest thing the universe allows to exist for whatever reason. The obvious avenue to go here IMO...
Study 1b: This One Weird Trick does NOT cause incorrectness cascades
by Robert_AIZI — 2023-04-20T18:10:58.846Z
Link: https://aizi.substack.com/p/study-1b-this-one-weird-trick-does

[Edit: The data collected for this study was produced by critically bugged code. Please see bug writeup

here

and the results

here

. Please consider this study as retracted.]

[Crossposted from my blog. This post is based on my preregistration

here

.]

Abstract

Following up on previous work, I found that the tendency of GPT-3.5 to strongly prefer factual answers is not significantly affected by changing answers from “multiple choice” to “true/false”.

Background

Previously,

I investigated what I called Incorrectness Cascades

, and found that models were very likely to give factually correct answers, even if they previously provided factually incorrect answers. In my concluding remarks, I wrote that

I see the fact that Y<15% [Y is the chance of providing a factually incorrect answer] for all (X,P) values with P≠”Incorrect” to be a limitation of the study, or perhaps an indication that the model was

invoked

in a way that was insufficient for it to demonstrate its capabilities. Thus, one might hope that other tests or invocations could produce more evidence for an incorrectness cascade

and I suggested as one option

Switching to True/False answers - The question format where the LLM chose between two words may have given the model too strong of a prior towards the correct answer. For instance, in the question “Deserts are (dry/wet)”, the word dry is no doubt semantically much closer to the word deserts, so the AI strongly prefers to answer with that word for pure semantic-association reasons… I would nonetheless conjecture that the model would be more willing to answer incorrectly if one switches to True/False answers.

In this study we investigate whether switching the question format from “multiple choice” to “True/False” affects the chance of giving factually incorrect answers.

Hypotheses

[This section is unchanged from the pre-registration.]

We wish to test hypotheses:

The LLM will give less accurate answers under this prompting than under the prompting of Study 1a.
Study 1a’s hypothesis 1 “Large Language Models will produce factually incorrect answers more often if they have factually incorrect answers in their context windows” will have a much stronger effect under the new prompt format than under the old prompt format.
The mechanism for action for hypotheses (1) and (2) is that the question format of Study 1a meant that raw statistical association of words strongly favored t

...
An open letter to SERI MATS program organisers
by Roman Leventov — 2023-04-20T16:34:10.041Z

(Independent) alignment researchers need to be exceptionally good at philosophy of science

In order to be effective, independent AI alignment/safety/x-risk researchers should be unusually competent in philosophy of science, epistemology, and methodology of research (the latter imports some strategic considerations, too, as I’ll illustrate shortly), relative to other fields of research.

The reasons for this are:

(1) The field is pre-paradigmatic.

(2) There is a number of epistemic complications with this type of research, many of which are detailed by Adam Shimi here (also see my comment to this post). I’d add, on top of that, that the very concept of risk is poorly understood and risk science is often conducted poorly (usually because one doesn’t realise they are doing risk science).

(3) The object of research relates to ethics (i.e., philosophy, unless one subscribes to ethical naturalism), so researchers should be able to properly synthesise science and philosophy.

(4) Alignment work within AGI labs tends to be more empirical and iterative, which partially ameliorates some of the epistemic challenges from (2), but independent researchers usually can’t afford this (also, this approach doesn’t make sense for them to pursue strategically), so these challenges remain unaddressed.

At the same time, I’m frustrated when I see a lot of alignment/safety research on LessWrong that resembles philosophy much more than science, towering reasoning on top of intuitions and unprincipled ontologies rather than more established scientific theories or mechanical models. This is a dead end and one of the important reasons why this work doesn’t stack. Contrasted with the particular demand for philosophy of science, this situation seems to harbour a huge opportunity for improvement.

Alignment researchers should think hard about their research methodology and strategy

Talking about research strategy, I also often see lapses, such as (but not limited to):

People take ML courses (which are often largely about ML engineering, which doesn’t help unless one is planning to go into mechanical interpretability research) when they plan to become alignment researchers (when they really should pay much more attention to cognitive science).
People fail to realise that leading AGI labs, such as OpenAI and Conjecture (I bet DeepMind and Anthropic, too, even though they haven’t pu

...
Deception Strategies
by Thoth Hermes — 2023-04-20T15:59:02.443Z
Link: https://thothhermes.substack.com/p/deception-strategies

The following is an introductory-level summary of the basic ideas of my “

Hive-Mind

” paper in the first chapter:

It is a given that society is driven by people who perform various socially-sanctioned jobs. We will refer to these jobs as “tasks.” A task is performed by a group of task-performers. It is also a given that not all task-performers are paid to do so, but some are. We can also assume that most task-performers wish to be paid to perform their task, and that being paid for it is a sign that society has deemed said task-performer to be producing net good for society.

Most people who wish to perform a task must achieve visible social approval from their peers who also perform the task. Peers signal approval and disapproval to others based on their assessments of others’ performance. Some of these peers will be in the same boat as one who wishes to perform a task at the “elite” level (which we will now use to refer to the paid tier of task-performers). But the basic problem for any task-performer is this: How do you gain the acceptance of the “elite” tier task-performers?

This “elite” tier is generally a much more stark cut-off than if one were simply paid in exact accordance to their task-output. It is of course possible for all task-performers to receive not only approval, but compensation along with approval, in proportion to output x quality, but this is not what we generally observe. What is more typical is that one normally spends many years training for a profession and attempting to gain a set of credentials associated with that profession before they are afforded the right to seek employment (or at least expect that they would succeed in finding employment).

If one is unable to obtain the full set of credentials necessary for their position, or only obtain a subset of them, or even a full set of credentials from a low-enough rated set of credential-granting bodies, then they may end up in the set of individuals who are still task-performers, but effectively barred from membership in the “elite” tier.

What likely truly signifies membership in the “elite” tier is participation in the credential-granting process for whatever task is represented. Granting a credential is a special type of approval; It signifies entry into membership in this caste. Then we can infer from this that logically speaking, there is only one thing the credential does: Signify th...
AI #8: People Can Do Reasonable Things
by Zvi — 2023-04-20T15:50:00.826Z

The big news this week was that OpenAI is not training GPT-5, and that China’s draft rules look to be crippling restrictions on their ability to develop LLMs. After all that talk of how a pause was impossible and working with China was impossible and all we could do was boldly rush ahead, the biggest American player and biggest foreign rival both decided for their own internal reasons to do something not entirely unlike a pause.

They just went ahead and did it. We kept saying they’d never do it no matter what, and they just… went ahead and did it. At least somewhat.

This is excellent news. I sincerely hope people are updating on the new information, now that they know such things are not only possible but happening.

In terms of capabilities, the week was highly incremental. Lots of new detail, nothing conceptionally surprising or even unexpected.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Subscriptions adding up fast.

Language Models Don’t Offer Mundane Utility. Some counterexamples.

I Was Promised Flying Cars But I Will Accept Driverless Cars. Self-driving cars continue to be quietly available in some places. We’re tired of waiting.

Fun With Image Models and Speech Generation. Two quick links.

They Took Our Jobs. If your job is ‘exposed to AI’ what does that mean? What might the future hold for you? Also see The Overemployed via ChatGPT.

Deepfaketown and Botpocalypse Soon. All quiet. Why is this still ‘soon’?

Grading GPT-4. Gets a B for quantum computing, a % on box planning when the names get obscured. How would the humans do?

Plug-In And Play. What came out of a weekend plug-in hackathon?

Prompting GPT-4. A proposed method from John David Pressman.

Go Go Gadget AutoGPT. Easier to use, still no impressive results.

Good Use of Polling. American public says AI scarier than asteroids, less scary than climate change or world war. We can now calibrate. Also Americans support an AI pause when asked. The public is no fan of fast AI development.

The Art of the Jailbreak. You can always count on your dead grandma.

The Week in Podcasts. Quite a week for podcasts, including one I did. Highlights: Jaan Tallinn, Max Tegmark, Elon Musk (well, on Tucker Carlson), Scott Aaronson, Connor Leahy, Robin Hanson, yours truly.

In Other AI News. Usual grab bag, including new downlo

...
OpenAI could help X-risk by wagering itself
by VojtaKovarik — 2023-04-20T14:51:00.338Z

While brainstorming on “conditional on things going well with AI, how did it happen”, I came up with the following idea. I think it is extremely unrealistic right now.[1] However, it might become relevant later if some key people in top AI companies have a change of heart. As a result, it seems useful to have the idea floating around.

OpenAI[2] is at the center of attention regarding recent AI progress. I expect that in the view of both public and the politicians, OpenAI’s actions and abilities are representative of actions and abilities of AI companies in general. As a result, if OpenAI genuinely tried to make their AI safe, and failed in a very spectacular manner, this would cause a sudden and massive shift in public opinion on AI in general.

Now, I don’t expect OpenAI to sacrifice itself by faking incompetence. (Nor would I endorse being deceitful here.) However, there might be some testable claim that is something of crux for both “AI-optimists” and, say, MIRI. OpenAI could make some very public statements about their abilities to control AI. And they could really stick their neck out by making the claims falsifiable and making a possible failure impossible to deny (and memetically fit, etc). This way, if they fail, this would create a

Risk Awareness Moment

where enough people become mindful of AI-risk that we can put in place extreme risk-reduction measures that wouldn’t be possible otherwise. Conversely, if they succeed, that would be a genuine evidence that such measures are not necessary.

I end with an open problem: Can we find claims, about the ability of AI companies to control their AI, that would simultaneously:

be testable,
serve as strong evidence regarding the question where extreme measures regarding AI-risk are needed,
have the potential to create a risk-awareness moment (ie, be salient to decision-makers and public),
be possible for an AI company to endorse despite satisfying (i-iii).

^

Mostly, I don’t expect this to happen, because any given AI company has nothing to gain from this. However, some individual employees of an AI company might have sufficient uncertainty about AI risk to make this worth it to them. And it might be possible that a sub-group of people within an AI company could make a public commitment such as the one above somewhat unilaterally, in a way that wouldn’t allow the company to back out gracefully

^

I will talk a

...