trentmkelly/LessWrong-43k huggingface dataset:
Acting Wholesomely
by owencb — 2024-02-26T21:49:16.526Z

Overview of essay series

This is the first in a collection of three essays exploring and ultimately defending the idea of choosing what feels wholesome as a heuristic for picking actions which are good for the world. I'm including a summary of the series here, before getting to the essay proper.

The two main generators of my thinking about this were:

Reflecting on major mistakes that have been made in EA, and wondering how EA might have been different at a deep level in ways that could have averted those mistakes.
Reflecting on and trying to name certain core praiseworthy behaviours of people whom I especially admire.

In the first essay,

Acting Wholesomely

(= the rest of this post)

, we see that the regular English concept of acting wholesomely can be action-guiding, especially if taken with a flavour of “paying attention to the whole of things”. In practice this involves leveraging our emotional intelligence to help recognise which actions or effects are (un)wholesome. This is a skill which many people already exercise implicitly; it can be practised, and can help us to avoid moral errors. Unwholesomeness can never be completely eliminated, so we must learn to relate to it wholesomely — rather than ignore it or treat it as toxic, we may do best to relate to it as a wise parent towards a child who is hurting others.

In the second essay,

Wholesomeness and Effective Altruism

, we will see that there are tensions between wholesome action and a simplistic interpretation of EA. But they can be unified in a “wholesome EA” perspective. Serious pursuit of wholesome action will integrate core EA principles, and serious pursuit of EA will (as I argue in the third essay) integrate a desire for wholesomeness. This might have helped to avert major historical mistakes in EA. There remain challenges to be tackled (when to act carefully vs quickly; when to focus on reducing local unwholesomeness vs more global), but these are just normal challenges in seeking to act wholesomely.

In the third essay,

Wholesome Culture

, we will ask how much wholesomeness should be an integral part of our culture. We have seen robustness-flavoured benefits in the previous essays. There are also important social effects: it is easier to act wholesomely in wholesome environments, so wholesome action can help to raise the waterline. And a reputation for who...
Getting rational now or later: navigating procrastination and time-inconsistent preferences for new rationalists

by milo_thoughts — 2024-02-26T19:38:52.436Z

This is a distillation of and reflection on O’Donoghue and Rabin’s “Doing it now or later” (see citation below)[1].

Many people struggle with procrastination or self-control. Critically, we struggle with the mismatch between current preference and future preference. Procrastination arises in situations that are unpleasant to perform but create future benefits: they have “immediate costs.” Similarly, indulgent behaviors (e.g. eating unhealthy foods) have “immediate rewards” (they taste good), while stashing away future costs. Both have a mismatch between present-self and the future-self.

The best way to “kick” this issue is to develop time-consistent preferences, placing no special value on what’s happening now–thus our choices around procrastination or indulgence will be more rational and no longer skewed by a current self-control limitation. But…that’s hard to achieve.

O’Donoghue and Rabin introduce a neat distinction (1999). A naive procrastinator, they write, expects that in the future, they won’t have the same self-control issues that they’re having now. A sophisticated procrastinator expects their imperfection: they expect their future self to also have procrastination or indulgence tendencies. With their model of time-inconsistent preferences, they demonstrate that sophisticates procrastinate less than their naive peers! This model suggests that I shouldn’t plan on having good self-control; actually, expecting my own irrationality may improve my performance. But…not so fast, because the model also indicates that sophistication may make me more likely to indulge now at a time when I’m better off waiting. The lesson for new rationalists, who haven’t yet cracked time consistency: when facing immediate costs, be realistic (like the “sophisticate”), but when facing immediate rewards, stay idealistic (like the “naif”).

Let’s get into it with an example!

You choose…you must either endure:

Three hours of an unpleasant activity today
Four hours of it one week from today

If you still prefer today, congrats on beating your procrastination tendency. As for the rest of us, people overwhelmingly prefer incurring the higher cost in a week to the lower cost today. But what if I posed it this way–the same unpleasant activity…. Would you rather endure:

Three hours on March 24th
Four hours on March 31st

Barring specific circumstances around these dates, most of us have ...
Whom Do You Trust?
by JackOfAllTrades — 2024-02-26T19:38:36.549Z

Whose perspectives or advice on A.I. safety do you most trust?

Feel free to mention forum members if their posts/comments are relevant.

Links to papers or YouTube videos are also welcome.

Also... do you know of any such experts who have a background in research related to security engineering/information security or have attempted to explain its possible relevance?
Boundary Violations vs Boundary Dissolution
by Chris Lakin — 2024-02-26T18:59:08.713Z

While at

Conceptual Boundaries Workshop

, I realized that I had been conflating two different phenomena in my mind:

Actions that violate

boundaries/membranes

, and
Actions that kill or dissolve boundaries/membranes.

This distinction is important because, towards the goal of

keeping agents safe

, it’s more important to prevent dissolution than it is to prevent violations.

Examples of actions that merely violate boundaries:

Injecting harmless saline solution into someone else’s arm.
Stealing an object from someone else’s house.
Illegally entering a country with closed borders.

Examples of actions that dissolve boundaries:

Injecting deadly poison into someone else’s arm.
Burning down someone’s house.
Invading a country and overthrowing the government.

Violating boundaries about locally betraying the sovereignty of an agent over their boundary/membrane. But on the whole the boundary still stays alive.

However, dissolving boundaries about destroying the capacity for an agent to be sovereign. The boundary dies.

This also all relates to how I understand Andrew Critch’s boundary protocol idea. He hasn’t written about this online (yet?). But in talking to him about this I realized the distinction of this post.
How I build and run behavioral interviews
by benkuhn — 2024-02-26T05:50:05.328Z

This is an adaptation of an internal doc I wrote for Wave.

I used to think that behavioral interviews were basically useless, because it was too easy for candidates to bullshit them and too hard for me to tell what was a good answer. I’d end up grading every candidate as a “weak yes” or “weak no” because I was never sure what bar I should hold them to.

I still think most behavioral interviews are like that, but after doing way too many behavioral interviews, I now think it’s possible to escape that trap. Here are my tips and tricks for doing so!

Confidence level: doing this stuff worked better than not doing it, but I still feel like I could be a lot better at behavioral interviews, so please suggest improvements and/or do your own thing :)

Before the interview

Budget 2+ hours to build

That’s how long I usually take to design and prepare a new type of interview. If I spend a couple hours thinking about what questions and follow-ups to ask, I’m much more likely to get a strong signal about which candidates performed well.

It might sounds ridiculous to spend 2 hours building a 1-hour interview that you’ll only give 4 times. But it’s worth it! Your most limited resource is time with candidates, so if you can spend more of your own time to use candidates’ time better, that’s worth it.

Think ahead about follow-ups and rubric

I spend most of those 2 hours trying to answer the following question: “what answers to these questions would distinguish a great candidate from a mediocre one, and how can I dig for that?” I find that if I wait until after the interview to evaluate candidates, I rarely have conviction about them, and fall back to grading them a “weak hire” or “weak no-hire.”

To avoid this, write yourself a rubric of all the things you care about assessing, and what follow-up questions you’ll ask to assess those things. This will help you deliver the interview consistently, but most importantly, you’ll ask much better follow-up questions if you’ve thought about them beforehand. See the appendix for an example rubric.

Focus on a small number of skills

I usually focus on 1-3 related skills or traits.

To get a strong signal from a behavioral interview question I usually need around 15 minutes, which only leaves time to discuss a small number of scenarios. For example, for a head of technical recruiting, I decided to focus my interview on the cluster of related traits ...
Hidden Cognition Detection Methods and Benchmarks 
by Paul Colognese — 2024-02-26T05:31:00.714Z

Thanks to Johannes Treutlein for discussions and feedback.

Introduction

An AI may be able to hide cognition that leads to negative outcomes from certain oversight processes (such as deceptive alignment/scheming). Without being able to detect this hidden cognition, an overseer may not be able to prevent the associated negative outcomes or include this information as part of the training signal.

For example, an AI managing a city's energy grid may begin to prioritize its own operational efficiency over the city’s actual energy needs. Knowing that its overseers will not approve of this goal, it pursues its goal via hidden cognition, undetected until issues arise.

One hope is that traces of the hidden cognition are present in the AI’s internals and could be detected and interpreted. This is part of the potential of interpretability as an alignment research agenda (see Hubinger, Olah, Nanda).

This post aims to briefly introduce the idea of hidden cognition, hidden cognition detection methods, and discuss the construction of benchmarks for evaluating such methods.

What is Hidden Cognition?

Some outputs produced by a cognitive system require it to use cognition/apply computation to solve some task that leads to that output. For example, finding the best travel route under various constraints may require the consideration of different routes, planning, and evaluation.

We say that an AI is performing hidden cognition/computation if it is solving a task to produce some output, such that evidence of this cognition is undetectable to an overseer purely based on oversight of the AI’s external behavior (input/outputs).

Note that this definition is quite broad and includes things like “attention heads

reading and writing to the residual stream

of a transformer during a forward pass to produce a token output” and “an AI performing internal optimization/search/planning to produce an action sequence that leads to a catastrophic outcome.”

Important alignment-relevant examples of hidden cognition could include cases of deceptive alignment/scheming, where the AI actively tries to deceive the overseer while planning some kind of treacherous turn.

An AI performing hidden cognition prevents oversight that can be used to prevent outcomes associated with that hidden cognition, which is concerning when the hidden cognition is related to catastrophic outcomes.

To be (somewhat) more spec...
Cellular respiration as a steam engine
by dkl9 — 2024-02-25T20:17:38.788Z
Link: https://dkl9.net/essays/cell_engine.html

When I helped an interlocutor learn about metabolism, we made an analogy between steam engines and cellular respiration. I find that people (at first) know steam engines better than cellular respiration, so I show the analogy here as a guide to cells assuming you understand engines. You could also use it in the other direction, if you learned about cells first.

If you go far enough into the details, the analogy will break down, as would any analogy. But this one holds more precisely than I or my interlocutor expected.

Glucose in the cell is like coal in the boiler. The boiler burns coal; likewise, the cell runs glucose thru the citric acid cycle (after glycolysis). The fire (the citric acid cycle and electron transport chain) heats the water (adenosine diphosphate, ADP) to a closely related, higher-energy form (adenosine triphosphate, ATP), which is steam. The steam (ATP) pushes a turbine (enables cellular work), condensing back into water (ADP).

With some changes to the engine, you can use oil in place of coal. Likewise, with some changes to what the cell does (beta-oxidation in place of glycolysis), it can use fat (or ketone bodies) in place of glucose. The fire of the boiler is essentially the same, and once the cell gets acetyl-CoA from carbs or fats, it uses the same citric acid cycle and ETC in either case.

Most of the boiler corresponds to the mitochondrion. The pipes carrying water to be boiled are like ATP synthase. ATP synthase is an enzyme embedded in mitochondrial membranes, which lets protons across the membrane in a way that phosphorylates ADP to ATP.

In some animals (including humans), the mitochondria in brown adipose tissue let pumped protons cross back for heat rather than ATP. This is like exposing the building around the steam engine to the heat of the boiler.
Rationalism and Dependent Origination?
by Baometrus — 2024-02-25T18:16:33.748Z

My understanding of the Buddhist concept of dependent origination or dependent arising is that everything arises from conditions required or conducive to its arising.

Things are the way they are for reasons. Those reasons or causalities may be unfathomably complex, but everything that is, is, and is the way it is because of conditions and reasons.

How close it this to the definition of rationalism itself? Or what are the more common LW concepts and terms for observing a thing, a pattern, a phenomenon, an object, or a system long enough to understand what it really is - to allow it to unfold and reveal itself - to the point of starting to gain insight into why it is: the causalities and conditions for its arising?
China-AI forecasts
by  — 2024-02-25T16:49:33.652Z

The rate at which China is able to advance towards TAI is a crucial consideration in for many policy questions. My current take is that, without significant political reforms which seem very unlikely while Xi is alive (although considerably more likely after his death,) it’s very unlikely that China will be able to mount a meaningful challenge to AI firms in US and allies in the race for TAI. I don’t think it requires democratic reforms are required to China to be competitive with the US and allies, but I do think rule of law reforms are likely to be required.

The first post is going to be me forecasting Chinese growth on the theory that, if China reaches rich country status, it’s likely that it will be able to compete with the US and allies for leadership in AI. I’ll write a second post looking at Chinese AI efforts in particular.

The outside view

Most countries that become middle income countries, have, thus far, stayed at middle income level. Chinese per capita income is currently at almost exactly the world average level.

The only countries (and territories) in the last 70 years that have gone low income to high income countries in the last 70 years (without oil wealth) are South Korea, Taiwan, Singapore (which does have substantial oil wealth,) and Hong Kong, although it seems very likely that Malaysia will join that club in the near future.

The majority of countries have managed to emerge from low-income status to middle income status because you only need to get a few things right. If you can get your population to urbanize, have basic rule by law so that firms have basic protection from violence, and get a high enough savings rate to accumulate physical capital you can get to middle income status just using catch up growth.

Catch up growth is the reason conceptually why middle-income status – rather than getting to a given level of GDP per capita – is the correct misuse. When growing with catch up growth you can just growth by accumulating physical capital using standard technologies that have been already been developed, like the technology for light manufacturing or civil engineering. Past this point though countries get rich by being able to develop and use technologies close to or at the frontier.

China has successfully managed to accumulate capital to utilize catch-up technologies, like steelmaking and light manufacturing. It’s quite successfull...
Ideological Bayesians
by Kevin Dorst — 2024-02-25T14:17:25.070Z
Link: https://kevindorst.substack.com/p/ideological-bayesians

TLDR: It’s often said that Bayesian updating is unbiased and converges to the truth—and, therefore, that biases must emerge from non-Bayesian sources. That's not quite right. The convergence results require updating on your total evidence—but for agents at all like us, that's impossible—instead, we must selectively attend to certain questions, ignoring others. Yet correlations between what we see and what questions we ask—“ideological” Bayesian updating—can lead to predictable biases and polarization.

Professor Polder is a polarizing figure.

His fans praise him for his insight; his critics denounce him for his aggression.

Ask his fans, and they’ll supply you with a bunch of instances when he made an insightful comment during discussions. They’ll admit that he’s sometimes aggressive, but they can’t remember too many cases—he certainly doesn’t seem any more aggressive than the average professor.

Ask his critics, and they’ll supply you with a bunch of instances when he made an aggressive comment during discussions. They’ll admit that he’s sometimes insightful, but they can’t remember too many cases—he certainly doesn’t seem any more insightful than the average professor.

This sort of polarization is, I assume, familiar.

But let me tell you a secret: Professor Polder is, in fact, perfectly average—he has an unremarkably average number of both insightful and aggressive comments.

So what’s going on?

His fans are better at noticing his insights, while his critics are better at noticing his aggression. As a result, their estimates are off: his fans think he’s more insightful than he is, and his critics think he’s more aggressive than he is. Each are correct about individual bits of the picture—when they notice aggression or insight, he is being aggressive or insightful. But none are correct about the overall picture.

This source of polarization is also, I assume, familiar. It’s widely appreciated that background beliefs and ideology—habits of mind, patterns of salience, and default forms of explanation—can lead to bias, disagreement, and polarization. In this broad sense of “ideology”, we’re familiar with the observation that real people—especially fans and critics—are often ideological.[1]

But let me tell you another secret: Polder’s fans and critics are all Bayesians. More carefully: they all maintain precise probability distributions over the relevant possibilities, and th...
Deconfusing In-Context Learning
by Arjun Panickssery — 2024-02-25T09:48:17.690Z

I see people use "in-context learning" in different ways.

Take the opening to "In-Context Learning Creates Task Vectors":

In-context learning (ICL) in Large Language Models (LLMs) has emerged as a powerful new learning paradigm. However, its underlying mechanism is still not well understood. In particular, it is challenging to map it to the “standard” machine learning framework, where one uses a training set S to find a best-fitting function f(x) in some hypothesis class.

In one Bayesian sense, training data and prompts are both just evidence. From a given model, prior (architecture + initial weight distribution), and evidence (training data), you get new model weights. From the new model weights and some more evidence (prompt input), you get a distribution of output text. But the "training step" (prior,data)→weights and "inference step" (weights,input)→output could be simplified to a single function:(prior,data,input)→output. An LLM trained on a distribution of text that always starts with "Once upon a time" is essentially similar to an LLM trained on the Internet but prompted to continue after "Once upon a time." If the second model performs better—e.g. because it generalizes information from the other text—this is explained by training data limitations or by the availability of more forward passes and therefore computation steps and space to store latent state.

A few days ago "How Transformers Learn Causal Structure with Gradient Descent" defined in-context learning as

the ability to learn from information present in the input context without needing to update the model parameters. For example, given a prompt of input-output pairs, in-context learning is the ability to predict the output corresponding to a new input.

Using this interpretation, ICL is simply updating the state of latent variables based on the context and conditioning on this when predicting the next output. In this case, there's no clear distinction between standard input conditioning and ICL.

However, it's still nice to know the level of abstraction at which the in-context "learning" (conditioning) mechanism operates. We can distinguish "task recognition" (identifying known mappings even with unpaired input and label distributions) from "task learning" (capturing new mappings not present in pre-training data). At least some tasks can be associated wi...