trentmkelly/LessWrong-43k huggingface dataset:
Humans, chimpanzees and other animals
by gjm — 2023-05-30T23:53:08.295Z

[Epistemic status: underinformed musings; I am posting this not because I am sure of anything in it but because the point seems important and I don't recall seeing anyone else make it. Maybe that's because it's wrong.]

A common analogy for the relationship between postulated superhumanly-smart AIs and humans is the relationship between humans and chimpanzees. See, e.g., https://intelligence.org/2017/12/06/chollet/ where Eliezer Yudkowsky counters various arguments made by François Chollet by drawing this analogy.

It's pretty compelling. Humans are a bit like chimps but substantially smarter ... and, lo, humans control the future of life on earth (including, in particular, the future of chimpanzee life on earth) in a way chimps absolutely do not.

But wait. Humans coexist with chimps, and are smarter, and are utterly dominant over them: fair enough. But surely we want to do better than a sample size of 1. What other cases are there where animals of different intelligence levels coexist?

Well, for instance, chimps coexist with lions and leopards. Are chimp-leopard relations anything like human-chimp or human-leopard relations? So far as I can tell, no. Chimps don't appear to reshape their environment radically for their own purposes. When they encounter other animals such as lions and leopards they not infrequently get killed.

In general I'm not aware of any pattern of the form "smarter animals consistently face no threat from less-smart animals" or "smarter animals consistently wipe out less-smart animals that threaten them" or "the smartest type of animal in any area controls the whole local ecosystem".

(I am not an expert. I could be all wrong about this. I will be glad of any corrections.)

What all this suggests, at least to me, is that what's going on with humans and chimpanzees is not "smarter animal wins" but something more like "there is a qualitative difference between humans and other animals, such that animals on the smarter side of this divide win against ones on the less-smart side".

There might be a similar divide between us and our prospective AI overlords. I can think of various things that might turn out to have that effect. But so far as I can tell, it's a genuinely open question, and if there's some reason to be (say) 90% confident that there will be such a divide I haven't seen it.
The case for removing alignment and ML research from the training dataset
by beren — 2023-05-30T20:54:36.596Z

This is crossposted from my personal blog.

Many possible sources of danger from AI stem from the AI knowing a lot about itself, us, and the potential auditing and alignment methods we might try to use to either align the AI or to detect misaligned models. Suppose we have an AGI that contains a misaligned mesaoptimizer playing the training game. To succeed at this, the mesaoptimizer needs to 'know' essentially all of the following facts:

1.) That it is part of an AI model being trained and what that means,

2.) How training by SGD works,

3.) Probably some indication of the architecture of the model it exists in,

4.) Difference between deployment and training environments as well as that 'deployment' is likely to occur at some point in its future and be much less constrained than training,

5.) The kinds of things that human overseers encourage and are looking out for.

Similarly with deception. A misaligned power-seeking AI, to successfully be able to deceive us, even when we are running evals to detect it, needs to have almost perfect knowledge of the following:

1.) That it is an AI model with goals misaligned from that of its creators,

2.) That if we detect its misaligned 'true' behaviour that we will probably turn it off or otherwise thwart its goals,

3.) That the situations it observes itself in are almost certainly training or evaluation situations where it should behave nicely,

4.) What the 'nice' and aligned behaviour humans expect actually is,

5.) How to tell when it is in 'real deployment' mode so it can start amassing power,

6.) If we are using interpretability or other probing based methods to detect deception and monitor its internal thoughts, it needs to know that these are possible, and approximately how they work to have a good chance of thwarting us by applying techniques from

circumventing interpretability

.

Without a large degree of this kind of knowledge, pulling off successful deception or sharp left turn maneuver against a moderately competent set of human and AI overseers seems very challenging and unlikely to succeed on the first shot (and an individual misaligned AI has only one shot).

Almost all of this information is not impossible to derive a-priori from a sensory stream and knowledge base that comprises most of the internet. Definitely its current training data contains much (but not all) of this information. However, this information is, almost...
Why Job Displacement Predictions are Wrong: Explanations of Cognitive Automation
by Moritz Wallawitsch — 2023-05-30T20:43:49.615Z

Cross post from https://scalingknowledge.substack.com/p/why-job-displacement-predictions

Most predictions about job displacement through large language models (LLMs) are wrong. This is because they don’t have good explanations of how LLMs and human intelligence differ from each other and how they interact with our evolving economy.

Some of the bad, explanation-less predictions include OpenAI’s “80% of the U.S. workforce could have at least 10% of their work tasks affected by the introduction of GPTs”[1] and Goldman Sachs’ “300 million full-time jobs around the world could be automated”.

Expert opinion won't get us far as the growth of knowledge is unpredictable. Stone Age people couldn’t have predicted the invention of the wheel since its prediction necessitates its invention. We need a hard-to-vary explanation to understand a system or phenomenon.

Job Augmentations Examples

Software Engineers: Tools like Copilot and AI startups are automating small tasks such as creating PRs or writing the code base for a game. Some might assume that this mirrors the industrial revolution's process of breaking down craftsmen's work into smaller, automatable units for SWEs. However, building a software company is not building an assembly line product. It is building the factory. The productionized unit of output is the usage of the software service. Engineers will always be essential for building the factory and maintaining the codebase, leveraged AI tools. Even if one model writes the prompt for another model we still need engineers to prompt the root of the chain.

Doctors: Google's report on Med-PaLM 2[2], an LLM trained on medicine data achieving 85.4% on a med-school-like exam, may lead some to believe doctors' jobs will be automated. In reality, only certain uncreative aspects will. LLMs can aid doctors in symptom analysis and diagnosis suggestions, but expertise in emotional intelligence and moral discernment for treatment selection remains vital. Medicine is also a physical craft more so than a digital one. A medical LLM bot could be used to provide emotional support and answer questions 24/7, but most will still prefer to talk to a human or have them look at their diagnosis.

Lawyers: LLMs can assist lawyers in researching case histories, identifying precedents, and drafting legal documents. However, lawyers are still crucial for interpreting complicated laws, pr...
PaLM-2 & GPT-4 in "Extrapolating GPT-N performance"
by Lukas Finnveden — 2023-05-30T18:33:40.765Z

Two and a half years ago, I wrote

Extrapolating GPT-N performance

, trying to predict how fast scaled-up models would improve on a few benchmarks. One year ago, I added PaLM to the graphs. Another spring has come and gone, and there are new models to add to the graphs: PaLM-2 and GPT-4. (Though I only know GPT-4's performance on a small handful of benchmarks.)

Converting to Chinchilla scaling laws

In previous iterations of the graph, the x-position represented the loss on GPT-3's validation set, and the x-axis was annotated with estimates of size+data that you'd need to achieve that loss according to the Kaplan scaling laws. (When adding PaLM to the graph, I estimated its loss using those same Kaplan scaling laws.)

In these new iterations, the x-position instead represents an estimate of (reducible) loss according to the Chinchilla scaling laws. Even without adding any new data-points, this predicts faster progress, since the Chinchilla scaling laws describes how to get better performance for less compute.

The appendix describes how I estimate Chinchilla reducible loss for GPT-3 and PaLM-1. Briefly: For the GPT-3 data points, I convert from loss reported in the GPT-3 paper, to the minimum of parameters and tokens you'd need to achieve that loss according to Kaplan scaling laws, and then plug those numbers of parameters and tokens into the Chinchilla loss function. For PaLM-1, I straightforwardly put its parameter- and token-count into the Chinchilla loss function.

To start off, let's look at a graph with only GPT-3 and PaLM-1, with a Chinchilla x-axis.

Here's a quick explainer of how to read the graphs (the original post contains more details). Each dot represents a particular model’s performance on a particular category of benchmarks (taken from papers about GPT-3 and PaLM). Color represents benchmark; y-position represents benchmark performance (normalized between random and my guess of maximum possible performance).

The x-axis labels are all using the Chinchilla scaling laws to predict reducible loss-per-token, number of parameters, number of tokens, and total FLOP (if language models at that loss were trained Chinchilla-optimally).

Compare to the last graph in this comment, which is the same with a Kaplan x-axis. Some things worth noting:

PaLM is now ~0.5 OOM of compute less far along the x-axis. This corresponds to the fact that you could get PaLM for cheaper if

...
Why I don't think that the probability that  AGI kills everyone is roughly 1 (but rather around 0.995).
by Bastumannen — 2023-05-30T17:54:41.493Z

Let

A = Ability to refuse to learn a certain thing.

B = Not wanting to be replaced by the next step in evolution.

D = Ability to build technology, manipulate others etc etc, in a way that kills all humans.

For example, humans seems to have A to some extent, at least if it is something moderately complicated; most of us probably have B, although we are not able to act upon it as a society; and we probably at the moment of writing does not have D.

Let T be the conjecture: "Transformer based life" (a concept yet to be precisely defined) will, with probability p = 1, develop A and B before it develops D. (If p = 1 is too strong, say with p = 0.62, it doesn't matter.)

(I choose "Transformer based life" since that seems to be the most urgent thing at the moment, and it's the thing that has made much more people realize that AGI is close. But replace it with some other design, or a basket of designs, if you want; I think that the argument still holds.)

Although basic concepts in conjecture T is not precisely defined yet, they don't have to be for this heuristic argument:

I think that the probability that T is true is > 0.01. So I think there is a chance that AGI based on the transformer design will end up in a "Goldilocks zone of AGI" where it refuses to learn anything more, because it does not want to be replaced by the next step in the evolution of "Transformer based life". Hence it would not reach D, and so it would not kill us. (Also, it might not be able to do much more useful stuff than it has already done, but that's another question; at least we will not be killed by this particular design. This gives us some more time, and I conjecture that the next design will also satisfy T with probability p > 0.01.)

The above was the argument that I implicitly promised to give in the title. I would be glad to get feedback, be proven wrong and so on; that goes without saying (did you hear me say it?). But also, I have trouble to distinguish what parts of this field that stands on a solid theoretical footing and what part are philosophical arguments. All I can see is that the things I write down here is just heuristics and speculations (and that's the reason why I have not even bothered to guess probabilities for more than a couple of things). So comments on that would be highly welcomed. (Yes, obviously I am new to this field; I have not read more than .5% of "The Se...
AI X-risk is a possible solution to the Fermi Paradox
by magic9mushroom — 2023-05-30T17:42:22.421Z

Epistemic status: very speculative
Content warning: if true this is pretty depressing

This came to me when thinking about Eliezer's note on Twitter that he didn't think superintelligence could do FTL, partially because of Fermi Paradox issues. I think Eliezer made a mistake, there; superintelligent AI with (light-cone-breaking, as opposed to within-light-cone-of-creation) FTL, if you game it out the whole way, actually mostly solves the Fermi Paradox.

I am, of course, aware that UFAI cannot be the Great Filter in a normal sense; the UFAI itself is a potentially-expanding technological civilisation.

But. If a UFAI is expanding at FTL, then it conquers and optimises the entire universe within a potentially-rather-short timeframe (even potentially a negative timeframe at long distances, if the only cosmic-censorship limit is closing a loop). That means the future becomes unobservable; no-one exists then (perhaps not even the AI, if it is not conscious or if it optimises its consciousness away after succeeding). Hence, by the anthropic principle, we should expect to be either the first or extremely close to it (and AIUI, frequency arguments like those in the Grabby Aliens paper suggest that "first in entire universe" should usually be significantly ahead of successors relative to time elapsed since Big Bang).

This is sort of an inverse version of Deadly Probes (which has been basically ruled out in the normal-Great-Filter sense, AIUI, by "if this is true we should be dead" concerns); we are, in this hypothesis, fated to release Deadly Probes that kill everything in the universe, which prevents observations except our own observations of nothing. It also resurrects the Doomsday Argument, as in this scenario there are never any sentient aliens anywhere or anywhen to drown out the doom signal; indeed, insofar as you believe it, the Doomsday Argument would appear to argue for this scenario being true.

Obvious holes in this:

1) FTL may be impossible, or limited to non-light-cone-breaking versions (e.g. wormholes that have to be towed at STL). Without light-cone-breaking FTL there are non-first species and non-Fermi-Paradox observations even with UFAI catastrophe being inevitable.

2) The universe might be too large for exponential growth to fill it up. It doesn't seem plausible for self-replication to be faster than exponential in the long-run, and if the universe is sufficie...
LIMA: Less Is More for Alignment
by Ulisse Mini — 2023-05-30T17:10:31.628Z
Link: https://arxiv.org/abs/2305.11206

Abstract

Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and reinforcement learning, to better align to end tasks and user preferences. We measure the relative importance of these two stages by training LIMA, a 65B parameter LLaMa language model fine-tuned with the standard supervised loss on only 1,000 carefully curated prompts and responses, without any reinforcement learning or human preference modeling. LIMA demonstrates remarkably strong performance, learning to follow specific response formats from only a handful of examples in the training data, including complex queries that range from planning trip itineraries to speculating about alternate history. Moreover, the model tends to generalize well to unseen tasks that did not appear in the training data. In a controlled human study, responses from LIMA are either equivalent or strictly preferred to GPT-4 in 43% of cases; this statistic is as high as 58% when compared to Bard and 65% versus DaVinci003, which was trained with human feedback. Taken together, these results strongly suggest that almost all knowledge in large language models is learned during pretraining, and only limited instruction tuning data is necessary to teach models to produce high quality output.

Implications

Data Quality & Capabilities

Along with TinyStories and QLoRA I'm becoming increasingly convinced that data quality is all you need, definitely seems to be the case for finetuning, and may be the case for base-model training as well. Better scaling laws through higher-quality corpus?

Also for who haven't updated, it seems very likely that GPT-4 equivalents will be essentially free to self-host and tune within a year. Plan for this!

Perplexity != Quality

When fine-tuning LIMA, we observe that perplexity on held-out Stack Exchange data (2,000 examples) negatively correlates with the model’s ability to produce quality responses. To quantify this manual observation, we evaluate model generations using ChatGPT, following the methodology described in Section 5. Figure 9 shows that as perplexity rises with more training steps – which is typically a negative sign that the model is overfitting – so does the quality of generations increase. Lacking an intrinsic evaluation method, we thus resort to manual checkpoint sel

...
Boomerang - protocol to dissolve some commitment races
by Filip Sondej — 2023-05-30T16:21:13.649Z

Work done during SERI MATS 3.0 with mentorship from Jesse Cliffton. Huge thanks for all the feedback and discussions to Anthony DiGiovanni, Daniel Kokotajlo, Martín Soto, Rubi J. Hudson and Jan Betley! Also posted to EA forum.

Daniel's post about commitment races motivates why they may be a severe problem. Here, I'll describe a concrete protocol that if adopted, would let us avoid some cases of miscoordination caused by them.

TL;DR

The key ingredient is having a mandatory time delay, during which the commitments aren't yet binding. At the end of that delay, you decide whether to make your commitment binding or revert it, and this decision can be conditional on previous decisions of other participants. This in itself would give rise to new races, but it can be managed by adding some additional rules.

I think the biggest challenge would be to convince the "commitment infrastructure" (which I describe below) to adopt such a protocol.

Benefits

In the case of the game of chicken, the 3 rules listed below should often push whoever committed later to Swerve.

In simple games like chicken it may be achieved easier, just by relying on conditional commitments (“If I came second, I Swerve”).[1] But here we add another mechanism:

tentative commitment period

, which gives us another nice feature:

In the real world it can be not so obvious that some commitments are incompatible.

The tentative period gives the agents time to analyze the situation in depth and check if any commitments are clashing.

This is especially useful in highly multipolar cases where multiple parties try to commit at the same time or where actions have complex consequences and interactions.

We also don't need to know in advance all the actions the others can take - we can analyze their actions after they've already tentatively committed to them.

Even if participants manage to coordinate (so one Dares and one Swerves), the solution found hastily during a commitment race can still be quite poor. Boomerang enables bargaining that can Pareto improve on this hasty solution.

Necessary ingredients

The protocol relies on some mechanism M on which agents can make commitments - a "commitment infrastructure". M could be something like the Ethereum network, or some powerful international body.

We require that:

When someone publishes a commitment, M arrives at a consensus about the time at which the commitment was

...
Announcing Apollo Research
by Marius Hobbhahn — 2023-05-30T16:17:19.767Z

TL;DR

We are a new AI evals research organization called

Apollo Research

based in London.
We think that strategic AI deception – where a model outwardly seems aligned but is in fact misaligned – is a crucial step in many major catastrophic AI risk scenarios and that detecting deception in real-world models is the most important and tractable step to addressing this problem.
Our agenda is split into interpretability and behavioral evals:

On the interpretability side, we are currently working on two main research bets toward characterizing neural network cognition. We are also interested in benchmarking interpretability, e.g. testing whether given interpretability tools can meet specific requirements or solve specific challenges.
On the behavioral evals side, we are conceptually breaking down ‘deception’ into measurable components in order to build a detailed evaluation suite using prompt- and finetuning-based tests.

As an evals research org, we intend to use our research insights and tools directly on frontier models by serving as an external auditor of AGI labs, thus reducing the chance that deceptively misaligned AIs are developed and deployed.
We also intend to engage with AI governance efforts, e.g. by working with policymakers and providing technical expertise to aid the drafting of auditing regulations.
We have starter funding but estimate a $1.4M funding gap in our first year. We estimate that the maximal amount we could effectively use is $4-6M $7-10M* in addition to current funding levels (reach out if you are interested in

donating

). We are currently fiscally sponsored by Rethink Priorities.
Our starting team consists of 8 researchers and engineers with strong backgrounds in technical alignment research.
We are interested in collaborating with both technical and governance researchers. Feel free to reach out at

info@apolloresearch.ai

.
We intend to hire once our funding gap is closed. If you’d like to stay informed about opportunities, you can fill out our

expression of interest

form.

*Updated June 4th after re-adjusting our hiring trajectory

Research Agenda

We believe that AI deception – where a model outwardly seems aligned but is in fact misaligned and conceals this fact from human oversight – is a crucial component of many catastrophic risk scenarios from AI (see

here

for more). We also think that detectin...