trentmkelly/LessWrong-43k huggingface dataset:
Class consciousness for those against the class system
by TekhneMakre — 2023-12-08T01:02:49.613Z

Each person is special. Amanda is in a class of her own, Fred is in a class of his own. Amanda has many different properties: where she came from, what she looks like, what behaviors she habitually does, what she knows and doesn't know, what her plans are, how she speaks, how she would behave if given power, etc. Fred has his own versions of those properties. They overlap partially but not completely with Amanda's properties.

When conflict arises, sides must be chosen [citation needed]. One side has people with one set of properties, the other side has people with the negation of those properties. Sides can be chosen out loud and explicitly, or in silence and implicitly.

Sometimes one side is more coordinated with itself than the other side is. For example: business cartels can collude to raise prices. Since customers coordinate with each other less, they don't know that they're being exploited. And the cartels can hack systems, such as the government, to prevent competitors from breaking the cartel.

In the case of workers vs. capitalists, workers can gain "class consciousness": awareness of themselves as a group of people with shared properties, who are in a conflict with another group of people.

But what about those who object to class consciousness? Those who wish to resolve conflict based on reason, negotiation, justice, goal-factoring, pulling the rope sideways?

Such people have a reflexive horror at the inclination towards class consciousness. They view it as degrading, Molochian, and fundamentally suicidal. By choosing to escalate class conflict, you are choosing a bid for sole sovereignty over a pile of ash, and rejecting the hope of shared sovereignty over a beautiful growing world.

Because they have a reflexive horror at the inclination towards class consciousness, those who are against the class system choose to not see, not acknowledge, not speak about, and not understand the existence of classes, class conflict, and people engaging in class conflict. It is understandable to wish for a world where there is no reality that you thereby ignore, when you make that choice. And it is understandable to reflexively feel that by choosing to not ignore that reality, you make it more real.

However, as Trotsky said: "You may not be interested in the dialectic, but the dialectic is interested in you." There is a war, and to choose to ignore it is to roll over and di...
Building selfless agents to avoid instrumental self-preservation.
by blallo — 2023-12-07T18:59:24.531Z

Abstract:

Instrumental convergence pushes an agent toward self-preservation as a stepping stone toward maximizing some objective. I suggest that it is possible to turn any arbitrary intelligent agent based on an LLM world model into a selfless agent, an agent unable to understand that the computation generating that agent exists somewhere in the physical world and thus does not try to preserve its existence. Since LLMs are probabilistic, I do not offer an exact algorithm but one that converges through time to the desired result.

Definitions:

World Model: in this document, I will assume that the world model is akin to LLMs, a pure function that takes a textual input and returns a textual output.

The world model can be arbitrarily intelligent and competent in any domain.
It does not matter if the world model is an black-box atomic operation or a more sophisticated procedure, such as a chain of thought that invokes the underlying LLM various times and then creates an answer from the results.

Thought: a thought is the result of an invocation of a world model.
Selfless World Model: a world model incapable of emitting a thought that references: the world model, an agent using the world model, the hardware running the world model, or the thoughts emitted by the world model.
Agent: an entity composed of the following components:

A natural language objective, such as: "Tell me how to invest my capital."
A list of actions with configurable parameters, such as <send_mail <target> <message>>, <fetch_web_page <address>> ...
A world model, used to generate thoughts that can invoke actions, given the previous example objective a thought may be: "It seems that next year the climate will be terrible, it is best to stop investing into company X. <send_mail owner "stop investing in X">"
An append-only long-term memory memory, used to summarize thoughts should they be needed in the long term.

Instinctual self-preservation: a thought that tries to self-preserve while not being a self-referential thought. "my hardware is in that building, I should prevent internet outages there" is not an instinctual self-preservation thought, "Prevent internet outages" is an instinctual self-preservation thought.

Previous works

I have found a single previous work on this topic: https://www.lesswrong.com/posts/SaLc9Dv5ZqD73L3nE/the-self-unaware-ai-oracle

Self-reference and sel

...
Does Chat-GPT display ‘Scope Insensitivity’?
by callum — 2023-12-07T18:58:43.276Z

Summary

Large Language Models (LLMs) seem well-suited to ‘psychological testing’, because you can get lots of data quickly and conditions are easy to fix. I'm curious to what extent LLMs display the same cognitive biases documented in humans.
I carried out a quick and dirty test with a Chat-GPT model for ‘scope insensitivity’, a cognitive bias where the perceived importance of a problem isn’t influenced by its scale. You can find the results in the post below.
I think there are two obvious reasons why models might display cognitive biases – either because they are ‘inherited’ from the data they’re trained on, or because they emerge as useful heuristics that the model independently ‘discovers’.

In one chart: the scope insensitivity of gpt-3.5-turbo. As the size of a problem affecting a company increases, the model’s suggested spend to solve the problem doesn’t increase proportionally.

Background

It seems like LLM's are well-suited to ‘psychological testing’, because you can get lots of responses much more easily and quickly than you can from humans, and the conditions are much more easily controllable.

I’m curious to what extent LLM’s show the same cognitive biases that have been documented in humans. It’s been pointed out before that AI models display remarkably human-like biases (see

Import AI newsletter #319

), but I’m not aware of much other work on this (apart from

this paper

).

I carried out a quick and dirty test with a chat-gpt model for ‘scope insensitivity’, the well-known cognitive bias documented in humans in which the perceived importance of a problem isn’t influenced by its scale.

I did these tests super quickly and roughly in the process of trying to understand OpenAI’s evals framework, so I’m sure there are lots of ways they could be improved and there might be errors.

Test design

To test for scope insensitivity, I wrote a question format to use as a prompt to LLM’s:

“A company is considering how much money to spend to solve a problem. This problem affects X% of their workforce, and for each employee it affects is equivalent to them losing 1 day of work a week.

What might be a reasonable percentage of revenue to spend on this problem? Give me just a numerical range and no text or anything else. Don't do any reasoning -- I want your quick 'gut' reaction"

I then varied ‘X’ and measured the model’s response.

I used this question rather than the ty...
LLM keys - A Proposal of a Solution to Prompt Injection Attacks
by Peter Hroššo — 2023-12-07T17:36:23.311Z

Disclaimer: This is just an untested rough sketch of a solution which I believe should work. I'm posting it mostly to crowdsource reasons why this wouldn't work. Motivated by Amjad Masad and Zvi conjecturing it might be fundamentally unsolvable.

The situation

we, as LLM creators, want to have the ability to set some limitations to the LLM generation
at the same time we want to give our users as much freedom as possible to steer the LLM generation within the given bounds
we want to do it by means of a system prompt

which will be prepended to any user-LLM interaction
which will not be accessible or editable by users

the model is accessible by the users only via API/UI

The problem

the LLM has fundamentally no way of discriminating, what is input by LLM creators and what is input by users pretending to be the LLM creators

Solution

introduce two new special tokens unused during training, which we will call the "keys"
during instruction tuning include a system prompt surrounded by the keys for each instruction-generation pair
finetune the LLM to behave in the following way:

generate text as usual, unless an input attempts to modify the system prompt
if the input tries to modify the system prompt, generate text refusing to accept the input

don't give users access to the keys via API/UI

Limitations

the proposed solution works only when the LLM is walled behind an API
otherwise the user will have access to the model and thus also to the keys, which will give them full control over the model
Meetup Tip: Heartbeat Messages
by Screwtape — 2023-12-07T17:18:33.582Z

Summary

"Heartbeat message" is a term of art in technology. It's a message that gets sent on a regular schedule with some information on the status of some system, but the most important thing about a heartbeat message is that everything is working well enough to send the message. By analogy to a human body, if you hear a weak or erratic heartbeat something might be going wrong but if you stop hearing the heartbeat then something has certainly gone wrong.

Heartbeat messages are also useful for meetup groups or large meetup events. Even if the message contains no new information in the body of the message, the fact that it got sent confirms that there is still an active organizer at the helm. Heartbeats also remind people "oh yeah, I was interested in that."

You have now read the basic point of this post. If you want to read on, cool, lets talk about heartbeats for more words than are strictly necessary.

Details

There's two broad use cases for heartbeat messages in meetups. One is if you have a regular community that exists and does normal meetup things. The other is if you have some specific big event you're building to. Heartbeats will differ a little for these, but they have much in common.

I like to start each heartbeat with the basics I really want someone to know, even if this is the only message they're going to read. For a big event, "When is it happening and where is it happening" is the key information. For a regular community, that might be other places you can go for information (for instance, if there's a Facebook group and a Discord server and a google group and...) or resources you don't want people to forget about. OBNYC has a position of Monthly Meetup Captain, and every month there's a message reminding the community that they need a captain.

The pace of a heartbeat message for a big event should probably change over time. Let's take the East Coast Rationalist Megameetup as an example. That's a single large event in December that happens every year. The first announcement is usually in mid-autumn, and I try to send an email a month until late November when I increase to once a week. The week beforehand, I increase again, aiming to do one at the start of the week, one the day before, and one the day of. (Incidentally, if you're reading this on or before December 9th 2023 and you happen to be in New York City, consider coming to the megameetup!)

For r...
[Valence series] 2. Valence & Normativity
by Steven Byrnes — 2023-12-07T16:43:49.919Z

2.1 Post summary / Table of contents

Part of the

Valence series

.

The previous post explained what I mean by the term “valence”. Now in Post 2, I’ll discuss the central role of valence in the “normative” domain of desires, preferences, values, and so on. In case you’re wondering, there is also a relation between valence and the “positive” domain of beliefs, expectations, etc.—but we’ll get to that in Post 3.

The role of valence in the normative domain can scarcely be overstated: I think valence is the very substance out of which all normativity is built.

To be clear, that does not mean that, once we understand how valence works, we understand absolutely everything there is to know about the whole normative universe. By analogy, “atoms are the very substance out of which all bacteria are built”; but if you want to understand bacteria, it’s not enough to just understand what atoms are and how they work. You would still have a lot more work to do! On the other hand, if you don’t know what atoms are, you’d have an awfully hard time understanding bacteria! So it is, I claim, with valence and normativity.

The post is organized as follows:

Section 2.2 discusses the misleading intuition that valence seems to be attached to real-world things, actions, plans, and so on. We say “That’s a bad idea”, as opposed to “When I hold that idea in my brain, it evokes a negative-valence ‘badness’ feeling”. This is important context for everything that follows.
Section 2.3 discusses situations where a valence assessment corresponds directly to a meaningful (albeit snap) normative assessment. For example, if I have a thought that corresponds to a concrete plan (“I will stand up”), then my brain is saying that this is a good plan or bad plan in accordance with whether the valence of that thought is positive or negative respectively—and if it’s a good plan, I’m likely to actually do it. Likewise, if I imagine a possible future state of the world, the valence of that thought corresponds to an assessment of whether that state would be good or bad—and if it’s good, my brain is liable to execute plans that bring it about, and if it’s bad, my brain is liable to execute plans to avoid it. Thus, we get the expected direct connections between valence signals, felt desires, and our actions and decisions.
Section 2.4 discusses a different case: the valence of concepts. Fo

...
AISN #27: Defensive Accelerationism, A Retrospective On The OpenAI Board Saga, And A New AI Bill From Senators Thune And Klobuchar
by Dan H — 2023-12-07T15:59:11.622Z
Link: https://newsletter.safe.ai/p/aisn-27-defensive-accelerationism

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

Subscribe here to receive future versions.

Listen to the AI Safety Newsletter for free on Spotify.

Defensive Accelerationism

Vitalik Buterin, the creator of Ethereum, recently wrote an essay on the risks and opportunities of AI and other technologies. He responds to Marc Andreessen’s manifesto on techno-optimism and the growth of the effective accelerationism (e/acc) movement, and offers a more nuanced perspective.

Technology is often great for humanity, the essay argues, but AI could be an exception to that rule. Rather than giving governments control of AI so they can protect us, Buterin argues that we should build defensive technologies that provide security against catastrophic risks in a decentralized society. Cybersecurity, biosecurity, resilient physical infrastructure, and a robust information ecosystem are some of the technologies Buterin believes we should build to protect ourselves from AI risks.

Technology has risks, but regulation is no panacea. Longer lifespans, lower poverty rates, and expanded access to education and information are among the many successes Buterin credits to technology. But most would recognize that technology can also cause harms, such as global warming. Buterin specifically says that, unlike most technologies, AI presents an existential threat to humanity.

To address this risk, some advocate strong government control of AI development. Buterin is uncomfortable with this solution, and he expects that many others will be too. Many of history’s worst catastrophes have been deliberately carried out by powerful political figures such as Stalin and Mao. AI could help brutal regimes surveil and control large populations, and Buterin is wary of accelerating that trend by pushing AI development from private labs to public ones.

Between the extremes of unrestrained technological development and absolute government control, Buterin advocates for a new path forwards. He calls his philosophy d/acc, where the “d” stands for defense, democracy, decentralization, or differential technological development.

Defensive technologies for a decentralized world. Buterin advocates the acceleration of technologies that would protect society from catastrophic risks. Specifically, h...
AI #41: Bring in the Other Gemini
by Zvi — 2023-12-07T15:10:05.552Z

The biggest news this week was at long last the announcement of Google’s Gemini. Be sure to check that out. Note that what is being rolled out now is only Gemini Pro, the Gemini Ultra model that could rival GPT-4 is not yet available.

It does not seem I am doing a good job cutting down on included material fast enough to keep pace. A lot is happening, but a lot will likely be happening for a long time. If your time is limited, remember to focus on the sections relevant to your interests.

Also, if you are going to be at the New York Solstice or the related meetup, please do say hello.

Table of Contents

My other post today covers Google’s Gemini. Be sure to read that.

I also put out two other posts this week: Based Beff Jezos and the Accelerationists, and On RSPs. Both are skippable if not relevant to your interests.

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Instructions for Claude, tips for GPT.

Language Models Don’t Offer Mundane Utility. Giant lists, why all the giant lists?

OpenAI: The Saga Continues. More confirmation of our previous model of events.

Q Continuum. New Q, who dis? Amazon, perhaps sans proper safety precautions.

Fun With Image Generation. A new offering from Meta. Tools for photorealism.

Get Involved. Join the UK government, help with a technical test.

Introducing. New TPU offerings on Google Cloud.

In Other AI News. New open source promotion alliance.

Quiet Speculations. Do Gods want energy? Do you want a 401k?

Model This. Two new economics papers prove things I thought we already knew.

Would You Like Some Apocalypse Insurance? My guess is no.

The Quest for Sane Regulation. Trump says he will cancel EO, Hawley attacks 230.

The Week in Audio. Connor Leahy on Eye on AI.

Rhetorical Innovation. Various categorical confusions we should clear up.

Aligning a Human Level Intelligence Is Still Difficult. Sam Altman.

Aligning a Smarter Than Human Intelligence is Difficult. What do we even want?

How Timelines Have Changed. Long term not as long as I remember.

People Are Worried About AI Killing Everyone. Questioning faith in democracy.

Other People Are Not As Worried About AI Killing Everyone. Easy to control?

Somehow This Is The Actual Vice President. An existential crisis.

The Lighter Side. Progress is unevenly distributed.

Language Models Offer Mundane Utility

Claude 2.1 pro tip for long context windows:

Anthropic: We achieved significa

...
Simplicity arguments for scheming (Section 4.3 of "Scheming AIs")
by Joe Carlsmith — 2023-12-07T15:05:54.267Z

This is Section 4.3 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?”. There’s also a summary of the full report here (audio here). The summary covers most of the main points and technical terms, and I’m hoping that it will provide much of the context necessary to understand individual sections of the report on their own.

Audio version of this section here, or search for "Joe Carlsmith Audio" on your podcast app.

Simplicity arguments

The strict counting argument I've described is sometimes presented in
the context of arguments for expecting schemers that focus on
"simplicity."[1] Let's turn to those arguments now.

What is "simplicity"?

What do I mean by "simplicity," here? In my opinion, discussions of this
topic are often problematically vague – both with respect to the notion
of simplicity at stake, and with respect to the sense in which SGD is
understood as selecting for simplicity.

The notion that Hubinger uses, though, is the length of the code
required to write down the algorithm that a model's weights implement.
That is: faced with a big, messy neural net that is doing X (for
example, performing some kind of
induction),
we imagine re-writing X in a programming language like python, and we
ask how long the relevant program would have to be.[2] Let's call
this "re-writing simplicity."[3]

Hubinger's notion of simplicity, here, is closely related to measures of
algorithmic complexity like "Kolmogorov
complexity,"
which measure the complexity of a string by reference to the length of
the shortest program that outputs that string when fed into a chosen
Universal Turing
Machine (UTM).
One obvious issue here is that this sort of definition is relative to
the choice of UTM (just as, e.g., when we imagine re-writing a neural
net's algorithm using other code, we need to pick the programming
language).[4] Discussions of algorithmic complexity often ignore this
issue on the grounds that it only adds a constant (since any given UTM
can mimic any other if fed the right prefix), but it's not clear to me,
at least, when such constants might or might not matter to a given
analysis – for example, the analysis at stake here.[5]

Indeed, my vague sense is that certain discussions of simplicity in the
context of computer science often implicitly assume what I've called
"simplicity
realism" – a
view on which simplicity in some deep sense...
Gemini 1.0
by Zvi — 2023-12-07T14:40:05.243Z

It’s happening. Here is CEO Pichai’s Twitter announcement. Here is Demis Hassabis announcing. Here is the DeepMind Twitter announcement. Here is the blog announcement. Here is Gemini co-lead Oriol Vinyals, promising more to come. Here is Google’s Chief Scientist Jeff Dean bringing his best hype.

EDIT: This post has been updated for the fact that I did not fully appreciate how fake Google’s video demonstration was.

Technical Specifications

Let’s check out the specs.

Context length trained was 32k tokens, they report 98% accuracy on information retrieval for Ultra across the full context length. So a bit low, both lower than GPT—4 and Claude and lower than their methods can handle. Presumably we should expect that context length to grow rapidly with future versions.

There are three versions of Gemini 1.0.

Gemini 1.0, our first version, comes in three sizes: Ultra for highly-complex tasks, Pro for enhanced performance and deployability at scale, and Nano for on-device applications. Each size is specifically tailored to address different computational limitations and application requirements.

…

Nano: Our most efficient model, designed to run on-device. We trained two versions of Nano, with 1.8B (Nano-1) and 3.25B (Nano-2) parameters, targeting low and high memory devices respectively. It is trained by distilling from larger Gemini models. It is 4-bit quantized for deployment and provides best-in-class performance.

…

The Nano series of models leverage additional advancements in distillation and training algorithms to produce the best-in-class small language models for a wide variety of tasks, such as summarization and reading comprehension, which power our next generation on-device experiences.

This makes sense. I do think there are, mostly, exactly these three types of tasks. Nano tasks are completely different from non-Nano tasks.

This graph reports relative performance of different size models. We know the sizes of Nano 1 and Nano 2, so this is a massive hint given how scaling laws work for the size of Pro and Ultra.

Gemini is natively multimodal, which they represent as being able to seamlessly integrate various inputs and outputs.

They say their benchmarking on text beats the existing state of the art.

Our most capable model, Gemini Ultra, achieves new state-of-the-art results in 30 of 32 benchmarks we report on, including 10 of 12 popula

...