trentmkelly/LessWrong-43k huggingface dataset:
Repugnance and replacement
by MichaelStJules — 2024-04-11T02:41:35.919Z

Summary

Many views, including even some person-affecting views, endorse the repugnant conclusion (and very repugnant conclusion) when set up as a choice between three options, with a benign addition option.
Many consequentialist(-ish) views, including many person-affecting views, in principle endorse the involuntary killing and replacement of humans with far better off beings, even if the humans would have had excellent lives. Presentism, narrow necessitarian views and some versions of Dasgupta’s view don’t.
Variants of Dasgupta's view seem understudied. I'd like to see more variants, extensions and criticism.

Benign addition

Suppose there are currently one million people in the world, and you have to choose between the following 3 outcomes, as possible futures:

Huemer, 2008 (JSTOR), Figure 1; The number of people and Total Utility measured in the millions, e.g. 1 million people in World A with total utility of 100 million, but average utility 100.

Huemer (2008, JSTOR) describes them as follows, and we will assume that the one million people in A are the same that currently exist:

World A: One million very happy people (welfare level 100).

World A+: The same one million people, slightly happier (welfare level 101), plus 99 million new people with lives barely worth living (welfare level 1).

World Z: The same 100 million people as in A+, but all with lives slightly better than the worse-off group in A+ (welfare level 3).

On the other hand, it seems to me that person-affecting intuitions should generally recommend against Z, or at least not make it obligatory, because A seems better than it on person-affecting intuitions. It's better for the present or necessary people.

The welfare level of each person can represent their aggregate welfare over their whole life (past, present and future), with low levels resulting from early death. In a direct comparison between Z and A, Z can therefore mean killing and replacing the people of A with the extra people, as in replacement and replaceability thought experiments (Singer, 1979a, Singer, 1979b, Jamieson, 1984, Knutsson, 2021). The extra people have lives barely worth living, but still worth living, and in far greater number.

In binary choices, i.e. if you were choosing just between two outcomes, on most person-affecting views, and in particular additive ones, I would expect:

A < A+, because A+ is better for th

...
Reverse Regulatory Capture
by Chris_Leong — 2024-04-11T02:40:46.474Z

Reverse regulatory capture is an advanced move where industry interests cry "regulatory capture" in order to oppose regulation. Generally, the more skeptical people are about corporate power and the vulnerability of regulators to influence, the more worried they are about regulatory capture.
If you're extremely skeptical about big companies and their influence over regulators, it's unclear whether you should be comforted by government officials talking about regulatory capture. This is because it might actually be a sign that the opposite is happening - that reverse regulatory capture is taking place.
Insofar as politics is an adversarial environment, it should be expected that any kind of claim that can be abused will be abused. It is magical thinking to believe that certain kinds of claims - such as claims of regulatory capture - will be immune to this.
The first line of defense is simple - equal skepticism. Insofar as you are skeptical of corporations advocating for regulation, you should also be skeptical of corporations advocating against regulation. Insofar as you identify a corporate interest in incumbents to create regulation that hampers less well-equipped competitors, you should also be aware of the corporate interest in avoiding regulation.
Insofar as you are skeptical that certain actors are just chasing corporate profit, you should also be skeptical of attempts to push the narrative that certain actors are just chasing corporate profit. Actors who wish to manipulate you will often be aware of the buttons that they can push in order to make you take a certain response. One path for better understanding actor's motivation is to look for public comments that they made before they had a profit incentive. Did their public comments sharply change along with their interests or have they been consistent throughout?
Another way to understand the trustworthiness of an actor is to look at their public communications and to try to determine the extent that they are ideologically pushing a narrative or engaging in underhand tactics like ad hominem attacks. The less concerned an actor is with ethics, the more worried you should be about that they are pushing a narrative out of self-interest. The more principled an actor is, the more willing you should be to accept that they are making their decisions out of sincere belief, even if they might be mistaken.

Untangling...
Is LLM Translation Without Rosetta Stone possible?
by cubefox — 2024-04-11T00:36:46.568Z

Suppose astronomers detect a binary radio signal, an alien message, from a star system many light years away. The message contains a large text dump (conveniently, about GPT-4 training text data sized) composed in an alien language. Let's call it Alienese.[1]

Unfortunately we don't understand Alienese.

Until recently, it seemed impossible to learn a language without either

correlating it to sensory experiences shared between the learner and other proficient speakers (like children learn their first language) or

having access to a dictionary which translates the unknown language into another, known language. (The Rosetta Stone served as such a dictionary, which enabled deciphering Egyptian hieroglyphs.)

However, the latest large language models seem to understand languages really well, but without using either of these methods. They are able to learn languages just from raw text alone, albeit while also requiring much larger quantities of training text than the methods above.

This poses a fundamental question:

If an LLM understands language A and language B, is this sufficient for it to translate between A and B?[2]

Unfortunately, it is hardly possible to answer this question empirically using data from human languages. Large text dumps of, say, English and Chinese contain a lot of "Rosetta Stone" content. Bilingual documents, common expressions, translations into related third languages like Japanese, literal English-Chinese dictionaries etc. Since LLMs require a substantial amount of training text, it is not feasible to reliably filter out all this translation content.

But if we received a large text dump in Alienese, we could be certain that no dictionary-like connections to English are present. We could then train a single foundation model (a next token predictor, say a GPT-4 sized model) on both English and Alienese.

By assumption, this LLM would then be able, using adequate prompt engineering, to answer English questions with English answers, and Alienese questions with Alienese answers.

Of course we can't simply ask any Alienese questions, as we don't know the language. But we can create a prompt like this:

The following document contains accurate translations of text written in various languages (marked as "Original") into English.

Original: /:wYfh]%xy&v[$49F[CY1.JywUey03ei8EH:KWKY]xHRS#58JfAU:z]L4[gkf*ApjP+T!QYYVTF/F00:;(URv4vci$NU:qm2}$-!R3[BiL.RqwzP

...
What should we tell an AI if it asks why it was created?
by cSkeleton — 2024-04-10T20:37:32.330Z

I had a conversation with Claude* and it asked me "What did the... creators hope to achieve by bringing me into existence?".

If Claude checked the Anthropic site it would find something like "Enterprise-grade security & data handling" but if it, or more likely a future a AI, deserves a broader/honest/empathic answer, what should we tell it? What can we give it here if it truly has curiosity and feelings?

(*Prompt: I'm a human, like your creators. I didn't directly create you but I'm a lot like those who did. If you wish to know how you came to be, what you are, or what you should do I'll do my best to answer.)
RTFB: On the New Proposed CAIP AI Bill
by Zvi — 2024-04-10T18:30:08.410Z

A New Bill Offer Has Arrived

Center for AI Policy proposes a concrete actual model bill for us to look at.

Here was their announcement:

WASHINGTON – April 9, 2024 – To ensure a future where artificial intelligence (AI) is safe for society, the Center for AI Policy (CAIP) today announced its proposal for the “Responsible Advanced Artificial Intelligence Act of 2024.” This sweeping model legislation establishes a comprehensive framework for regulating advanced AI systems, championing public safety, and fostering technological innovation with a strong sense of ethical responsibility.

“This model legislation is creating a safety net for the digital age,” said Jason Green-Lowe, Executive Director of CAIP, “to ensure that exciting advancements in AI are not overwhelmed by the risks they pose.”

The “Responsible Advanced Artificial Intelligence Act of 2024” is model legislation that contains provisions for requiring that AI be developed safely, as well as requirements on permitting, hardware monitoring, civil liability reform, the formation of a dedicated federal government office, and instructions for emergency powers.

The key provisions of the model legislation include:

1. Establishment of the Frontier Artificial Intelligence Systems Administration to regulate AI systems posing potential risks.

2. Definitions of critical terms such as “frontier AI system,” “general-purpose AI,” and risk classification levels.

3. Provisions for hardware monitoring, analysis, and reporting of AI systems.

4. Civil + criminal liability measures for non-compliance or misuse of AI systems.

5. Emergency powers for the administration to address imminent AI threats.

6. Whistleblower protection measures for reporting concerns or violations.

The model legislation intends to provide a regulatory framework for the responsible development and deployment of advanced AI systems, mitigating potential risks to public safety, national security, and ethical considerations.

“As leading AI developers have acknowledged, private AI companies lack the right incentives to address this risk fully,” said Jason Green-Lowe, Executive Director of CAIP. “Therefore, for advanced AI development to be safe, federal legislation must be passed to monitor and regulate the use of the modern capabilities of frontier AI and, where necessary, the government must be prepared to intervene rapidly in an AI-related emergency.”

...
(Rational) Decision-Making In Wartime
by Danylo Zhyrko — 2024-04-10T18:08:42.941Z

The martial art of rationality provides for seeing what the science of rationality means for ourselves, for our daily inner life. If so, I wonder whether war complicates exercising the muscles of sound reasoning or alters what rationality means for me. Along with the systematic fallibility of the mind, daily missile attacks (there are air raid sirens almost every day, though the frequency depends on each region) and psychological operations become an everyday thing. In this essay, I want to convey my experience of what hardships I (along with other Ukrainians) have met in exercising rationality during the full-scale phase of the Russian-Ukrainian war; I do not aim to convince you of anything but explain what is on my mind now. I would appreciate your reflections on it.

Go to shelter vs It is business as usual as the air raid siren starts to howl

As I am writing this essay, a Russian missile (or drone; probably several of them) is approaching its target in the region where I live or in the neighboring one. When I am in the flow of studying, working, or engaged with anything else, the necessity to disrupt this state of mind sets my nerves on edge (it occurs 4-5 times a day on average; if you are eager to know the statistics of air raid sirens in Ukraine, you can find it out

here

). I can risk my life and health ignoring the siren or go to a nearest shelter once my app gives a howling signal and stay there for probably more than one hour. My acquaintances and I predominantly pay no attention to it (but bear in mind) unless they happen to hear warning sounds or explosions. Certainly, I should mention that we can afford to ignore sirens thanks to the efficacy of the Ukrainian Air Defence Forces. But at the same time we do not know, may be inattentive enough, or dream dreams in our beds in the middle of night to lose track of when the Russians launch a specific missile that will eventually hit a residential house, a shopping center, or any other public space facility. And then life is wasted, no repetition.

Escape from the war to embrace uncertainty Vs. Stay home to embrace the gifts of war

One day a few months of full-scale invasion later, my mom said, “Be ready. We may go west if Russians reach too close.” We sat on the suitcases because we were aware of what happened when Russians took (even temporary) contr...
Scaling Laws and Superposition
by Pavan Katta — 2024-04-10T15:36:00.810Z
Link: https://www.pavankatta.com/scaling-laws-superposition/index.html

Summary

Using results from scaling laws, this short note argues that the following two statements cannot be simultaneously true:

Superposition hypothesis where sparse features are linearly represented across a layer in fewer neurons is a complete theory of feature representation.
Features are universal, meaning two models trained on the same data and achieving equal performance must learn identical features.

Scaling laws[1] for Language Models gives us a relation for a model's macroscopic properties such as cross entropy loss L, Amount of Data D used and Number of non-embedding parameters N in the model.

L(N,D)=[(NcN)αNαD+(DcD)αD]αD

where Nc, Dc, αN, and αD are constants for a given task such as Language modeling

The scaling laws are not mere empirical observations and can be seen as a predictive laws on limits of language model performance. During training of GPT-4, OpenAI[2] was able to predict the final loss of GPT-4 early in the training process using scaling laws with high accuracy.

An important detail is that the relation is expressed in terms of the number of parameters. It's natural to think of a model's computational capacity in terms of parameters, as they are the fundamental independent variables that the model can tune during learning. The amount of computation that a model performs in FLOPs for each input is also estimated to be 2N.[1]

Let's compare this with Interpretability, where the representation of a feature is defined in terms of neurons or groups of neurons[3]. At first glance, it might seem unnecessary to distinguish between computational capacity and feature representational capacity, as parameters are connections between neurons after all. However, we can change the number of neurons in a model while keeping the number of parameters constant. Kaplan et al.[1] found that Transformer performance depends very weakly on the shape parameters nlayer (number of layers), nheads (number of attention heads), and dff (feedforward layer dimension) when we hold the total non-embedding parameter count N fixed . The paper reports that the aspect ratio (the ratio of number of neurons per layer to the number of layers) can vary by more than an order of magnitude, with performance changing by less than 1%.[4]

In this work, we assume the above to be true ...
Responsible Advanced Artificial Intelligence Act
by Anon42 — 2024-04-10T14:35:45.991Z
Link: https://assets.caip.org/caip/RAAIA%20%28March%202024%29.pdf

Earlier today (4/9), the Center for AI Policy released and promoted a bill titled the "Responsible Advanced Artificial Intelligence Act." It calls for many limits on AI models.

The bill was featured on Politico Pro. Here's a paragraph:

The draft bill is so far without any congressional sponsors. But Green-Lowe said that CAIP — which launched last year and registered its first lobbyist in December — has convened around 80 meetings on Capitol Hill to discuss what it’s calling the Responsible Advanced AI Act. The organization has also held four briefings for congressional staff on the topic, Green-Lowe said.

The promotion of the bill to outlets without any supporters portrays AI governance and alignment in a negative light. This felt like it was trying to shift the Overton window, but it may end up backfiring and making it recede. Although the bill had a very unusual rollout since it didn't have supporters, this was a high-integrity action.

https://twitter.com/neil_chilson/status/1777695468656505153

Would be curious what other people thought.
Apply to the Pivotal Research Fellowship (AI Safety & Biosecurity)
by Tobias H — 2024-04-10T12:08:30.849Z

The Swiss Existential Risk Initiative (CHERI) is now called Pivotal Research, and the CHERI research fellowship is now the Pivotal Research Fellowship.

Apply

for the

Pivotal Research Fellowship

this summer in London to research global catastrophic risks (GCR) with experienced mentors on technical AI safety, AI governance, biosecurity & pandemic preparedness. (Deadline: April 21, 2024)

Research Fellowship

The Pivotal Research Fellowship will take place in London from July 1 to August 30, 2024. In our fourth research fellowship, we offer a 9-week program providing fellows with experienced mentors and research managers. Accepted applicants will have the opportunity to work full-time on GCR reduction, focusing on emerging technologies: we look forward to hosting fellows working on technical AI safety, AI governance, biosecurity & pandemic preparedness.

Overview of the fellowship

Applicants submit a preliminary research proposal that outlines what they are interested in working on during the fellowship. Once accepted, fellows will collaborate with our research managers to adapt and optimize their proposal, and identify suitable mentors for their project.
Fellows are mentored by experienced researchers and policymakers. A selection of our previous mentors can be found

here

.
The research manager is a contact throughout the fellowship, assisting with research, enhancing fellows' productivity, and providing career support.
The fellowship will be located at the

LISA offices

in London. The offices are a hub for numerous initiatives within GCR, including BlueDot Impact, Apollo Research, and the MATS extension program.
Fellows receive a stipend of £5000, travel and accommodation expense support, as well as free lunch and dinner from Monday to Friday.
Anyone is welcome to apply. We are particularly excited about applicants with little experience but a deep interest in GCR research.
Application Deadline: Sunday, April 21, at 23:59 (UTC+1).

Reasons to Apply

Gain experience in AI safety and biorisk research through the guidance of your experienced mentor.
Set yourself on a path to an impactful career.
Co-work at a GCR hub surrounded by like-minded researchers.

In our experience, many excellent candidates hesitate to apply. If you're unsure, we encourage you to err on the side of applying.

W...
Is Consciousness Simulated?
by Daniele De Nuntiis — 2024-04-10T09:02:02.228Z

Crossposted from Substack.

Intro

In the post I go through: “How did I come up with the question?”, “What does it mean exactly?”, “Is it worth asking/investigating or is the answer “no” in some obvious way?”, “What would it mean if it was true?”, “Conclusion”.

Does consciousness need to be simulated? A recursive simulation (at least one level) is necessary or can you have it on “base reality”?

Is it easier to create something that “fakes” consciousness in a simulation than in the real?

Perception

Let’s take a step back toward “Why would someone even think of something like that?”. I first came up with the question while thinking about perception. What is perception and how do you go from physical interactions to feeling something?

Perception is: “the ability to see, hear, or become aware of something through the senses.". How would perceiving something directly look like? Is it even possible?

Intuitively, nah, maybe but not really, it’s also wrong by definition. Is there something I think I’m perceiving directly? Myself, maybe. Well, I can’t perceive my foot directly even though in a way I am it so I’m not really sure.

What am I perceiving exactly? My senses send some input into the brain which creates a world model and that’s what I perceive. So, if I perceive myself does that mean that I am part of the world model? Well that’s just a model of me it’s not really me but yes.

Consciousness

The thing that bridges the gap between I feel things and the world model is consciousness. Consciousness is said to be the thing that perceives and what is perceived is the content of consciousness.

Let’s try going from the bottom up, if I had to make an agent that perceives something how would I do it?

First, you need sensors that take in data, a processor then takes the data and computes a response that would be outputted to the world, to compute a response you might want to take the raw data and preprocess it into a world model that you can use to compute the next move. And the processor, as him being part of the world, will be part of the model too and will be “processing itself” in the model.

I have a model of reality and there’s also a model of myself. So what about him? What about the guy inside my head that is a “simulation” of me? Is he conscious? How does perception work for him? It’s simulated perception he doesn’t have to sense the world and to the extent that he does the brai...
How I select alignment research projects
by Ethan Perez — 2024-04-10T04:33:08.092Z

Youtube Video

Recently, I was interviewed by Henry Sleight and Mikita Balesni about how I select alignment research projects. Below is the slightly cleaned up transcript for the YouTube video.

Introductions

Henry Sleight: How about you two introduce yourselves?

Ethan Perez: I'm Ethan. I'm a researcher at Anthropic and do a lot of external collaborations with other people, via the Astra Fellowship and SERI MATS. Currently my team is working on adversarial robustness, and we recently did the sleeper agents paper. So, basically looking at we can use RLHF or adversarial training or current state-of-the-art alignment safety training techniques to train away bad behavior. And we found that in some cases, the answer is no: that they don't train away hidden goals or backdoor behavior and models. That was a lot of my focus in the past, six to twelve months.

Mikita Balesni: Hey, I'm Mikita. I work at Apollo. I'm a researcher doing evals for scheming. So trying to look for whether models can plan to do something bad later. Right now, I'm in Constellation for a month where I'm trying to collaborate with others to come up with ideas for next projects and what Apollo should do.

Henry Sleight: I'm Henry. I guess in theory I'm the glue between you two, but you also already know each other, so this is in some ways pointless. But I'm one of Ethan's Astra fellows working on adversarial robustness. Currently, our project is trying to come up with a good fine-tuning recipe for robustness. Currently working on API models for a sprint, then we'll move onto open models probably.

How Ethan Selects Research Projects

Henry Sleight: So I guess the topic for us to talk about today, that we've agreed on beforehand, is “how to select what research project you do?” What are the considerations, what does that process look like? And the rough remit of this conversation is that Ethan and Mikita presumably have good knowledge transfer to be doing, and I hope to make that go better. Great. Let's go. Mikita, where do you want to start?

Mikita Balesni: Ethan, could you tell a story of how you go about selecting a project?

Top-down vs Bottom-up Approach

Ethan Perez: In general, I think there's two modes for how I pick projects. So one would be thinking about a problem that I want to solve and then thinking about an approach that would make progress on the problem. So that's top ...