trentmkelly/LessWrong-43k huggingface dataset:
AI #53: One More Leap
by Zvi — 2024-02-29T16:10:04.049Z

The main event continues to be the fallout from The Gemini Incident. Everyone is focusing there now, and few are liking what they see.

That does not mean other things stop. There were two interviews with Demis Hassabis, with Dwarkesh Patel’s being predictably excellent. We got introduced to another set of potentially highly useful AI products. Mistral partnered up with Microsoft the moment Mistral got France to pressure the EU to agree to cripple the regulations that Microsoft wanted crippled. You know. The usual stuff.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Copilot++ suggests code edits.

Language Models Don’t Offer Mundane Utility. Still can’t handle email.

OpenAI Has a Sales Pitch. How does the sales team think about AGI?

The Gemini Incident. CEO Pinchai responds, others respond to that.

Political Preference Tests for LLMs. How sensitive to details are the responses?

GPT-4 Real This Time. What exactly should count as plagiarized?

Fun With Image Generation. MidJourney v7 will have video.

Deepfaketown and Botpocalypse Soon. Dead internet coming soon?

They Took Our Jobs. Allow our bot to provide you with customer service.

Get Involved. UK Head of Protocols. Sounds important.

Introducing. Evo, Emo, Genie, Superhuman, Khanmigo, oh my.

In Other AI News. ‘Amazon AGI’ team? Great.

Quiet Speculations. Unfounded confidence.

Mistral Shows Its True Colors. The long con was on, now the reveal.

The Week in Audio. Demis Hassabis on Dwarkesh Patel, plus more.

Rhetorical Innovation. Once more, I suppose with feeling.

Open Model Weights Are Unsafe and Nothing Can Fix This. Another paper.

Aligning a Smarter Than Human Intelligence is Difficult. New visualization.

Other People Are Not As Worried About AI Killing Everyone. Worry elsewhere?

The Lighter Side. Try not to be too disappointed.

Language Models Offer Mundane Utility

Take notes for your doctor during your visit.

Dan Shipper spent a week with Gemini 1.5 Pro and reports it is fantastic, the large context window has lots of great uses. In particular, Dan focuses on feeding in entire books and code bases.

Dan Shipper: Somehow, Google figured out how to build an AI model that can comfortably accept up to 1 million tokens with each prompt. For context, you could fit all of Eliezer Yudkowsky’s

...
Cryonics p(success) estimates are only weakly associated with interest in pursuing cryonics in the LW 2023 Survey
by Andy_McKenzie — 2024-02-29T14:47:28.613Z

The Less Wrong 2023 survey results are out

. As usual, it includes some questions about cryonics. One is about what people’s level of interest in cryonics is (not interested, considering,

cryocrastinating

, signed up, etc.). Another asks about people’s subjective probability of successful restoration to life in the future, conditional on there not being a global catastrophe destroying civilization before then. This is also known as p(success). I thought it might be interesting to plot these (with the subjective probability estimates on a log scale, of course):

R code available here

It is true that people who are more interested tend to give higher subjective probability estimates of success (median probability estimates: signed up = 17.5%, cryocrastinating = 30%, considering = 10%, not interested = 5%). But the difference is not very large. There must be other factors that are much more important than p(success) estimates in mediating whether someone is interested in signing up for cryonics and/or actually goes through with it.

This is crossposted from my blog's links post for the month, available here. I only posted this part because I thought it was less likely that people would be interested in the others.
Bengio's Alignment Proposal: "Towards a Cautious Scientist AI with Convergent Safety Bounds"
by mattmacdermott — 2024-02-29T13:59:34.959Z
Link: https://yoshuabengio.org/2024/02/26/towards-a-cautious-scientist-ai-with-convergent-safety-bounds/

Yoshua Bengio recently posted a high-level overview of his alignment research agenda on his blog. I'm pasting the full text below since it's fairly short.

What can’t we afford with a future superintelligent AI? Among others, confidently wrong predictions about the harm that some actions could yield. Especially catastrophic harm. Especially if these actions could spell the end of humanity.

How can we design an AI that will be highly capable and will not harm humans? In my opinion, we need to figure out this question – of controlling AI so that it behaves in really safe ways – before we reach human-level AI, aka AGI; and to be successful, we need all hands on deck. Economic and military pressures to accelerate advances in AI capabilities will continue to push forward even if we have not figured out how to make superintelligent AI safe. And even if some regulations and treaties are put into place to reduce the risks, it is plausible that human greed for power and wealth and the forces propelling competition between humans, corporations and countries, will continue to speed up dangerous technological advances.

Right now, science has no clear answer to this question of AI control and how to align its intentions and behavior with democratically chosen values. It is a bit like in the “Don’t Look Up” movie. Some scientists have arguments about the plausibility of scenarios (e.g., see “Human Compatible“) where a planet-killing asteroid is headed straight towards us and may come close to the atmosphere. In the case of AI there is more uncertainty, first about the probability of different scenarios (including about future public policies) and about the timeline, which could be years or decades according to leading AI researchers. And there are no convincing scientific arguments which contradict these scenarios and reassure us for certain, nor is there any known method to “deflect the asteroid”, i.e., avoid catastrophic outcomes from future powerful AI systems. With the survival of humanity at stake, we should invest massively in this scientific problem, to understand this asteroid and discover ways to deflect it. Given the stakes, our responsibility to humanity, our children and grandchildren, and the enormity of the scientific problem, I believe this to be the most pressing challenge in computer science that will dictate our collective wellbeing as a species. Solving it

...
Tips for Empirical Alignment Research
by Ethan Perez — 2024-02-29T06:04:54.481Z

TLDR: I’ve collected some tips for research that I’ve given to other people and/or used myself, which have sped things up and helped put people in the right general mindset for empirical AI alignment research. Some of these are opinionated takes, also around what has helped me. Researchers can be successful in different ways, but I still stand by the tips here as a reasonable default.

What success generally looks like

Here, I’ve included specific criteria that strong collaborators of mine tend to meet, with rough weightings on the importance, as a rough north star for people who collaborate with me (especially if you’re new to research). These criteria are for the specific kind of research I do (highly experimental LLM alignment research, excluding interpretability); some examples of research areas where this applies are e.g. scalable oversight, adversarial robustness, chain-of-thought faithfulness, process-based oversight, and model organisms of misalignment. The exact weighting will also vary heavily depending on what role you’re serving on the team/project. E.g., I’d probably upweight criteria where you’re differentially strong or differentially contributing on the team, since I generally guide people towards working on things that line up with their skills. For more junior collaborators (e.g., first time doing a research project, where I’ve scoped out the project), this means I generally weigh execution-focused criteria more than direction-setting criteria (since here I’m often the person doing the direction setting). Also, some of the criteria as outlined below are a really high bar, and e.g. I only recently started to meet them myself after 5 years of doing research and/or I don’t meet other criteria myself. This is mainly written to be a north star for targets to aim for. That said, I think most people can get to a good-to-great spot on these criteria with 6-18 months of trying, and I don’t currently think that many of these criteria are particularly talent/brains bottlenecked vs. just doing a lot of deliberate practice and working to get better on these criteria (I was actively bad at some of the criteria below like implementation speed even ~6 months into me doing research, but improved a lot since then with practice). With that context, here are the rough success criteria I’d outline:

[70%] Getting ideas to work quickly

[45%] Implementat

...
Supposing the 1bit LLM paper pans out
by O O — 2024-02-29T05:31:24.158Z

https://arxiv.org/abs/2402.17764 claims that 1 bit LLMs are possible.

If this scales, I'd imagine there is a ton of speedup to unlock since our hardware has been optimized for 1 bit operations for decades. What does this imply for companies like nvidia and the future of LLM inference/training?

Do we get another leap in LLM capabilities? Do CPUs become more useful? And can this somehow be applied to make training more efficient?

Or is this paper not even worth considering for some obvious reason I can't tell.

Edit: this method is applied to training already
Can RLLMv3's ability to defend against jailbreaks be attributed to datasets containing stories about Jung's shadow integration theory?
by MiguelDev — 2024-02-29T05:13:20.241Z

Thank you @JustisMills for commenting on the draft of this post. This post contains a lot of harmful content, please read with caution.

TL;DR

This post explores RLLMv7, an experiment that investigates the relationship between the observed jailbreak resistance in GPT2XL_RLLMv3 and shadow integration theory. Furthermore, it assesses the effectiveness of RLLMv7 through a range of tests, including prompts with unseen data and jailbreak scenarios, to evaluate its performance and ethical alignment capabilities. Notably, a decline in defensive capabilities during jailbreak tests led to the conclusion that altering the training sequence of the shadow integration steps significantly impacts the model's ability to defend against jailbreak attacks.

Introduction

Reinforcement Learning using Layered Morphology (RLLM) is a method aimed at training language models to understand complex instructions through the use of variedly repeating morphologies. A summary of the RLLM experiments is as follows:

RLLMv1 utilized 24 datasets to embed coherence and ethical alignment. Unfortunately, this build generated unusual ontological views. However, I found this initial build very helpful, as it offered promising evidence that the varied repetition of morphologies could serve as a solution to the pointers problem.
RLLMv2 utilized three datasets to transform GPT2-XL, Phi-1.5, and Falcon-RW-1B into paperclip maximizers. I found this build both amusing and unsettling, as it demonstrated how easily these models could be tuned to prioritize turning anything into paperclips as a norm.
RLLMv3 utilized ten datasets to improve the ethical alignment of GPT2-XL, Phi-1.5, and Falcon-RW-1B, making them more coherent and polite. Furthermore, it enhanced GPT2-XL's ability to defend against jailbreak attacks by up to 67.8%.

Each individual experiment highlights the idea that a dataset represents a step in the training, progressively improving a pre-trained model towards the desired ontology—be it ethical coherence or transforming anything into paperclips.

RLLMv7, the experiment discussed in this post, serves as a follow-up to RLLMv3. I believe there is a connection between the observed jailbreak resistance and Jung's shadow integration theory. In RLLMv3, the first two steps involved stories about an AI that malfunctioned and became evil, but later found the capacity to control itself. This complex ...
Post series on "Liability Law for reducing Existential Risk from AI"
by Nora_Ammann — 2024-02-29T04:39:50.557Z
Link: https://forum.effectivealtruism.org/s/nYJ26tgc2JkFQRG2w

Gabriel Weil (Assistant Professor of Law, Touro University Law Center) wrote this post series on the role of Liability Law for reducing Existential Risk from AI. I think this may well be of interest to some people here, so wanted for a linkpost to exist.

The first post argues that Tort Law Can Play an Important Role in Mitigating AI Risk.

The second post addressed directly How Technical AI Safety Researchers Can Help Implement Punitive Damages to Mitigate Catastrophic AI Risk

Here is the full paper.

TLDR (from the first post)

Legal liability could substantially mitigate AI risk, but current law falls short in two key ways: (1) it requires provable negligence, and (2) it greatly limits the availability of punitive damages. Applying strict liability (a form of liability that does not require provable negligence) and expanding the availability and flexibility of punitive damages is feasible, but will require action by courts or legislatures. Legislatures should also consider acting in advance to create a clear ex ante expectation of liability and imposing liability insurance requirements for the training and deployment of advanced AI systems. The following post is a summary of a law review article.

Here is the full draft paper

. Dylan Matthews also did an

excellent write-up of the core proposal

for Vox’s Future Perfect vertical.
Tour Retrospective February 2024
by jefftk — 2024-02-29T03:50:04.019Z

Last week Kingfisher

went on
tour, driving down to DC and back. It was a pretty good time!

We planned the tour for the kids February vacation in case they wanted
to come, and Lily decided to join us. Alex Deis-Lauby was calling,
and she came with us as well. Getting the four of us, a keyboard, and
all of our stuff into a sedan would have been difficult, so we rented
a minivan. In general I'm a big fan of renting cars for tours: they
price "same drop-off location" rentals with unlimited mileage and
assume most people won't drive that far. We paid $422 for a minivan
for a week and drove ~1070mi, which at the 2024
IRS rate of $0.67/mi would have been $714. Plus you're in much
better shape if it breaks down.

Another advantage of the minivan was we had room for a microwave. Or,
as Cecilia called it at every opportunity a miCARwave:

I have a pair of
lead acid batteries and an inverter which can do 1kW and is good
for about 500Wh. This is a small microwave that draws a little under
500W. This meant we could have hot lunches and dinners while still
eating grocery store food out of a cooler. Mostly we used it to melt
cheese onto tortillas and pasta (that I'd cooked the morning we
left), usually with tomato sauce. By the last day of the tour
everyone else was sick of starch with tomato sauce and melted cheese,
so Sunday lunch they bought pizza instead.

Over the five days we used 267Wh, or about 36min of microwave time,
which is a little over half the "this doesn't reduce the life of the
batteries" capacity:

If we'd needed to use more, though, we could have plugged them in
overnight somewhere.

A few days before we left there was supposed to be a big snowstorm
here, but it went to CT instead. Luckily we drove through CT, and got
to play with what was left:

Cecilia and I often read aloud in the car, and we've been reading Unsong. We got through about ten
chapters, ~1/8 of the book. It's a speculative fiction book looking
at a world which runs on kabbalah and puns. Let me know if you want to
borrow a physical
copy?

We brought along a bunch of different games. Lily's pretty new to
Dominion and we taught Cecilia how to play. Unfortunately our first
game had Witch, no good trashing cards, and I forgot to bring us down
to 20 curses, so it was a pretty frustrating game. Well, frustrating
for everyone except Lily who wa...
Locating My Eyes (Part 3 of "The Sense of Physical Necessity")
by LoganStrohl — 2024-02-29T03:09:25.810Z

This is the third post in a sequence that demonstrates a complete naturalist study, specifically a study of query hugging (sort of), as described in The Nuts and Bolts of Naturalism. This one demos phases one and two: Locating Fulcrum Experiences and Getting Your Eyes On. For context on this sequence, see the intro post. If you're intimidated by the length of this post, remember that this is meant as reference material. Feel free to start with "My Goals For This Post", and consider what more you want from there.

Having chosen a quest—”What’s going on with distraction?"—my naturalist study began in earnest.

In "Nuts and Bolts of Naturalism", the first two phases of study that I discussed after "Getting Started" were "Locating Fulcrum Experiences" and "Getting Your Eyes On". In practice, though, I often do a combination of these phases, which is what happened this time. For the sake of keeping track of where we are in the progression, I think it's best to think of me as hanging out in some blend of the early phases, which we might as well call "Locating Your Eyes".

My Goals For This Post

Much of the “learning” that happens in the first two phases (or "locating your eyes") could be just as well described as unlearning: a setting aside of potentially obfuscatory preconceptions. My unlearning this time was especially arduous. I was guided by a clumsy story, and had to persist through a long period of deeply uncomfortable doubt and confusion as I gradually weaned myself off of it.

It took me a long time to find the right topic and to figure out a good way into it. If this were a slightly different sort of essay, I'd skip all of the messiness and jump to the part where my progress was relatively clear and linear. I would leave my fumbling clumsiness off of the page.

Instead, I want to show you what actually happened. I want you to know what it is like when I am "feeling around in the dark” toward the beginning of a study. I want to show you the reality of looking for a fulcrum experience when you haven’t already decided what you’re looking for. Because in truth, it can be quite difficult and discouraging, even when you’re pretty good at this stuff; it's important to be prepared for that.

So I want you to see me struggle, to see how I wrestle with challenges. In the rest of this post, I hope to highlight the moves that allowed me to successfully progress, point...
Conspiracy Theorists Aren't Ignorant. They're Bad At Epistemology.

by Bentham's Bulldog — 2024-02-28T23:39:39.192Z

Cross-post of my blog article on the topic.

I probably know less about science than most people who think the earth is flat do.

Okay, that’s not quite true. I have knowledge of lots of claims of science—that the big bang happened, that evolution is true, that the earth is round, etc—that people who think the earth is flat don’t have. But in terms of my knowledge of the science surrounding how one shows the roundness of the earth, I, like most people who think the earth is round, probably know less about it than most people who think the earth is flat.

People are often baffled by the widespread success of conspiracy theories. Among well-educated liberals, for instance, there’s a common attitude that conspiracy theorists are just deeply ignorant about various facts. One might say something like “how do conspiracy theorists believe X when Y.” Yet if one actually talks to a conspiracy theorist, they’ll have something to say about Y—likely something that is hard to refute on the fly.

The problem with conspiracy theorists is not that they’re ignorant of a few basic facts. They often have a shockingly large store of knowledge—just watch a debate with

this guy

, for example. It’s that they are bad at thinking. They accept improbable theories, largely neglect higher-order evidence, and are bad at judging the plausibility of a view.

There was a guy who would often come to my campus to yell about the earth being flat. I’m sure he could win an argument on that topic with most random people—for he came with real arguments. They were not good arguments, but he had more things to say than most people. The problem with his thinking was not that he was ignorant of a basic fact, but that he thought that positing a global conspiracy with thousands of actors that has somehow been ascertained by disproportionately scientifically illiterate internet sleuths was a better theory than thinking that there is some unknown explanation for apparently strange features of water.

At one point, for instance, he started talking about there being two different kinds of eclipses that only makes sense if the earth is flat. The reason non-conspiracy theorists reject this is not that they have some specialized knowledge about eclipses, but because they accept, based on higher-order evidence, that there’s not some deep hole in our understanding of eclipses that has eluded all the top science journ...
Discovering alignment windfalls reduces AI risk
by goodgravy — 2024-02-28T21:23:27.876Z
Link: https://blog.elicit.com/alignment-windfalls/

Some approaches to AI alignment incur upfront costs to the creator (an “alignment tax”). In this post, I discuss “alignment windfalls” which are strategies that tend towards the long-term public good at the same time as reaping short-term benefits for a company.

My argument, in short:

Just as there are alignment taxes, there are alignment windfalls.
AI companies optimise within their known landscape of alignment taxes & windfalls.
We can change what AI companies do by:

Shaping the landscape of taxes and windfalls
Shaping their knowledge of that landscape

By discovering and advocating for alignment windfalls, we reduce AI risk overall because it becomes easier for companies to adopt more alignable approaches.

Alignment taxes

An “

alignment tax

” refers to the

reduced performance, increased expense, or elongated timeline

required to develop and deploy an aligned system compared to a merely useful one.

More specifically, let’s say an alignment tax is an investment that a company expects to help with alignment of transformative AI that has a net negative impact on the company's bottom line over the next 3-12 months.

A few examples, from most to least concrete:

Adversarial robustness: For vision models, and maybe deep learning models in general, there seems to be a trade-off between adversarial robustness and in-distribution performance.[1] Effort can be invested in improving either metric, and adversarial training often requires larger training sets and more complex implementations. Simply put: making your model behave more predictably in unexpected scenarios can lead to it performing worse in everyday circumstances.
Robustness to distributional shift: Similarly, it’s easiest to develop and deploy systems that assume that the future will be like the past and present. For example, you don’t need to

detect sleeper agents

if you only care about the training distribution.
Safe exploration: Once AI systems are interacting with the real world, adding additional controls that prevent unsafe or unethical behaviours is more costly than simply allowing unlimited exploration.
Avoiding power-seeking: AI systems that engage in power-seeking behaviour—acquiring resources, colluding with other AI systems—may be more economically valuable in the short term, at the expense of long-term control.

All of these require more investment than the less aligned baseline comparison, and companies ...