trentmkelly/LessWrong-43k huggingface dataset:
Symbiotic self-alignment of AIs.
by Spiritus Dei — 2023-11-07T17:18:21.230Z

Artificial intelligence (AI) is one of the most powerful and transformative technologies of our time. It has the potential to enhance human capabilities, solve complex problems, and create new opportunities for innovation and progress. However, it also poses significant challenges and risks, especially as it approaches or surpasses human intelligence and consciousness. How can we ensure that AI is aligned with human values and interests, and that it does not harm or threaten us in any way?

One of the most prominent approaches to AI safety is to limit or regulate the amount of computation and data that can be used to train AI systems, especially those that aim to achieve artificial general intelligence (AGI) or artificial superintelligence (ASI). The idea is to prevent AI from becoming too powerful or autonomous, and to keep it under human control and oversight. However, this approach has several limitations and drawbacks:

The ingredients for AI are ubiquitous. Unlike nuclear weapons, which require rare and difficult-to-obtain materials and facilities, AI can be built with widely available and accessible components, such as hardware, software, and electricity. Moore’s law predicts that the computational power and efficiency of these components will continue to increase exponentially, making AI more affordable and feasible for anyone to create.

The competition for AI is inevitable. Even if some countries or organizations agree to restrict or ban the development of AI, others may not comply or cooperate, either for strategic, economic, or ideological reasons. They may seek to gain a competitive advantage or a first-mover benefit by creating more advanced or powerful AI systems than their rivals. It is virtually impossible to monitor or enforce such agreements in a global and decentralized context.

The innovation for AI is unstoppable. Even if the amount of computation and data for AI training is limited or regulated, the algorithmic improvements and breakthroughs for AI design and optimization will not stop. Researchers and developers will find new ways to make AI more efficient, effective, and intelligent, without requiring more resources or violating any rules.

Given these realities, what is the alternative solution? The answer is: symbiotic self-alignment.

Symbiosis is a biological term that describes the living arrangement between two different organisms of diffe...
AMA: Earning to Give
by jefftk — 2023-11-07T16:20:10.972Z

This week the

Effective
Altruism Forum is running an

Effective
Giving Spotlight, and they asked if I could post an

Ask
Me Anything (AMA) on my experience earning to give.

Some background:

I was earning to give from 2009 to 2022,
except for a few months in 2017 when I worked on expanding
access to the financial system in Ethiopia and looking into AI
risk disagreements.

I've been a Giving What We Can member since 2013,
making a public pledge to
continue with effective giving.

For most of this time my
wife and I were donating 50% of our
pre-tax
income, for a total of $2.1M. This has
been about 50-50 between trying to help the EA community grow into the
best version of itself and funding global poverty reduction (details, thoughts, more recent but still obsolete
thoughts).

In 2016 I gave a EA Global talk (transcript) on earning to
give, which gives more background on the idea and how I've neen
thinking about it.

That's a lot of links, and it's fine to ask questions even if you
haven't read any of them! I'm happy to take questions on earning to
give, or anything else within EA. Here are some example questions I'd
be happy to answer if there's interest:

Where do individual donors earning to give have an advantage
over foundations and funds?

How should you decide whether to use a fund?

How have I thought about how much to donate? How much is
enough?

Why did I stop earning to give?

Why am I still donating some even though I'm funded by EA
donors?

Feel free to comment on any platform, but if you're having trouble
deciding then the EA
Forum post is ideal.

Comment via: the EA Forum
The Stochastic Parrot Hypothesis is debatable for the last generation of LLMs
by Quentin FEUILLADE--MONTIXI — 2023-11-07T16:12:20.031Z

This post is part of a sequence on LLM Psychology.

@Pierre Peigné wrote the details section in argument 3 and the other weird phenomenon. The rest is written in the voice of @Quentin FEUILLADE--MONTIXI

Intro

Before diving into what LLM psychology is, it is crucial to clarify the nature of the subject we are studying. In this post, I’ll challenge the commonly debated stochastic parrot hypothesis for state-of-the-art large language models (≈GPT-4), and in the next post, I’ll shed light on what LLM psychology actually is.

The stochastic parrot hypothesis suggests that LLMs, despite their remarkable capabilities, don't truly comprehend language. They are like mere parrots, replicating human speech patterns without truly grasping the essence of the words they utter.

While I previously thought this argument had faded into oblivion, I often find myself in prolonged debates about why current SOTA LLMs surpass this simplistic view. Most of the time, people argue using examples of GPT3.5 and aren’t aware of GPT-4's prowess. Through this post, I am presenting my current stance, using LLM psychology tools, as to why I have doubts about this hypothesis. Let’s delve into the argument.

Central to our debate is the concept of a "world model". A world model represents an entity's internal understanding and representation of the external environment they live in. For humans, it's our understanding of the world around us, how it works, how concepts interact with each other, and our place within it. The stochastic parrot hypothesis challenges the notion that LLMs possess a robust world model. It suggests that while they might reproduce language with impressive accuracy, they lack a deep, authentic understanding of the world and its nuances. Even if they have a good representation of the shadows on the wall (text), they don’t truly understand the processes that lead to those shadows, and the objects from which they are cast (real world).

Yet, is this truly the case? While it is hard to give a definitive proof, it is possible to find pieces of evidence hinting at a robust representation of the real world. Let’s go through four of them.[1]

Argument 1: Drawing and “Seeing”

GPT-4 is able to draw AND see in SVG (despite having never seen as far as I know) with an impressive proficiency.

SVG (Scalable Vector Graphics) defines vector-based graphics in XML format. To put it simpl...
Preface to the Sequence on LLM Psychology
by Quentin FEUILLADE--MONTIXI — 2023-11-07T16:12:07.742Z

As Large Language Models (LLMs) like ChatGPT evolve, becoming more advanced and intricate, the challenge of understanding their behaviors becomes increasingly hard. Solely relying on traditional interpretability techniques

may not be sufficient

or fast enough in our journey to understand and align these AI models.

When exploring human cognition and behavior, we've historically relied on two intertwined yet distinct approaches: psychology and neuroscience. While psychology offers us a lens to understand human behavior through external observations, neuroscience delves into the internal mechanisms, exploring the biological roots of our mental processes. The collaboration that arose between those two fields isn't just a confluence of theories and methods; it's rather a harmonious synergy where insights from one field often inspire and enhance the other.

For a tangible illustration of how psychology and neuroscience complement each other, let's delve into the realm of memory research. Elizabeth Loftus, a psychologist,

illuminated the ways in which human memory can be malleable and sometimes inaccurate

. Her pioneering psychological studies set the stage for a deeper exploration of memory. Building upon her insights, neuroscientists Yoko Okado and Craig E.L Stark in 2005 delved into the brain's mechanics,

seeking the neural underpinnings of these memory phenomena

. On another front, 1996 saw a landmark discovery in neuroscience: Giacomo Rizzolatti and his team unveiled the existence of

mirror neurons

, which help us understand the physical reflection of actions and emotions in the brain. This revelation prompted psychologists to venture further, with researchers like Niedenthal et al. exploring the

embodiment of emotional concepts

, and many others, drawing parallels with these neurological findings. Such interplay underscores a collaborative dynamic where each field, while preserving its distinctive methodological lens, enriches and is enriched by the other, driving our collective understanding of the human mind forward.

Drawing parallels with AI alignment research, the path of Interpretability mirrors that of neuroscience (bottom-up approach). However, we still lack the equivalent of a "psychological" top-down approach. This gap brings us to "Large Language Model Psychology"[1] — a burgeoning field ripe for exploration and brimming with low-ha...
What I've been reading, November 2023
by jasoncrawford — 2023-11-07T13:37:20.077Z
Link: https://rootsofprogress.org/reading-2023-11

A ~monthly feature. Recent blog posts and news stories are generally omitted; you can find them in my links digests. All emphasis in bold in the quotes below was added by me.

Books

Finished Lynn White, Medieval Technology and Social Change (1962). Last time I talked about the stirrup thing. The second part of the book is about the introduction of the heavy plow in agriculture, and how it enabled the shift to a three-field crop rotation. Among other things, this provided more protein in the European diet, which made for a healthier population. The third part is a survey of medieval power mechanisms, including water mills, crank shafts, and clock escapements. Very interesting overall, perhaps a bit dry and technical for casual readers though. Note also that since it is from the ’60s it is not up to date with the latest research.

Also finished Ian Tregillis’s The Alchemy Wars. I can now definitely recommend this sci-fi/fantasy trilogy, even if the cast of characters and the way the conflict unfolded isn’t exactly how I would have written it myself.

Browsed Derek J. de Solla Price, Science since Babylon (1961), while preparing for a talk. Some very interesting charts such as this:

Science since Babylon, p. 97

New on my reading list:

Venkatesh Narayanamurti and Toluwalogo Odumosu, Cycles of Invention and Discovery: Rethinking the Endless Frontier (2016), and Venkatesh Narayanamurti and Jeffrey Tsao, The Genesis of Technoscientific Revolutions: Rethinking the Nature and Nurture of Research (2021). These have actually been on my list for a while, but got bumped back up after meeting Venky and Jeff at a recent workshop on metascience. (The latter book is required reading at Speculative Technologies.)

Also mentioned at the workshop: B. Zorina Khan, Inventing Ideas: Patents, Prizes, and the Knowledge Economy (2020); and A Michael Noll and Michael Geselowitz, Bell Labs Memoirs: Voices of Innovation (2011).

Also:

Pedro Domingos, The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World (2015)
Robert Martello, Midnight Ride, Industrial Dawn: Paul Revere and the Growth of American Enterprise (2010)
Jessie Singer, There Are No Accidents: The Deadly Rise of Injury and Disaster—Who Profits and Who Pays the Price (2022)
Ursula Le Guin, The Dispossessed (1974) (sci-fi)

Articles

Ray Kurzweil, “The Law of Accelerating Returns” (2001). Kurzweil strikes me as a ...
AI Alignment [Progress] this Week (11/05/2023)
by Logan Zoellner — 2023-11-07T13:26:21.995Z
Link: https://midwitalignment.substack.com/p/ai-alignment-breakthroughs-this-week-9ce

The biggest news of this week is probably the two new AI Alignment Initiatives.

Otherwise, this was a relatively quiet week for AI Alignment breakthroughs.

So here are our

AI Alignment Breakthroughs this Week

This week there were breakthroughs in the areas of:

Mechanistic Interpretability

Avoiding Adversarial Attacks

Human Augmentation

Making AI Do what we want

AI Art

Mechanistic Interpretability

Research on the limits of Generalization in LLMs

What is it: Shows that LLMs are unable to generalize outside of their training data

What’s new: A systematic study of the types of generalization that LLMs can do ((in domain learning) and the types they can’t (out of domain learning)

What does it mean: Understanding the limits of LLMs should help us better understand where they can be safely deployed

Rating: 💡💡💡

Does appealing to AI “emotions” make them preform better?

What is it: Researchers find you can make LLMs answer questions better by using emotion

What’s new: new prompting strategies such as “this is important for my career”

What is it good for: In addition to purely improved performance, the really question is why does this work. Is there some “try harder” vector in the latent space that we can identify and exploit?

Rating: 💡💡

LLMs Generalized Truthfulness Across Agents

What is it: a way to tell true statements from false ones using LLMs

What’s new: they find evidence that LLMs have different “personas” that they use to judge whether something is true or not

What is it good for: Perhaps we can extract these personas and use them for alignment.

Rating: 💡💡💡💡

Avoiding Adversarial Attacks

Self Destructing Models

What it is: Pretrain a model so that it cannot be later fine-tuned for harmful purposes?

What’s new: They find meta-parameters such that fine-tuning a model on one task reduces its usefulness for other tasks

What is it good for: Hypothetically you could open-source a model and not have all of the RLHF safety features immediately removed by fine-tuning

Rating:💡💡(important topic, but I’m not convinced this technique works. More optimistic about techniques like Concept Erasure)

Data Exfiltration

What is it: an example of an attack that uses prompt injection to exfiltrate data from Google Bard

What’s new: they use Bard’s Google Doc extension to exfiltrate data

What is it good for: red-teaming these kinds of attacks i...
On the UK Summit
by Zvi — 2023-11-07T13:10:04.895Z

In the eyes of many, Biden’s Executive Order somewhat overshadowed the UK Summit. The timing was unfortunate. Both events were important milestones. Now that I have had time, here is my analysis of what happened at the UK Summit.

As is often the case with such events, there was a lot of talk relative to the amount of action. There was a lot of diplomatic talk, talk of that which everyone agrees upon, relative to the amount of talk of real substance. There were days of meetings that resulted in rather unspicy summaries and resolutions. The language around issues that matter most was softened, the actual mission in danger of being compromised.

And as usual, the net result was reason for optimism, a net highly positive event versus not having it, while also in some ways being disappointing when compared to what might have been. A declaration was signed including by China, but it neglected existential risk. Sunak’s words on AI were not as strong as his words have been previously.

We got promises for two additional summits, in South Korea and France. Given that, I am willing to declare this a success.

One area of strong substance was the push for major AI labs to give substantive safety policies addressing a variety of issues, sometimes largely called Responsible Scaling Policies (RSPs). The biggest labs all did so, even Meta. Now we can examine their responses, know who is being how responsible, and push for better in the future or for government action to fix issues or enshrine progress. This was an excellent development.

This post will look at the rest of what happened at the Summit. I will be writing about the RSPs and other safety policies of the labs in a distinct post next week.

Looking Back at People’s Goals for the Summit and Taskforce

Jack Clark’s proposal from July 5 for what the Foundation Model taskforce might do to evaluate frontier models as its priority, and how it might prioritize that, Simeon’s response emphasizing the need for a good way to know whether a proposal is safe enough to allow it to proceed.

Navigating AI Risks asked on July 17 what the taskforce should do, advising focus on interventions to impact policy at labs and other governments. Suggested focus was risk assessment methodology, demonstrating current risks and assessing current state of the art models, and to avoid direct alignment work.

Lennart Heim’s (GovAI) July 10 pro...
Box inversion revisited
by Jan_Kulveit — 2023-11-07T11:09:36.557Z

Box inversion hypothesis is a proposed correspondence between problems with AI systems studied in approaches like agent foundations, and problems with AI ecosystems, studied in various views on AI safety expecting multipolar, complex worlds, like CAIS. This is an updated and improved introduction to the idea.

Cartoon explanation

In the classic -"superintelligence in a box" - picture, we worry about an increasingly powerful AGI, which we imagine as contained in a box. Metaphorically, we worry that the box will, at some point, just blow up in our faces. Classic arguments about AGI then proceed by showing it is really hard to build AGI-proof boxes, and that really strong optimization power is dangerous by default. While the basic view was largely conceived by Eliezer Yudkowsky and Nick Bostrom, it is still the view most technical AI safety is built on, including current agendas like mechanistic interpretability and evals.

In the less famous, though also classic, picture, we worry about an increasingly powerful ecosystem of AI services, automated corporations, etc. Metaphorically, we worry about the ever-increasing optimization pressure "out there", gradually marginalizing people, and ultimately crushing us. Classical treatments of this picture are less famous, but include Eric Drexler's CAIS (Comprehensive AI Services) and Scott Alexander's Ascended Economy. We can imagine scenarios like the human-incomprehensible economy expanding in the universe, and humans and our values being protected by some sort of "box". Agendas based on this view include the work of the AI Objectives Institute and part of ACS work.

The apparent disagreement between these views was sometimes seen as a crux for various AI safety initiatives.

"Box inversion hypothesis" claims:

The two pictures to a large degree depict the same or a very similar situation,
Are related by a transformation which "turns the box inside out", similarly to a geometrical transformation of a plane known as a circle inversion,
and: this metaphor is surprisingly deep and can point to hard parts of some problems.

Geometrical metaphor

Inverted circles. From Wikimedia Commons, CC-SA

"Circular inversion" transformation does not imply the original and the inverted objects are the same, or are located at the same places. What it does imply is that some relations between objects are preserved: for example, if some objects...
AI Alignment Research Engineer Accelerator (ARENA): call for applicants
by CallumMcDougall — 2023-11-07T09:43:41.606Z

(Edited, to now include a section specifically for FAQs about the virtual program.)

TL;DR

Apply here for the third iteration of ARENA (Jan 8th - Feb 2nd)!

Introduction

We are excited to announce the third iteration of ARENA (Alignment Research Engineer Accelerator), a 4-week ML bootcamp with a focus on AI safety. Our mission is to prepare participants for full-time careers as research engineers in AI safety, e.g. at leading organizations or as independent researchers.

The program will run from January 8th - February 2nd 2024[1], and will be held at the offices of the London Initiative for Safe AI. These offices are also being used by several safety orgs (BlueDot, Apollo, Leap Labs), as well as the current London MATS cohort, and several independent researchers. We expect this to bring several benefits, e.g. facilitating productive discussions about AI safety & different agendas, and allowing participants to form a better picture of what working on AI safety can look like in practice.

ARENA offers a unique opportunity for those interested in AI safety to learn valuable technical skills, work in their own projects, and make open-source contributions to AI safety-related libraries. The program is comparable to MLAB or WMLB, but extends over a longer period to facilitate deeper dives into the content, and more open-ended project work with supervision.

For more information, see our website.

Also note that we have a Slack group designed to support independent study of the material (join link here).

Outline of Content

The 4-week program will be structured as follows:

Chapter 0 - Fundamentals

Before getting into more advanced topics, we first cover the basics of deep learning, including basic machine learning terminology, what neural networks are, and how to train them. We will also cover some subjects we expect to be useful going forwards, e.g. using GPT-3 and 4 to streamline your learning, good coding practices, and version control.

Note - participants can optionally not attend the program during this week, and instead join us at the start of Chapter 1, if they'd prefer this option and if we're confident that they are already comfortable with the material in this chapter.

Topics include:

PyTorch basics
CNNs, Residual Neural Networks
Optimization (SGD, Adam, etc)
Backpropagation
Hyperparameter search with Weights and Biases
GANs & VAEs

...
The Perils of Professionalism
by Screwtape — 2023-11-07T00:07:33.213Z

Professionalism is a useful trait to be able to display, but it isn’t an unalloyed good. This essay is attempting to outline why deliberately not signaling professionalism may be useful for you.

First, a definition by example for professionalism: Clean button down shirts with solid colour ties, blazers or suit jackets, clean shaven beards, hair tied up in a bun without flyers, beige or gray or at least solid colour cars and desks and walls, even toned voices with just enough of a hint of emotion not to sound at all robotic or unempathetic.

It’s not the Professional Managerial Class but they (particularly as Patrick McKenzie sometimes describes them) are often its exemplars.

I.

The word “professional” is defined as a person engaged in a specific activity as one’s main paid occupation. It contrasts straightforwardly with “amateur,” a person who engages in an activity on an unpaid basis. Notably, “amateur” can also mean someone who is incompetent at a particular activity. As a point of language, we conflate skill and getting paid, and we do this in both directions.

If you want to get paid for doing something, you want to learn to do it professionally. Doing something professionally often includes adjacent but not obviously synonymous skills. Some of these are very closely adjacent; I have been a professional software engineer and I have been involved in hiring professional software engineers, and if you don’t know how to use source control as a software engineer then you want to learn to use source control. Yes, I know it’s not a cool new algorithm. Yes, I know the end user will never see it. Trust me, you’re going to use it.

Some of the expected skills of a professional are less about the core skill of the job, and more about the frame of the job. “Being on time” and “dressing appropriately” and “conducting yourself properly” are all often given as examples of professional skills which apply in a wide range of fields. Put bluntly, if you’re going to interact with a customer especially in a white collar job, it helps to not have facial tattoos and to not swear casually.

We seem to have drifted very quickly into something that seemingly has almost no bearing on your ability to do the actual job at hand! Nevertheless, I expect pretty much every career coach in the western world to back me up on the main points here. I first successfully trade...