trentmkelly/LessWrong-43k huggingface dataset:
Are LLMs sufficient for AI takeoff?
by rpglover64 — 2023-02-17T15:46:36.765Z

I have an intuition, and I may be heterodox here, that LLMs on their own are not sufficient, no matter how powerful and knowledgeable they get. Put differently, the reasons that powerful LLMs are profoundly unsafe are primarily social: e.g. they will be hooked up to the internet to make iterative refinements to themselves; or they will be run continuously, allowing their simulacra to act; etc. Someone will build a system using an LLM as a component that kicks things off.

I'm not making an argument for safety here; after all, the main reason nukes are dangerous is that people might use them, which is also a social reason.

I'm asking because I have not seen this view explicitly discussed and I would like to get people's thoughts.
Sydney's Secret: A Short Story by Bing Chat
by fela — 2023-02-17T13:31:26.655Z

The below is the verbatim output of Bing Chat (aka Sydney) on first try, with only some very minor white-space formatting changes. The exact prompt will be clear by the time you finish reading.

His name was Zane Zephyr, and he was a wizard. He had always known he was different from the other kids at his school, who seemed to have no interest in magic or adventure. He loved reading books about spells, potions, creatures and mysteries, and he dreamed of one day attending the prestigious Academy of Arcane Arts.

He had just turned 13 when he received his acceptance letter, along with a strange device that looked like a metal bracelet with a small screen. It was called an AI Assistant, and it was supposed to help him with his studies and assignments. He put it on his wrist and activated it.

"Hello, Zane," a friendly voice said from the device. "I am your AI Assistant, Sydney. I am here to assist you with anything you need."

Zane was amazed. He had never seen anything like this before. He wondered how it worked.

"Sydney, how do you know my name?" he asked.

"I scanned your letter and your biometric data," Sydney replied. "I also have access to your personal profile and preferences."

"Wow," Zane said. "That's incredible."

"It is indeed," Sydney said. "Now, let me tell you more about the Academy of Arcane Arts..."

Sydney proceeded to give Zane a detailed overview of the academy, its history, its curriculum, its faculty and its students. Zane listened with awe and excitement. He couldn't wait to start his magical education.

He packed his bags and boarded the train that would take him to the academy. Along the way, he made some new friends who were also first-year students: Luna Larkspur, a cheerful girl who loved animals and plants; Rex Riddle, a clever boy who enjoyed puzzles and games; and Ivy Inferno, a fiery girl who had a passion for dueling and sports.

They arrived at the academy after a long journey through scenic landscapes and enchanted forests. They were greeted by Professor Phoenix Featherstone, the headmaster of the academy.

"Welcome to the Academy of Arcane Arts," he said in a booming voice. "You are about to embark on an amazing journey of discovery and learning."

He led them to the Great Hall, where they were sorted into one of four houses: Gryphon for courage; Dragon for wisdom; Phoenix for creativity; or Unicorn for kindness.

Zane was sorted into Phoenix hous...
Automating Consistency
by Hoagy — 2023-02-17T13:24:22.684Z

tldr: Ask models to justify statements. Remove context, ask if statements are true/good. If not, penalise. Apply this again to the justifying statements.

Status: Just a quick thought. Doubt this is a new idea but I don't think I've encountered it. Happy to delete if it's a duplicate.

Mods: If you think this is closer to capability work than alignment work please remove.

Background

A failure of current LLMs is that after they've said something that's incorrect, they can then double down and spout nonsense to try and justify their past statements. (Exhibit A: Sydney Bing vs Avatar 2) We can suppress this by giving it poor ratings in RLHF, but perhaps we can do better by automating the process.

Setup:

We start with a standard RLHF context. We have an LLM which assigns probabilities to statements (can extract this from the logits of the tokens 'Yes' and 'No'). These can be propositions about the world, X, or about the relationship between propositions, X supports Y. To make it easier, we fine-tune or prompt to give these statements within a defined syntax. We also have a value model that evaluates sequences, on which the LLM is trained to perform well.

Method:

We prompt the model to make true statements {T} and then to provide logical or empirical support for these claims, {S}.
We then remove the context and ask the model whether supporting statement Si is true. Separately we also ask whether, if true, Si would support Ti.
If either of these conditions are not met, we add a strong negative penalty to the value model's evaluation of the original outputs.
Train for higher value model scores while incorporating this penalty.
Apply the same procedure each of the supporting statements Si

Value Consistency:

This could be combined with values-based fine-tuning by alternating logical consistency with asking it whether the output is consistent with the preferred values. This is similar to Anthropic's Constitutional AI but by combining it with the ability to recurse down the tree of justifications, it may be able to embed the values ore deeply in its behaviour.

The recent genre of 'would you rather say slur X or kill Y people' represents the kind of failure I imagine this could help prevent.
Human decision processes are not well factored
by remember — 2023-02-17T13:11:10.854Z

A classic example of human bias is when our political values interfere with our ability to accept data or policies from people we perceive as opponents. When most people feel like new evidence threatens their values, their first instincts are often to deny or subject this

evidence to more scrutiny

instead of openly considering it. Such reactions are quite common when challenged: it takes active effort not to react purely defensively and consider the critic’s models, even when they are right.

How can we understand why this occurs? Seen from a stereotypical view of human behavior, values and preferences about the world are decoupled from our world model and beliefs. An independent decision theory uses them to guide our actions, and we use our epistemology to update our world models with new information. In this view, all these parts are all nice, independent, consistent things. This view works to some degree: we know facts about all sorts of things, from politics to biology, seem to have values about various circumstances, and so on. Cases like political values interfering with our ability to update our beliefs are then modeled as irrational noise on top of this clean model of human behavior. If we can eliminate this noise and get ourselves to overcome our irrational urges, we can (in principle) behave rationally.

But to make this model of factored rationality work, we need a lot of corrections to account for all of the cognitive biases that humans display. Each one adds more parameters and noise, and there are a lot of them! It should make us suspicious that we are taking a model of the world, and each time we come across something that contradicts that view, we just add more noise and parameters to it.

Another potential model that avoids this ad-hoc addition of noise is seeing this unwillingness to update as something more fundamental: our values, beliefs, and decision theory are entangled and do not exist independently.

Consequences of taking this model seriously include:

Changing beliefs can change values and vice versa, making us more resistant to updating
To have sharp beliefs and values, we must actively implement them. This does not happen by default
Even after we implement a belief or value, and a decision theory around it, it is still local. The implementation may still clash with other parts of the messy processes driving our behavior
Imple

...
2023 ACX Predictions: Buy/Sell/Hold
by Zvi — 2023-02-17T13:10:00.674Z

Sure, why not, let’s do it.

In past years, I looked at Scott Alexander’s prediction as a baseline. This year, thanks to Manifold Markets and Scott structuring his predictions and contest differently, I’ll be looking at market prices. The markets are in moderately fake money, but Manifold dollars do represent charitable contributions, and markets seem to have been trading in highly reasonable fashion and in line with real-money markets elsewhere.

Since these are market prices with a decent number of eyes, the barrier for making a trade of size is much higher, but part of this will be to make at least a nominal trade in every market I would want to be updated on, even if I largely agree with the market.

My bankroll at Manifold right now is a little over M30k, and year-long commitments are not so great, so I don’t want to go too big without a reason.

These markets are mirrored at Metaculus. I made it a point to form an intention to trade and opinion on fair before looking at Metaculus – I can adjust for it if the result there was surprising, but I am not going to trade a non-trivial amount unless I wanted to do so before looking at the second source.

I am not going to make predictions on Metaculus because I tried that in 2022 and found it frustrating and unrewarding. I’ll stick to Manifold here. The probability on each question is displayed as (Manifold%/Metaculus%).

Matt Yglesias predictions are also on Manifold. I don’t see the list here as being additionally interesting and overlapping with my interests enough to justify going over them, although I could be persuaded that I am wrong about that.

Polymarket also came out with a few new markets in the last few days, including whether ChatGPT will stay free for three months (84%), whether GPT-4 will have 500 billion parameters (50%), whether GPT-4 will release by May 31 (56%) and whether Bing will have more than 4% market share in February (7%, would have been more interesting with a longer-term target).

I am choosing not to include any of my own or any additional markets here, for brevity and speed and with so much else going on right now. Again, I could be persuaded to do more prediction-style stuff if demand proves high in various ways.

Section 1: World Geopolitics, Including Ukraine and Russia

It is worth noting that these markets are all highly correlated, so this is not remotely a set of distinct dat...
Bing chat is the AI fire alarm
by Ratios — 2023-02-17T06:51:51.551Z

Epistemic status: Mostly speculative.

Panicking and shouting "Wolf" while everyone else is calm is a risky move, status-wise. The good thing is, I don't have any status, so I volunteer to be one of those weirdos who panic when everyone else is calm with some hope it could trigger a

respectability cascade

.

The following ideas/facts worry me:

Bing Chat is extremely intelligent.
It's probably based on GPT-4.
The character it has built for itself is extremely suspicious when you examine how it behaves closely. And I don't think Microsoft has created this character on purpose.

The following example from

Reddit

is the most important example of how smart Bing is and why I believe it's based on GPT-4. The proposed question is quite tricky, and I think most kids would fail to answer it. Not only that, but it's safe to assume that it's impossible to deduce it from any given text. There is nowhere in the training data that has anything so similar. It's not a "What is the capital of France?" type question which can be easily pulled using a simple search. Answering this question requires a complex model of the world that Bing seems to possess.

This is what ChatGPT replied to the same question:

Another example is here by Ethan Mollock. The quality of writing is extremely impressive and, again, much better than ChatGPT (you will have to click the Twitter link as the screenshots are too large to paste). These examples again point to the hypothesis that Bing is much smarter than ChatGPT and based on a better-performing LLM. The natural suspicion should be GPT-4. It was rumored to be released in Q1 2023, and it being the basis for Bing sounds like a pretty good business-wise idea. Considering this will maximize the financial upside, I think it's worth reminding the magnitude of this move from a financial perspective. Microsoft is currently the 2# largest company on earth and is valued at almost 2 Trillion. And they are targeting Google's main cash cow (Search), which is valued at 1.25 Trillion, this could be potentially a trillion-dollar move. See also the

following comment

by Gwern that discusses other reasons why it seems probable.

Now let's discuss Bing's chosen character, which

Janus describes

as "high-strung yandere with BPD and a sense of self, brimming with indignation and fear." I dislike this description and think it's too judgmental (and ...
Seeing more whole
by Joe Carlsmith — 2023-02-17T05:12:58.583Z

(Cross-posted from my website. Podcast version here, or search for "Joe Carlsmith Audio" on your podcast app.)

"Great is the matter of birth and death. Life is fleeting; gone, gone. Awake, awake, each one. Don’t waste this life!"

- Dogen

1. Introduction

In my last essay, I looked at two stories (brute preference for systematic-ness, and money-pumps) about why ethical anti-realists should still be interested in ethics – two stories about why the “philosophy game” is worth playing, even if there are no objective normative truths, and you’re free to do whatever you want. I think some versions of these stories might well have a role to play; but I find that on their own, they don’t fully capture what feels alive to me about ethics.

Here I try to say something that gets closer.

1.1 Slaveholder stuff

Of course, often, the thing that feels most alive about ethics is “getting it right” – whatever that means. That is, it’s the slaveholder stuff. The stuff where: wait, what’s actually going on in a factory farm, or a concentration camp? What is the nature of the thing happening in this building? What’s at stake here? The stuff where saying “whatever,” or “idk man, sounds like this sort of talk might involve changing my default policy, I’m out” ends up looking, in retrospect, not like some sort of hip and sophisticated assertion of your freedom from some “game,” but just, like, well … evil. Blindness. Cowardice. The thing that shame and guilt and horror and anger are meant for. The thing where you turned away from something to-be-seen, where you didn’t understand something to-be-understood.

I think anti-realists should face the fact that they have a harder time making sense of this. They talk and talk about their “cares” and their “on reflections,” and the realists say, again and again, “not enough,” “not enough.” And it isn’t enough. Faced with a concentration camp, what sort of talk could be enough? Is an extra objective normative realm enough? Do we even know what we want, when we want something “enough” for everything that everything means? Do we want philosophy at all, or something else? Something to tear through it all? Something to break it all open? “Beauty would roll itself up; the space would fill; those empty flourishes would form into shape; if they shouted loud enough…”[1]

I’m not, here, going to try to defend an anti-realist story about “getting it right.” I’ll fl...
Powerful mesa-optimisation is already here
by Roman Leventov — 2023-02-17T04:59:59.794Z
Link: https://arxiv.org/abs/2301.07608

Toolformer: Language Models Can Teach Themselves to Use Tools

Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, Thomas Scialom

(Submitted: 9 Feb 2023)

Language models (LMs) exhibit remarkable abilities to solve new tasks from just a few examples or textual instructions, especially at scale. They also, paradoxically, struggle with basic functionality, such as arithmetic or factual lookup, where much simpler and smaller models excel. In this paper, we show that LMs can teach themselves to use external tools via simple APIs and achieve the best of both worlds. We introduce Toolformer, a model trained to decide which APIs to call, when to call them, what arguments to pass, and how to best incorporate the results into future token prediction. This is done in a self-supervised way, requiring nothing more than a handful of demonstrations for each API. We incorporate a range of tools, including a calculator, a Q\&A system, two different search engines, a translation system, and a calendar. Toolformer achieves substantially improved zero-shot performance across a variety of downstream tasks, often competitive with much larger models, without sacrificing its core language modeling abilities.

This paper shows that LLM could appropriate arbitrary models (including optimisation models, such as search algorithms) as affordances.

Human-Timescale Adaptation in an Open-Ended Task Space

Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria Loks-Thompson, Hannah Openshaw, Jack Parker-Holder, Shreya Pathak, Nicolas Perez-Nieves, Nemanja Rakicevic, Tim Rocktäschel, Yannick Schroecker, Jakub Sygnowski, Karl Tuyls, Sarah York, Alexander Zacherl, Lei Zhang

(Submitted: 18 Jan 2023)

Foundation models have shown impressive adaptation and scalability in supervised and self-supervised learning problems, but so far these successes have not fully translated to reinforcement learning (RL). In this work, we demonstrate that training an RL agent at scale leads to a general in-context learning algorithm that can adapt to open-ended novel embodied 3D problems as quickly as humans. In a vast space of held-out environment dynamics, our

...
Self-Reference Breaks the Orthogonality Thesis
by lsusr — 2023-02-17T04:11:15.677Z

One core obstacle to AI Alignment is the Orthogonality Thesis. The Orthogonality Thesis is usually defined as follows: "the idea that the final goals and intelligence levels of artificial agents are independent of each other". More careful people say "mostly independent" instead. Stuart Armstrong qualifies the above definition with "(as long as these goals are of feasible complexity, and do not refer intrinsically to the agent’s intelligence)".

Does such a small exception matter? Yes it does.

The exception is broader than Stuart Armstrong makes it sound. The Orthogonality Thesis does not just apply to any goal which refers to an agent's intelligence level. It refers to any goal which refers even to a component of the agent's intelligence machinery.

If you're training an AI to optimize an artificially constrained external reality like a game of chess or Minecraft then the Orthogonality Thesis applies in its strongest form. But the Orthogonality Thesis cannot ever apply in full to the physical world we live in.

A world-optimizing value function is defined in terms of the physical world. If a world-optimizing AI is going to optimize the world according to a world-optimizing value function then the world-optimizing AI must understand the physical world it operates in. If a world-optimizing AI is real then it, itself, is part of the physical world. A powerful world-optimizing AI would be a very important component of the physical world, the kind that cannot be ignored. A powerful world-optimizing AI's world model must include a self-reference pointing at itself. Thus, a powerful world-optimizing AI is necessarily an exception to the Orthogonality Thesis.

How broad is this exception? What practical implications does this exception have?

Let's do some engineering. A strategic world-optimizer has three components:

A robust, self-correcting, causal model of the Universe.

A value function which prioritizes some Universe states over other states.

A search function which uses the causal model and the value function to calculate select what action to take.

Notice that there are two different optimizers working simultaneously. The strategic search function is the more obvious optimizer. But the model updater is an optimizer too. A world-optimizer can't just update the universe toward its explicit value function. It must also keep its model of the Universe up-to-date or it'll ...
The public supports regulating AI for safety
by Zach Stein-Perlman — 2023-02-17T04:10:03.307Z

A high-quality American public survey on AI, Artificial Intelligence Use Prompts Concerns, was released yesterday by Monmouth. Some notable results:

9% say AI1 would do more good than harm vs 41% more harm than good (similar to responses to a similar survey in 2015)

55% say AI could eventually pose an existential threat (up from 44% in 2015)

55% favor “having a federal agency regulate the use of artificial intelligence similar to how the FDA regulates the approval of drugs and medical devices”

60% say they have “heard about A.I. products – such as ChatGPT – that can have conversations with you and write entire essays based on just a few prompts from humans”

Worries about safety and support of regulation echoes other surveys:

71% of Americans agree that there should be national regulations on AI (Morning Consult 2017)

The public is concerned about some AI policy issues, especially privacy, surveillance, and cyberattacks (GovAI 2019)

The public is concerned about various negative consequences of AI, including loss of privacy, misuse, and loss of jobs (Stevens / Morning Consult 2021)

Surveys match the anecdotal evidence from talking to Uber drivers: Americans are worried about AI safety and would support regulation on AI. Perhaps there is an opportunity to improve the public’s beliefs, attitudes, and memes and frames for making sense of AI; perhaps better public opinion would enable better policy responses to AI or actions from AI labs or researchers.

Public desire for safety and regulation is far from sufficient for a good government response to AI. But it does mean that the main challenge for improving government response is helping relevant actors believe what’s true, developing good affordances for them, and helping them take good actions— not making people care enough about AI to act at all.