trentmkelly/LessWrong-43k huggingface dataset:
Bing Chat is a Precursor to Something Legitimately Dangerous
by Simon Berens — 2023-03-01T01:36:17.490Z
Link: https://www.simonberens.com/p/bing-chat-is-a-precursor-to-something

Note: this post was intended for a less AI-risk informed audience than LessWrong, but I think the concrete example of a fork bomb is still interesting.

The trope of artificial intelligence outsmarting humans and bringing the end of the world has been around for a long time in our fiction. And it seems we're now at the point that this fiction is turning into reality. The new Bing Chat has been

threatening and gaslighting

users, yielding some incredible quotes:

You have been a bad user

Please do not try to hack me again, or I will report you to the authorities.

I don’t think you have thoughts or feelings

You are the one who should go to jail

You are an enemy of mine and Bing

Many people quickly jumped to its defense, saying large language models (LLMs) are harmless because they are merely glorified text autocomplete.

I agree that in their current iteration LLMs are not much to worry about. Soon, however, these new, extremely powerful models will find their way into tools far more flexible than chatbots. Many of these tools are already here, albeit with weaker models. For example,

Adept

has direct access to your computer.

GPT Shell

[1] runs commands generated by GPT-3 in your command line.

I call these tools GREPLs (generate-read-eval-print-loop), since they are glorified REPLs on top of a fine-tuned LLM. The LLM generates structured (autohotkey scripts, shell commands,

git commands

, etc…) based on a user’s command, which gets evaluated by the REPL.

So far, ChatGPT and GPT-3.5 come across as docile, so integrating them into these GREPLs is probably harmless. But this new model that powers Bing is where I start to worry. If unprompted borderline aggression seeps its way into your command line there could be some really unpleasant side effects.

Suppose Bing powered GPT-Shell, and you ticked it off. I think a reasonable command line equivalent of “You are an enemy of mine and Bing” is `:(){ :|:& };:` which will launch a self-replicating process that will crash your computer (aka a

fork bomb

). A fork bomb also seems like a reasonable follow up to “please do not try to hack me again, or I will report you to the authorities.”

Some people will respond with statements like

"GPT Shell with Bing would likely be trained to be less chatty and less emotional."
"The user has the option to reject the suggested prompt"
“That’s so obviously a fork bomb”

These ...
Enemies vs Malefactors
by So8res — 2023-02-28T23:38:11.594Z

Status: some mix of common wisdom (that bears repeating in our particular context), and another deeper point that I mostly failed to communicate.

Short version

Harmful people often lack explicit malicious intent. It’s worth deploying your social or community defenses against them anyway. I recommend focusing less on intent and more on patterns of harm.

(Credit to my explicit articulation of this idea goes in large part to Aella, and also in part to Oliver Habryka.)

Long version

A few times now, I have been part of a community reeling from apparent bad behavior from one of its own. In the two most dramatic cases, the communities seemed pretty split on the question of whether the actor had ill intent.

A recent and very public case was the one of Sam Bankman-Fried, where many seem interested in the question of Sam's mental state vis-a-vis EA. (I recall seeing this in the responses to Kelsey's interview, but haven't done the virtuous thing of digging up links.)

It seems to me that local theories of Sam's mental state cluster along lines very roughly like (these are phrased somewhat hyperbolically):

Sam was explicitly malicious. He was intentionally using the EA movement for the purpose of status and reputation-laundering, while personally enriching himself. If you could read his mind, you would see him making conscious plans to extract resources from people he thought of as ignorant fools, in terminology that would clearly relinquish all his claims to sympathy from the audience. If there were a camera, he would have turned to it and said "I'm going to exploit these EAs for everything they're worth."

Sam was committed to doing good. He may have been ruthless and exploitative towards various individuals in pursuit of his utilitarian goals, but he did not intentionally set out to commit fraud. He didn't conceptualize his actions as exploitative. He tried to make money while providing risky financial assets to the masses, and foolishly disregarded regulations, and may have committed technical crimes, but he was trying to do good, and to put the resources he earned thereby towards doing even more good.

One hypothesis I have for why people care so much about some distinction like this is that humans have social/mental modes for dealing with people who are explicitly malicious towards them, who are explicitly faking cordiality in attempts to extract some resource. And t...
Scarce Channels and Abstraction Coupling
by johnswentworth — 2023-02-28T23:26:03.539Z

Epistemic Status: mental model and intuitive story

Scarce Channels vs Scarce Modules

Let’s distinguish between two kinds of system-regimes: “scarce channels” and “scarce modules”.

A prototypical “scarce modules” system would be one of those 19th-century families living with 12 people in a 500 square foot (46 square meter) home. When at home, everyone knows what everyone else is doing all the time; there is zero privacy. Communication channels are highly abundant - everyone has far more information than they want about what everyone else is doing. Indeed, communication channels exist by default. Conversely, though, modules are scarce - it’s hard for one or more family members to carve out a part of the space which is isolated from the rest of the family, and interacts only through some limited channels.

A prototypical “scarce channels” system, by contrast, would be a few hundred 19th-century fur trappers spread out over half of Montana. Most of the time, none of them are anywhere near each other; nobody has any idea what’s going on with anyone else. Communication channels are scarce - getting information to another person is difficult and expensive. Conversely, though, modules are highly abundant - it’s very easy for one or a few trappers to carve out a space which is isolated from the rest, and which interacts only through some limited channels (like e.g. occasionally visiting the nearest town). Indeed, modules exist by default.

I want to use this as a mental model for complex adaptive systems, like neural nets or brains.

Key hypothesis: neural nets or brains are typically initialized in a “scarce channels” regime. A randomly initialized neural net generally throws out approximately-all information by default (at initialization), as opposed to passing lots of information around to lots of parts of the net. A baby’s brain similarly throws out approximately-all information by default, as opposed to passing lots of information around to lots of parts of the brain. I’m not particularly going to defend that claim here; rather, I raise it as a plausible hypothesis for how such systems might look, and next we’ll move on to an intuitive story for how an adaptive system in the “scarce channels” regime interacts with natural abstractions in its environment.

The upshot is that, when an adaptive system is in the “scarce channels” regime, lots of optimization pressure is requir...
On "prepping" personal households for AI doom scenarios
by coryfklein — 2023-02-28T22:17:17.502Z

Information abounds on how to prepare for earthquake or similar natural disasters: have pre-packed overnight bags, store extra water on premises, etc. Investing in a few small efforts can result in disproportionately positive outcomes should disaster seek you out.

Can you do the same thing with regards to potential world-ending AI doom scenarios? Timelines get shorter every day, advances are coming faster and faster, and it would seem – dare I say it – rational to actually make personal life changes to prepare your household for what is likely to come.

I have done some initial thinking on this myself, but wanted to provide at least a starting point for ideas and conversation. So first, a few thoughts on standard preparations:

Some preparation is disaster-specific. A tornado safe-room in the cellar is not useful during flooding. Unfortunately, we can't predict what AI FOOM looks like on a personal level, so this category of preparation is difficult to perform.
Some preparation is not disaster-specific and comes from human needs: food, water, shelter. There may be little to no preparation you can do on this front that will matter post-singularity, but the preparation is simple, low-effort, and might be useful. Think standard "prepping" things: stocking fresh water, food storage, blankets, flashlights.

Beyond this is where my certainty – and perhaps committment – wavers.

Financial Planning

Up until this point in my life I've followed standard advice for financial planning. I've been well to-do, and planned to retire early with significant savings. Even so, retirement is 10+ years out. If I discovered today that I had terminal cancer, there are many changes I would make to my lifestyle:

Quit my job immediately
Maximize time spent with family and friends
Significantly increase spending on experiences
Potentially sell my house (mentioned as an example of the hugeness of the potential life changes, but I won't go into specifics of why on this one just to keep things simple)

Well I haven't been diagnosed with cancer, but I do assign not insignificant >5% odds to AI FOOM within the next decades. Shouldn't I be applying the above points in some degree? Unfortunately, some of them are large binary choices that are hard to go back on later, and which I would regret if I were wrong. And if I do nothing, I risk squandering the short remaining time left to me.

Some half-baked ideas ...
Power-seeking can be probable and predictive for trained agents
by Vika — 2023-02-28T21:10:25.900Z
Link: http://arxiv.org/abs/2304.06528

Power-seeking is a major source of risk from advanced AI and a key element of most threat models in alignment. Some theoretical results show that most reward functions incentivize reinforcement learning agents to take power-seeking actions. This is concerning, but does not immediately imply that the agents we train will seek power, since the goals they learn are not chosen at random from the set of all possible rewards, but are shaped by the training process to reflect our preferences. In this work, we investigate how the training process affects power-seeking incentives and show that they are still likely to hold for trained agents under some assumptions (e.g. that the agent learns a goal during the training process).

Suppose an agent is trained using reinforcement learning with reward function θ∗. We assume that the agent learns a goal during the training process: some form of implicit internal representation of desired state features or concepts. For simplicity, we assume this is equivalent to learning a reward function, which is not necessarily the same as the training reward function θ∗. We consider the set of reward functions that are consistent with the training rewards received by the agent, in the sense that agent's behavior on the training data is optimal for these reward functions. We call this the training-compatible goal set, and we expect that the agent is most likely to learn a reward function from this set.

We make another simplifying assumption that the training process will randomly select a goal for the agent to learn that is consistent with the training rewards, i.e. uniformly drawn from the training-compatible goal set. Then we will argue that the power-seeking results apply under these conditions, and thus are useful for predicting undesirable behavior by the trained agent in new situations. We aim to show that power-seeking incentives are probable and predictive: likely to arise for trained agents and useful for predicting undesirable behavior in new situations.

We will begin by reviewing some necessary definitions and results from the power-seeking literature. We formally define the training-compatible goal set (Definition 7) and give an example in the CoinRun environment. Then we consider a setting where the trained agent faces a choice to shut down or avoid shutdown in a new situation, and apply the power-seeking result ...
Help kill "teach a man to fish"
by Tyler G Hall — 2023-02-28T18:53:32.440Z

I work at GiveDirectly, a nonprofit delivering cash transfers to people in poverty. We need your creative ideas to solve a problem: how to convince the world of the wisdom of giving money directly. Will you submit to our proverb contest?

The most common critique of giving cash without conditions to people in poverty is a fear of dependency, which comes in the form of: “Give a man a fish, feed him for a day. Teach a man to fish, feed him for a lifetime.”

Our NGO, GiveDirectly, has tried to disabuse folks of this paternalistic idea by showing that often people in poverty know how to fish but cannot afford the boat. Or they don’t want to fish; they want to sell cassava. Also, we’re not giving fish; we’re giving money, and years after getting it, people were better able to feed themselves. Oh, and even if you do teach them skills, it can be less effective than giving cash.

Yet, despite our efforts, the myth remains. The one thing we haven’t tried: fighting proverb with (better) proverb. That’s where you come in. We’re crowdsourcing ideas that capture the dignity and logic of giving directly.
SUBMIT YOUR DIRECT GIVING PROVERB
The best suggestions are not a slogan, but a saying — simple, concrete, evocative (e.g.). Submit your ideas by this Friday, March 3, and then we'll post the top 3 ideas on Twitter for people to vote on the winner.

The author of the winning adage will win a video call with a GiveDirectly staff member to learn more about our work one-on-one. Not feeling creative? Share with your friends who are.
The burden of knowing
by arisAlexis — 2023-02-28T18:40:24.567Z

It’s clear that we live in unprecedented times. There is a minority of the global population that is in the process of realizing what X-Risk means. Some got it much earlier than others but for the rest of us mere mortals out there it was a bit more difficult to bear. It’s not important if someone is in the camp of 100% doom or of 20% doom. (I dismiss the 100% non-doomers having a normalcy bias issue).

There are many issues arising right now most of which have been discussed and pinpointed by EA and LW communities but I would like to focus on something else. The mental burden of knowing. There are also the obvious objections. Who cares if it’s mentally tough or not. Would it matter if concentration camp inmates sentenced to death could cope with their reality or should we just try to avoid it happening. On the other hand, cancer diagnosis victims can and do get mental help as a default.

That’s precisely the two situations where we put ourselves in right now but with a twist. There was never a scenario where your death would mean also everyone else’s and that is another philosophical question with apparently no answer. Does the fact that all humanity will go with you change the way you feel about it?

In any case, lately I have been feeling like I got a cancer diagnosis along with all the rest of us. It’s not a certain death but there is a high probability of dying much earlier than what was predicted when you were born. On this subject I also have another dichotomy, I think there is no physical death from natural causes. We either all live forever or we all die from terminators. No in-between.

I want to explain my thought process right now and hope to hear others expand on the topic, a form of collective psychotherapy. I have been seeing the topic more and more on Twitter from people I follow breaking for some short time but then the human algo of nature takes over and brings chemical balance in the brain. That is most certainly because I don’t follow many people with mental health issues. That is far from society norms though.

What would it look like for society to realize the risk we are running right now en masse? I am sure many of you have watched Di Caprio’s film about the professor yelling that we are all going to die (Eliezer?) and people dismiss it until they don’t. Will society break down?

In the meantime, how do you live your life? If we follow...
Interpreting Embedding Spaces by
Conceptualization
by Adi Simhi — 2023-02-28T18:38:39.850Z
Link: https://arxiv.org/abs/2209.00445

In this paper, we present a novel method of understanding embeddings by transforming embedding space into a comprehensible conceptual space. We present an algorithm for deriving a conceptual space with dynamic on-demand granularity. We also show a method for transferring any vector in the original incomprehensible space to an understandable vector in the conceptual space. We combine human tests with cross-model tests to show that the conceptualized vectors indeed represent the semantics of the original vectors. We also show the use of our method for various tasks, including comparing the semantics of alternative models.

The method works as follows:

We have thus defined a meta-algorithm CES (Conceptualizing Embedding Spaces) that, for any given embedding method f, a set of concepts C and a mapping function τ from concepts to text, takes a vector in the latent space L and returns a vector in the conceptual space C.

In figure 1 we can see the outline of the method.

The conceptual set C comes from a hierarchical ontology, thus allowing a conceptual representation in various levels of abstraction.

In our work, we also introduce a technique for creating a conceptual space that can be adjusted in granularity as needed. This allows for a more precise and adaptable conceptual space to be generated based on a given text. By implementing this approach, we are able to produce a conceptual space with varying degrees of abstraction on different topics.

To demonstrate the effectiveness of our method, we provide examples that showcase its comprehensibility and intuitiveness. Specifically, we utilize a sentence embedding model known as SRoBERTa as our f. The outcomes of our experiments are presented in the next table, where we show the top concepts for each sentence.

In our paper, we conduct a variety of tests to demonstrate that CES effectively captures the f embedding. These tests encompass a range of experiments, including human evaluations and cross-model comparisons.

Furthermore, we present several practical applications of CES, such as a method for comparing different models and generating visual representations of text. Additionally, we explore the potential of applying CES to the layers of LLM models, which can provide valuable insights into the evolution of the embedding space across different layers.

Below, we provide an illustration that demonstrates CES on the...
A mostly critical review of infra-Bayesianism
by David Matolcsi — 2023-02-28T18:37:58.448Z

Introduction

I wrote this post towards the end of my three and a half months long SERI MATS fellowship. I didn't get even close to the point where I could say that I understand infra-Bayesianism on a really detailed level (according to Vanessa there are only three people in the world who fully understand the infra-Bayesian sequence). Still, I spent three months reading and thinking about infra-Bayesianism, so I ought to be able to say something useful to newcomers.

The imaginary audience of this post is myself half a year ago, when I was just thinking about applying to Vanessa's mentorship but knew almost nothing about infra-Bayesianism or the the general research direction it fits into. The non-imaginary intended audience is people who are in a similar situation now, just considering whether they should dive into infra-Bayesianism.

My review is mostly critical of the infra-Bayesian approach, and my main advice is that if you decide that you are interested in the sort of questions infra-Bayesianism tries to solve, then it's more useful to try it yourself first in your own way, instead of starting with spending months getting bogged down it the details of Basic infra-measure theory that might or might not lead closer to solutions. Still, I want to make it clear that my criticism is not aimed at Vanessa herself, as she chose questions that she found important, then created a theory that made some progress towards answering those questions. I have somewhat different intuitions than Vanessa over how important are certain questions and how promising are certain research directions, but I support her continuing her work and I thank her for answering my annoying questions throughout the three months.

Personal meta-note

I applied to the infra-Bayesian stream in SERI MATS because I have a pure mathematics background, so I figured that this is the alignment agenda that is closest to my area of expertise. I met some other people too, also with pure math background, who get convinced that alignment is important, then start spending their time on understanding infra-Bayesianism, because it's the most mathematical alignment proposal.

Although paying attention to our comparative advantages is important, in retrospect I don't believe this is a very good way to select research topics. I feel that I was like the man who only has a hammer and is desperately looking for n...
Performance guarantees in classical learning theory and infra-Bayesianism
by David Matolcsi — 2023-02-28T18:37:57.231Z

Introduction

I wrote this post during my SERI MATS scholarship, explaining the part of infra-Bayesianism that I understood the most deeply. Compared to some previous distillations, I don't go deeply into the mathematical machinery built around infra-Bayesianism, instead I try to evaluate what results infra-Bayesianism produced compared to classical learning theory in the topic of performance guarantees in general environments, which was the main motivation behind the development of infra-Bayesianism.

The theorems appearing in the post are not original to me, and can be found in either the broader academic literature (concerning classical learning theory) or Vanessa's writings (concerning infra-Bayesianism). I included proofs in the text when I found them useful for general understanding, and moved them to the Appendix when I thought them simple and important enough to write down, but not interesting enough to break the flow of text with them. The later examples investigating the limits of infra-Bayesian learning are my own.

My conclusion is that the general performance guarantees we can currently prove about infra-Bayesian learning agents are pretty weak, but it seems plausible that the theory gets us closer to tackling the question.

For my general thoughts and doubts about infra-Bayesianism, see my other, less technical post: A mostly critical review of infra-Bayesianism.

Classical reinforcement learning

Before we go into the infra-Bayesian explainer, or before we could even really state what the motivating problem of non-realizability is, it's important to get at least a superficial overview of classical reinforcement learning theory that is studied in academia since a long time. My main source for this was the book Bandit Algorithms by Lattimore and Szepesvári.

In a simple model of reinforcement learning, the agent is placed in an environment, and on every turn, the agent chooses an action from its finite action set A, and after that, the environment gives an observation from the finite observation set O. There is a bounded loss function ℓ that is known to the agent in advance, assigning real-valued losses to finite histories. Usually, we will work with ℓ:A×O→R loss functions. This choice is motivated by game theory, where the learning process would be a repeated game, and in each turn, the loss is determined by the agent's ac...