trentmkelly/LessWrong-43k huggingface dataset:
Causality: A Brief Introduction
by tom4everitt — 2023-06-20T15:01:39.377Z

Post 2 of

Towards Causal Foundations of Safe AGI

,

see also Post 1 Introduction.

By Lewis Hammond, Tom Everitt, Jon Richens, Francis Rhys Ward, Ryan Carey, Sebastian Benthall, and James Fox, representing the Causal Incentives Working Group. Thanks also to Alexis Bellot, Toby Shevlane, and Aliya Ahmad.

Causal models are the foundations of our work. In this post, we provide a succinct but accessible explanation of causal models that can handle interventions, counterfactuals, and agents, which will be the building blocks of future posts in the sequence. Basic familiarity with (conditional) probabilities will be assumed.

What is causality?

What does it mean for the rain to cause the grass to become green?

Causality

is a philosophically intriguing topic that underlies many other concepts of human importance. In particular, many concepts relevant to safe AGI, like influence, response, agency, intent, fairness, harm, and manipulation, cannot be grasped without a causal model of the world, as we mentioned in

the intro post

and will discuss further in subsequent posts.

We follow

Pearl

and adopt an

interventionist

definition of causality: the sprinkler today causally influences the greenness of the grass tomorrow, because if someone intervened and turned on the sprinkler, then the greenness of the grass would be different. In contrast, making the grass green tomorrow has no effect on the sprinkler today (assuming no one predicts the intervention). So the sprinkler today causally influences the grass tomorrow, but not vice versa, as we would intuitively expect.

Interventions

Causal Bayesian Networks

(CBNs) represent causal dependencies between aspects of reality using a

directed acyclic graph

. An arrow from a variable A to a variable B means that A influences B under some fixed setting of the other variables. For example, we draw an arrow from sprinkler (S) to grass greenness (G):

A causal graph representing our running example. The sprinkler (S) influences the greenness of the grass (G).

For each node in the graph, a causal mechanism of how the node is influenced by its parents is specified with a

conditional probability distribution

. For the sprinkler, a distribution p(S) specifies how commonly it is turned on, e.g. P(S=on)=30%. For the grass, a conditional distribution p(G∣S) specifies how likely ...
Lightning Post: Things people in AI Safety should stop talking about
by Prometheus — 2023-06-20T15:00:24.479Z

This is experimenting with a new kind of post which is meant to convey a lot of ideas very quickly, without going into much detail for each.

Things I wish people in AI Safety would stop talking about

A list of topics people concerned about x-risk from AI spend, in my opinion, way too much time talking about to those outside the community. It's not that these things aren't real, they just likely won't actually end up mattering that much.

How an AI could persuade you to let it out of the box

WRONG!

Keeping AIs in boxes was never a thing companies were seriously going to do. An AI in a box isn’t useful. These aren’t academics carefully studying a new species. This is an industry, with everyone trying to get ahead, get consumer feedback, and training on parallel cloud computing.

How an AI could become an agent

WRONG!

Agency is the obvious next step that people will try to make their AIs into. An AI “tool” is simply inferior in almost every possible way to an agent. You don’t need to specify a special prompt, constantly click “Approve Plan”, or any of the supervised, time-consuming requirements from mere tools. The economic advantage is simply too staggering for people not to do this. Everyone can even agree that it’s dangerous, a totally bad idea, and still have to do it anyway if they want to (economically) survive.

How an AI could get ahold of, or create, weapons

WRONG!

The military advantages of fully-autonomous weapons is just too great for any large-scale government not to do, especially for democracies where losing troops abroad results in massive political backlash. Humans using a remote controller is just too slow a process, because it still means humans have to make very fast, split-second decisions. Fully autonomous warfare would result in tactical decisions occurring faster than any possible human could perform. Look at how Alpha Zero played millions of games against itself in 70 hours. AIs can make decisions faster, which is all that will matter.

How an AI might Recursively Self Improve without humans noticing

WRONG!

RSI is an ace in the hole for any company or government. They will try to do this. As AI continues to expand faster, and the stakes get higher, the paranoia that someone else will achieve it first will drive players to compete to create RSI. It’s the gift that keeps on giving. You don’t just get momentarily ahea...
Having a headache and not having a headache
by Jim Pivarski — 2023-06-20T14:59:33.615Z

Yesterday at 9am, I had a headache. I tried to put it out of mind and focus on other things, and I guess I was successful because the next time I thought about it, it was 10am and I didn't have a headache. (I'm simplifying and making up times that are approximately right.)

So it went away, but when did it go away? Did it end at 9:10am and I just didn't notice until 50 minutes later, or did it end at 9:50am? In physics-speak, it was an unobserved variable between the two measurements.

Taking the word "head-ache" literally, I didn't have a headache at all from 9:01am through 9:59am because I wasn't noticing it, not perceiving it, and therefore not feeling it. A "pain" that you don't feel is not pain. How can it make sense to say, "This hurts but I can't feel it"?

If I noticed I had a headache at 8am and I noticed I had a headache at 9am and the physical mechanism causing it didn't change over that interval, but I wasn't noticing it, did I really have a headache from 8:01am through 8:59am?

Suppose that the physical mechanism can be detected by an fMRI machine, and I was being monitored all morning. A neuroscientist could point to an angry blob on the computer screen and say, "There's your headache. It definitely existed from 7:30am until 9:30am, when the aspirin kicked in. Whether you were aware of it or paying attention to it is another matter."

The headache that the neuroscientist is talking about is a different thing from the "head-ache" that I'm talking about. In fact, all of our words about sense perceptions and thoughts are doubled like this: my sense of taste is one thing, chemical interactions on my tongue are another, my feeling of excitement is one thing, adrenaline in my blood is another, and my feeling that I understand a mathematical concept is distinct from my ability to answer questions about it[1]. Sure, they're highly correlated, but that doesn't make them identical. If your measurements of all the objective criteria disagree with my assertions of my subjective experiences, you might wonder if I'm lying or don't understand the common usage of the words, but they're not 100% guaranteed to go hand-in-hand.

This is an elaboration on Reality #5 in my previous essay on Reality and reality-boxes. My point is that the physical reality of chemical interactions on my tongue, adrenaline in my blood, and my quiz results on a mathematical concept are not all the...
Never Fight The Last War
by ChristianKl — 2023-06-20T12:35:26.038Z

Lieutenant Colonel J. L. Schley, of the Corps of Engineers wrote:

“There is a tendency in many armies to spend the peace time studying how to fight the last war”

After the Chinchilla paper, everyone got focused on the idea that large language models can be more effective.

The GPT3.5 we saw in ChatGPT was scary and got people to think about how we can prevent bigger models from being trained. There was a lot of capability gain when OpenAI released GPT4 but if we can trust Sam Altman, most of it was not because of an increase in model size but by doing a lot of different improvements.

From that perspective, many people think about whether or not GPT5 or GPT6 will be scary and dangerous by improving along the same axis as the GPT3.5 ->GPT4 improvement.

Right now, it seems that most of the danger is not in model improvement but in model utilization.

Palantir argues that models are commodities and released an AI product without providing any models that go beyond the existing open-source models. The value that Palantir does provide is that it provides interfaces of how a model can interact with the existing data a company has and how to create multi-step chains of data transformation. It provides a system to manage agents that use models to make decisions.

I read some people argue that Palantir's product is worse than GPT4 and thus should not be seen as an advance in AI, but that's flawed thinking. Autonomous agents are more dangerous than anything you can do in ChatGPT.
Why didn't virologists run the studies necessary to determine which viruses are airborne?
by ChristianKl — 2023-06-20T11:58:30.756Z

If virology would be a reasonable scientific field I would expect them to care about the road viruses use to infect humans and other animals.

The experiment to determine whether or not a virus is airborne seems pretty straightforward to me. Get a bunch of animals and house them in a way where they would transmit the virus to each other if and only if the virus is airborne.

One hypothesis would be that virologists don't really care about understanding transmissions but care about registering patents from which they profit when drugs based on the patents get developed.

People in the field of environmental health seemed to be able to know that COVID was airborne and wrote letters to the WHO saying so only to be ignored.

What's wrong with the virologists that they didn't manage to do the necessary science?
A Friendly Face (Another Failure Story)
by Karl von Wendt — 2023-06-20T10:31:24.655Z

Edit: Based on the comment by Daniel Kokotajlo, we extended the dialog in the chapter "Takeover from within" by a few lines.

The perfect virtual assistant

The year is 2026 and the race for human-level artificial general intelligence (AGI) draws to a close. One of the leading AI companies, MegaAI, committed the last year and a half to training a new large language model (LLM). They employ advanced algorithms that use the available compute more efficiently than earlier models. A comprehensive range of tests establish that the model surpasses the average human in all conventionally accepted intelligence benchmarks, and exceeds expert level in most of them.

In contrast to earlier LLMs, the new AI is not designed to be a mere question-answering tool. Under mounting pressure from the open-source community and their efforts to develop an agentic AGI capable of acting in the real world, MegaAI decides to imbue their new model with a specific purpose: to provide universal, helpful assistance that improves the quality and ease of life for all. They name this assistant "Friendlyface".

To improve upon the assistant's functionality, they endow it with limited agentic capabilities. Friendlyface has a complex, detailed world model, can make long-term plans, and has access to certain tools that enable it to achieve objectives in the real world. For example, it can write messages and book flights, but will reliably and consistently ask the user to confirm before executing an action. It can write programs for nearly any purpose imaginable with superhuman ingenuity, but is prohibited from executing them volitionally. Unlike previous generations of LLMs, it is multimodal, communicating with users in text and spoken language, accepting pictures and videos as input, and interacting directly with smart devices and the “internet of things”. The users may also customize Friendlyface's appearance and personality to their liking.

Most importantly, Friendlyface is designed to assume the role of a personalized smart advisor. In a world where users are regularly inundated with a torrent of irrelevant or false information, it is able to successfully discern and present what is important to the user while filtering out fake news, preventing spear phishing attempts, and more. Beyond merely answering questions, it proactively offers users advice on advancing their careers, improving their relatio...
DSLT 3. Neural Networks are Singular
by Liam Carroll — 2023-06-20T08:20:55.516Z

TLDR; This is the third main post of Distilling Singular Learning Theory which is introduced in DSLT0. I explain that neural networks are singular models because of the symmetries in parameter space that produce the same function, and introduce a toy two layer ReLU neural network setup where these symmetries can be perfectly classified. I provide motivating examples of each kind of symmetry, with particular emphasis on the non-generic node-degeneracy and orientation-reversing symmetries that give rise to interesting phases to be studied in DSLT4.

As we discussed in DSLT2, singular models have the capacity to generalise well because the effective dimension of a singular model, as measured by the RLCT, can be less than half the dimension of parameter space. With this in mind, it should be no surprise that neural networks are indeed singular models, but up until this point we have not exactly explained what feature they possess that makes them singular. In this post, we will explain that in essence:

Neural networks are singular because there are often ways to vary their parameters without changing the function they compute.

In the case where the model and truth are both defined by similar neural network architectures, this fact means that the set of true parameters W0 is non-trivial (i.e. bigger than the regular case where it is a single point), and often possesses many symmetries. This directly implies that neural networks are singular models.

The primary purpose of this post is to show with examples why neural networks are singular, and classify the set of true parameters W0 in the case where the model and truth are simple two layer feedforward ReLU networks. In doing so, we will lay the groundwork for understanding the phases present in the setup so that we can then study relevant phase transitions in DSLT4. Feel free to jump ahead to the slightly more exciting DSLT4 Phase Transitions in Neural Networks and refer back to this post as needed.

Outline of Classification

To understand the different regions that minimise the free energy (and thus, as we'll see in DSLT4, the phases), one needs to first understand the singularities in the set of optimal parameters of K(w).

In the realisable regression case with a model neural network f(x,w) and true neural network defined by f0(x)=f(x,w(0)) for som...
10 quick takes about AGI
by Max H — 2023-06-20T02:22:08.087Z

I have a bunch of loosely related and not fully fleshed out ideas for future posts.

In the spirit of 10 reasons why lists of 10 reasons might be a winning strategy, I've written some of them up as a list of facts / claims / predictions / takes. (Some of the explanations aren't exactly "quick", but you can just read the bold and move on if you find it uninteresting or unsurprising.)

If there's interest, I might turn some of them into their own posts or expand on them in the comments here.

Computational complexity theory does not say anything practical about the bounds on AI (or human) capabilities. Results from computational complexity theory are mainly facts about the limiting behavior of deterministic, fully general solutions to parameterized problems. For example, if a problem is NP-hard (and P≠NP), that implies[1] that there is no deterministic algorithm anyone (even a superintelligence) can run, which accepts arbitrary instances of the problem and finds a solution in time steps polynomial in the size of the problem. But that doesn't mean that any particular, non-parameterized instance of the problem cannot be solved some other way, e.g. by exploiting a regularity in the particular instance, using a heuristic or approximation or probabilistic solution, or that a human or AI can find a way of sidestepping the need to solve the problem entirely.
Claims like "ideal utility maximisation is computationally intractable" or "If just one step in this plan is incomputable, the whole plan is as well." are thus somewhat misleading, or at least missing a step in their reasoning about why such claims are relevant as a bound on human or AI capabilities. My own suspicion is that when one attempts to repair these claims by making them more precise, it becomes clear that results from computational complexity theory are mostly irrelevant.
From here on, capabilities research won't fizzle out (no more AI winters). I predict that the main bottleneck on AI capabilities progress going forward will be researcher time to think up, design, implement, and run experiments. In the recent past, the compute and raw scale of AI systems was simply too little for many potential algorithmic innovations to work at all. Now that we're past that point, some non-zero fraction of new ideas that smart researchers think up and spend the time to test will "just work" at least somewhat, and th

...
OpenAI introduces function calling for GPT-4
by mic — 2023-06-20T01:58:47.881Z
Link: https://openai.com/blog/function-calling-and-other-api-updates

OpenAI’s latest

update

unlocks function calling for GPT-4 and GPT-3.5 Turbo. This enables developers to "describe functions to gpt-4-0613 and gpt-3.5-turbo-0613, and have the model intelligently choose to output a JSON object containing arguments to call those functions." This might seem like just another technical update, but it's more like handing a magic wand to AI.

We’re entering a world where large language models (LLMs) can not only chat with users but take actions in pursuit of goals. With function calling, developers can create competent AI systems that perform tasks by integrating external tools. These tools enable broad capabilities, from sending email to facilitating cyberattacks or

synthesizing toxic chemicals in a lab

.

This newfound power doesn’t come without its risks. Margaret Mitchell, most known for her work on automatically removing demographic biases from AI models, voices concerns about large language models (LLMs) taking actions autonomously. According to

Mitchell

, enabling LLMs to perform actions in the real world could edge us towards extinction-level events. Yoshua Bengio has argued for a “policy banning powerful autonomous AI systems that can act in the world […] unless proven safe.”

We’ve seen concerns raised earlier with the release of

ChatGPT plugins

in March 2023. Red teamers found that they could “send fraudulent or spam emails, bypass safety restrictions, or misuse information sent to the plugin,” and the computational linguist Emily Bender

cautioned

, “Letting automated systems take action in the world is a choice that we make.” Even so, the release of ChatGPT plugins claimed to have “safety as a core principle” through the following requirements:

User confirmation is needed before taking actions (specifically, making POST requests)
Plugins are reviewed before being made accessible through the plugin store
ChatGPT only calls plugins in response to user prompts, rather than continually making observations, plans and taking actions
Web browsing provides read-only access to the web, and code execution is sandboxed
Rollout was slow and gradual

None of these safeguards exist with function calling, through which GPT-4 can take a sequence of actions without humans in the loop. Developers can connect it with their own tools, with no oversight regarding whether such tools are used safely. Overnight, on June 27, fun...
Approaches to Thump
by jefftk — 2023-06-20T01:50:01.006Z

In playing for contra dances there are logistical and economic
pressures that push for

smaller
bands. While the

older larger
bands would often include drumkits, if you're thinking about how
to allocate roles among three—or especially two—people
you're

usually (but

not always) going to do
without a drummer. Even without a drummer, though, bands often want
something in the general direction of a kick drum to provide a thumpy
beat on sets that need it. What are the approaches you see?

The most straightforward is for one of the musicians to stamp their
foot on the stage or a board, and amplify that. This is best suited
for musicians who are interested in more than just stamping out out a
downbeat: it doesn't have a huge amount of low-end punch but in
exchange the musician can make complex and varied
rhythms using both their feet. A French Canadian "one,
two, and" beat is a common place to start, but it can be taken
much farther. Here's Noah VanNorstrand in
Buddy System:

John Cote in Perpetual
e-Motion":

Matt Turino in the Mean Lids:

If you want something simple, cheap, and portable, with a consistent
thump, analog piezo boxes (ex)
can work well. Here's Alex
Cumming playing one with Carol Bittenson:

And Claude Ginsberg with KGB:

A more flexible option is the Roland SPD::ONE
Kick Pad, which is an electronic instrument. Instead of capturing and
filtering the vibrations induced by your stomps it measures how hard
you're tapping and uses that to trigger drum sounds. This means you
can choose exactly what it sounds like, including a traditional bass
drum kick. Here's Guillaume Sparrow-Pepin playing one with Nova, adding some thump to
Everest's higher-pitched but
more expressive acoustic feet:

Instead of an all-in-one pad you can also combine the brain (sound
synthesis module) from an electronic drum kit with a trigger,
including homemade ones. If you get the
brain used (which isn't hard—a lot of people get electronic
drum kits and then abandon them) this is a lot cheaper than a Pad at
the cost of carting around more gear. It's especially a lot cheaper
if you want multiple pedals for different sounds, because the brain is
the main cost. Here's Guillaume again:

Instead of making a trigger you can also buy a beaterless kick pedal
like the Yamaha
KU-100. This is what I use these days, and is a g...