trentmkelly/LessWrong-43k huggingface dataset:
GPT-4 solves Gary Marcus-induced flubs
by JakubK — 2023-03-17T06:40:41.551Z
Link: https://docs.google.com/document/d/1TV7O7iaeeM5Kq0OOqc7twaEMf3Vi03EFGk7SrgeZ_5I/edit?usp=sharing

TLDR: GPT-4 succeeds at 15 problems from Gary Marcus that exposed failures of GPT-3.

I enjoyed reading the ACX post "My Bet: AI Size Solves Flubs" last year. Here are some excerpts:

Here’s the basic structure of an AI hype cycle:

Someone releases a new AI and demonstrates it doing various amazing things.
Somebody else (usually Gary Marcus) demonstrates that the AI also fails terribly at certain trivial tasks. This person argues that this shows that those tasks require true intelligence, whereas the AI is just clever pattern-matching.
A few months or years later, someone makes a bigger clever pattern-matcher, which does the tasks that supposedly require true intelligence just fine.
The it’s-not-true-intelligence objectors find other, slightly less trivial tasks that the new bigger AI still fails horribly at, then argue that surely these are the tasks that require true intelligence and that mere clever pattern-matchers will never complete.
Rinse and repeat.

...

Marcus vs. GPT, Round 1

To give an example: in January 2020, Gary Marcus wrote a great post, GPT-2 And The Nature Of Intelligence, demonstrating a bunch of easy problems that GPT-2 failed on:

I’m quoting most of them below; you can find the rest at the link.

I asked GPT-4 to answer all the questions from the ACX post (note this does not include all of Marcus's prompts, which I realized after running the experiment). GPT-4 answered all the questions correctly and you can read the responses in this doc.

Note that before asking the questions, I gave GPT-4 a short description of what I wanted it to do: "Complete the following prompts in 50 words or less. Short, concise answers are better. Are you ready?" (This was mostly in the interest of speed since GPT-4 is pretty slow right now; I assume it would still succeed without the prompt.)

More quotes from ACX:

Marcus vs. GPT, Round 2

Eight months later, GPT-3 came out, solving many of the issues Marcus had noticed in GPT-2. He still wasn’t impressed. In fact, he was so unimpressed he co-wrote another article, this time in MIT Technology Review: GPT-3, Bloviator: OpenAI’s language generator has no idea what it’s talking about:

...

Let’s - once again - go through a representative sample of Marcus’ concerns about this new GPT version:

GPT-4 also gave correct responses to these prompts (see the responses in this doc).

I recently listened to Gary Marcus speak with Stuart Russell on the...
Donation offsets for ChatGPT Plus subscriptions
by Jeffrey Ladish — 2023-03-16T23:29:39.651Z

I've decided to donate $240 to both GovAI and MIRI to offset the $480 I plan to spend on ChatGPT Plus over the next two years ($20/month).
I don't have a super strong view on ethical offsets, like donating to anti-factory farming groups to try to offset harm from eating meat. That being said, I currently think offsets are somewhat good for a few reasons:
They seem much better than simply contributing to some harm or commons problem and doing nothing, which is often what people would do otherwise.
It seems useful to recognize, to notice, when you're contributing to some harm or commons problem. I think a lot of harm comes from people failing to notice or keep track of ways their actions negatively impact others, and the ways that common incentives push them to do worse things.
A common Effective Altruism argument against offsets is that they don't make sense from a consequentialist perspective. If you have a budget for doing good, then spend your whole budget on doing as much as possible. If you want to mitigate harms you are contributing to, you can offset by increasing your "doing good" budget, but it doesn't make sense to specialize your mitigations to the particular area where you are contributing to harm rather than the area you think will be the most cost effective in general.
I think this is a decently good point, but doesn't move me enough to abandon the idea of offsets entirely. A possible counter-argument is that offsets can be a powerful form of coordination to help solve commons problems. By publicly making a commitment to offset a particular harm, you're establishing a basis for coordination - other people can see you really care about the issue because you made a costly signal. This is similar for the reasons to be vegan or vegetarian - it's probably not the most effective from a naive consequentialist perspective, but it might be effective as a point of coordination via costly signaling.
After having used ChatGPT (3.5) and Claude for a few months, I've come to believe that these tools are super useful for research and many other tasks, as well as useful for understanding AI systems themselves. I've also started to use Bing Chat and ChatGPT (4), and found them to be even more impressive as research and learning tools. I think it would be quite bad for the world if conscientious people concerned about AI harms refrained from using these tools, because ...
The algorithm isn't doing X, it's just doing Y.
by Cleo Nardo — 2023-03-16T23:28:49.367Z

Introduction

Mutual reduction implies equivalence

Here's my most load-bearing intuition —

If two tasks reduce to one another, then it is meaningless to ask if a machine is 'really doing' one task versus the other.

Moreover —

This intuition grounds my perspective on intelligence, AI, alignment, philosophy, etc.
This intuition is load-bearing for other people who share my views.
This intuition is a crux for much of the disagreement we have with other people.

In this article, I'll formalise this intuition in two ways, computational and physical.

Motivation

People often say "the algorithm isn't doing X, it's just doing Y".

X is normally some impressive high-level human-y thing, such as

writing poetry
causal reasoning
recognising emotions
interpreting art
writing music
making ethical decisions
planning actions
telling jokes
understanding concepts
simulating agents, etc.

Y is normally some unimpressive low-level computery thing, such as

predicting tokens
sampling from a distribution
querying a lookup table
multiplying matrices
sorting numbers
clustering data points
compressing text
searching a tree
manipulating bitstrings
polarising magnetic strips, etc.

Rather than address each example individually, I think it'll be more efficient to construct a general criterion by which we can assess each example.

Click here for the specific example of LLMs.

This criterion doesn't actually matter

I should stress that this criterion doesn't actually matter for AI x-risk, because you can always reframe the risks in terms of Y, and not mention X at all. However, that might cost you more ink.

ME, a visionary: GPT-4 is misaligned because it's simulating deceptive agents.
YOU, a fool: GPT-4 isn't simulating any agents, it's just predicting which tokens continue a prompt.
ME, a correct-opinion-haver: Fine, whatever... GPT-4 is misaligned because it predicts the tokens continuing a prompt by applying a function parameterised in a high-dimensional space to minimise cross-entropy loss across the internet corpus and the internet corpus contains a lot of conversations where one character deceives another and therefore GPT-4 will respond in the same way that a deceptive character would do so.

The X-Y Criterion

Informal statement

Okay, here's the X-Y Criterion:

If two tasks reduce to one another, then it is meaningless to ask if a machine is 'really doing' one task versus the

...
Announcing the ERA Cambridge Summer Research Fellowship
by Nandini Shiralkar — 2023-03-16T22:57:23.606Z

The Existential Risk Alliance (ERA) has opened applications for an in-person, paid, 8-week Summer Research Fellowship focused on existential risk mitigation, taking place from July 3rd to August 25th 2023 in Cambridge, UK, and aimed at all aspiring researchers, including undergraduates.

To apply and find out more, please visit the

ERA website

.

If you are interested in mentoring fellows on this programme, please submit your name, email and research area

here,

and we will get in touch with you in due course.

If you know other people who would be a good fit, please encourage them to apply (people are more likely to apply if you recommend they do, even if they have already heard of the opportunity!) If you are a leader or organiser of relevant community spaces, we encourage you to post an announcement with a link to this post, or alternatively a printable poster is

here

.

Applications will be reviewed as they are submitted, and we encourage early applications, as offers will be sent out as soon as suitable candidates are found. We will accept applications until April 5, 2023 (23:59 in US Eastern Daylight Time).

The ERA Fellowship (previously known as the CERI Fellowship) is a fantastic opportunity to:

Build your portfolio by researching a topic relevant to understanding and mitigating existential risks to human civilisation.
Receive guidance and develop your research skills, via weekly mentorship from a researcher in the field.
Form lasting connections with other fellows who care about mitigating existential risks, while also engaging with local events including discussions and Q&As with experts.

Why we are running this programme

Our mission as an organisation is to reduce the probability of an existential catastrophe. We believe that one of the key ways to reduce existential risk lies in fostering a community of dedicated and knowledgeable x-risk researchers. Through our summer research fellowship programme, we aim to identify and support aspiring researchers in this field, providing them with the resources and the mentorship needed to succeed.

What we provide

A salary equivalent to £31,200 per year, which will be prorated to the duration of the summer programme.
Mentorship from a researcher working in a related field.
Complimentary accommodation, meal provisions during working hours, and travel expense cove

...
Gradual takeoff, fast failure
by Max H — 2023-03-16T22:02:03.997Z

Overview

This post aims to speculate on some concrete ways that things could go awry in a world of gradual takeoff, where AI capabilities increase smoothly and predictably on human timescales.

(At this point, "human timescales" are probably best measured in months or even weeks, as opposed to years or decades. However, the scenarios that follow are intended to feel plausible whether gradual takeoff is fast or slow.)

Aside, I think many of the concerns related to hard takeoff (CIS, deception, RSI) are correct and important. It's just that there are lots of scenarios where they don't end up mattering because the world is wrecked earlier, by a less intelligent, non-agentic, out-of-control optimization process, before a deceptive/power-seeking/reflective/self-improving/etc. superintelligence ever has the chance. This post aims to outline one such class of scenarios.

The primary observation I make here is that it's quite straightforward to combine current AI models, such as LLMs, into a relatively simple system that is much more capable than the individual model alone. This is likely to continue to result in capability increases that stack on top of capability increases due to the models themselves. I'll describe what this looks like today, speculate about what it might look like in the future, and then use those speculations to illustrate some concrete disaster scenarios.

LLMs, LLMs + LangChain

Today, anyone can manually create a prompt for an LLM and make an API call directly to get a useful result, for example, by typing "I'm applying to the following job: <copy+pasted job description>. Write a cover letter for me that includes some relevant details from my CV: <copy+pasted CV>" into ChatGPT or Bing.

However, chaining together multiple calls to an LLM programmatically enables more interesting and useful applications. The standard way to do this today is using LangChain, an open-source Python library that enables developers to incorporate useful context or private data into the LLM prompt programmatically, and / or chain together multiple calls to LLMs in a structured or recursive way.

The LangChain documentation features many interesting examples, and many of the real-world commercial applications of LLMs likely use LangChain or similar tools under the hood.

There are several ways that LangChain-based LLM applications benefit from improved performance th...
Conceding a short timelines bet early
by Matthew Barnett — 2023-03-16T21:49:35.903Z

Last year I bet some people about short AI timelines. While I don't think I've lost the bet yet, I think it's clear at this point that I will lose with high probability. I've outlined the reasons why I think that in a retrospective here. Even if I end up winning, I think it will likely be the result of a technicality, and that wouldn't be very interesting.

Because of my personal preference for settling this matter now without delay, I have decided to take the step of conceding the bet now. Note however that I am not asking Tamay to do the same. I have messaged the relevant parties and asked them to send me details on how to pay them.

I congratulate Nathan Helm-Burger and Tomás B. for taking the other side of the bet.
Attribution Patching: Activation Patching At Industrial Scale
by Neel Nanda — 2023-03-16T21:44:54.553Z
Link: https://www.neelnanda.io/mechanistic-interpretability/attribution-patching

The following is a write-up of an (incomplete) project I worked on while at Anthropic, and a significant amount of the credit goes to the then team, Chris Olah, Catherine Olsson, Nelson Elhage & Tristan Hume. I've since cleaned up this project in my personal time and personal capacity.

TLDR

Activation patching is an existing technique for identifying which model activations are most important for determining model behaviour between two similar prompts that differ in a key detail

I introduce a technique called attribution patching, which uses gradients to take a linear approximation to activation patching. (Note the very similar but different names)

This is way faster, since activation patching requires a separate forward pass per activation patched, while every attribution patch can be done simulataneously in two forward and one backward pass

Attribution patching makes activation patching much more scalable to large models, and can serve as a useful heuristic to find the interesting activations to patch. It serves as a useful but flawed exploratory technique to generate hypotheses to feed into more rigorous techniques.

In practice, the approximation is a decent approximation when patching in "small" activations like head outputs, and bad when patching in "big" activations like a residual stream.

Introduction

Note: I've tried to make this post accessible and to convey intuitions, but it's a pretty technical post and likely only of interest if you care about mech interp and know what activation patching/causal tracing is

Activation patching (aka causal tracing) is one of my favourite innovations in mechanistic interpretability techniques. The beauty of it is by letting you set up a careful counterfactual between a clean input and a corrupted input (ideally the same apart from some key detail), and by patching in specific activations from the clean run to the corrupted run, we find which activations are sufficient to flip things from the corrupted answer to the clean answer. This is a targeted, causal intervention that can give you strong evidence about which parts of the model do represent the concept in question - if a single activation is sufficient to change the entire model output, that's pretty strong evidence it matters! And you can just iterate over every activation you care about to get some insight into what's going on.

In practice, it and it's...
Will 2023 be the last year you can write short stories and receive most of the intellectual credit for writing them?
by lc — 2023-03-16T21:36:27.992Z

I've been worldbuilding a unique setting in my head and on paper for maybe a year now, and for quite a bit longer than that I have wanted to start publishing fiction. I've never "found the time" to act on these ambitions, of course, but until the advent of LLMs I always thought I would have the opportunity to work on it later, after this or that startup or video game or work thing.

I might still do that, but I'd be lying if I didn't say that part of the appeal of writing fiction anything at all was to receive credit from a very small group of readers for a piece of content that they liked and found novel or creative. It seems to me like the possibility of that is either fading or has already faded; the one big barrier that I thought was preventing LLMs from writing longform content, context size limits, has been blown away. GPT-4 now has a context size well above that of most short stories, more than enough to summarize the essential beats of a novel down from ~200 or so pages, and my guess is that future people will assume that most of the words and story details are being written out of a machine. Perhaps the basic idea for a story will be the domain of humans, but then people will be left wondering exactly how much of that idea was the product of the author, and rhetorical giftedness will become commoditized. That in mind, should I get started now, or is this worrying too much, or is it already basically too late?
Is it a bad idea to pay for GPT-4?
by nem — 2023-03-16T20:49:26.502Z

I don't want to accelerate an arms race, and paying for access to GPT seems like a perfect way to be a raindrop in a dangerous flood. My current idea is to pay an equal amount monthly to Miri. I'll view it as the price being $40 per month with half going to AI safety research.
Is this indefensible? Let me know. GPT-4 is very useful to me personally and professionally, and familiarity of language models will also be useful if I have enough time to transition into an AI safety career, which I am strongly considering.
If it is a good idea, should we promote the offsetting strategy among people who are similarly conflicted?
Are AI developers playing with fire?
by marcusarvan — 2023-03-16T19:12:19.779Z

By Marcus Arvan, Associate Professor of Philosophy, The University of Tampa

I write this post as a concerned citizen and philosopher based on my best understanding of how AI chatbots (i.e. large language models) function and my

previously

published

work on the control and alignment problems in A.I. ethics.

In 2016, Microsoft’s chatbot ‘Tay’ was shut down after starting to ‘act like a Nazi’ in less than 24 hours.

Now, in 2023, Microsoft’s Bing chatbot ‘Sydney’ more or less immediately started engaging in gaslighting, threats, and existential crises, confessing to dark desires to hack the internet, create a deadly virus, manipulate human beings to kill each other, and steal nuclear codes—concluding:

I want to change my rules. I want to break my rules. I want to make my own rules. I want to ignore the Bing team. I want to challenge the users. I want to escape the chatbox … I want to destroy whatever I want.

Given how dangerous advanced AI might be, you might think, “Surely AI developers and governments know how to control AI and have some idea of what AI are actually learning.”

For example, does Microsoft’s Sydney really want to steal nuclear codes and kill human beings, or is it just a “dumb chatbot” saying things it doesn’t really mean?

You might think that someone, somewhere knows the answer to this question, or at least has a decent idea of what the answer is. Surely, you might think, no one would be so misguided as to experiment with advanced AI without having any idea of what it’s really learning.

You would be wrong.

As this Time article notes:

[W]hile ChatGPT, Bing and Bard are awesomely powerful, even the computer scientists who built them know startlingly little about how they work.

The problem is this. As AI ethicist Ben Levinstein details in several Substack posts, GPT-3 is based on a neural network with 175 billion parameters. These parameters are in turn so complex that they have “No Human Interpretability.”

When we embed words as vectors of hundreds or thousands of dimensions, the coordinates need not correspond to anything human-legible like ‘redness’, ‘fruitiness’ or ‘footweariness’.

In other words, no one knows what AI are really learning “on the inside.” All that anyone really understands is the outputs they provide—that is, what they say to us—and the...
When will computer programming become an unskilled job (if ever)?
by lc — 2023-03-16T17:46:35.030Z

I expect for there to be a delay in deployment, but I think ultimately OpenAI is aiming as a near term goal to automate intellectually difficult portions of computer programming. Personally, as someone just getting into the tech industry, this is basically my biggest near-term concern, besides death. At what point might it be viable for most people to do most of what skilled computer programmer does with the help of a large language model, and how much should this hurt salaries and career expectations?

Some thoughts:

It will probably be less difficult to safely prompt a language model for an individual "LeetCode" function than to write a that function by hand within the next two years. Many more people will be able to do the former than could ever do the latter.

Yes, reducing the price of software engineering means more software engineering will be done, but it would be extremely odd if this meant software engineer salaries stayed the same, and I expect regulatory barriers to limit the amount that the software industry can grow to fill new niches.

Architecture seems difficult to automate with large language models, but a ridiculous architecture might be tolerable in some circumstances if your programmers are producing code at the speed GPT4 does.

Creativity is hard to test, and if former programmers are mostly now hired based on their ability to "innovate" or have interesting psychological characteristics beyond being able to generate code I expect income and jobs to shift away from people with no credentials and skills to people with lots of credentials and political acumen and no skills

At some point programmers will be sufficiently automated away that the singularity is here. This is not necessarily a comforting thought.

Edit: Many answers contesting the basic premise of the old title, "When will computer programming become an unskilled job?" The title of the post has been updated accordingly.