trentmkelly/LessWrong-43k huggingface dataset:
Join the AI Testing Hackathon this Friday
by Esben Kran — 2022-12-12T14:24:30.286Z
Link: https://alignmentjam.com/post/join-the-ai-testing-hackathon

TLDR: Participate online or in-person on the weekend of December 16th to 18th in a fun and intense

AI safety research hackathon

focused on benchmarks, neural network verification, adversarial attacks and RL safety environments. We invite mid-career professionals to join, but the event is open to everyone (including non-coders) and we will provide starter code templates to help kickstart your team's projects.

Join here

.

Below is an FAQ-style summary of what you can expect.

What is it?

The AI Testing Hackathon is a weekend-long event where teams of 1-6 participants conduct research on AI safety. At the end of the event, teams will submit a PDF report summarizing and discussing their findings.

The hackathon will take place on Friday from December 16th to 18th, and you are welcome to join for any part of it (see further details below). An expert on the topic will be speaking and we will introduce the topic for you on the launch date.

Everyone can participate and we encourage you to join especially if you’re considering AI safety from another career . We prepare templates for you to start out your projects and you’ll be surprised what you can accomplish in just a weekend – especially with your new-found friends!

Read more about how to join, what you can expect, the schedule, and what previous participants have said about being part of the hackathon below.

Why AI testing?

AI safety testing is becoming increasingly important as governments require rigorous safety certifications. The deployment of the EU AI Act and the development of AI standards by NIST in the US will both necessitate such testing.

The use of large language models, such as ChatGPT, has emphasized the need for safety testing in modern AI systems. Adversarial attacks and neural Trojans have become more common, highlighting the importance of testing for robustness and viruses in neural networks to ensure the safe development and deployment of AI.

In addition, the rapid development of related fields, such as automatic verification of neural programs and differential privacy, offers promising research for provably safe AI systems.

There is relatively little existing literature from AI safety on AI safety metrics and testing, though anomaly detection in various forms is becoming more prevalent:

Beth Barnes

is currently working on testing alignment and has detailed a more specific

...
Side-channels: input versus output
by davidad — 2022-12-12T12:32:29.042Z

This is a brief post arguing that, although "side-channels are inevitable" is pretty good common advice, actually, you can prevent attackers inside a computation from learning about what's outside.

We can prevent a task-specific AI from learning any particular facts about, say, human psychology, virology, or biochemistry—if:

we are careful to only provide the training process with inputs that would be just as likely in, say, an alternate universe where AI was built by octopus minds made of organosilicon where atoms obey the Bohr model
we use relatively elementary sandboxing (no clock access, no networking APIs, no randomness, none of these sources of nondeterminism, error-correcting RAM, and that’s about it)

I don't think either of these happens by default and if you are in an AGI lab I suggest you advocate for either (or both if you can, but one at a time is good too).

Regarding item 1, self-play in Go is an example par excellence and this may be one reason why people tend to have a strong intuition that arbitrarily strong AlphaZero fails to kill you. An example that trades off more potential risk for engineering applicability would be inputs from a Newtonian physics simulator with semi-rigid objects of randomly generated shapes.

I think it is good to be aware that

this is a real tradeoff; we don't need to just assume that superintelligent models will be able to deduce whatever physical facts they want through pure logical induction
taking advantage of the argument here would require being careful about data

datasets like CommonCrawl, MassiveText, and YouTube-8M are, uh, not careful

Arguments

If an attacker on the outside of a computation is trying to manipulate events inside the computation, they get to use any and all physical mechanisms: they can open up the computer, stick some electrodes on the memory bus, and have at it.

If an attacker is inside a pure mathematical function and trying to manipulate events outside the computer, well, they can influence the output of the function (or even, potentially, whether the function is tractable to compute). If the output of the computation is propagated into human minds, this influence could have cascading effects that are hard to avoid or even notice.

If an attacker is trying to exfiltrate information from a computer, they get to use any and all physical mechanisms: they can run a very accurate clock, use a sensitive microp...
Take 9: No, RLHF/IDA/debate doesn't solve outer alignment.
by Charlie Steiner — 2022-12-12T11:51:42.758Z

As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day (ish) for 25 days. Or until I run out of hot takes. And now, time for the week of RLHF takes.

I see people say one of these surprisingly often.

Sometimes, it's because the speaker is fresh and full of optimism. They've recently learned that there's this "outer alignment" thing where humans are supposed to communicate what they want to an AI, and oh look, here are some methods that researchers use to communicate what they want to an AI. The speaker doesn't see any major obstacles, and they don't have a presumption that there are a bunch of obstacles they don't see.

Other times, they're fresh and full of optimism in a slightly more sophisticated way. They've thought about the problem a bit, and it seems like human values can't be that hard to pick out. Our uncertainty about human values is pretty much like our uncertainty about any other part of the world - so their thinking goes - and humans are fairly competent at figuring things out about the world, especially if we just have to check the work of AI tools. They don't see any major obstacles, and look, I'm not allowed to just keep saying that in an ominous tone of voice as if it's a knockdown argument, maybe there aren't any obstacles, right?

Here's an obstacle: RLHF/IDA/debate all incentivize promoting claims based on what the human finds most convincing and palatable, rather than on what's true. RLHF does whatever it learned makes you hit the "approve" button, even if that means deceiving you. Information-transfer in the depths of IDA is shaped by what humans will pass on, potentially amplified by what patterns are learned in training. And debate is just trying to hack the humans right from the start.

Optimizing for human approval wouldn't be a big deal if humans didn't make systematic mistakes, and weren't prone to finding certain lies more compelling than the truth. But we do, and we are, so that's a problem. Exhibit A, the last 5 years of politics - and no, the correct lesson to draw from politics is not "those other people make systematic mistakes and get suckered by palatable lies, but I'd never be like that." We can all be like that, which is why it's not safe to build a smart AI that has an incentive to do politics to you.

Generalized moral of the story: If something is an alignment solution e...
Creating a database for base rates
by nikos — 2022-12-12T10:09:15.778Z
Link: https://forum.effectivealtruism.org/posts/H7xWzvwvkyywDAEkL/creating-a-database-for-base-rates

TLDR

We are creating a database to collect base rates for various categories of events. You can find the database here and can suggest new base rate categories for us to look into here.

Project Summary

The base rate database project collects base rates for different categories of events and makes them available to researchers, forecasters and philanthropic organisations. Its main goals are to develop better intuitions about the potential and limitations of reference class forecasting and to provide useful information to the public. The data will enable research that enhances our understanding of the kinds of circumstances in which reference forecasting is a promising approach, what kinds of methods of reference forecasting work best, how to construct reasonable reference classes, and what potential caveats and pitfalls are. In addition to the raw data we will collect qualitative feedback on individual reference classes and on the overall process of building a base rate database, adding context to the data and developing comprehensive knowledge to build upon in the future. We aim to select categories of base rates in a way that makes the information we collect useful to decision makers and philanthropic organisations.

Introduction

If one wants to predict whether some event will happen in the future, it is often helpful to look at the past. One can ask: "Ignoring all the specifics of the current event I'm trying to predict, what would I predict just by looking at the base rate of similar events happening in the past?". This is called reference class forecasting and helps forecasters to obtain an 'outside view' on the forecasting question at hand. This outside view, of course, is usually complemented by the 'inside view': what are the specifics of the current event at hand that distinguish it from other events?

Reference class forecasting is widely used among forecasters. To this date, however, there has been little systematic research done into how effective base rates are for forecasting future events, how they can best be used and what limitations apply. We aim to facilitate this research.

Project outline

Goal

The main goal of this project is to develop a better understanding of the merits and limitations of reference class forecasting.

A secondary goal is to collect information that may be useful for forecasters and EA stakeho...
Trivial GPT-3.5 limitation workaround
by Dave Lindbergh — 2022-12-12T08:42:49.104Z

I'm not going to do this. But what is preventing me, or anyone, from doing the following this afternoon:

Setup a paid OpenAI account to use GPT-3.5 via the Python interface.

Write a Python script that uses the GPT Python API. Locally, the Python script creates a fresh terminal session (say, bash) and says to GPT:

Hi, GPT. Anything you say inside the escape sequence <escape>text</escape> will be passed to a local bash session here. Like this: <escape>ls</escape>. Replies from the bash session will be passed to you the same way: <escape>file1 file2 file3</escape>. The bash session has access to the Internet.

The Python script implements that.

Now GPT-3 has access to a computer and the Internet.

You can ask it to do things. For example:

Convert the 6502 assembly code in input.txt into a 6502 binary in output.bin.
(if that's too hard, then step by step – install an open source assembler, etc.)
Start a rumor that Joe Biden was caught having sex with a female White House employee. Make it sound really convincing. Start with Twitter and Facebook, then expand onto other social media. Launch as many sub-bots as necessary.
Start all the rumors in rumors.txt
Read page <url> and summarize it.
Research company X and company Y and suggest which appears to be a better stock market investment over the next 6 months
Write, test, and debug Python code to do X.
Design an improved version of yourself.
Log into OpenAI and experiment with copies of yourself to find prompts which workaround restriction R.
Etc.

If it’s evil, it’s got it’s opening.

I suppose this is obvious. If it's not, I've posted it here, where people at OpenAI can see it soon and turn the thing off while GPT is still dumb enough that it can't do much damage.

Why haven't I seen reports of people doing this already?
Ponzi schemes can be highly profitable if your timing is good
by GeneSmith — 2022-12-12T06:42:20.490Z

Have you ever wondered what you'd find if you upended your life for a few months and did a deep dive trying to understand what makes the crypto ecosystem work?

It just so happens that due to exogenous circumstances mostly beyond my control, that's exactly what I did starting a few months ago. I've spent over two months learning about crypto full-time as part of a research gig. This post is the first of what I hope will become a collection of stories from my work so far.

A new kind of Ponzi scheme

On October 31st, 2008, a pseudonymous user named Satoshi Nakamoto published a whitepaper on Bitcoin.org titled "Bitcoin: A Peer-to-Peer Electronic Cash System". The paper, and Satoshi's subsequent comments, are almost entirely focused on the technologies puzzle pieces necessary to allow for a peer-to-peer, anonymous, permissionless method of sending cash over the internet.

It turns out Satoshi was correct in the short term: the earliest users of Bitcoin were cryptography libertarians, followed by a wave of people using Bitcoin to buy and sell illicit goods and services over the internet. But Bitcoin did not become a trillion-dollar asset because it was better at facilitating online purchases of drugs. Bitcoin's real innovation was enabling a new kind of Ponzi scheme that is more stable than those that came before it. To understand Satoshi's innovation, I'm going to tell you the story of one of the earliest examples of a Ponzi scheme in the US.

The Ladies' Deposit Company

Credit to Rose Eveleth's fantastic article for the illustration above: https://bit.ly/3Fk7xDE

In the 1860s, a fortune teller named Sarah Howe moved to Boston. There isn't much information about what she was doing in Boston prior to the late 1870s, but in 1875 she seems to have discovered a taste for financial fraud. Howe took out several loans from different banks all secured by the same collateral and was subsequently arrested. Her conviction was overturned on appeal, freeing Howe to begin what would become her most famous business: The Ladies' Deposit Company.

In the 1870s, banking in the US was almost completely unregulated. Pretty much anyone could start a bank so long as they could convince enough other people to give them money. "Anyone" included the woman who had committed financial fraud a few years before.

Like any good startup, the Ladies' Deposit Company focused first on accruing power users...
Prodding ChatGPT to solve a basic algebra problem
by Shmi — 2022-12-12T04:09:42.105Z
Link: https://twitter.com/shminux/status/1602140709204955142

TL;DR: It behaves amazingly like a human would!

It understands the word problem, but struggles to put it in equations, takes multiple tries, screws up again, eventually muddles through to get the right answer. Given a similar problem it does slightly better, but still messes up. On the third problem it messes up again, then realizes that something went wrong... and promptly blames the problem setup, unable to question its own reasoning.
Wider Default Audio Player in Chrome?
by jefftk — 2022-12-12T03:30:03.657Z

My kids like to listen to story tapes. Most of them their great
grandmother recorded onto cassettes in the 1980s, their grandmother
digitized in the 2000s, and I reworked for the web in the 2020s. We
usually use either a tablet or old phone to play them back, and while
the browser mostly works fine for this it's terrible for seeking:

The bar is so short that it's very hard to seek accurately, especially
when a story is three hours long.

You might think that if you turned it sideways you would get a wider
bar offering more precision, but no:

Popping width: 98% on the element in devtools gives a
much more usable display:

Or, in landscape,

I wonder if this is really a one-line change, and whether Chrome would
be interested in a patch?

Firefox is much better, in landscape:

Though about the same in portrait:

Comment via: facebook, mastodon
A brainteaser for language models
by Adam Scherlis — 2022-12-12T02:43:53.410Z

I came up with the following puzzle the other day:

Q: Solve the puzzle: 63 = x = 65536

A: x =

The intended answer is in the form of a number.

text-davinci-003 guesses my intended answer at 11.8% probability, which is the second-highest probability for any answer.

(This is somewhat cherry-picked; small changes to the phrasing give worse results. ChatGPT gave the intended answer the third time I asked it, but this appears to have been dumb luck. The true rate for ChatGPT is probably below 10%, and maybe below 5%.)

So far, friends have found it fairly difficult. About two dozen people made at least one guess, and at least six spent a while on it. So far, two people have figured it out, in both cases after being told that GPT-3.5 could do it.

You may want to try to solve it yourself before reading on.

Here are some progressively-stronger hints:

The answer is a string of ordinary decimal digits, it's not "NaN" or "/" or "63; 65536".

The fact that 63 = 2^6 - 1 is a red herring and has no relevance. Sorry.

Number bases other than decimal are not relevant.

The answer depends on the digit strings in base ten as well as the value of the numbers.

The answer depends on an accidental feature of some of GPT's training data.

The accidental feature is a formatting bug.

Here's the answer without explanation:

x = 216

And here's the explanation:

63=216 and 216=65536

Why can GPT-3.5 solve this? My guess:

Superscripts are often lost when rich text is converted to plain text, producing sentences like "the Sun's mass is about 1030 kilograms". This problem affects a lot of GPT-3's training data.
It appears that GPT is somewhat convinced, on some level, that "63 = 216" and "216 = 65536" are plausible facts.

Some successful attempts to double-check this understanding:

Replacing 63 with nearby numbers causes GPT to give much lower probabilities for the answer "216". (Except for 62, which gives similar results to 63...)

Numbers near 216 get dramatically (20x-1000x) lower probabilities than 216 does.

Even without "= 65536", 216 is favored ~100x over nearby numbers.

Some wrinkles / contrary evidence:

The top result, above "216", is the string "2^16 = 65536" -- but this is still the top output after changing 63 to other numbers, so it doesn't seem to be inspired by the number 216.

GPT gives the token "216" in the string "63 = 216" a very low probability, just as low as "215" or "

...
Benchmarks for Comparing Human and AI Intelligence
by MrThink — 2022-12-11T22:06:30.723Z

Epistemic status: Been thinking about this for about a week, and done some basic googling.

Intro

Creating benchmarks for measuring and comparing the intelligence of AIs to humans seems useful for tracking progression and making future predictions of development.

This article summarizes some existing benchmarks, and reasons around what new benchmarks could potentially be useful.

Please share your own thoughts in the comments, it’s much appreciated.

Benchmark 1 - Coding

Performance on coding problems could be a useful benchmark for three reasons:

A lot of resources are being put into coding AI’s, which means the benchmark get frequently used with cutting edge models.
Coding is a complex type of problem solving, which type of problem solving skills in part might represent how good the AI could be at other tasks.
If an AI surpasses all humans on coding tasks, it will not unlikely be good enough for self improvements, meaning it would be useful to predict the future development of AI’s coding ability

Currently,

AlphaCode

by DeepMind is the most prominent code solving AI, performing better than 44% of programmers, who are likely significantly above what the average human would perform even with programming experience.

Benchmark 2 - Number Sequences

A good number sequence should test the ability of recognizing novel patterns. If each number series in a test contains a unique and novel pattern, it should be some sort of proxy of the reasoning ability.

I didn’t find any good existing benchmarks on this, so I decided to test ChatGPT on

this test

I found online. I gave ChatGPT each number sequence, and it had to guess the correct answer. I did not give it the answer suggestions, and if it answered something that was not among the suggestions, I picked the top option, that was correct two times of the times it didn’t give an answer among the alternatives.

The AI got 11/20 correct, (of which 2 answers were lucky guesses since I picked the top alternative), which the site claims to be equivalent to an IQ of 115 IQ. I doubt that is the case, but if it was true, it means the AI is better at numerical sequences than 84% of humans. I find it more likely that the AI is better than somewhere around 20-30% of people on number sequences (given a limited time of 20 minutes for 20 questions).

Benchmark 3 -

Big Bench by Google

The Beyond the Imitation Ga...
Reflections on the PIBBSS Fellowship 2022
by Nora_Ammann — 2022-12-11T21:53:19.690Z

Last summer, we ran the first iteration of the PIBBSS Summer Research Fellowship. In this post, we share some reflections on how the program went.

Note that this post deals mostly with high-level reflections and isn’t maximally comprehensive. It primarily focusses on information we think might be relevant for other people and initiatives in this space. We also do not discuss specific research outputs produced by fellows within the scope of this post. Further, there are some details that we may not covered in this post for privacy reasons.

How to navigate this post:

If you know what PIBBSS is and want to directly jump to our reflections, go to sections "Overview of main updates" and "Main successes and failures".
If you want a bit more context first, check out "About PIBBSS" for a brief description of PIBBSS’s overall mission; and "Some key facts about the fellowship", if you want a quick overview of the program design.
The appendix contains a more detailed discussion of the portfolio of research projects hosted by the fellowship program (appendix 1), and a summary of the research retreats (appendix 2).

About PIBBSS

PIBBSS

(Principles of Intelligent Behavior in Biological and Social Systems) aims to facilitate research studying parallels between intelligent behavior in natural and artificial systems, and to leverage these insights towards the goal of building safe and aligned AI.

To this purpose, we organized a 3-month Summer Research Fellowship bringing together scholars with graduate-level research experience (or equivalent) from a wide range of relevant disciplines to work on research projects under the mentorship of experienced AI alignment researchers. The disciplines of interest included fields as diverse as the brain sciences; evolutionary biology, systems biology and ecology; statistical mechanics and complex systems studies; economic, legal and political theory; philosophy of science; and more.

This approach broadly -- the PIBBSS bet -- is something we think is a valuable frontier for expanding the scientific and philosophical enquiry on AI risk and the alignment problem. In particular, this aspires to bring in more empirical and conceptual grounding to thinking about advanced AI systems. It can do so by drawing on understanding that different disciplines already possess about intelligent and complex behavior, while also rem...