trentmkelly/LessWrong-43k huggingface dataset:
Black Box Investigation Research Hackathon
by Esben Kran — 2022-09-12T07:20:34.966Z

TLDR;

Join the Black Box Investigation Research Hackathon

to participate with others in a weekend of alignment research on language models using a

cognitive psychology-style approach

.

Purpose

In the Black Box Investigation Research Hackathon, the goal is to explore where black box language models work well and where they break down. It is inspired by

this post by Buck Shlegeris

which calls for investigators of language models in a cognitive psychology style fashion. Become the AI psychologist of the future.

The meta-goal of this

Alignment Jam

is to investigate if hackathons work well for 1) providing real research results and ideas, 2) increasing the research experience of the participants, and 3) being a scalable solution to exciting ways of joining the alignment community to work on interesting and well-defined problems.

Outcome

Participants are encouraged to discover interesting behavior of black box language models. At the end of the hackathon each group / participant should have created a short (<5 min) presentation showcasing their results. A jury will then vote for their favorite projects according to the review rubric specified in the appendix.

After the hackathon, we will write a LessWrong post (or series of posts) describing the main findings and the overall experience. We hope to encourage more people to do this kind of work and we believe the hackathon format works well for generating novel research perspectives on alignment.

If this hackathon is successful, we expect to repeat the success with other

research agendas

.

Participation instructions

It is

hosted on the hackathon platform itch.io

where you will follow the instructions to sign up (see

here

): Make an itch.io account, click “Join jam”, and come along on the 30th of September!

We will work all weekend in Gathertown in groups of 2-6. The link to Gathertown will be announced in the

itch.io community tab

and sent to your itch.io email when we get closer to the date. You can ask questions

here

.

When the hackathon starts, you will be able to continually upload new PDF / Google Doc / Colab / Github repositories during the extent of the research competition.

For interacting with the black box models, we will use

20b.eleuther.ai

,

beta.openai.com/playground

, and you will receive research API credits to work directly with the APIs of OpenAI. There ...
Argument against 20% GDP growth from AI within 10 years [Linkpost]
by aog — 2022-09-12T04:08:03.901Z
Link: https://twitter.com/cameronfen1/status/1567205153937457153

Mohammed Bavarian, a research scientist at OpenAI, tweeted this thread arguing that he could see "the overall US GDP growth rising from recent avg 2-3% to 20+% in 10 years." Feel free to check out those arguments, though they'll probably be familiar to you: GPT-3, GitHub Copilot, and image synthesis will drive unprecedented improvements.

Cameron Fen, an economics PhD student at the University of Michigan, responded with this thread disagreeing with Bavarian's argument. I wanted to share some of the arguments that I found novel and interesting.

Argument #1: The impacts of previous transformative technologies

There have been 3 industrial revolutions in history, mechanization, electricity and mass production, and IT and the internet. China went through all 3 at the same time and was barely able to go above 7% growth annually.

The newest industrialization will be big, but will it be bigger than moving 95% of the population working in agriculture to working factories, not to mention all 3 combined?

In particular, the growth of the first industrial revolution accelerated growth from 1.5% to 3% a year in the UK (Source). Growth from electrification was 1.5% a year on average (source) and growth during the information industrialization was 3.5% a year (source).

Given these growth rates, it seems unlikely that a single industrial revolution can move the needle to such an extent that US GDP growth accelerates from 2% to 7%.

Argument #2: The size of the tech industry

Perhaps you can argue that the industrial revolution on the horizon is going to be 3x bigger than any industrial revolution in the past. Let’s see what that would imply:

7% growth implies 5% growth over our current trend rate of about 2%. This comes out to be about 1.15 trillion GDP additional growth this year and increasing as the base gets larger.

According to this article, Facebook contributes 100 billion to US economic activity. While much of this is uncounted because GDP methodology is imperfect, I’ll go with that number.

To get to the 1.15 trillion additional growth you need 11.5 Facebook created every year. This comes out to a market cap of 5.5 trillion dollars created every year in tech to get something like a 5% growth. The market cap of the entire tech sector is 13.5 trillion dollars.

Call adjacent markets another 6.5 trillion dollars. Thus a 20 trillion dollar market cap sector needs to add 6 tr

...
AI Safety field-building projects I'd like to see
by Orpheus16 — 2022-09-11T23:43:32.031Z

People sometimes ask me what types of AIS field-building projects I would like to see.

Here’s a list of 11 projects.

Background points/caveats

But first, a few background points.

These projects require people with specific skills/abilities/context in order for them to go well. Some of them also have downside risks. This is not a “list of projects Akash thinks anyone can do” but rather a “list of projects that Akash thinks could Actually Reduce P(Doom) if they were executed extremely well by an unusually well-qualified person/team.”
I strongly encourage people to reach out to experienced researchers/community-builders before doing big versions of any of these. (You may disagree with their judgment, but I think it’s important to at least have models of what they believe before you do something big.)
This list represents my opinions. As always, you should evaluate these ideas for yourself.
If you are interested in any of these, feel free to reach out to me. If I can’t help you, I might know someone else who can.
Reminder that you can apply for funding from the

long-term future fund

. You don’t have to apply to execute a specific project. You can apply for career exploration grants, grants that let you think about what you want to do next, and grants that allow you to test out different hypotheses/uncertainties.
I sometimes use the word “organization”, which might make it seem like I’m talking about 10+ people doing something over the course of several years. But I actually mean “I think a team of 1-3 people could probably test this out in a few weeks and get something ambitious started here within a few months if they had relevant skills/experiences/mentorship.
These projects are based on several assumptions about AI safety, and I won’t be able to articulate all of them in one post. Some assumptions include “AIS is an extremely important cause area” and “one of the best ways to make progress on AI safety is to get talented people working on technical research.” If I’m wrong, I think I’m wrong because I’m undervaluing non-technical interventions that could buy us more time (e.g., strategies in AI governance/strategy or strategies that involve outreach to leaders of AI companies). I plan to think more about those in the upcoming weeks.

Some projects I am excited about

Global Talent Search for AI Alignment Researchers

Purpose: Raise awareness about AI safet...
Fermi Paradox: Iron Age Milky Way
by Rofel Wodring — 2022-09-11T20:32:01.762Z

I have some observations about pre-industrial civilization and their relationship to why we don't see evidence of aliens. These observations, which in my opinion were consistent across human history despite very diverse cultures, can apply even to alien cultures.

Because this Great Filter isn't biological, temporal, or even technological -- it's structural and inherent to how civilizations develop in the first place.

The road from Stone Age to Iron Age seems pretty obvious. Humans have independently experienced this phenomenon several times in history. So is the road from Industrial Age to Information Age; not as obvious or dramatic, but going from steamships to spaceships seems to be a pretty inevitable progression, done by polities of varying political and economic situations.

The problem is that event between Stone Age and Information Age, the Industrial Age. Unlike pre-industrialization or post-industrialization societies, which seem to come about in all sorts of political conditions, industrialization only came from a very specific set of circumstances that may be impossible to replicate on other planets.

The Industrial Revolution was not kicked off just from having a handful or even plethora of inventions. We had the technological powder keg for the Industrial Revolution for centuries, what was needed was a political situation that allowed society to use the pieces in a way to change our mode of production. Because the lead-up to the Industrial Revolution saw some deeply, deeply profound changes that I think were both necessary and also very unlikely. And not just improbable, but 'straight-up most civilizations will never have the prerequisites, because the path to Iron Age civilization opposes the path to industrialization'.

Any pre-industrial civilization of sufficient size relative to its neighbors was an autocracy, and even the smaller ones such as the North American Confederations and the Swiss Republic had more autocratic than democratic elements.

You cannot begin the process of industrialization without a sizeable middle class, but throughout history the middle class was very small and under the thumb of the nobility. Even in times of surplus the middle class barely grew. And note that you can't technologize your way to the middle class.

Pre-industrial autocracies don't want to innovate or change. They want to expand, yes, but not in a way...
You Don't Have To Click The Links
by Simon Berens — 2022-09-11T18:13:24.817Z

There are a lot of blog posts that have >5 links to other blog posts. I used to click on all of these, and upon seeing that I hadn't read them, I would embark on a several hour long journey of reading until exhaustion, often times without even finishing the original post I started.

Recently, I had the simple realization that I could usually read a blog straight through without ever reading one of the linked posts while still understanding the content of the blog.

I wanted to share this with other lurkers in hopes that I could make reading on LessWrong more enjoyable and efficient.

Exercise for the reader: don't click the links!
The Ultimate Step-by-Step Hiring Playbook
by intellectronica — 2022-09-11T14:39:48.628Z
Link: https://www.intellectronica.net/the-ultimate-step-by-step-hiring-playbook/

Mullah Nasrudin Hodja was crawling on his hands and knees under the streetlight when he was spotted by a passer-by. “What are you doing, Mullah? Why are you crawling?” asked the man. The mullah answered “I am searching for my key. I have lost it and need it to get back into my home.” The man offers Nasrudin his help, and soon they are both crawling on their hands and knees, under the streetlight. The key, alas, is nowhere to be found. “Are you sure you lost the key here, mullah?”, asks the man. “Oh no, not here,” says Nasrudin, “I must have dropped the key in the park.” Bewildered, the man asks, “then why are you looking for it here?!” “The park is dark”, explains Nasrudin, “here there is light.”

Earlier drafts of this playbook started with the steps, followed by commentary and reflection, but on the advice of my editor I have flipped the two sections. The playbook, you see, is in the correct order, one step after another, but the correct order is counterintuitive, and might make your head explode, or at the very least result in the misapplication the steps.

The most dangerous trap when making hiring decisions (or when making any low certainty / high stakes decision in work and life) is that we become overly focused on what we know at present. Beginnings are relatively uninteresting. We are only going to hire someone once, and when we do, we’ll then end up working with them for years to come. But like Nasrudin, we are too easily tempted to look under the proverbial streetlight. We focus on the candidate’s resumé, their performance in a limited number of interviews, our gut feelings about their personality. We take into consideration the information we have most readily available at the time of making the decision and ignore everything that really matters - the candidate’s future if they are to join our team.

To make a better decision, we can start from the end. Recognize what we know (and especially what we don’t know) about the future and move step-by-step back to present time. Following this order can help us avoid the instinct to tell ourselves a coherent narrative, fooling ourselves into believing that we can be more certain than is reasonable. Instead, we recognize at every step of the way how little we can confidently predict.

Our default answer to any question about the future should be that we simply don’t know. The world is complicated and chaotic, and ...
In forecasting, how do accuracy, calibration and reliability relate to each other?
by amarai — 2022-09-11T12:04:18.531Z

Forecasters vary on at least three dimensions:

accuracy- as measured in (e.g.) average brier score over time (brier score is a measure of error where if you think (say) p is 0.7 likely and p turns out to be true, then your brier score on this forecast is (1 - 0.7)^2).
calibration - how close are they to perfect calibration where for any x, if they assign a probability of x% to a given statement, in x% of cases, they are right?
reliability - how much evidence does a given forecast of yours provide for the proposition in question being true? I think of this as "for a given confidence level c, whats the bayesfactor P(you say the probability of x is c|x)/P(you say the probability of x is c|not-x)?"

I wonder how these three properties relate to each other.

(A) Assume that you are perfectly calibrated at 90% and you say "It will rain today with 90% probability" - how should I update on your claim given I know your perfect calibration? My first intuition is that, given your perfect calibration,
P(you say rain with 90%|rain) is 90% and P(you say rain with 90%| no rain) is 10% likely. But that doesn't follow from the fact that you are perfectly calibrated, does it? Does your calibration have any bearing at all on your reliability (apart from the fact that both positively correlate with forecasting competence)? If it doesn't - why do we care about being calibrated?
(B) How does accuracy relate to reliability? Can infer something about your reliability from knowing your over-time brier score?
Briefly thinking through some analogs of debate
by Eli Tyre — 2022-09-11T12:02:35.857Z

[This post is an almost direct summary of a conversation with John Wentworth.]

Note: This post assumes that the reader knows what AI Safety via Debate is. I don't spend any time introducing it or explaining how it works.

The proposal of AI safety via debate depends on the a critical assumption that "in the limit of argumentative prowess, the optimal debate strategy converges to making valid arguments for true conclusions."

If that assumption is true, then a sufficiently powerful AI-debate system is a safe oracle (baring other possible issues that I'm not thinking about right now). If this assumption is false, then the debate schema doesn't add any additional safety guarantee.

Is this assumption true?

That's hard to know, but one thing that we can do is try to compare to other, analogous, real world cases, and see if analogous versions of the "valid arguments for true conclusions" holds. Examining those analogous situations is unlikely to be definitive, but it might give us some hints, or more of a handle on what criteria must be satisfied for the assumption to hold.

In this essay I explore some analogs, an attempted counterexample and an attempted example.

Attempted counterexample: Public discourse

One analogous situation in which this assumption seems straightforwardly false is in the general discourse (on twitter, in "the media", around the water cooler, etc.). Very often, memes that are simple and well fit to human psychology, but false, have a fitness advantage over more complicated ideas that are true.

For instance, the core idea of minimum wage can be pretty attractive: lots of people are suffering because the have to work very hard, but they don't make much money, sometimes barely enough to live on. That seems like an inhumane outcome, so we should mandate that everyone be paid at least a fair wage.

This simple argument doesn't hold [1], but the explanation for why it is false is a good deal longer than the initial argument. To show what's wrong with it, one has to back up and explain the basic principles of supply and demand, and demonstrate that those principles apply in this case. You might have to write a slim textbook to make the counterargument in a compelling way.

And in the actual real world, it seems like false-but-simple-and-attractive ideas very often win out over true ones.

This seems like it doesn't bode well for AI safety via debate, but ...
Making a New Table Leaf
by jefftk — 2022-09-11T11:40:02.714Z

We got our dining room table used, and it didn't come with any leaves.
I made one out of plywood when we first got it, and it's been pretty
helpful over the past ~7y:

With the leaf we can comfortably fit eight: three on the long sides
and one on the short. If we have more people two on the short works,
but when you get above ten people it's pretty squishy. With our new housemate, starting to have dinner
guests again, and Nora getting close to sitting at the table with us,
I wanted to make the table bigger again.

The telescoping mechanism opens to 36", and the current leaf was half
that. I considered making a new leaf the same size as the previous,
but decided to make a full-size one instead since it would be
sturdier. It should also look a bit better, since it will only fail
to match the surrounding table and not additionally the earlier leaf.

I cut some 3/4" plywood to the table width and rounded off the
corners. I needed to drill four holes for pegs that the table uses to
keep the leaf aligned. Here's the initial fit:

I sanded it with 80/120/150/220 grit on the random
orbit sander until it was very smooth, almost soft to the touch.
I used Minwax wood finish ("Early American 230"), left over from an
earlier project, to stain it. I did two rounds, because last time it
ended up lighter than the rest of the table. If you want a better
match you can test out a bunch of different things in parallel on the
back of the piece or a scrap of identical wood, but that didn't feel
worth it here: I'm not even using the same type of wood as the
rest of the table. Then three coats of oil-based polyurethane:

It was still slightly rough so I sanded it again with 220:

Another coat of poly, and here's the finished leaf:

It will get less shiny with a bit of use, and fit in better. You can
see that the outer edge is a simple rounded corner, and not the more
complex routed design on the original. I'd considered trying to match
it, but it would be a lot of work and I don't think most people
sitting at the table will care.

Now it's back outside to air out (smells like polyurethane) until we
need a big table!
AI Risk Intro 1: Advanced AI Might Be Very Bad
by CallumMcDougall — 2022-09-11T10:57:12.093Z

This is a submission for the AI Safety Public Materials contest.

Introduction

If human civilisation is destroyed this century, the most likely cause is advanced AI systems. This is perhaps a surprising conclusion, but one that many people who think about the topic keep coming to. While it is not easy to describe the case for risks from advanced AI in a single piece, here we make an effort that assumes no prior knowledge. Rather than try to argue from theory straight away, we approach it from the angle of what computers actually can and can’t do.

The Story So Far

Image generated by OpenAI's DALL-E 2. Prompt: artist's impression of an artificial intelligence thinking about chess, digital art, artstation. All images below with italic captions are generated by DALL-E.

(This section can be skipped if you understand how machine learning works and what it can and can’t do today)

Let’s say you want a computer to do some complicated task, for example learning chess. The computer has no understanding of high-level things like “chess”, “board”, “piece”, “move”, or “win” - it only understands how to do a small set of things. Your task as the programmer is to break down the high-level goal of “beat me at chess” into simpler and simpler steps, until you arrive at a simple mechanistic description of what the computer needs to do. If the computer does beat you, it’s not because it had any new insight into the problem, but rather because you were clever enough to find some

set of steps

that, carried out blindly in sufficient speed and quantity, overwhelms whatever cleverness you yourself can apply during the game. This is how Deep Blue beat Kasparov, and more generally how most software and the so-called “Good Old-Fashioned AI” (GOFAI) paradigm works.

Programs of this type can be powerful. In addition to

beating humans at chess

, they can

calculate shortest routes

on maps,

prove maths theorems

,

mostly fly airplanes

, and

search all human knowledge

. Programs of this type are responsible for the stereotypical impression of computers as logical, precise, uncreative, and brittle. They are essentially executable logic.

Many people hoped that you could write programs to do “intelligent” things. These people were right - after all, ask almost anyone before Deep Blue won whether playing chess counts as “intelligence”, they’d have said yes. But “classical” p...