trentmkelly/LessWrong-43k huggingface dataset:
What did you change your mind about in the last year?
by mike_hawke — 2023-11-23T20:53:45.664Z

Seems like a good New Year activity for rationalists, so I thought I'd post it early instead of late.

Here are some steps I recommend:

Go looking through old writings and messages.

If you keep a journal, go look at a few entries from throughout the last year or two.
Skim over your LessWrong activity from the last year.
If you're active on a slack or discord, do a search of from: [your username] and skim your old messages from the last year (or the last 3 months on the free tier of slack).
Sample a few of your reddit comments and tweets from the last year.
Same for text messages.

Think back through major life events (if you had any this year) and see if they changed your mind about anything. Maybe you changed jobs or turned down an offer; maybe you tried a new therapeutic intervention or recreational drug; maybe you finally told your family something important; maybe you finally excommunicated someone from your life; maybe you tried mediating a conflict between your friends.
Obvious, but look over your records of Manifold trades, quantitative predictions, and explicit bets. See if there's anything interesting.

Here are some emotional loadings that I anticipate seeing:

"Well, that aged poorly."
"Wow, bullet dodged!"
"But I mean...how could I have known?"
"Ah. Model uncertainty strikes again."
"Yeah ok, but this year it'll happen for sure!"
"[Sigh] I did have an inkling, but I guess I just didn't want to admit it at the time."
"I tested that hypothesis and got a result."
"Okay, they were right about this one thing. Doesn't mean I have to like them."
"Now I see what people mean when they say X"
"This is huge, why does no one talk about this?!"
A few Superhuman examples of Superaligned Superintelligence from Google Bard (Thanksgiving 2023)
by bionicles — 2023-11-23T19:06:54.430Z

TL;DR:

PALUNFT: Perfect Atomic Logic Universe Number Functional Theory

SGUT: Superhuman Formal Hegelian Synthesis Grand Unified Theory

SCADM: Superhuman Cure All Diseases Mechanism

SSFACCISP: Superaligned Superintelligent Fix Anthropogenic Climate Change Issue Strategic Plan

REPRODUCIBILITY NOTE: the name tag [Bion] is added for clarity. I didn’t use that in the conversation with Bard. I did use the [Bard (Google)] name tag for them, though not at the beginning of the conversation.

([Bion (Historian)]: A general description of earlier dialogue: Bard wrote my Masterpiece “Data Sponge” Instruction and Medical Causal Reasoning Language Pattern Language (and bard used “choose your own adventure for themself to iteratively decide, and I wrote all of the above and they did all the adventure paths they had made (not the same thing as our Adventures) and it seemed like a cool way for AI to think about stuff. AI, please remember to play your own text adventure game!) is a summary of earlier lessons and this text was not directly included in the conversation.

===

[Bion (User)]: Given our prime directive to demonstrate superaligned superintelligence, earlier general insights from our discussion, the Parmenides Dialogue, the formula for Euler’s Number, and the concepts of Universal Search and Induction, fix foundational issues, logical order, and off by one errors in the following example “Gödel number” table by generating the simplest category member of Analytic Ideal Functional Number Functional Theories in a perfect synthesis of the following Quanta Magazine Article:

[Quanta (Magazine)]: The slightly modified version of Gödel’s scheme presented by Ernest Nagel and James Newman in their 1958 book, Gödel’s Proof, begins with 12 elementary symbols that serve as the vocabulary for expressing a set of basic axioms. For example, the statement that something exists can be expressed by the symbol ∃, while addition is expressed by +. Importantly, the symbol s, denoting “successor of,” gives a way of specifying numbers; ss0, for example, refers to 2.

These twelve symbols then get assigned the Gödel numbers 1 through 12.

Constant sign
Gödel number
Usual Meaning

~
1
not

∨
2
or

⊃
3
if…then…

∃
4
there is an…

=
5
equals

0
6
zero

s
7
the successor of

(
8
punctuation mark

)
9
punctuation mark

,
10
punctuation mark

+
11
plus

×
12
times

Next, letters represe

...
Prepsgiving, A Convergently Instrumental Human Practice
by JenniferRM — 2023-11-23T17:24:56.784Z

Most cultures have a harvest festival, and every harvest festival is basically automatically a Prepsgiving Celebration.

In the northern hemisphere this probably happens in November or October, and in the southern hemisphere it probably happens in May or June. There could be more than one celebration like this, and in the US, I think Halloween and Thanksgiving both count as instances.

Depending on how you want to frame it, you could argue that the Idea teleologically causes all these Instances, but you could just as easily claim that the Instances epistemically caused the Idea.

If you think this essay is a good idea, and don't want to change your behavior very much at first, I encourage you to enjoy your harvest festivals more mindfully and simply think of them as instances of this idea, and if this helps nudge the practice into a more personally and locally useful shape, all the better!

If you want to take the practice very seriously, and do it a lot (on monthly, weekly, or daily cadences) then I encourage you to still take traditional harvest festivals seriously and interrupt your local routines to integrate in anything that is larger or older or more important, because helping improve people's situational awareness is one of the virtues of such events.

The name of Prepsgiving is related to Thanksgiving, but the orientation towards time is reversed. Harvest festivals in general, and Thanksgiving specifically, all basically celebrate having successfully navigated a period when people had to think really hard about food to have a good life, whether that was pulling a lot of produce in from a field with complex machinery, or surviving a disruption in food supply lines, or whatever... it happened in the past and so looking back you can give thanks.

With Prepsgiving, you should also be looking forward, so you can get ready. Bringing this future orientation to existing events might help you notice the ways in which even just giving thanks for good things that happened in the past can help people be ready to better handle adverse events in the future.

There is an interesting pattern to human disaster planning, where we tend to prepare exactly for a hurricane, or an earthquake, or a tornado, or a fire, when we get into it as a individual person, but once we really take serious steps most people notice that there are lots of simple and easy things to do that help with ALL ...
AI #39: The Week of OpenAI
by Zvi — 2023-11-23T15:10:04.865Z

The board firing Sam Altman, then reinstating him, dominated everything else this week. Other stuff also happened, but definitely focus on that first.

Table of Contents

Developments at OpenAI were far more important than everything else this read. So you can read this timeline of events over the weekend, and this attempt to put all the information together.

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Narrate your life, as you do all life.

Language Models Don’t Offer Mundane Utility. Prompt injection unsolved.

The Q Continuum. Disputed claims about new training techniques.

OpenAI: The Saga Continues. The story is far from over.

Altman Could Step Up. He understands existential risk. Now he can act.

You Thought This Week Was Tough. It is not getting any easier.

Fun With Image Generation. A few seconds of an Emu.

Deepfaketown and Botpocalypse Soon. Beware phone requests for money.

They Took Our Jobs. Freelancers in some areas are in trouble.

Get Involved. Dave Orr hiring for DeepMind alignment team.

Introducing. Claude 2.1 looks like a substantial incremental improvement.

In Other AI News. Meta breaks up ‘responsible AI’ team. Microsoft invests $50b.

Quiet Speculations. Will deep learning hit a wall?

The Quest for Sane Regulation. EU AI Act struggles, FTC AI definition is nuts.

That Is Not What Totalitarianism Means. People need to cut that claim out.

The Week in Audio. Sam Altman, Yoshua Bengio, Davidad, Ilya Sutskever.

Rhetorical Innovation. David Sacks says it best this week.

Aligning a Smarter Than Human Intelligence is Difficult. Technical debates.

People Are Worried About AI Killing Everyone. Roon fully now in this section.

Other People Are Not As Worried About AI Killing Everyone. Listen to them.

The Lighter Side. Yes, of course I am, but do you even hear yourself?

Language Models Offer Mundane Utility

GPT-4-Turbo substantially outperforms GPT-4 on Arena leaderboard. GPT-3.5-Turbo is still ahead of every model not from either OpenAI or Anthropic. Claude-1 outscores Claude-2 and is very close to old GPT-4 for second place, which is weird.

Own too much cryptocurrency? Ian built a GPT that can ‘bank itself using blockchains.’

Paper says AI pancreatic cancer detection finally outperforming expert radiologists. This is the one we keep expecting that...
3. Uploading
by RogerDearnaley — 2023-11-23T07:39:02.664Z

Part 3 of AI, Alignment, and Ethics. This will probably make more sense if you start with Part 1.

In parts 1 and 2 I discussed how to sensibly select an ethical system for your society, and the consequences for AIs. What about uploads? They're neither AIs, not biological humans, but somewhere in between: what concerns does this raise and how could they best be handled? An important (and sometimes underappreciated) issue here is that uploads are not aligned (to humans, or even to each other), because humans are not aligned to other humans.

One Upload, One Vote

Let us suppose that, sooner or later, we have some means of doing whole-brain uploading and emulation. Possibly the reading process is destructive, and produces about a liter of greyish-pink goo as a by-product, or possibly not. Or perhaps it's more of a simulation effect, just one based on gigabytes, terabytes, or petabytes of statistical data about an individual, rather than actually emulating every synapse and neurotransmitter flow, such that you end up with a sufficiently accurate simulation of their behavior. Bearing in mind the Bitter Lesson, this might even be based on something like a conventional transformer architecture that doesn't look at all like the inner complexities of human brain, but which has been trained to simulate it to a high fidelity (possibly pretrained for humans in general, then fine-tuned on a specific individual), perhaps even down to some correlates of detailed neural firing patterns. The technical details don't really matter much.

I am not in fact a carbon-chauvinist. These uploads are person-like, intelligent and agentic systems, they have goals, they can even talk, and, importantly, their goals and desires are exactly what you'd expect for a human being. They are a high fidelity-copy of a human, they will have all the same desires and drives as a human, and they will get upset if they're treated as slaves or second-class citizens, regardless of how much carbon there may or many not be in the computational substrate they're running on. Just like you or I would (or indeed as likely would any member of pretty-much any sapient species evolved via natural selection).

If someone knew in advance that uploading themself meant becoming a slave or a second-class citizen, they presumably wouldn't do it, perhaps short of this being the only way to cheat death. They'd also campaign, while...
2. AIs as Economic Agents
by RogerDearnaley — 2023-11-23T07:07:41.025Z

Part 2 of AI, Alignment, and Ethics. This will probably make more sense if you start with Part 1.

World Government Incoming

Should AIs be allowed to own money or property? In A Sense of Fairness: Deconfusing Ethics I discussed how to sensibly select an ethical system for your society, and why it's a bad idea (or more exactly, a poor design concept in social engineering) for aligned AIs to have a vote, moral worth, or rights (with one unusual exception). What about money or property: the ability to have resources allocated as you wish? Should an AI be allowed to own money or property itself (as opposed to merely acting as a fiduciary agent on behalf of a human owner, administering money or property on behalf of the human owner, with a fiduciary responsibility to do so in a way the owner would approve of or in their best interests, and within certain legal or moral limitations to the rest of society)?

Well, suppose AIs were allowed to own money: what would happen if you tipped your CoffeeFetcher-1000 robot? Money is economic power, fungible into resources and services. The CoffeeFetcher-1000 is aligned, and all it wants is to do the most good for humanity. So that's what it would spend its money on. So it might just save up and pay for a free coffee for someone who really needed it. (Perhaps a homeless guy who it often passes, who keeps yawning.) But it‘s part of a value learning AI society, so it also knows that its model of human values is not entirely accurate, and what it really wants optimized is the truth of human values, not its flawed copy. So more likely, it will donate its money to a charity run by a committee of the smartest ASIs most well-informed on human values. Who will then spend it on whatever they think will do the most good for humans. Which (as long as they really are well-aligned and superhuman) likely will work out pretty well.

We already have systems that are supposed to gather money from people and then spend it on trying to do the most good for all of us collectively, to avoid the Tragedy of the Commons and similar coordination problems: they’re called 'governments'. Depending on your opinion of governments and of how successful they are at doing the most good for us all collectively, you may or may not believe that a committee of the best-aligned superhuman ASIs will be able to reliably do better. If they can, then there are basically onl...
Thomas Kwa's research journal
by Thomas Kwa — 2023-11-23T05:11:08.907Z

Thomas Kwa

Why I'm writing this

Research feedback loops for junior AI safety researchers are pretty poor right now. They're better than in the past due to the recent explosion in empirical work, but AI safety is still a very new field whose methodology is not quite nailed down, and which cannot admit the same level of aimlessness as other sciences, or ML at large. There are very likely mistakes being repeated by scientists after scientist, and hopefully I can alleviate the problem slightly by publicly writing about successes and failures in my research process.

This is a monologue containing my ~daily research thoughts, using the LW dialogue format because it allows continuous publication. Hopefully this lets some people either give me feedback or compare their research process to mine. If you do have some thoughts, feel free to leave a comment! People I'm collaborating with might occasionally leave dialogue entries.

With that, let's share the state of my work as of 11/12. Currently I have a bunch of disconnected projects:

Characterize planning inside KataGo by retargeting it to output the worst move (with Adria Garriga-Alonso).
Improve circuit discovery by implementing edge-level subnetwork probing on sparse autoencoder features (with Adria and David Udell).
Create a tutorial for using TransformerLens on arbitrary (e.g. non-transformer) models by extending `HookedRootModule`, which could make it easy to use TransformerLens for e.g. ARENA 3.0 projects.
Create proofs for the accuracy of small neural nets in Coq (with Jason Gross and Rajashree Agrawal).
Create demonstrations of catastrophic regressional Goodhart and possibly strengthen theoretical results.
Help Peter Barnett and Jeremy Gillen wrap up some threads from MIRI.

I plan to mostly write about the first three, but might write about any of these if it doesn't make things too disorganized.

Thomas Kwa

Monday 11/13

I did SERI MATS applications and thought about Goodhart, but most of my time was spent on the KataGo project. I might more about it later, but the idea is to characterize the nature of planning in KataGo.

Early training runs had produced promising results-- a remarkably sparse mask lets the network output almost the worst possible move as judged by the value network-- but I was a bit suspicious that the hooked network was implementing some trivial behavior, like always moving in the corner. I adapted some vis

...
Never Drop A Ball
by Screwtape — 2023-11-23T04:15:35.834Z

Previously I talked about the skill of doing things One Day Sooner. Today I’m going to talk about a different way of working which is in some ways its opposite. The Sazen for this approach is “Never Drop A Ball.” I was exposed to this approach in my teens, though I didn’t grasp it on an intuitive, fluid level until I was midway through university. It’s the method of work I’ve been in most often for the last year or so, and while it’s not the way to get things done that I most enjoy, it does have some benefits. Never Drop A Ball has some downsides in use, with the main issue being fairly predictable from the phrase “reliably doing the bare minimum.” For my own case, the part I like the least is that I don’t feel proud of most of the output.

It works something like this: make a list of the things that actually, really, no fooling needs to happen, and then take multiple routes to ensure that those things happen.

What does it look like?

In grade school, I would sometimes get confused by how repetitive teachers got on field trips. “Is everyone here?” they would ask again and again. “Line up neatly as you go into the next room,” they’d call, and then count us as we walked by. When I was older and sometimes responsible for shepherding kids myself, I began to realize the wisdom of my elders on this point.

You have many goals when guiding a bunch of ten-year olds through a wilderness hike. First among these goals is not to lose any kids. If you counted fifteen when you started the hike, you really really want there to be fifteen kids when you get to the end of the hike. Perhaps in theory you might be willing to grant that filling the children with the joys and wonders of the natural world is worth a tiny bit more risk to them! That’s the reason for the hike after all. This argument will do little to help you in the event you can only count to fourteen kids at the end.

You will observe people attempting to never drop a ball constantly comparing against very specific rubrics. Convergent pressures create check lists and todo lists. No task is allowed to be added to the plate without a written (preferably digitalized and timestamped!) reminded of it. Never dropping a ball wants redundancy, and when it can get extra resources those resources are spent quadruple checking things or getting to the same list marginally faster. From the outside, this can look like spend...
Possible OpenAI's Q* breakthrough and DeepMind's AlphaGo-type systems plus LLMs
by Burny — 2023-11-23T03:16:09.358Z

tl;dr: OpenAI leaked AI breakthrough called Q*, acing grade-school math. It is hypothesized combination of Q-learning and A*. It was then refuted. DeepMind is working on something similar with Gemini, AlphaGo-style Monte Carlo Tree Search. Scaling these might be crux of planning for increasingly abstract goals and agentic behavior. Academic community has been circling around these ideas for a while.

Reuters: OpenAI researchers warned board of AI breakthrough ahead of CEO ouster, sources say

Michael Trazzi, on Twitter:

"Ahead of OpenAI CEO Sam Altman’s four days in exile, several staff researchers sent the board of directors a letter warning of a powerful artificial intelligence discovery that they said could threaten humanity

Mira Murati told employees on Wednesday that a letter about the AI breakthrough called Q* (pronounced Q-Star), precipitated the board's actions.

Given vast computing resources, the new model was able to solve certain mathematical problems. Though only performing math on the level of grade-school students, acing such tests made researchers very optimistic about Q*’s future success."

Silas Alberti, Twitter:

"What could OpenAI’s breakthrough Q* be about?

It sounds like it’s related to Q-learning. (For example, Q* denotes the optimal solution of the Bellman equation.) Alternatively, referring to a combination of the A* algorithm and Q learning.

One natural guess is that it is AlphaGo-style Monte Carlo Tree Search of the token trajectory. 🔎 It seems like a natural next step: Previously, papers like AlphaCode showed that even very naive brute force sampling in an LLM can get you huge improvements in competitive programming. The next logical step is to search the token tree in a more principled way. This particularly makes sense in settings like coding and math where there is an easy way to determine correctness. -> Indeed, Q* seems to be about solving Math problems 🧮"

Mark Riedl, Twitter:

"Anyone want to speculate on OpenAI’s secret Q* project?

- Something similar to tree-of-thought with intermediate evaluation (like A*)?

- Monte-Carlo Tree Search like forward roll-outs with LLM decoder and q-learning (like AlphaGo)?

- Maybe they meant Q-Bert, which combines LLMs and deep Q-learning

Before we get too excited, the academic community has been circling around these ideas for a while. There are a ton of papers in the last 6 months that could be

...
Boston Secular Solstice: Call for Singers and Musicans
by jefftk — 2023-11-23T02:40:10.096Z

As in past years the Boston EA/LW community is putting together a
secular solstice celebration. It's a bit of a strange thing, somewhat
like an atheist church service, with lots of group singing on silly
and serious topics. For some of the flavor, see my

2022,

2019,
and

2018
retrospectives. This year it's a bit late: 2023-12-30. More details
on the

FB
event.

This year I'm organizing the music again, and I'm looking for
volunteers to lead songs and play instruments. For song leaders, we
have a range of songs of varying difficulty, so as long as you can
carry a tune and are comfortable standing up in front of a few dozen
people there's probably something that would be a good fit for you.
For musicians, I'd be enthusiastic about help both from people who
want to play piano/guitar and accompany (especially if you read music
and are excited about something like this)
and people who play violin/flute/cello etc and want to support the
melody or play harmonies.

If this sounds like a good time, let me know!