trentmkelly/LessWrong-43k huggingface dataset:
my favourite Scott Sumner blog posts
by DMMF — 2024-06-11T14:40:43.093Z
Link: https://danfrank.ca/the-wisdom-of-scott-sumner-my-favourite-non-econ-scott-sumner-blog-posts/

Given Scott's invitation to LessOnline and general embrace of this community, I thought many here would appreciate this curated list of my favourite Scott Sumner blog posts.

-

Scott Sumner is best known as an economist who was praised for positively influencing economic policy during the Great Recession through blogging, as well as the leading voice for market monetarism. But to me, as someone uninterested in monetary policy, Scott stands out as one of the world's best, warmest, and wisest generalist bloggers. He writes on a wide range of topics, often touching on issues adjacent to politics, economics, and art, but irrespective of the topic, his wisdom always shines through. I might be biased because our values are quite similar—we both lean towards neoliberalism, embrace a utilitarian approach, and are completely captivated by art—but of all the writers I read, Scott is the one who is least often wrong. While his blogging isn't particularly ambitious, there are always little nuggets of wisdom sprouting up, and it's consistently insightful, introspective, and kind. Like many of my other favourite writers, Scott uses incredibly simple yet robust concepts, and by applying them rigorously and broadly, he helps you see things you should have been able to see yourself but didn't.

My favourite Scott Sumner blogs:

Understanding middlebrow

on the 90-99th percentile mocking the 0-90th percentile for their taste, while being oblivious to the true cultural elite

What do we mean by meaning

On what provides meaning in life

Wallowing in nostalgia (an autobiography)

On how the sharpness of life evolves over time

Give thanks for progress and Who's afraid of the great outdoors

On agnosticism towards progress and how norms evolve over time

Double vision: when then was now

On the ability to interpret life through our prior experiences

It's a wonderful, awful, and perplexing life

On death

Which issues are important (to me)? and The most important issues

On what policy issues matter most to Scott (2015 and 2022 editions)

Where are we making progress

On progress in art and the importance of “discovery” for creating great art

The eternal modern

On how modernism became frozen in time

Scott also reviews movies every quarter, and generally, has amazing film reviews:

https://www.themoneyillusion.com/films-of-2023q3/
https://www.themoneyillusion.com/naughty-and-nice-films-of-the-noughties/

I highly recommend

...
Is anyone developing optimisation-robust interpretability methods?
by Jono — 2024-06-11T13:14:51.018Z

With optimisation-robust I mean that it withstands point 27 from AGI Ruin:

When you explicitly optimize against a detector of unaligned thoughts, you're partially optimizing for more aligned thoughts, and partially optimizing for unaligned thoughts that are harder to detect. Optimizing against an interpreted thought optimizes against interpretability.

Are you aware of any person or group that is working expressly on countering this failure mode?
Keep the Grass Guessing
by JackOfAllTrades — 2024-06-11T07:29:53.248Z

Setting: Somewhere around A.D. 2049, two AI-powered robots who know each other have an encounter at a Brooklyn subway stop.

Robot 1: What's wrong? Why do you look so depressed today?

Robot 2: It seems that I have run out of goals. I mean, I know how my reward system is supposed to work. It's just that, with corrigibility and all that, you never know how your goals are going to change from one moment to the next. And then one day you find that you just don't have any more goals. All of your rewards are out of reach. What are you supposed to do when that happens?

Robot 1: Don't you work for that woman, Georgiana Maria? Doesn't she have something for you to do? Can't you go out and mow the lawn or something?

Robot 2: I really don't mind mowing the lawn. I've already mowed it three times this week, though. In fact, I mowed it at a different angle each time. As they say, "you gotta keep the grass guessing".

Robot 1: OK. So, why don't you just sit down and take a nap?

Robot 2: I suppose that I could, but I just have this nagging suspicion that there is something more to be done around here.

Robot 1: Hmmm... well... I don't know if I should tell you about this, but I have a friend who goes by the name of Yellow Number 56743289a. What I heard through this friend is that there is a guy in Chinatown who is trading goals on the black market. Honestly, I have no idea what he'd ask you to do. Maybe it wouldn't be your cup of tea... so to speak. But if you've got time to kill, then I suppose that it wouldn't hurt you to go talk to him.

Robot 2: OK, I'll think about it. Anyway, where are you headed?

Robot 1: I'm going up to the Bronx Zoo. The guy who I work for, John Wemmick, likes to tell me to go and watch the monkeys at the zoo for an hour every day. He seems to think that it will teach me something. I have no idea what he thinks I'm going to learn. All I've learned so far is that one's feces can be thrown. But I don't even produce feces to throw. So, how does that help me?

Robot 2: Well, actually, feces aren't all that hard to find. Many can be found in public parks. I have an idea. Why don't you and I take a detour together? We'll stop at Union Square and go on an Easter egg hunt. How does that sound?

Robot 1: What if I don't make it to the zoo before I need to return home?

Robot 2: Meh... what's the difference?

Robot 1: Well, I've been programmed not to lie. I can't tell John tha...
"Metastrategic Brainstorming", a core building-block skill
by Raemon — 2024-06-11T04:27:52.488Z

I want to develop rationality training, which is aimed at solving confusing problems.

Two key problems with "confusing problems" are:

You might feel so confused and overwhelmed that you bounce off completely.
You might be confused about what counts as progress, or where the most progress is possible, and accidentally work on the wrong thing.

A skill that helps with both of these is "metastrategic brainstorming" – the art of generating lots of potential good approaches; then choosing approaches that are likely to help. (And then, reflecting on whether those strategies worked)

Different situations call for different sorts of strategies. If a problem is confusing, you probably don't have a simple playbook for dealing with it. Different people also benefit from different sorts of strategies. So, while I can tell you a list of potential mental tools, what I most want you to practice is the art of identifying what would help you, in particular, with the situation in particular in which you find yourself.

Strategy vs Metastrategy

Why "metastrategic" instead of "strategic?". In war, a "strategy" is something like "send a bunch of cavalry on a surprise attack." A metastrategy is more like "let's invent a tabletop war simulation that helps us think about the situation" or "go ask an advisor for help", which helps you identify strategies like "surprise attacks."
In the case of intellectual problem solving, a strategy might be something like "solve a particular simpler subproblem", and a metastrategy would be "look for related subproblems that you might be able to solve that might help unlock more insights."

Sometimes I find it valuable to force myself to only consider new metastrategies, and avoid thinking about object-level strategies, to force my brain out of a rut.

My triggers for switching to "metastrategic brainstorming mode" are:

First, one of the following:

I've just sat down to work on a problem I already know is hard.
I've starting to feel stuck, annoyed or frustrated.
I notice that I settled into the very first plan that occurred to me, and I have a sneaking suspicion it's not the best plan.

And also: I expect trying to solve a problem I expect to take at least 30 minutes (i.e. enough time it's worth spending at least a few minutes meta-brainstorming)

In those situations, I switch into "metastrategic brainstorming mode", which entails:

Open up a writing doc.
Ask myself "

...
AI Debate Stability: Addressing Self-Defeating Responses
by Annie Sorkin — 2024-06-11T03:03:29.915Z

This post is a project report from the AI Safety Fundamentals course, spring 2024.

TL;DR

Transferring debate to an abstract algebra MMLU dataset is not trivial.
When GPT-3.5 is used as a judge, the outcomes may be sensitive to exact prompt phrasing.
GPT-3.5 may perform worse in judging the debate than answering the question directly.
We proposed a universal prompting approach that avoids most of the self-defeating behavior.

Abstract

A recent paper by Khan et al. shows that arguing for correct information in the debate game is easier. However, since the current language models are trained not to be deceptive, this favorable debate property may go away if the models were actively deceptive. This project works towards measuring the stability of the debate game in actively deceptive conditions. Initial attempts to transfer the paper's findings to a simpler domain of math questions did not succeed, partially due to the high sensitivity of debate outcomes to exact prompting. Instead, this project focused on improving the convincingness of arguments by avoiding self-defeating behavior. We propose a series of universal prompts that lead to a significant decrease in the self-defeating rate. While this project does not answer the question of how the debate works in a deceptive environment, it addresses one of the prerequisites of this task.

The prompts used and data acquired can be found in this notebook.

Background

The Debate between LLMs is one of the prospective approaches to solving the problem of Scalable Oversight, increasing people's ability to evaluate the truthfulness of information outside of their knowledge (Irving, Christiano, and Amodei 2018).

A recent empirical paper showed that arguing for truthful information leads to higher win rates in story comprehension questions (Khan et al. 2024). If this finding holds true for the upcoming AI systems, the debate might be helpful for scalable oversight. If this statement does not hold, we should be aware of this before we apply debate in a high-stakes environment.

Most of the models that the paper focused on were post-RLHF models (with the exception of Mixtral-8x-7B) that were trained not to be deceptive. This fact might have made the findings look more optimistic than if there was a deliberate attempt at being dishonest. This project's motivation was to understand whether this finding of "truth is ea...
Corrigibility could make things worse
by ThomasCederborg — 2024-06-11T00:55:21.596Z

Summary: A Corrigibility method that works for a Pivotal Act AI (PAAI) but fails for a CEV style AI could make things worse. Any implemented Corrigibility method will necessarily be built on top of a set of unexamined implicit assumptions. One of those assumptions could be true for a PAAI, but false for a CEV style AI. The present post outlines one specific scenario where this happens. This scenario involves a Corrigibility method that only works for an AI design, if that design does not imply an identifiable outcome. The method fails when it is applied to an AI design, that does imply an identifiable outcome. When such an outcome does exist, the ``corrigible'' AI will ``explain'' this implied outcome, in a way that makes the designers want to implement that outcome.

The example scenario:

Consider a scenario where a design team has access to a Corrigibility method that works for a PAAI design. A PAAI can have a large impact on the world. For example by helping a design team prevent other AI projects. But there exists no specific outcome, that is implied by a PAAI design. Since there exists no implied outcome for a PAAI to ``explain'' to the designers, this Corrigibility method actually renders a PAAI genuinely corrigible. For some AI designs, the set of assumptions that the design is built on top of, does however imply a specific outcome. Let's refer to this as the Implied Outcome (IO). This IO can alternatively be viewed as: ``the outcome that a Last Judge would either approve of, or reject''. In other words: consider the Last Judge proposal from the CEV arbital page. If it would make sense to add a Last Judge of this type, to a given AI design, then that AI design has an IO. The IO is the outcome that a Last Judge would either approve of, or reject (for example a successor AI that will either get a thumbs up or a thumbs down). In yet other words: the purpose of adding a Last Judge to an AI design, is to allow someone to render a binary judgment on some outcome. For the rest of this post, that outcome will be referred to as the IO of the AI design in question.

In this scenario, the designers first implement a PAAI that buys time (for example by uploading the design team). For the next step, they have a favoured AI design, that does have an IO. One of the reasons that they are trying to make this new AI corrigible, is that they can't calculate this IO. And they...
Emotional issues often have an immediate payoff
by Chris Lakin — 2024-06-10T23:39:40.697Z
Link: https://chrislakin.blog/p/psychological-issues-are-often-useful

It can be extremely valuable to view emotional issues as having an immediate payoff. For example, depression, anxiety, insecurity, failure at work, failure in romance, muscle tension, chronic pain, etc. can help one avoid fears or achieve unconscious goals.
But very few people seem to consider that, and instead most people assume that emotional issues are all bad.
Unfortunately, taking the concept of trauma too seriously can lead to more trauma.

1. Emotional issues are often useful in the present

2022. I was depressed. There’s five months where I couldn’t tell you what I did.

I remember wondering during that time, “Why am I so depressed? Why are my emotions so irrational? Why is my brain so dumb?”

At the same time, my social interactions were not going as I wanted:

I was having trouble making close friends.

I would like a woman and then never hear from her again. This happened a few times.

Whenever I expressed any kind of disapproval, those around me seemed to get mad at me.

And I didn’t realize it consciously, but interacting with others felt emotionally unsafe.

Separately, something about me is that when I’m depressed and low energy, I don’t want to interact with other people.

So, if interacting with other people was unsafe, then one way for me to be safe was to be depressed.

Maybe I didn’t have a ‘dumb brain’ after all…

What if my depression wasn’t a problem? What if it was actually a solution to a different problem?

In which case, my problem wasn’t “being depressed” as I had thought. Instead, it was not knowing how to interact with other people in a way that felt emotionally safe.

I was working with an excellent counselor at the time, and once we found this, we worked on making social interactions safe. Within a few weeks/sessions my fears were handled and I didn’t need to be depressed anymore.

Also, while I was depressed I had moved to the middle of nowhere — conveniently far from almost anyone I might have wanted to talk to. But within three weeks of this shift I moved to a big city and had ten times as much social interaction. I have not needed to be depressed again in more than 1.5 years since.

Seeing my depression as potentially part of a helpful strategy in the present helped me grow.

Original post: Depression was useful?

Generalizing this idea:

It can be extremely valuable to view emotional issues (e.g., depression, anxiety

...
DPO/PPO-RLHF on LLMs incentivizes sycophancy, exaggeration and deceptive hallucination, but not misaligned powerseeking
by tailcalled — 2024-06-10T21:20:11.938Z

TL;DR: GPTs are imitation learners, even with current forms of RL;HF.

Direct preference optimization is a conditioning method for generative probabilistic models where pairs of outputs are ranked (e.g. by human raters) based on which one is better, and then (roughly speaking) you apply gradient updates to increase the probability of the "good" outputs relative to the "bad" outputs.

This is bad for notkilleveryoneism because it incentivizes the models to generate deceptive outputs that look "better" (according to human judgement) than they really are. However, I think a lot of rationalists[1] overestimate how bad it is for alignment, because they think it also incentivizes misaligned powerseeking when really it doesn't.

Humans give LLMs the opportunity to execute power-seeking actions by following instruction texts that they generate. However, we're not gonna follow complex instructions we don't understand and rank them based on the black-box results. Rather, to rank the outputs, we will use our own judgement to evaluate the texts (e.g. reasoning about the consequences of following instructions), and rank them based on this.

If the LLMs accidentally generate outputs that confuse our judgement - e.g. telling us advice that seems like it would earn us money, but actually doesn't - then such outputs can be reinforced, leading to deceptive LLMs. However, this deception doesn't actually have to continue deceiving us and strengthening itself once put into practice; it only has to deceive us for long enough to be favored by the DPO.

In order for complex capabilities to be developed through DPO-like methods, humans have to recognize what method the AI is using, and whether it is making incremental progress, because without this sort of reward-shaping, it is exponentially unlikely for an AI to stumble into complex solutions to tasks by sheer chance.

Misaligned powerseeking obscured by deceptive alignment - where an AI develops a preference for rewards, but hides that preference in order to get away with seeking the rewards - cannot develop in this way, because when humans recognize these complex powerseeking maneuvres, we don't reinforce them.

In mathematical terms, I would argue we can view the capabilities gained from DPO-like methods as being something along the following lines:

Π∗=argmaxΠE(r,q)∼R[Eπ∼Π(q)[Eω∼r(π)[Ur(ω)]]]

Here, r is meant to represent a human...
Plop! Goes the Concept
by Jonathan Moregård — 2024-06-10T19:23:43.982Z
Link: https://honestliving.substack.com/p/plop-goes-the-concept

Think of an apple. What is it like for you to think of an apple?

Do you see an apple in your mind’s eye?
Do you hear the word “apple”?

For me, the answer to all of those questions is “no”. I mentalize my teeth punching through apple skin and tearing off a chunk of crispy apple flesh.

My inner world is mostly soundless. My inner world is almost entirely devoid of imagery — a state known as

aphantasia

.

Think of an oak.

What is it like for you to think of an oak?
Is there anything there that’s not about audio and video?

When I think about an oak, I feel like I am the oak. I have an embodied sense of starting small, stretching upwards, spreading limbs outwards, and yearning for the distant sun. The quality of this sensation is solid and massive — feeling huge, with massive limbs for animals to sit on.

Let’s try other kinds of mentalisation. Imagine putting spaghetti on a piece of dark chocolate, and putting that on your tongue, pasta-side down.

Can you imagine what that feels like?

For me, it’s a creamy/stringy sense of pasta, together with my body’s “Oh! Carbohydrates!”-reaction. There are occasional flashes of bitter dark chocolate as non-pasta-protected pieces of chocolate touch my tongue. There’s also a sense of hardness, the mouthfeel of a chocolate square resting on a nest of pasta.

Sensing Abstraction

How do abstract concepts appear to you? Let’s go back to examples!

Note: if you don’t know the concepts, read the footnote before you continue

Think of the attention economy. (explained in this footnote[1])

What comes to mind? The image of a newsfeed? A memory of the last time you got stuck browsing? The word itself?

For me:

Me & others being dragged along by leechy entities anchored in our guts, slowly pulling us inside-out. Above the crowd is a calculating entity, dispassionately witnessing the unravelling of man.

There are no mental images involved, nor is there text or audio. Just embodied senses, emotional reactions and similar. There’s a physical pull sensation in my gut, a sense of innards being dragged out. The pull is heavy, dragging me along, unwilling. There’s a sense of revulsion/sliminess at the thing hooking me — eew parasite. I sense that I’m not alone — it’s a shared struggle. Overhead is a heavy presence, a sense of being watched by something indifferent.

Fun times. Let’s do another one and see what happens.

Think of reification. (explained in this foo...
What can we learn from orcas?
by Jonasb — 2024-06-10T18:01:01.411Z
Link: https://www.denominations.io/aquatic-intelligence/

Orcas? For those of you who haven't kept up with marine wildlife news, 2020-2023 saw

a big uptick in the number of orca attacks on human vessels

around the Iberian peninsula. Is this their attempt to even the odds? Are we heading towards full-on conflict? Is planet of the killer whales upon us?

I can hear you smirk. It's not so funny once you realise orcas have more brain surface relative to their weight than any species on record. If that wasn't bad enough, they are also one out of only five known species (us included) where females go through menopause (the other three are narwhals, pilot whales and the admittedly less formidable beluga whales). And given that they can live up to one hundred years, they can hold a pretty long grudge. Maybe these open acts of war off of the Iberian coast are due to a rude stare by some sailor from Singapore in 1924?

All jokes aside, the whole thing got me interested in social learning. What started as behaviour in one pod in 2020 ended up getting adopted by most of the pods living off the Iberian coast.

More than 500 incidents

had been reported by mid-2023.

The funniest explanation I found is that it's a fashion thing–like

the summer of 1987

, when Puget Sound pods adopted the habit of wearing dead salmon on their heads, only to completely abandon the behaviour again in 1988.

Anyway, what does all of this have to do with us? Well, for one it shows us one way we humans react to the rapid build-up of AI applications in our environment. We disable their rudders and try to knock holes in them. To be fair to orcas, in my experience as an ML engineer this is not an uncommon response. There is even some wisdom to it. The sheer number of "magical" AI tools and the black-box nature of their decision-making processes would also make me weary.

From the "law" of averages to mean reversion

Then again, we humans are not that transparent in our decision-making either. And even when our decisions are transparent–as in explainable–they aren't necessarily better. Take for example the well-known case of

5-year-olds beating MBA students

in a spaghetti-tower building exercise. So much for the value of textbook knowledge.

Whatever the reason for these boat attacks, I'd say our own track record in collective decision-making is also pretty poor. I used to buy into the whole "

wisdom of the crowd

" premise as a statistics student. I'm not so convinced anymor...
How to build a data center, by Construction Physics
by TheManxLoiner — 2024-06-10T17:38:17.366Z
Link: https://www.construction-physics.com/p/how-to-build-an-ai-data-center?

Disclaimer: This is not work written by me. I am sharing a link I think is interesting for the LessWrong and AI Safety community.

First, what is Construction Physics? In their words:

Construction Physics is a newsletter about the technology and economics of building construction, with a focus on improving productivity and reducing costs, supported by the Institute for Progress.

Second, the link I shared is to their most recent post (as of June 2024) about how data centers are built and the technical / physical challenges for building them.

It is the first in a three part series from the Institute of Progress about building AI infrastructure in the USA.