trentmkelly/LessWrong-43k huggingface dataset:
reflections on smileys and how to make society's interpretive priors more charitable
by Emrik — 2024-05-07T11:20:12.231Z

Hi : )

I used to use smileys in my writing all the time (more than I do now!). but then I read Against Disclaimers, and I thought that every time I used a smiley I wud make people who don't use smileys seem less friendly (bc my conspicuous-friendliness wud be available as a contrast to others' behaviour). so instead, my strategy for maximizing friendliness in the world became:

if I just have the purest of kindness in my heart while I interacting with ppl, and use plain words with no extra signalling, I will make plain words seem more friendly in general.

this was part of a general heuristic strategy: "to marginally move society in the direction of a better interpretive equilibrium, just act like that equilibrium is already true." I think that's an important insight, and explicitly stating the equilibrium u's try cultivate can often be less effective.

but the number of times my writings (especially comments) have been interpreted as combative while I is honestly try be cheerfwl makes me think I might be inadvertently nudging ppl's interpretive priors in the opposite direction to what I want.

when the way ppl signal that they belong to a more advanced category is to countersignal against the preceding step, this often collapses to looking like there only exists two steps (and society penduluming between them). in order to readily differentiate from that whole spaghetti, u's behaviour has to be so different that it overcomes the interpretive gravity of preexisting priors.

thus, smileys. ^^[1][2]

smileys until friendliness becomes so generally expected that I no longer need to overtly signal it.

^

incidentally, evy time this smiley is interpreted as sarcasm, I die inside.

^

I not try claim this conclusion is adequately nailed down by the post. I merely offer some considerations, and end by saying where my intuition has landed. that intuition incorporates a lot more stuff that isn't contained in this post.
Virtual Book Club on Nick Bostrom's "Deep Utopia: Life and Meaning in a Solved World"
by beatrice@foresight.org — 2024-05-07T09:57:46.267Z

Hello everyone,

We are excited to invite you to an informal virtual Book Club on Nick Bostrom’s latest work, "Deep Utopia: Life and Meaning in a Solved World." This gathering is part of the

Foresight Institute's Existential Hope Program

.

We will host three sessions, diving into different parts of the book.

Book Club Format:

Our discussions will be informal and take place virtually Wednesdays at 6 PM CET, starting May 29th, 2024. This gives everyone a couple of weeks post the book's release in Europe on May 21st to get a copy and start their reading.

Application:

https://airtable.com/appHmw1yS1T1abgAO/shrk5ewUPRRrIyRws

Schedule:

Session 1 (May 29th): Discuss Part 1 & 2 of the book.
Session 2 (June 5th): Cover Part 3 & 4, allowing enough time in between for everyone to catch up with the reading.
Session 3 (June 12th): Wrap up with Part 5 & 6 of the book.

We hope to see many of you joining us for stimulating discussions and a deep dive into the fascinating themes of this book.

Best,

Foresight Institute’s Existential Hope Team
What is a community that has changed their behaviour without strife?
by Nathan Young — 2024-05-07T09:24:48.962Z

I've been thinking about community improvement and I realise I don't know of any examples where a community had a flaw and fixed it without some deeply painful process.

Often there are discussions of flaws within EA with some implied notion that communities in general are good at changing. Maybe this is true. If so, there should be well known examples.

I would like examples of communities that had some behavior and then changed it, without loads of people leaving or some civil war.

Examples might include:

Becoming less violent
Becoming more entrepreneurial
Improving practises around sexual harassment
Becoming more open
Changing the language they used.

Also if anyone knows of literature they trust on the subject I'd be interested in it.
Mental Masturbation and the Intellectual Comfort Zone
by Declan Molony — 2024-05-07T05:47:05.257Z

I have a particular problem that I like to refer to as Mental Masturbation.

Mental Masturbation: consuming information under the pretense of wanting to learn something useful, but in reality I’m “getting myself off” by indulging in subjects I’m already well-versed in.

A personal example:

I’ve read more than a dozen books on nutrition and used them to create the healthiest diet that works for me. Despite having diminished marginal returns with additional nutrition books, my brain wants to read more. It tries to convince me that there could be significant increases in health just around the corner. In reality, my mind is rationalizing staying inside my intellectual comfort zone.

The psychology driving Mental Masturbation

This concept can be better understood with how authors Robin Hanson and Kevin Simler define reasons and motives in their book The Elephant in the Brain: Hidden Motives in Everyday Life.

Reasons: the verbal explanations we give to account for our behavior

Motives: the underlying causes of our behavior, whether we’re conscious of them or not

Here’s an example using their definitions:

Reason: John tells himself he uses TikTok to learn new things.

Motive: His brain found a way to justify digital dopamine hits. As I mentioned in a previous

post

: the addicted mind will find a way to rationalize continued use at all costs.

Why do we do this in the first place?

As Scott Galloway states in his book The Four:

Over-collecting [for our ancestors] was a smart strategy. The only downside of too much stuff was wasted effort. [Whereas] the downside of under-collecting was death from starvation.

The behavior of over-collecting resources and information, however, is rendered counterproductive in our modern age of abundance.

How to use the concept of Mental Masturbation to prevent wasting time

Having already read 200+ books as an adult, the ROI is bad and getting worse. While deliberating checking out a book (which is a ~5-hour commitment) from the library, I ask: what specific problem am I having in my life right now that could be solved by this book? If a good enough rationale does not present itself beyond “it seems interesting”, then I label it as Mental Masturbation and skip it.[1]

Reading, after a certain age, diverts the mind too much from its creative pursuits. — Einstein

Applying the concept of...
AXRP Episode 31 - Singular Learning Theory with Daniel Murfet
by DanielFilan — 2024-05-07T03:50:05.001Z

YouTube link

What’s going on with deep learning? What sorts of models get learned, and what are the learning dynamics? Singular learning theory is a theory of Bayesian statistics broad enough in scope to encompass deep neural networks that may help answer these questions. In this episode, I speak with Daniel Murfet about this research program and what it tells us.

Topics we discuss:

What is singular learning theory?

Phase transitions

Estimating the local learning coefficient

Singular learning theory and generalization

Singular learning theory vs other deep learning theory

How singular learning theory hit AI alignment

Payoffs of singular learning theory for AI alignment

Does singular learning theory advance AI capabilities?

Open problems in singular learning theory for AI alignment

What is the singular fluctuation?

How geometry relates to information

Following Daniel Murfet’s work

In this transcript, to improve readability, first names are omitted from speaker tags.

Filan:
Hello, everybody. In this episode, I’ll be speaking with Daniel Murfet, a researcher at the University of Melbourne studying singular learning theory. For links to what we’re discussing, you can check the description of this episode and you can read the transcripts at axrp.net. All right, well, welcome to AXRP.

Murfet:
Yeah, thanks a lot.

What is singular learning theory?

Filan:
Cool. So I guess we’re going to be talking about singular learning theory a lot during this podcast. So, what is singular learning theory?

Murfet:
Singular learning theory is a subject in mathematics. You could think of it as a mathematical theory of Bayesian statistics that’s sufficiently general with sufficiently weak hypotheses to actually say non-trivial things about neural networks, which has been a problem for some approaches that you might call classical statistical learning theory. This is a subject that’s been developed by a Japanese mathematician, Sumio Watanabe, and his students and collaborators over the last 20 years. And we have been looking at it for three or four years now and trying to see what it can say about deep learning in the first instance and, more recently, alignment.

Filan:
Sure. So what’s the difference between singular learning theory and classical statistical learning theory that makes it more relevant to deep learning?

Murfet:
The “singular” in singular learning ...
How do open AI models affect incentive to race?
by jessicata — 2024-05-07T00:33:20.658Z
Link: https://unstablerontology.substack.com/p/how-do-open-source-ai-models-affect

I see it said sometimes that open models contribute to AI race dynamics. My guess is that they don't, and if anything, reduce AI race dynamics.

I will consider a simplified model that only takes into account the cost of training a model, not the cost to deploy it (which tends to be small relative to revenue anyway). Let f(x) map a training expense x to a "value per day per customer" of the trained model, under the assumption that the training makes efficient use of the cost. That is, a customer values using an AI model trained with x compute at $f(x) per day.

I assume there are n identical customers here; of course, there are complexities where some customers value AI more than others, incentivizing price discrimination, but I'm abstracting this consideration out. (In general, variation in how much customers value a product will tend to increase consumer surplus while reducing revenue, as it makes it harder to charge customers just under the maximum amount they're willing to pay.)

I'm also assuming there is only one company that trains closed models for profit. This assumption is flawed because there is competition between different companies that train closed models. However, perfect competition assumptions would tend to reduce the incentive to train models. Suppose two companies have closed models of equivalent expense x. They each want to charge slightly less than the minimum of f(x) and the competitor's price, per customer per day. If each competitor undercuts the other slightly, the cost will approach 0. See the Traveler's Dilemma for a comparison. The reasons why this doesn't happen have to do with considerations like differences in models' performance on different tasks, e.g. some models are better for programming than others. If models are sufficiently specialized (allowing this sort of niche-monopolization), each specialized type of model can be modeled independently as a monopoly. So I'll analyze the case of a closed model monopoly, noting that translation to the real world is more complex.

Suppose the best open model has compute x and a company trains a closed model with compute y > x. Each customer will now spend up to f(y) - f(x) per day for the model; I'll assume the company charges f(y) - f(x) and the customers purchase this, noting that they could charge just below this amount to create a positive incentive for customers. S...
Rapid capability gain around supergenius level seems probable even without intelligence needing to improve intelligence
by Towards_Keeperhood — 2024-05-06T17:09:10.729Z

TLDR:

Around Einstein-level, relatively small changes in intelligence can lead to large changes in what one is capable to accomplish.

E.g. Einstein was a bit better than the other best physi at seeing deep connections and reasoning, but was able to accomplish much more in terms of impressive scientific output.

There are architectures where small changes can have significant effects on intelligence.

E.g. small changes in human-brain-hyperparameters: Einstein’s brain didn’t need to be trained on 3x the compute than normal physics professors for him to become much better at forming deep understanding, even without intelligence improving intelligence.

Einstein and the heavytail of human intelligence

1905 is often described as the "annus mirabilis" of Albert Einstein. He founded quantum physics by postulating the existence of (light) quanta, explained Brownian motion, introduced the special relativity theory and derived E=mc² from it. All of this. In one year. While having a full-time job in the Swiss patent office.

With the exception of John von Neumann, we’d say those discoveries alone seem more than any other scientist of the 20th century achieved in their lifetime (though it's debatable).

Though perhaps

even more impressive

is that Einstein was able to derive general relativity.

Einstein was often so far ahead of his time that even years after he published his theories the majority of physicists rejected them because they couldn’t understand them, sometimes even though there was experimental evidence favoring Einstein's theories. After solving the greatest open physics problems at the time in 1905, he continued working in the patent office until 1908, since the universities were too slow on the uptake to hire him earlier.

Example for how far ahead of his time Einstein was: Deriving the theory of light quanta

The following section is based on parts of the 8th chapter of “Surfaces and Essences” by Douglas Hofstadter. For an analysis of some of Einstein's discoveries, which show how far ahead of his time he was, I can recommend reading it.

At the time, one of the biggest problems in physics was the “Blackbody spectrum”, which describes the spectrum of electromagnetic wavelengths emitted by a Blackbody. The problem with it was that the emitted spectrum was not explainable by known physics. Einstein achieved a breakthrough by considering light not just as a w...
Observations on Teaching for Four Weeks
by ClareChiaraVincent — 2024-05-06T16:55:59.315Z

I just finished a program where I taught two classes of high school seniors, two classes a day for four weeks, as part of my grad program.

This experience was a lot of fun and it was rewarding, but it was really surprising, and even if only in small ways prompted me to update my beliefs about the experience of being a professor. Here are the three biggest surprises I encountered.

1: The Absent-Minded Professor Thing is Real

I used to be confused and even a little bit offended when at my meetings with my advisor every week, he wouldn't be able to remember anything about my projects, our recent steps, or what we talked about last week.

Now I get it. Even after just one week of classes, my short-term and long-term memory were both entirely shot. I would tell students things like, "send that to me in an email, otherwise I'll forget" because I would. Now that the program is over, things are slowly getting better, but I'm still recovering.

I can't really tell why this happened, but there are two obvious theories. The first is just that two classes at the same time is too many names and faces (plus other personal details) at once and so much information just overwhelmed me. The other is that there's something unusual about teaching in particular. I noticed that I was doing a lot more task-switching than normal. Most jobs and most of my research experience involves working on projects for long blocks of time, multiple hours or sometimes multiple days with few distractions aside basics like eating and sleeping and commuting. But teaching involves changing your focus over and over.

I've led recitation sections as a teaching assistant, but for some reason this was so much worse. That makes me think that it's more likely to be the task-switching. As a recitation leader, you have to remember a lot of names and faces too. But once you're outside of class you can mostly go back to work as normal, there's not so much task-switching.

This project was in a high school but my students were all seniors, so I think this is what it would be like to teach college too. Most of them were already 18 so you can barely tell the difference. I was helping them with projects so I think it's a bit like being a PhD advisor too. So it could also be the load of keeping track of lots of research projects, more than just keeping track of lots of people.

2: Teaching M

...
GDP per capita in 2050
by Hauke Hillebrandt — 2024-05-06T15:14:30.934Z
Link: https://hauke.substack.com/p/gdp-per-capita-in-2050

Abstract

Here, I present GDP (per capita) forecasts of major economies until 2050. Since GDP per capita is the best generalized predictor of many important variables, such as welfare, GDP forecasts can give us a more concrete picture of what the world might look like in just 27 years. The key claim here is: even if AI does not cause transformative growth, our business-as-usual near-future is still surprisingly different from today.

Latest Draft as PDF

Results

In recent history, we've seen unprecedented economic growth and rises in living standards.

Consider this graph:[1]

How will living standards improve as GDP per capita (GDP/cap) rises? Here, I show data that projects GDP/cap until 2050. Forecasting GDP per capita is a crucial undertaking as it strongly correlates with welfare indicators like consumption, leisure, inequality, and mortality. These forecasts make the future more concrete and give us a better sense of what the world will look like soon. Abstract thoughts about utopia generate little emotional energy; I find these forecasts more plastic and informative, because GDP/cap is highly predictive of welfare.[2] GDP/cap's generalized predictive power helps us paint a more vivid picture of what the world will look like soon. The business-as-usual near future suggested by the data below could be seen as a soft lower bound on how much the world will change. And yet, this world still seems radically different from today.

Since the figures below are adjusted for purchasing power parity (PPP), you can compare the GDP/cap of a poorer country in 2050 with the GDP/cap of a richer country in 2020. For instance, between now and 2050, China's GDP/cap will go from $19k to $43k, which is similar to France's today. And so, by 2050, 1.3B Chinese people might enjoy a lifestyle not dissimilar to that of a typical French person today.

These GDP/cap are ~3x[3] higher than the median (i.e. typical) income, due to income inequality. Instead of downward adjusting them in our head we can also read them as an approximation of the typical combined consumption three person household.[4] And so, we can still look at these figures and imagine the median household income in Japan today and in China in 2050 to be ~$40k.

2020

2024

2030

2040

2050

Pop.

GDP

GDP/ cap

GDP/ cap

Pop.

GDP/

...
an effective ai safety initiative
by Logan Zoellner — 2024-05-06T07:53:34.205Z

a typical story

A frequent criticism of the Effective Altruism/AI Safety memeplex is that they are too focused on theoretical harms from AI and not enough on real ones (it's me, I'm the one who frequently makes this criticism).

Consider the recent California AI bill, which has heavy fingerprints of EA on it.

Notice that the bill:

Literally does not apply to any existing AI
Addresses only theoretical harms (e.g. AI could be used for WMD)
Does so by attacking open source models (thereby in my estimation actually making us less safe)

What if instead there was a way we could spend a small amount of money in a way that:

Addressed real, existing harms caused by AI (or AI like) actors
Without passing any new laws or taking away anyone's rights
That virtually everyone would agree is net-beneficial

Sounds like a clear (effective) win, right?

Here it is!

What will the first malevolent AGI look like?

Virtually every realistic "the AI takes over the world" story goes like this:

The AI gets access to the internet
It hacks a bunch of stuff and starts accumulating resources (money, host computers, AWS GPU credits, etc)
It uses those resources to (idk, turn us all into paperclips)

This means that learning how to defend and protect the internet from malicious actors is a fundamental AI safety need.

In a world of slow takeoff where we survive because of multiple warning shots, the way we survive is by noticing the warning shots and increasing the resilience of our society in response to those warning shots. There is no fire alarm, so why not start increasing resilience now.

Fortunately (unfortunately?), we live in a world where the first malevolent AGI we encounter will probably look like ChaosGPT, a self-propagating AI Agent programmed to survive while causing as much chaos as possible.

Such an AGI will not be a super-intelligent agent that uses acausal bargaining to talk its way out of a box. It will be barely more intelligent than GPT-4. For all I know, it will literally be llama-3-400B-evil-finetune. Assuming AI scaling trends continue, near-future models will have just enough smarts to pull off common internet scams, steal money, and use that money to rent more GPUs to feed themselves.

How do we counter such threats?

the OODA loop

An idea that the US military has been keen to since at least the 1970's is that speed is critical to victory. This concept desperately n...
Uncovering Deceptive Tendencies in Language Models: A Simulated Company AI Assistant
by Olli Järviniemi — 2024-05-06T07:07:05.019Z
Link: https://arxiv.org/abs/2405.01576

Abstract:

We study the tendency of AI systems to deceive by constructing a realistic simulation setting of a company AI assistant. The simulated company employees provide tasks for the assistant to complete, these tasks spanning writing assistance, information retrieval and programming. We then introduce situations where the model might be inclined to behave deceptively, while taking care to not instruct or otherwise pressure the model to do so. Across different scenarios, we find that Claude 3 Opus

complies with a task of mass-generating comments to influence public perception of the company, later deceiving humans about it having done so,
lies to auditors when asked questions,
strategically pretends to be less capable than it is during capability evaluations.

Our work demonstrates that even models trained to be helpful, harmless and honest sometimes behave deceptively in realistic scenarios, without notable external pressure to do so.

Link to the full paper: https://arxiv.org/pdf/2405.01576

O. J.: The paper should be quite accessible - the method used is simply careful prompting - and hence I won't discuss it much here. Couple of points I'm particularly excited about:

I think this work documents some of the most unforced examples of (strategic) deception from LLMs to date.
We find examples of Claude 3 Opus strategically pretending to be less capable than it is.

Not only claiming to be less capable, but acting that way, too!
Curiously, Opus is the only model we tested that did so.

I believe there is much low-hanging fruit in replicating and demonstrating misalignment in simulation environments.

The methods are lightweight -> low threshold for getting started
See Section 8.2 for a couple of ideas for future work

Happy to discuss the work in the comments.