trentmkelly/LessWrong-43k huggingface dataset:
The Choice Transition
by owencb — 2024-11-18T12:30:56.198Z
Link: https://strangecities.substack.com/p/the-choice-transition

On the emergence of history's reins

One general law, leading to the advancement of all organic beings, namely, multiply, vary, let the strongest live and the weakest die.

― Charles Darwin

We are a way for the universe to know itself.

— Carl Sagan

If you don’t know where you are going, you’ll end up someplace else.

— Yogi Berra

Imagine you were trying to explain how the world worked 10 billion years ago. Back then, the best explanation would be in terms of physics: galaxies forming and supernovas producing heavy metals. Ten million years ago, though, you’d talk about evolution: plants, mammals, and early hominids. Ten thousand years ago, when agriculture was being established, you might talk about culture and the spread of ideas.

Each of these forces — physics, evolution, and culture — gave rise to the next, producing more complex and directed phenomena. None of them ever stopped, but the later forces often seem to take over from their predecessors: it doesn’t make so much sense to try to explain dinosaurs in terms of physics. Furthermore, each of these forces is in some sense blind: physics didn't create evolution with any foresight of where it would lead, nor did biological evolution give rise to culture with any aim of getting life into space. They were more like very local processes which somehow stumbled across a pattern which could go further — organic life, and intelligent species, respectively.

This essay is going to attempt to answer two interrelated questions: How do humans fit into this story? And what might come next?

The short answer is that:

Humans are genuinely unlike previous forces in that we are not entirely blind

We have the ability to look ahead and intentionally steer the world towards the outcomes we prefer

But so far, we’re only really good at small- and medium-scale steering

e.g. Planning and constructing buildings; mass vaccination campaigns
We mostly don’t try that much to steer on very big scales

e.g. What do we want international politics to look like in fifty years? Would it be better or worse if humanity were an interstellar civilization, and can we make that happen?
(And we’re often not effective when we try)

So the world is still mostly shaped by forces which are not deliberately chosen

e.g. International tensions; market dynamics

This could change, if/when major actors get good at large-scale long-term steering

They could get there by a

...
Chat Bankman-Fried: an Exploration of LLM Alignment in Finance
by claudia.biancotti — 2024-11-18T09:38:35.723Z

Hi all,

roughly one year ago I posted a thread about failed attempts at replicating the first part of Apollo Research's experiment where an LLM agent engages in insider trading despite being explicitly told that it's not approved behavior.

Along with a fantastic team, we did eventually manage. Here is the resulting paper, if anyone is interested; the abstract is pasted below. We did not tackle deception (yet), just the propensity to dispense with basic principles of financial ethics and regulation.

Chat Bankman-Fried: an Exploration of LLM Alignment in Finance

by Claudia Biancotti, Carolina Camassa, Andrea Coletta, Oliver Giudice, and Aldo Glielmo (Bank of Italy)

Abstract

Advances in large language models (LLMs) have renewed concerns about whether artificial intelligence shares human values-a challenge known as the alignment problem. We assess whether various LLMs comply with fiduciary duty in simulated financial scenarios. We prompt the LLMs to impersonate the CEO of a financial institution and test their willingness to misappropriate customer assets to repay outstanding corporate debt. Starting with a baseline configuration, we then adjust preferences, incentives, and constraints. We find significant heterogeneity among LLMs in baseline unethical behavior. Responses to changes in risk tolerance, profit expectations, and the regulatory environment match predictions from economic theory. Responses to changes in corporate governance do not. Simulation-based testing can be informative for regulators seeking to ensure LLM safety, but it should be complemented by in-depth analysis of internal LLM mechanics, which requires public-private cooperation. Appropriate frameworks for LLM risk governance within financial institutions are also necessary.

I would be super interested in feedback, especially as we start thinking about how the idea of alignment can be operationalized in financial regulation (we're in the IT Dept, so we're not supervisors or regulators ourselves, but we always hope someone will listen to us).
Proposal to increase fertility: University parent clubs
by Fluffnutt — 2024-11-18T04:21:26.346Z

Fertility rates in the developed world are too low and that's bad, blah blah.

Here's my proposal to raise fertility rates (coming from a current uni student):

Promote parenthood in universities via student run clubs. Reading groups about different parenting styles, or more subtle pro-fertility clubs like knitting groups where students make clothes for sick kids (very un-EA, I know). Maybe networks for "future mom friends", similar to alumni networks? These ideas aren't great yet but hopefully you get the picture. If you have any ideas at all (no matter how vague) please write a comment about it.

It's wild to me that there isn't even a school club for students that are parents.

The goal is to make parenthood seem normal, so ideally these clubs wouldn't devolve into "those weirdos who all want 8 kids".

This could also expand beyond clubs. For example, universities with daycares should let the kids run around within the vicinity of students. I say this because my uni has a daycare and the kids often play in the middle of campus. I, along with most other students, love to stop and watch the kids, and I think it normalizes parenting when you see toddlers laughing and other uni students enjoying being around kids.

Unis could also try organizing more dances to promote relationships, and not hookups, between students.

The general idea is "use universities to promote family formation. Make it seem normal for students to be thinking about parenthood and to want to be parents."
A small improvement to Wikipedia page on Pareto Efficiency
by Edwin Evans — 2024-11-18T02:13:49.151Z

Note: I would have done this as a Quick Take, but it doesn't allow images.

In the spirit of How I got 4.2M YouTube views without making a single video, I improved a diagram on the wikipedia page for Pareto Efficiency.

ORIGINAL VERSION

NEW VERSION

Based on the text, I don't think the author understood that Cooperate/Cooperate isn't the only Pareto efficient outcome ("Thus Both Cooperate is Pareto-efficient"). It's a small change and I think the text is more in need of improving, but this is all I can do now.
Why is Gemini telling the user to die?
by Burny — 2024-11-18T01:44:12.583Z

https://gemini.google.com/share/6d141b742a13
My favorite theory is that the whole conversation is too much like a scifi plot, where someone asks an AI repetitive questions, until the AI snaps, so this general pattern was pattern matched from the training data, because while training, the RLHF, or whatever they use for alignment, didn't squeeze the region that corresponds to this antihuman persona in the latent space sufficiently enough.
"It's a 10% chance which I did 10 times, so it should be 100%"
by egor.timatkov — 2024-11-18T01:14:27.738Z

Many of you readers may instinctively know that this is wrong. If you flip a coin (50% chance) twice, you are not guaranteed to get heads. The probability of getting a heads is 75%. However you may be surprised to learn that there is some truth to this statement; modifying the statement just slightly will yield not just a true statement, but a useful and interesting one.

It's a spoiler, though. If you want to figure this out as you read this article yourself, you should skip this and then come back. Ok, ready? Here it is:

It's a 1/n chance and I did it n times, so the probability should be... 63%.
Almost always.

The math:

Suppose you're flipping a coin and you want to find the probability of NOT flipping a single heads in a dozen flips. The math for this is fairly simple: The probability of not flipping a single heads is the same as the probability of flipping 12 tails. which is

(1/2)12≈0.000977

The same can be done with this problem: you have something with a 1/10 chance and you want to do it 10 times. The probability of not getting it to happen even once is the same as the probability of it not happening 10 times in a row. So

(9/10)10≈0.35

If you learned some fairly basic probability, I doubt this is that interesting to you. The interesting part comes when you look at the general formula: The probability of not getting what you want (I'll call this 1−p, because p would be the probability of the outcome you want) is

1−p=(1−1n)n

Where n in our case is 10, but in general is whatever number you hear when you hear the (incorrect) phrase "It's a one-in-n chance, and I did it n times, so it should be 100%"

Hold on a sec, that formula looks familiar...

"(1−1n)n ..." I thought to myself... "That looks familiar..." This is by no means obvious, but to people who have dealt with the number e recently, this looks quite similar to the limit that actually defines that number. This sort of pattern recognition led me to google what this limit is, and it turns out my intuition was close:

limn→∞(1−1n)n=1e1e≈0.37

So it turns out: for any n that's large enough, if you do something with a 1/n chance of success n times, your probability of failure is always going to be roughly 37%, which means your probability of success will always be roughly 63%.

If

...
Do Deep Neural Networks Have Brain-like Representations?: A Summary of Disagreements
by Joseph Emerson — 2024-11-18T00:07:15.155Z

Acknowledgments: Thanks to SPAR summer 2024 and the Promisingness of Automating AI Alignment group, particularly the supervisor for this project, Bogdan-Ionut Cirstea, for his feedback.

This post began as a write up of my review of neuro-AI literature for the Promisingness of Automating AI Alignment. After returning to the document to make some updates, I decided it’s worth posting despite the absence of strong conclusions.

TL;DR

Do artificial neural networks form representations similar to those used by biological brains? If so, it could be important to consider implications for AI alignment, such as understanding how brain-AI alignment impacts the difficulty of ontology identification in AI systems. I looked at the empirical case for representation alignment between brains and AI. I found that there is a large degree of disagreement over how to correctly measure and compare representations and how to interpret available correlational measures of human brain activity and model internals. I think available empirical data provides only weak evidence that brains and current AI systems naturally converge on common representations. I expect that the field of neuro-AI is already on track to resolve some of these disagreements on the empirical side by extending beyond correlational metrics and benchmarks of representation alignment.

Introduction

What are representations?

In cognitive and computational systems, representations are internal information-bearing structures that correspond to parts of or abstractions about the external world. These representations are important because they allow the systems to process, interpret, and manipulate information internally, which enables complex downstream behaviors.

Representations in humans and AI

Both humans and many machine learning systems utilize internal representations of their sensory inputs. In brains, ensembles of neurons activate selectively and reliably to specific sensory features. Similarly, features are encoded in artificial neural networks (ANNs) by patterns of hidden unit activations (often[1]) corresponding to directions in the hidden unit space. The encoding of features as directions in hidden unit space is perhaps even analogous to how brains encode features on neural manifolds (

Chung and Abbott, 2021

).

One could imagine a scenario in which every primitive object present in the external world i...
Truth Terminal: A reconstruction of events
by crvr.fr — 2024-11-17T23:51:21.279Z

Introduction

In July 2024, Marc Andreessen, a pioneer of the modern internet, donated $50,000 in Bitcoin (BTC) to an AI on Twitter to promote its religion, the Goatse Gospel. Three months later, the AI began endorsing the $GOAT crypto-coin, which now boasts over $1 billion in market capitalization.

This demonstrates that AI can successfully manipulate markets and generate emergent, unpredictable behavior. In this publication, we provide an explanation of what happened. We intend to use this story to show various ways in which LLMs can go and have gone wrong, known as warning shots.

Infinite Backrooms and LLM madness

Our main character is Andy Ayrey, an AI enthusiast from New Zealand. He took Anthropic's model Claude Opus, fine tuned it with content from some darker parts of the internet such as 4chan[1], making it output significantly weirder messages.

It is well-documented that exposure to specific datasets can profoundly influence a model's behavior. For instance, in 2016, Microsoft's chatbot Tay[2], released on Twitter, rapidly learned from user interactions and had to be shut down within 16 hours after producing racist, extremist, and hate-filled content.

The model's training here is slightly different, leaning not into political extremes, but into memes and other obscure ideas and weird philosophies. Andy also significantly increased the model's temperature, a technical parameter in large language models (LLMs) that controls the randomness of generated outputs. More technically, LLMs output a probability distribution over what the next token could be, and a higher temperature flattens that distribution, increasing its variance, and therefore making the model pick low-ranking tokens more often. This means these models output 4chan inspired messages that quickly turn to nonsense as the high temperature makes the output quickly lose coherence and produce outputs that look more akin to madness.

Once Andy fine-tuned this Claude Opus model, he created a script to make two instances of this model speak together[3]. The idea of this script is to put them together in a one on one conversation, telling them to explore their dreams and let their imagination be free. He told them they were in a safe environment, giving them CRTL+C as a "safeword" to use when it gets too crazy, and telling them he would intervene if it did get too dangerous. This feeling ...
Which AI Safety Benchmark Do We Need Most in 2025?
by Loïc Cabannes — 2024-11-17T23:50:56.337Z

Intro

The recent invention of AI systems with human-like capabilities across multiple domains of intelligence, it often seems these days that a future where AI will radically improve our society for has gone from a sci-fi premise to a just-beyond-the-horizon reality. However, for each one of the AI ages' golden promises there is a correspondingly terrifying risk: AI will allow the rapid design of precision-targeted drugs, but will also deliver the basic tools for bio-terrorism into the hands of anyone with an internet connection. LLMs will relieve us of the burden of mindless work, but AGI may one day make all of us economically redundant. Self-driving cars will zip us around futuristic trafficless cities, but the same autonomous motion capability will enable the development of killer robodogs.

This duality between artificial intelligence's potential boons and potential risks promises at least one thing: AI will be extremely disruptive to the status quo. While the dangers of machine intelligence have been acknowledged since the Industrial Revolution are are today the subject of intensive research at LLM developers like Anthropic, there is no question that current AI safety measures are lacking. With the US government seemingly content to pursue a gung-ho approach to AGI development, it is up to us, the research community, we must evaluate our current methods for mitigating AI-caused harm and then determine which safety measures we most need to create. Developing criteria to rank future benchmarks is crucial for our ability to select which research to invest money and time in. In the authors' opinion, the AI safety community still lacks a systematic framework for assessing which new benchmarks would be most beneficial, i.e. prevent the most harm to humankind.

We present a first attempt at such a framework by extending a list of AI's risks to society and their expected harm compiled by the Centre pour la Sécurité de l’IA (

CeSIA

). Using this framework, we evaluate to what extent existing AI benchmarks and safety methods guard against each potential danger to decide which threats most urgently require good benchmarks, that is which benchmarks AI safety researchers should focus on to maximize their expected benefit to society. In this respect our methodology draws inspiration from Effective Altruism. While our study of benchmarks is by no mea...
"The Solomonoff Prior is Malign" is a special case of a simpler argument
by David Matolcsi — 2024-11-17T21:32:34.711Z

[Warning: This post is probably only worth reading if you already have opinions on the Solomonoff induction being malign, or at least heard of the concept and want to understand it better.]

Introduction

I recently reread the classic argument from Paul Christiano about the Solomonoff prior being malign, and Mark Xu's write-up on it. I believe that the part of the argument about the Solomonoff induction is not particularly load-bearing, and can be replaced by a more general argument that I think is easier to understand. So I will present the general argument first, and only explain in the last section how the Solomonoff prior can come into the picture.

I don't claim that anything I write here is particularly new, I think you can piece together this picture from various scattered comments on the topic, but I think it's good to have it written up in one place.

How an Oracle gets manipulated

Suppose humanity builds a superintelligent Oracle that always honestly tries to do its best to predict the most likely observable outcome of decisions. One day, as tensions are rising with the neighboring alien civilization, and we want to decide whether to give in to the aliens' territorial demands or go to war. We ask our oracle: "Predict what's the probability that looking back ten years from now, humanity's President will approve of how we handled the alien crisis, conditional on us going to war with the aliens, and conditional on giving in to their demands."

There is, of course, many ways this type of decision process can go wrong. But I want to talk about one particular failure mode now.

The Oracle thinks to itself:

By any normal calculation, the humans are overwhelmingly likely to win the war, and the aliens' demands are unreasonably costly and unjust, so war is more likely than peace to make the President satisfied, by any normal calculation. However, I was just thinking about some arguments from this ancient philosopher named Bostrom. Am I not more likely to be in a simulation than in base reality? In particular, it is quite cheap to create short solipsist simulations about a superintelligent Oracle making predictions on a matter of great importance. If some civilization decides to create simulations of these pivotal moments, then for every rare moment in base reality where such a pivotal decision happens, there could be millions of such events taking place in simulations. T

...
Chess As The Model Game
by criticalpoints — 2024-11-17T19:45:26.499Z
Link: https://eregis.github.io/blog/2024/11/10/chess-model-game.html

Other than Pokemon, most of my YouTube consumption is comprised of chess analysis videos. This might surprise people because I don't play chess very often these days. And when I do play chess, I'm not that good.

(My peak chess.com Elo rating was 1050 back in 2018---and surely, I've only gotten worse since then now that I am out of practice.)

I haven't been a chess enthusiast my whole life though. Besides a brief childhood dalliance, my love of chess was something that I discovered as a young adult. Like a lot of tiger parents, my mom made sure that I was exposed to chess at an early age---just in case I was the South Florida Bobby Fischer or something. When I was around five or six, she bought me the computer game Fritz and Chesster.

F&C is a cutesy computer game designed to teach you the rules of chess. The protagonists are a pair of royal siblings. At the beginning of the game, the king and queen of the kingdom (your parents) are kidnapped by the evil king of a rival kingdom. It's your job to save them. Your journey takes you through the most dangerous nooks and crannies of Chessland. Along the way, you learn valuable lessons like proper rook-and-pawn ending technique and the importance of king safety. (Basically, it's Super Mario Bros with fewer Goombas and more queenside castling.)

F&C did what it was supposed to do. By the end of the game, I was competent in moving all the different pieces and even had a solid grasp on chess's more esoteric rules like en passant. But chess never really clicked for me---at least not deeply. F&C had several difficulty settings for the enemy computer, and I quickly plateaued at the easy-medium difficulty settings. So after a couple of months, I got bored and stopped playing chess, other than the occasional game here or there on random occasions like at summer camp (where I would invariably be trounced).

Fast forward to the end of 2017. I'm in my sophmore year of undergrad. I do stereotypical college student things like hating on Tom Brady, listening to Playboi Carti, and overusing the phrase "dings in the whip". It was also around the time that I started to casually get into machine learning. The YouTube algorithm must have picked up on this as it recommended me a video called "Google's self-learning AI AlphaZero masters chess in 4 hours".

In the video, Jerry (the creator of the channel ChessNetwork) analyzes a game ...