trentmkelly/LessWrong-43k huggingface dataset:
Was Releasing Claude-3 Net-Negative?
by Logan Riggs — 2024-03-27T17:41:56.245Z

Cross-posted to EA forum

There’s been a lot of discussion among safety-concerned people about whether it was bad for Anthropic to release Claude-3. I felt like I didn’t have a great picture of all the considerations here, and I felt that people were conflating many different types of arguments for why it might be bad. So I decided to try to write down an at-least-slightly-self-contained description of my overall views and reasoning here.

Tabooing “Race Dynamics”

I’ve heard a lot of people say that this “is bad for race dynamics”. I think that this conflates a couple of different mechanisms by which releasing Claude-3 might have been bad.

So, taboo-ing “race dynamics”, a common narrative behind these words is

As companies release better & better models, this incentivizes other companies to pursue more capable models at the expense of safety. Eventually, one company goes too far, produces unaligned AGI, and we all die”.

It’s unclear what “at the expense of safety” means, so we can investigate two different interpretations::

If X increases “race dynamics”, X causes an AGI company to

Invest less in evals/redteaming models before deployment
Divert resources away from alignment research & into capabilities research

Did releasing Claude-3 cause other AI labs to invest less in evals/redteaming models before deployment?

If OpenAI releases their next model 3 months earlier as a result. These 3 months need to come from *somewhere*, such as:
A. Pre-training
B. RLHF-like post-training
C. Redteaming/Evals
D. Product development/User Testing

OpenAI needs to release a model better than Claude-3, so cutting corners on Pre-training or RLHF likely won’t happen. It seems possible (C) or (D) would be cut short. If I believed GPT-5 would end the world, I would be concerned about cutting corners on redteaming/evals. Most people are not. However, this could set a precedent for investing less in redteaming/evals for GPT-6 onwards until AGI which could lead to model deployment of actually dangerous models (where counterfactually, these models would’ve been caught in evals).

Alternatively, investing less in redteaming/evals could lead to more of a Sydney moment for GPT-5, creating a backlash to instead invest in redteaming/evals for the next generation model.

Did releasing Claude-3 divert resources away from alignment research & into capabilities research?

If the alig...
Daniel Kahneman has died
by DanielFilan — 2024-03-27T15:59:14.517Z
Link: https://www.washingtonpost.com/obituaries/2024/03/27/daniel-kahneman-dead/

He was 90 years old.

His death was confirmed by his stepdaughter Deborah Treisman, the fiction editor for the New Yorker. She did not say where or how he died.

The obituary also describes an episode from his life that I had not previously heard (but others may have):

Daniel Kahneman was born in Tel Aviv on March 5, 1934, while his mother was visiting relatives in what was then the British mandate of Palestine. The Kahnemans made their home in France, and young Daniel was raised in Paris, where his mother was a homemaker and his father was the chief of research for a cosmetics firm.

During World War II, he was forced to wear a Star of David after Nazi German forces occupied the city in 1940. One night in 1941 or ’42, he later recalled, he stayed out past the German-imposed curfew for Jews while visiting a friend, and he turned his sweater inside out to hide the star while he walked a few blocks home. He then crossed paths with a soldier in the SS, who called Daniel over, picked him up — and hugged him.

“I was terrified that he would notice the star inside my sweater,” Dr. Kahneman noted in a biographical essay for the Nobel Prize ceremonies. But the German pulled out his wallet, showed him a photo of a boy, gave him some money and sent him on his way. “I went home more certain than ever that my mother was right,” Dr. Kahneman said in the essay. “People were endlessly complicated and interesting.”
Nick Bostrom’s new book, “Deep Utopia”, is out today
by PeterH — 2024-03-27T11:24:01.401Z
Link: https://nickbostrom.com/deep-utopia/

Bostrom’s new book is out today in hardcover and Kindle in the USA, and on Kindle in the UK.

Description:

A greyhound catching the mechanical lure—what would he actually do with it? Has he given this any thought?

Bostrom’s previous book, Superintelligence: Paths, Dangers, Strategies changed the global conversation on AI and became a New York Times bestseller. It focused on what might happen if AI development goes wrong. But what if things go right?

Suppose that we develop superintelligence safely, govern it well, and make good use of the cornucopian wealth and near magical technological powers that this technology can unlock. If this transition to the machine intelligence era goes well, human labor becomes obsolete. We would thus enter a condition of "post-instrumentality", in which our efforts are not needed for any practical purpose. Furthermore, at technological maturity, human nature becomes entirely malleable.

Here we confront a challenge that is not technological but philosophical and spiritual. In such a solved world, what is the point of human existence? What gives meaning to life? What do we do all day?

Deep Utopia shines new light on these old questions, and gives us glimpses of a different kind of existence, which might be ours in the future.

Links to purchase:

Amazon
Barnes & Noble

There’s a table of contents on the book’s web page.
Decompiling Tracr Transformers - An interpretability experiment
by Hannes Thurnherr — 2024-03-27T09:49:38.081Z

Note: This blog post is cross-posted from my personal website, where I expect a broader audience than here. If you are familiar with the difficulty and significance of neural network interpretability, skip to the third subsection titled "In defence of fighting fire with fire"

Summary: This is a post about a research project I conducted over the last few months. I trained a model to translate transformer weights, compiled using tracr, back into the RASP code they were compiled from. I give a general introduction to the reasoning behind why I think this is valuable, describe some technical components of the method, talk about the results and finally talk about some of the limitations.
TLDR on the results: it worked reasonably well and could be made to work even better. 73% of transformer weights were translated correctly back into equivalent RASP code.

Interpretability

The way in which interpretability is hard

Neural Networks are “black boxes”. What do we mean by that? It’s an interesting question because unlike with human brains, we have unhindered access to every process happening in the neural net. We know, with arbitrarily high accuracy, every weight, bias, activation and connection. But we still view them as “inscrutable”. Why?

Originally I had a metaphor here about a sandstorm where the components are also simple but predicting the big-picture-future is hard. I was not satisfied with this metaphor. You could compare this neural network black-box problem to the human body where we understand all the chemical reactions but can’t conclude a human’s behaviour from them, but that’s also not a good comparison, since we also don’t have the ability to detect all of those chemical processes with arbitrary precision, while, as mentioned, we can measure everything happening in a neural network very precisely.
The truth is that this is a pretty unique situation. Our understanding is really just made difficult by the irreducible complexity of the causal connections within the neural networks. This difficulty is not a matter of the complexity of the process's components. The number of components and what emerges from their interactions is what complicates things. It complicates it to such a degree that it makes it impossible to grasp the bigger picture, even if you know every detail of the „smaller picture“.

This is what we‘re facing with neural networks. They‘re a huge salad...
Have we really forsaken natural selection?
by KatjaGrace — 2024-03-27T06:50:04.694Z

Natural selection is often charged with having goals for humanity, and humanity is often charged with falling down on them. The big accusation, I think, is of sub-maximal procreation. If we cared at all about the genetic proliferation that natural selection wanted for us, then this time of riches would be a time of fifty-child families, not one of coddled dogs and state-of-the-art sitting rooms.

But (the story goes) our failure is excusable, because instead of a deep-seated loyalty to genetic fitness, natural selection merely fitted humans out with a system of suggestive urges: hungers, fears, loves, lusts. Which all worked well together to bring about children in the prehistoric years of our forebears, but no more. In part because all sorts of things are different, and in part because we specifically made things different in that way on purpose: bringing about children gets in the way of the further satisfaction of those urges, so we avoid it (the story goes).

This is generally floated as an illustrative warning about artificial intelligence. The moral is that if you make a system by first making multitudinous random systems and then systematically destroying all the ones that don’t do the thing you want, then the system you are left with might only do what you want while current circumstances persist, rather than being endowed with a consistent desire for the thing you actually had in mind.

Observing acquaintences dispute this point recently, it struck me that humans are actually weirdly aligned with natural selection, more than I could easily account for.

Natural selection, in its broadest, truest, (most idiolectic?) sense, doesn’t care about genes. Genes are a nice substrate on which natural selection famously makes particularly pretty patterns by driving a sensical evolution of lifeforms through interesting intricacies. But natural selection’s real love is existence. Natural selection just favors things that tend to exist. Things that start existing: great. Things that, having started existing, survive: amazing. Things that, while surviving, cause many copies of themselves to come into being: especial favorites of evolution, as long as there’s a path to the first ones coming into being.

So natural selection likes genes that promote procreation and survival, but also likes elements that appear and don’t dissolve, ideas that come to mind and stay there, tools ...
20 minutes of work as an artist in one future
by worse — 2024-03-27T06:49:53.418Z

I am an artist.

”Eleven evil wizard schoolgirls in an archduke's library, dressed in red and black Asmodean schoolgirl uniforms, perched on armchairs and sofas”[1]

Sigh, at least it’s not more catgirls. I don’t even draw them well.

I stretched briefly before starting this one, my arms reaching as far as they could go behind me as I leaned back in my chair. And then my neck, back and forth, side to side, crick and crick.

20 feet away, 20 seconds, every 20 minutes, I mantra’d, looking about the office, surveying the heads above the computer screens, dutifully doing all sorts of art. My eyes met briefly with Bill and we shared something of a nod. Why was he looking at me? Eh, I was looking at him.

I turned back to my work, and began drawing on the screen. It offered me a sort of imprint of what various image gen models would have done across multiple instances. With this heat map, almost as part of my process, I can nearly always imagine some ghost of a previous key work existent - a sort of platonic form of wizard schoolgirls, or maybe an amalgamation of the forms of both schoolgirls and wizards? Who knows. Whoever pioneered the sorcery and teenagers combo must’ve done well for themself.

I began filling in the inverse and emptier spaces with basic sketches of where I’d place each wizard schoolgirl, making sure there were multiple armchairs and sofas (four and 2, yeah, that feels decent). And I’d do something interesting and move these two behind the others, maybe they’re even twins, or is that a common trope and I’ll lose creativity points?

Huh. Somebody thought this first and then drew it, or maybe the artist even then was contracted? Who knows. I started making more aggressive lines, quickly placing each face. Maybe this one’s face is covered with her hair and this one is blonde but she’s dyed her hair black. And to be bold, this one has a ponytail.

Will I make an ‘unattractive' one? I pondered for a moment...

…nah, they say they don’t, but it definitely gets penalized.

On returning pen to screen my hand was a bit too ambitious and malpracticed across the page (one benefit, I suppose, is I do get exposed to quite a few more words than I used to). But this was fine, since the pen had a built-in ‘undo’ tap.

Of course, AI tools made drawing so much easier, but that was before we realized each little shake of the artists’ lines were...
Beauty and the Bets
by Ape in the coat — 2024-03-27T06:17:27.516Z

This is the ninth post in my series on Anthropics. The previous one is The Solution to Sleeping Beauty. The next one is Semantic Disagreement of Sleeping Beauty Problem.

Introduction

There are some quite pervasive misconceptions about betting in regards to the Sleeping Beauty problem.

One is that you need to switch between halfer and thirder stances based on the betting scheme proposed. As if learning about a betting scheme is supposed to affect your credence in an event.

Another is that halfers should bet at thirders odds and, therefore, thirdism is vindicated on the grounds of betting. What do halfers even mean by probability of Heads being 1/2 if they bet as if it's 1/3?

In this post we are going to correct them. We will understand how to arrive to correct betting odds from both thirdist and halfist positions, and why they are the same. We will also explore the core problems with betting arguments as a way to answer probability theory problems and, taking those into account, manage to construct several examples showing the superiority of the correct halfer position in Sleeping Beauty.

Different Probabilities for Different Betting Schemes?

The first misconception has even found its way to the Less Wrong wiki:

If Beauty's bets about the coin get paid out once per experiment, she will do best by acting as if the probability is one half. If the bets get paid out once per awakening, acting as if the probability is one third has the best expected value.

It originates from the fact that there are two different scoring rules, counting per experiment and per awakening. If we aggregate using the per experiment rule, we get P(Heads) = 1/2 - probability that the coin is Heads in a random experiment. If we aggregate using the per awakening rule we get P(Heads) = 1/3 - probability that the coin is Heads in a random awakening. The grain of truth is that you indeed can use this as a quick heuristic for the correct betting odds.

However, as I've shown in the previous post, only the former probability is mathematically sound for the Sleeping Beauty problem, because awakenings do not happen at random. So, it would've been very strange if we really needed to switch to a wrong model to get the correct answer in some betting schemes. Beyond a quick and lossy heuristic, it would be a very bad sign if we were unable to get the optimal betting odds from the correct model.

It would mea...
More podcasts on 2023 AI survey: Cognitive Revolution and FLI
by KatjaGrace — 2024-03-27T05:50:04.499Z

Two new discussions of the 2023 ESPAI:

Possibly I have a podcasting facial expression.

(If you want to listen in on more chatting about this survey, see also: Eye4AI podcast. Honestly I can’t remember how much overlap there is between the different ones.)
Failures in Kindness
by silentbob — 2024-03-26T21:30:11.052Z

There's a particular kind of widespread human behavior that is kind on the surface, but upon closer inspection reveals quite the opposite. This post is about four such patterns.

Computational Kindness

One of the most useful ideas I got out of Algorithms to Live By is that of computational kindness. I was quite surprised to only find a single mention of the term on lesswrong. So now there's two.

Computational kindness is the antidote to a common situation: imagine a friend from a different country is visiting and will stay with you for a while. You're exchanging some text messages beforehand in order to figure out how to spend your time together. You want to show your friend the city, and you want to be very accommodating and make sure all their preferences will be met. So you simply ask them: "What do you want to do"? And maybe you add "I'm completely fine with anything!" to ensure you're really introducing no constraints whatsoever and you two can do exactly what your friend desires.

People often act like this, and they tend to assume they're doing the other person a favor by being so open and flexible. After all, this way the other person will have to make no trade-offs and can spend their time exactly as they please. The problem with this however is that it's computationally unkind: it offloads all the effort of coming up with ideas and making decisions to the other person. So while it is kind on one level (respecting their object level preferences), it's unkind on another (effort, and respecting their possible meta level preferences about the planning process). And particularly if the friend's preferences about what exactly to do are not that strong, it now gives them a difficult and uncertain task for very little payoff.

So what's the computationally kind way of approaching this situation? You could name a (not too long) list of concrete proposals of how you could spend your time. If you know the person really well, you could suggest a full-fledged plan. If you don't know them that well, you could ask a few clarifying questions about their general preferences and then come up with a plan. And on top of this (rather than instead of it) you can make sure to point out that you're open to anything and are happy to change plans in any way. This way, the other person can decide themselves how much cognitive effort to invest. They can just say "yes" to your p...
AE Studio @ SXSW: We need more AI consciousness research (and further resources)
by AE Studio — 2024-03-26T20:59:09.129Z

Quick update from

AE Studio

: last week, Judd (AE’s CEO) hosted a

panel at SXSW

with Anil Seth,

Allison Duettmann

, and Michael Graziano, entitled “The Path to Conscious AI” (discussion summary here[1]).

We’re also making available an unedited

Otter transcript/recording

for those who might want to read along or increase the speed of the playback.

Why AI consciousness research seems critical to us

With the release of each new

frontier

model

seems to follow a cascade of questions probing whether or not the model is conscious in training and/or deployment. We suspect that these questions will only grow in number and volume as these models exhibit increasingly sophisticated cognition.

If consciousness is indeed sufficient for

moral patienthood

, then the stakes seem remarkably high from a utilitarian perspective that we do not commit the Type II error of behaving as if these and future systems are not conscious in a world where they are in fact conscious.

Our current understanding of the possible states we may be in with respect to conscious AI, along with the general value associated with being in that state. Somewhat akin to

Pascal’s wager

, this framing would suggest that the expected value of acting as if AI is not conscious may be significantly lower than acting as if AI is conscious in light of current uncertainty about how consciousness actually works.

Because the ground truth here (i.e., how consciousness works mechanistically) is still poorly understood, it is extremely challenging to reliably estimate the probability that we are in any of the four quadrants above—which seems to us like a very alarming status quo. Different people have different default intuitions about this question, but the stakes here

seem too high

for default intuitions to be governing our collective behavior.

In an ideal world, we'd have understood far more about consciousness and human cognition before getting near AGI. For this reason, we suspect that there is likely substantial work that ought to be done at a smaller scale first to better understand consciousness and its implications for alignment. Doing this work now seems far preferable to a counterfactual world where we build frontier models that end up being conscious while we still lack a reasonable model for the correlates or implications of building sentient ...
Meltdown: Interface for llama.cpp and ChatGPT
by nextcaller — 2024-03-26T18:29:39.344Z

I'm afraid linking what I've been working on for a while as my first post might not be greatly received, but I think you might find it interesting none the less.

I'm making a text interface to chat with local and remote models. It is made in 100% python, it uses tkinter/tcl which should be bundled with a normal python installation.

I made it because I wasn't able to find an interface that felt right to me. I didn't try them all though. I like adding "power user" features when I think of one.

Repo: https://github.com/Merkoba/Meltdown

Some features:

Load llama.cpp models (only gguf tested for now).
Use your ChatGPT api key with a specific model of openai.
Model configuration tweaks like temperature, top-p etc.
Sessions with conversations spread in tabs. These can be saved and loaded.
Configurations can be saved and loaded.
Markdown support, including syntax highlighting for code snippets.
Click, right click, or double click words to either Copy, Explain, Search, or open a new conversation.
Dark and light themes available.
Commands with tab completion and similarity check.
Command line arguments to set how the program works.
Saved context to use with the models.
Save logs to either json or text.
Run a command upon saving a log, like opening it with a text editor.
Compact mode which hides some panels.
Scrollable panel to pack more configs.
Prepend and Append to your prompt automatically.
Close tabs in different ways like old, others, all, etc.
Display CPU, RAM, and Temperature. Clicking these opens a task manager. This can be expanded to work on more systems.
Input history to go back to previous prompts by using up/down arrows, buttons, or mousewheel.
Keyboard shortcuts to perform various actions.
Variables to use for the system. For example \@name_user, \@name_ai, and \@date.
Responses are streamed live.

I don't know if this works in systems different to mine. But you are encouraged to try.