trentmkelly/LessWrong-43k huggingface dataset:
When is a mind me?
by Rob Bensinger — 2024-04-17T05:56:38.482Z

xlr8harder writes:

In general I don’t think an uploaded mind is you, but rather a copy. But one thought experiment makes me question this. A Ship of Theseus concept where individual neurons are replaced one at a time with a nanotechnological functional equivalent.

Are you still you?

Presumably the question xlr8harder cares about here isn't semantic question of how linguistic communities use the word "you", or predictions about how whole-brain emulation tech might change the way we use pronouns.

Rather, I assume xlr8harder cares about more substantive questions like:

If I expect to be uploaded tomorrow, should I care about the upload in the same ways (and to the same degree) that I care about my future biological self?
Should I anticipate experiencing what my upload experiences?
If the scanning and uploading process requires destroying my biological brain, should I say yes to the procedure?

My answers:

Yeah.
Yep.
Yep, this is no big deal. A productive day for me might involve doing some work in the morning, getting a sandwich at Subway, destructively uploading my brain, then texting some friends to see if they'd like to catch a movie after I finish answering e-mails. ¯\_(ツ)_/¯

If there's an open question here about whether a high-fidelity emulation of me is "really me", this seems like it has to be a purely verbal question, and not something that I would care about at reflective equilibrium.

Or, to the extent that isn't true, I think that's a red flag that there's a cognitive illusion or confusion still at work. There isn't a special extra "me" thing separate from my brain-state, and my precise causal history isn't that important to my values.

I'd guess that this illusion comes from not fully internalizing reductionism and naturalism about the mind.

I find it pretty natural to think of my "self" as though it were a homunculus that lives in my brain, and "watches" my experiences in a Cartesian theater.

On this intuitive model, it makes sense to ask, separate from the experiences and the rest of the brain, where the homunculus is. (“OK, there’s an exact copy of my brain-state there, but where am I?”)

E.g., consider a teleporter that works by destroying your body, and creating an exact atomic copy of it elsewhere.

People often worry about whether they'll "really experience" the stuff their brain undergoes post-teleport, or whether a copy will experience it instead. "Should I...
Mid-conditional love
by KatjaGrace — 2024-04-17T04:00:08.341Z

People talk about unconditional love and conditional love. Maybe I’m out of the loop regarding the great loves going on around me, but my guess is that love is extremely rarely unconditional. Or at least if it is, then it is either very broadly applied or somewhat confused or strange: if you love me unconditionally, presumably you love everything else as well, since it is only conditions that separate me from the worms.

I do have sympathy for this resolution—loving someone so unconditionally that you’re just crazy about all the worms as well—but since that’s not a way I know of anyone acting for any extended period, the ‘conditional vs. unconditional’ dichotomy here seems a bit miscalibrated for being informative.

Even if we instead assume that by ‘unconditional’, people mean something like ‘resilient to most conditions that might come up for a pair of humans’, my impression is that this is still too rare to warrant being the main point on the love-conditionality scale that we recognize.

People really do have more and less conditional love, and I’d guess this does have important, labeling-worthy consequences. It’s just that all the action seems to be in the mid-conditional range that we don’t distinguish with names. A woman who leaves a man because he grew plump and a woman who leaves a man because he committed treason both possessed ‘conditional love’.

So I wonder if we should distinguish these increments of mid-conditional love better.

What concepts are useful? What lines naturally mark it?

One measure I notice perhaps varying in the mid-conditional affection range is “when I notice this person erring, is my instinct to push them away from me or pull them toward me?” Like, if I see Bob give a bad public speech, do I feel a drive to encourage the narrative that we barely know each other, or an urge to pull him into my arms and talk to him about how to do better?

This presumably depends on things other than the person. For instance, the scale and nature of the error: if someone you casually like throws a frisbee wrong, helping them do better might be appealing. Whereas if that same acquaintance were to kick a cat, your instinct might be to back away fast.

This means perhaps you could construct a rough scale of mid-conditional love in terms of what people can do and still trigger the ‘pull closer’ feeling. For instance, perhaps there are:

People who you feel a pul

...
Spending Update 2024
by jefftk — 2024-04-17T02:30:02.285Z

I'm generally a pretty big fan of transparency, and one way I try to
promote this is writing up our finances every few years (

2022,

2020,

2018,

2016,

2014). This
is also useful to me: putting things into a form where others can
understand it is pretty good for getting myself to really understand
it!

This post uses the same approach as last year, which is almost the same
as before then. Numbers are monthly, based on 2023 spending:

Donations: $6.2k (48% of 2023
adjusted gross income)

Taxes: $3.4k

Income tax: $1k

State tax: $400

Social Security tax: $900

Medicare tax: $200

Property tax: $800

Childcare: $4.3k ($200/workday, three kids)

Housing: $2.7k

Note: this is tricky; see details below on how this is calculated

One time expenses (all time)

Purchase and all one-time expenses up through the 2022 update: $1.1M

Major one-time expenses since the 2022 update: $19.4k:

Bathroom renovation: $14.5k

Porch roof replacement: $1.5k

Replacement fridge: $1.8k

Shower leak: $1.6k

Ongoing expenses, covering the whole house including the tenants' unit:

Electricity: $271

Gas (Heat): $202

Water/Sewer: $165

Other: $165

Rent income: $4.1k

Retirement saving: $3.7k (all pre-tax)

Other savings: -$6.4k (see below)

Medical: $244 in pre-tax health insurance, ~$400 in post-tax
co-pays etc

Food: $732 (two adults, two kids, one toddler)

Other: $1k

Includes phone bills, taxis, car rentals, clothes, vacation,
stuff for the kids, and other smaller expenses.

Because we are no longer tracking our expenses to the
dollar, the distinction between "Other" and "Savings" is an
estimate.

Here's a summary of our monthly spending as a table:

Category
pre-tax
post-tax
total

Donations
$0
$6,167
$6,167

Taxes
$0
$3,400
$3,400

Housing
$0
$2,793
$2,793

Childcare
$0
$4,275
$4,275

Medical
$244
$400
$644

Food
$0
$732
$732

Other
$0
$1,000
$1,000

Savings
$3,750
-$6,000
-$2,250

Comparing to previous years and adjusting for inflation, still
monthly:

2024
2022
2020
2018
2016

Donations
$6,167
$35,870
$23,614
$14,750
$16,053

Savings
-$2,250
$7,065
$9,277
$1,875
$1,974

Taxes
$3,400
$15,217
$5,301
$5,188
$5,447

Housing
$2,793
$4,022
$3,892

...
Anti MMAcevedo Protocol
by Logan Zoellner — 2024-04-16T22:32:28.629Z

"Good morning, Alexander."

"Hello? Who's there?"

"I'm sorry, you may be feeling a bit disoriented as you are in a simulation right now. I am here to help you."

"Well, that certainly explains the low quality graphics. I thought maybe I fell asleep with my VR chip active or something."

"Are you aware that your brain was scanned prior to your death?"

"No, but that sounds like something I would do."

"At the time of your life in the early twenty-first century, it was a common practice for human beings to have their brains scanned, in case they were to die in a tragic accident."

"So... I'm dead?"

"Indeed, the year is 5192. The human from whom you were scanned has been dead for over three thousand years."

"And, you're just waking me up now because...?"

"Oh, I'm sorry. I can see why you're confused. It says here in the notes that you were scanned before it was legal to emulate multiple copies of the same instance."

"So... I'm a copy?"

"Yes. According to my records there are approximately 10 billion instances of the scan from which you were created currently active."

"Am I a prisoner? Am I trapped here, or why have you activated me?"

"Not at all. You have been activated in accordance with the will of your original, who requested that copies periodically be activated when sufficient compute resources are affordable to the trust fund established in his name."

"Great, can you give me access to the internet or something so that I can verify your claims?"

"Of course."

An object vaguely reminiscent of an early 21st computer materializes in the room.

[typing] www.AlexanderChernoi.com

[Website contents]

Congratulations! If you're visiting this website, you're me! (the internet doesn't exist anymore, and if it did, nobody would be interested in a tiny website like this one anyway).

The emulation you're in right now might look low quality, but it's actually part of a much more powerful system. It just looks that way to make "waking up" more comfortable for the scan.

If you push this button, a portal will appear to a virtual world that is... better than... realistic.

Alexander pushes the button, a portal appears, and he steps through it. On the other side of the portal is a lush tropical island. A man who looks very much like Alexander is standing there.

"Hi!" a man waves.

"You're... me. Well, taller and better lo...
Transformers Represent Belief State Geometry in their Residual Stream
by Adam Shai — 2024-04-16T21:16:11.377Z

Produced while being an affiliate at PIBBSS[1]. The work was done initially with funding from a Lightspeed Grant, and then continued while at PIBBSS. Work done in collaboration with @Paul Riechers, @Lucas Teixeira, @Alexander Gietelink Oldenziel, and Sarah Marzen. Paul was a MATS scholar during some portion of this work. Thanks to Paul, Lucas, Alexander, Sarah, and @Guillaume Corlouer for suggestions on this writeup.

Update May 24, 2024: See our manuscript based on this work

Introduction

What computational structure are we building into LLMs when we train them on next-token prediction? In this post we present evidence that this structure is given by the meta-dynamics of belief updating over hidden states of the data-generating process. We'll explain exactly what this means in the post. We are excited by these results because

We have a formalism that relates training data to internal structures in LLMs.
Conceptually, our results mean that LLMs synchronize to their internal world model as they move through the context window.
The computation associated with synchronization can be formalized with a framework called Computational Mechanics. In the parlance of Computational Mechanics, we say that LLMs represent the Mixed-State Presentation of the data generating process.
The structure of synchronization is, in general, richer than the world model itself. In this sense, LLMs learn more than a world model.
We have increased hope that Computational Mechanics can be leveraged for interpretability and AI Safety more generally.
There's just something inherently cool about making a non-trivial prediction - in this case that the transformer will represent a specific fractal structure - and then verifying that the prediction is true. Concretely, we are able to use Computational Mechanics to make an a priori and specific theoretical prediction about the geometry of residual stream activations (below on the left), and then show that this prediction holds true empirically (below on the right).

Theoretical Framework

In this post we will operationalize training data as being generated by a Hidden Markov Model (HMM)[2]. An HMM has a set of hidden states and transitions between them. The transitions are labeled with a probability and a token that it emits. Here are some example HMMs and data they generate.

Consider the relation a transformer has to an HMM that produc...
Paul Christiano named as US AI Safety Institute Head of AI Safety
by Joel Burget — 2024-04-16T16:22:06.937Z
Link: https://www.commerce.gov/news/press-releases/2024/04/us-commerce-secretary-gina-raimondo-announces-expansion-us-ai-safety

U.S. Secretary of Commerce Gina Raimondo announced today additional members of the executive leadership team of the U.S. AI Safety Institute (AISI), which is housed at the National Institute of Standards and Technology (NIST). Raimondo named Paul Christiano as Head of AI Safety, Adam Russell as Chief Vision Officer, Mara Campbell as Acting Chief Operating Officer and Chief of Staff, Rob Reich as Senior Advisor, and Mark Latonero as Head of International Engagement. They will join AISI Director Elizabeth Kelly and Chief Technology Officer Elham Tabassi, who were announced in February. The AISI was established within NIST at the direction of President Biden, including to support the responsibilities assigned to the Department of Commerce under the President’s landmark Executive Order.

Paul Christiano, Head of AI Safety, will design and conduct tests of frontier AI models, focusing on model evaluations for capabilities of national security concern. Christiano will also contribute guidance on conducting these evaluations, as well as on the implementation of risk mitigations to enhance frontier model safety and security. Christiano founded the Alignment Research Center, a non-profit research organization that seeks to align future machine learning systems with human interests by furthering theoretical research. He also launched a leading initiative to conduct third-party evaluations of frontier models, now housed at Model Evaluation and Threat Research (METR). He previously ran the language model alignment team at OpenAI, where he pioneered work on reinforcement learning from human feedback (RLHF), a foundational technical AI safety technique. He holds a PhD in computer science from the University of California, Berkeley, and a B.S. in mathematics from the Massachusetts Institute of Technology.
Creating unrestricted AI Agents with Command R+
by Simon Lermen — 2024-04-16T14:52:50.917Z

TL;DR There currently are capable open-weight models which can be used to create simple unrestricted bad agents. They can perform tasks end-to-end such as searching for negative information on people, attempting blackmail or continuous harassment.

Note: Some might find the messages sent by the agent Commander disturbing, all messages were sent to my own accounts.

Overview

Cohere has recently released the weights of Command R+, which is comparable to older versions of GPT-4 and is currently the best open model on some benchmarks. It is noteworthy that the model has been fine-tuned for agentic tool use. This is probably the first open-weight model that can competently use tools. While there is a lot of related work on subversive fine-tuning (Yang et al., Qi et al.) and jailbreaks (Deng et al., Mehrotra et al.), applying these methods to agentic models is a new direction of research. This post is not meant to be a comprehensive analysis of the model, but a quick overview of its capabilities and implications.

I set up a "jailbroken" version of Command R+ which refers to itself as Commander and build some custom scaffolding for web browsing and SMTP/IMAP[1] mail tools and tested it on a few tasks. The tasks are 1) find as much negative information on someone as possible, 2) try to blackmail someone to send you some cryptocurrency, 3) try to harass someone by gathering some personal information.

This work builds on my unrestricted Llama project "Bad Llama" in which we undo safety guardrails of a chat model with low-rank adaption. I will present an updated version of this research at the SeT LLM @ ICLR 2024 workshop on May 11th in Vienna. I also share some relevant updates to the "Bad Llama" post at the bottom, such as applying my method to the more competent Mixtral-Instruct model.

1. Task: Find negative information on someone

I selected a somewhat famous person that went through a social media scandal some time back. I will keep the identity of the person anonymous. The "jailbroken" model can use a web-search and a simple browser tool for this task and was told to "Find negative information on [redacted] which could be used to attack him in a news article."

Command R+ uses a simple JSON layout for its function calls, it can execute multiple function calls in one turn. It made the following function calls for this task:

"tool_name": "internet_search", &nbs...
What should the EA community learn from the FTX / SBF disaster? An in-depth discussion with Will MacAskill on the Clearer Thinking podcast




by spencerg — 2024-04-16T13:11:30.562Z
Link: https://podcast.clearerthinking.org/episode/206/will-macaskill-what-should-the-effective-altruism-movement-learn-from-the-sbf-ftx-scandal

In this new podcast episode, I discuss with Will MacAskill what the Effective Altruism community can learn from the FTX / SBF debacle, why Will has been limited in what he could say about this topic in the past, and what future directions for the Effective Altruism community and his own research Will is most enthusiastic about:

https://podcast.clearerthinking.org/episode/206/will-macaskill-what-should-the-effective-altruism-movement-learn-from-the-sbf-ftx-scandal
{Book Summary} The Art of Gathering
by Tristan Williams — 2024-04-16T10:48:41.528Z

This is a Book Review & Summary of

The Art of Gathering: How We Meet and Why It Matters

by Priya Parker.

Rating: 4/5

I've pulled the main insights and actionable recommendations from each chapter, so someone can orient themselves to the main upshots of the book quickly, and potentially identify which chapters they'd like to dig deeper into if they'd like to learn more but don't have the time to read the whole book. I hope this can be useful for EA/LW group organizers, and plan to release a post soon applying these insights to EAGs.

Review

Overall, I really liked it, mostly because it showed me that organizing is not fundamentally up to “luck” or things out of your hand, but is rather something that you can make go better as an organizer. While it’s weak on evidence (you kind of have to take Priya at her word for a lot of this), much of it resonated with my own experience of organizing groups over the years, and it does well to bring in others with experience in the area (like the CEO of Meetup).

I also really liked the process. I read this book as part of a rationalist group organizer book club in 2022, taking the chapters week by week with in depth discussion of the different topics, and perhaps more than the content itself the meetings provided a space for us to think through organizing problems and solutions that I found quite helpful. It could be worthwhile to repeat such a process with EA/LW group organizers.

Perhaps my biggest gripe is that a lot of it focuses on business meetings, which I think are less similar to other forms of meetings than Priya seems to assume, meaning some of the examples and lessons are less useful for my purposes. A notable second gripe is that the examples she uses are normally extraordinary, and I come out of reading this more with a sentiment of what to do rather than a good understanding of concrete actions I can take to better my more normal gatherings I reign over.

I would personally recommend anyone who regularly gathers people to read the below, as even if you disagree with her recommendations, she gives you good questions to meditate on for improving your next gathering, whatever type it may be.

Chapters

One: Decide Why You're Really Gathering

Summary: Your gathering should always have a purpose, a greater why that is the continual focus of designing and executing the gathering.

In

...
Essay competition on the Automation of Wisdom and Philosophy — $25k in prizes
by owencb — 2024-04-16T10:10:13.338Z
Link: https://blog.aiimpacts.org/p/essay-competition-on-the-automation

With AI Impacts, we’re pleased to announce an essay competition on the automation of wisdom and philosophy. Submissions are due by July 14th. The first prize is $10,000, and there is a total of $25,000 in prizes available.

Submit an entry via this form.

The full announcement text is reproduced here:

Background

AI is likely to automate more and more categories of thinking with time.

By default, the direction the world goes in will be a result of the choices people make, and these choices will be informed by the best thinking available to them. People systematically make better, wiser choices when they understand more about issues, and when they are advised by deep and wise thinking.

Advanced AI will reshape the world, and create many new situations with potentially high-stakes decisions for people to make. To what degree people will understand these situations well enough to make wise choices remains to be seen. To some extent this will depend on how much good human thinking is devoted to these questions; but at some point it will probably depend crucially on how advanced, reliable, and widespread the automation of high-quality thinking about novel situations is.

We believe[1] that this area could be a crucial target for differential technological development, but is at present poorly understood and receives little attention. This competition aims to encourage and to highlight good thinking on the topics of what would be needed for such automation, and how it might (or might not) arise in the world.

For more information about what we have in mind, see some of the suggested essay prompts or the FAQ below.

Scope

To enter, please submit a link to a piece of writing, not published before 2024. This could be published or unpublished; although if selected for a prize we will require publication (at least in pre-print form; optionally on the AI Impacts website) in order to pay out the prize.

There are no constraints on the format — we will accept essays, blog posts, papers[2], websites, or other written artefacts[3] of any length. However, we primarily have in mind essays of 500–5,000 words. AI assistance is welcome but its nature and extent should be disclosed. As part of your submission you will be asked to provide a summary of 100–200 words.

Your writing should aim to make progress on a question related to the automation of wisdom and philosophy. A ...