trentmkelly/LessWrong-43k huggingface dataset:
Roots of Progress is hiring an event manager
by jasoncrawford — 2024-12-03T20:46:42.929Z
Link: https://rootsofprogress.notion.site/Event-Manager-13f543614e9780458f61d528628a1473

The Roots of Progress Institute is hiring a full-time, in-house event manager to run our annual Progress Conference (at Lighthaven!) and other events. See the job ad below, crossposted in full from the link above.

Event Manager

Fully remote, full-time

The Role

We’re looking for a super-organized self-starter who loves bringing people together in person around a shared set of ideas and who is great at creating magical experiences.

The Roots of Progress Institute is a nonprofit dedicated to establishing a new philosophy of progress for the 21st century. We’re part of a larger progress and abundance movement, and one key role we play within this movement is to develop talent and to build community.

As the Event Manager, you’ll be in charge of our annual progress conference, which brings together 200-300 thinkers and doers in the progress community. Our first event in October 2024 was a huge success, with 200+ invitation-only attendees coming together at a unique venue for two days. Dozens of attendees shared that this was the best conference they ever attended, and that it was “THE network to connect with the founders, writers, academics, and activists working to build a better world.” You will be running the event next year, and of course get to attend it, too! You will also be in charge of other events, from smaller fundraising salon-type gatherings, to the in-person gathering at the end of our annual writer’s fellowship.

This role reports to Heike Larson, our Vice President of Programs. It is a full-time position that is fully remote within in the contiguous US or Canada, but ideally, you’ll be located in/near a city with a major airport as the role requires a couple of multi-day trips every quarter, and around ten days on-site during the time of the annual conference.

About You

You love organizing events that bring people together and enable them to learn and form communities. You are good at creating delightful experiences and working with a wide range of partners, remotely. You’re excited about working in a small team, where you iterate on programs, learn from feedback, and improve quickly. You are thrilled when an event you put on leads to a new project, an essay written, or a partnership formed.

Do you enjoy events and project management? You have a minimum of three years of experience with project and/or event management or another operational role that involves

...
Do simulacra dream of digital sheep?
by EuanMcLean — 2024-12-03T20:25:46.296Z

This is the third in a sequence of posts scrutinizing computational functionalism (CF). In a previous post, I defined a concrete claim that computational functionalists tend to make:

Theoretical CF: A simulation of a human brain on a computer, with physics perfectly simulated down to the atomic level, would cause the conscious experience of that brain.

I contrasted this with “practical CF”, the claim that a suitably low-fidelity simulation of a brain, like one that only captures functional properties, would be conscious. In the last post, I discussed practical CF. In this post, I’ll scrutinize theoretical CF.

To evaluate theoretical CF, I’m going to meet functionalists where they (usually) stand and adopt a materialist position about consciousness. That is to say that I’ll assume all details of a human’s conscious experience are ultimately encoded in the physics of their brain.

My assumptions

I assume realism about phenomenal consciousness: Given some physical process, there is an objective fact of the matter whether or not that process is having a phenomenal experience, and what that phenomenal experience is. I am in camp #2 of Rafael’s two camps.
I assume a materialist position: that there exists a correct theory of phenomenal consciousness that specifies a map between the third-person properties of a physical system and whether or not it has phenomenal consciousness (and if so, the nature of that phenomenal experience).
I assume that phenomenal consciousness is a sub-component of the mind.

Two ways to live in a simulation

First of all, I want to pry apart two distinct meanings of “living in a simulation” that are sometimes conflated.

Living in the matrix: Your brain exists in base reality, but you are hooked up to a bunch of sophisticated virtual reality hardware, such that all of the sensory signals entering your brain create a simulated world for you to live in. Consciousness lives in base reality.
Living in Tron: Your brain is fully virtual. Not only are your surroundings simulated but so are all the details of your brain. Consciousness lives in the simulation.

Many intuitions about the feasibility of living in a simulation come from the matrix scenario. I’ve often heard arguments like “Look at the progress with VR - it won’t be long until we also have inputs for tactile sensations, taste etc. There is no technological barrier stopping us from being hooked up to s...
Orca communication project - seeking feedback (and collaborators)
by Towards_Keeperhood — 2024-12-03T17:29:40.802Z

TLDR

It is currently plausible (352115%[1]23%) to me that average orcas have at least as high potential for being great scientists as the greatest human scientists, modulo their motivation for doing science[2]. To see why I think this, see my LW question (including the considerations in my answer). EDIT: See my post.

I now want to test how intelligent orcas are as soon as possible. For this, I am creating a very easily-learnable language, which is easy for orcas to vocalize and ideally also for humans (after some practice), along with a plan for how to teach the language to orcas.

I hope to start teaching orcas the language around the start of January, but I still need to plan where to best do this, buy equipment, and find at least one person who can operate a boat who is willing to do those experiments with me. EDIT: I now noticed that interacting with orcas is relatively strictly illegal basically everywhere without permit. I don't think my project has great chances to get permitted in most places, and generally need to reconsider how to best proceed and it might take longer because I need a permit etc.

ADDED: See here for an update.

Project plan summary

For a rough summary of how I'd go about teaching orcas language, you can either watch the overview video or read the text summary in the appendix below.

I expect we will already get quite useful evidence on orca intelligence from seeing how fast and accurately they can learn the language and the abstract concepts we're trying to convey. But I will probably also try to create some other tests for testing the orcas' intelligence, but haven't thought about it yet.

Video series showing how one might get started teaching a language to orcas

To get a better impression how the project might look like, see my other videos in my video series.

Seeking feedback and collaborators

EDIT: I now don't expect anymore that many people could speed me up significantly in the way described here. However, I'm looking for people who can help me in different ways: See here.

I think other smart people could probably contribute to this project, and in particular I think experience of people who know much about linguistics could be useful.

If you have feedback or ideas how you would design some part of the language, please comment. Also lmk of resources (e.g. useful wikipedia pages...) that might be useful to know for my project. (Also feel free...
Book a Time to Chat about Interp Research
by Logan Riggs — 2024-12-03T17:27:46.808Z

In the spirit of the season, you can book a call with me to help w/ your interp project (no large coding though)

Would you like someone to:

Review your paper or code?
Brainstorm ideas on next steps?
How to best communicate your results?
Discuss conceptual problems
Obvious Advice (e.g. being affected by SAD because it's winter, not exercising, not getting enough sleep)
[anything else that would be useful!]

When we're chatting, you can interrupt me to better focus on what you specifically want. We can set [5] minute timers to try to solve whatever problem you're having. You can steer the conversation or I can!

Who is this for?

Anyone working on interp research. My niche is SAE's, so that's likely where I'm most helpful.

This is for both junior and senior researchers. If you're unsure, feel free to dm me here on LW, and I'll let you know!

I'm Useful, I Swear

I worked on one of the earliest SAE papers, applied SAEs to Preference Modelss, and co-authored Decomposing The Dark Matter of SAEs (in fact, Josh asked me to work w/ him on this project because I was useful chatting about it!).

I'd like to donate at least 20 hours of my time this month, so please have a low bar for booking a time w/ me! (booking times are for 1 hour, but feel free to just use 15 minutes!).

Why are You Doing this?

I'm currently burnt out on coding on my edge-sparse SAE project (hence why I don't want to code atm); however, talking to folks for 8 hours a day about research isn't tiring for me!

So I'd still like to be useful to our research community, even it's not through novel research for these next few weeks.

How to Book

Here's the calendly link. Do send a quick blurb on the topic or a link to your paper or document or whatever you think is helpful for me to best help you!
Balsa Research 2024 Update
by Zvi — 2024-12-03T12:30:06.829Z

For our annual update on how Balsa is doing, I am turning the floor over to Jennifer Chen, who is the only person working full time on Balsa Research.

For my general overview of giving opportunities, see my post from last week.

Previously: The 2023 Balsa Research update post, Repeal the Jones Act of 1920.

tl;dr: In 2024, Balsa Research funded two upcoming academic studies on Jones Act impacts and published the Jones Act Post. In 2025, we’ll expand our research and develop specific policy proposals. Donate to Balsa Research here.

Today is Giving Tuesday. There are many worthy causes, including all of the ones highlighted by Zvi in a recent post. Of all of those orgs, there is one organization I have privileged information on – Balsa Research, where I’ve been working for the past year and a half.

Balsa Research is a tiny 501(c)(3) currently focused on repealing the Jones Act, a century-old law that has destroyed American domestic shipping for minimal gain. You can read the long Zvi post for details, or this Planet Money podcast transcript if you would like the arguments from someone who is not Zvi.

This is not the most urgent challenge facing humanity, but we believe that it’s one where relatively small investments have a chance to unlock fairly large economic benefits.

This post is an update on what we’ve been up to this year, and our plans for 2025.

Table of Contents

What We Did in 2024.

Looking Ahead to 2025.

Why Support Balsa.

What We Did in 2024

Our work this year focused on building a robust foundation for future policy change:

In March, we opened up an RFP for academic studies quantifying the costs of the Jones Act after our literature review revealed that it’s been several decades since someone has attempted to do this.

We’re funding studies for a few different reasons. For one, updated numbers are just nice to have, for understanding the state of the world and our likely impact. They’re also good for advocacy work in particular – numbers grow stale over time, and people like seeing numbers that are from the 2020s more than they like seeing numbers from the 1990s in their policy one-pagers. Lastly, we know that DC does occasionally pay attention to policy findings coming out of top econ journals, and this shapes their policy choices at times. We’re not counting on this happening, but who knows!

We have accepted proposals from two di...
First Solo Bus Ride
by jefftk — 2024-12-03T12:20:02.344Z

Our kids have been riding the bus since they were little, though less
often since we started sharing a car

almost four years
ago. For a while the oldest, age 10, has been asking when they
could start taking the bus on their own. A few days ago they did, and
it went well!

We worked up to it gradually, so they knew how to do it and we knew
they knew how to do it. They had a background experience of having
taken this specific bus many times, though mostly not paying
attention. Once they said this was something they wanted to work on
we switched to having them lead: they sat by themself in front, and we
sat in back. They handled keeping track of where we were and pressing
the "stop requested" button.

We did this twice, which they handled well. They got us on the right
bus, paid attention along the route, and got us off at the right
place. The only problem was that both times someone else happened to
press the button for the stop before they did, so I wasn't totally
sure they'd remember. But they thought they would, and I was pretty
confident that they would as well.

They day before they rode by themself for the first time we talked
through the process: walking to the bus stop (0.5mi), how they'd know
which bus to get on (94), how they'd know their stop was coming (first
crossing the tracks, then going up the hill), what they would do if
they missed their stop (walk back along the route), and what they'd do
if someone asked about them being alone (I'm going to my
grandfather's). Then we did it again that morning before they set
out. And it all went great!

Pictured: Lily at the Seashore
Trolley Museum, operating a vehicle that curiously lacked a
steering wheel.

They also had their watch in case
there was a problem, but they didn't end up needing to call us.

It's great that we and they now have this option for getting to their
grandfather's house!

I was wondering whether this was a typical age for people to start
being able to take the bus solo, and remembered that I did a survey on
this kind of thing:

It looks like 75% of people thought a mature 10yo could learn to
handle public transit solo, and 40% thought a typical 10yo could.

Comment via: facebook, lesswrong, mastodon, bluesky
How to make evals for the AISI evals bounty
by TheManxLoiner — 2024-12-03T10:44:45.700Z

TLDR

Last weekend, I attended

an AI evals hackathon organized by Arcadia

, where the outputs were proposals for

AISI’s evals bounty

. My main learnings:

Keep your evals simple, measuring one specific capability.
It is surprisingly easy to come up with ideas, once you have seen examples.
AISI extended the deadline for submissions by two weeks. Translation: they want more ideas!
Story of real-life example of LLM deducing it was in a test environment, due to a bug in the mock environment that could not occur in a real one.

Crossposted: https://lovkush.substack.com/p/how-to-make-evals-for-the-aisi-evals

Introduction

Last weekend, I attended

an AI evals hackathon organized by Arcadia

, where the outputs were proposals for

AISI’s evals bounty

.

In the morning there was a talk by

Michael Schmatz

from the autonomous systems team at AISI and by

Marius Hobbhahn

the Director and co-founder of

Apollo Research

. I summarize Michael’s talk below and have a separate post for Marius’s talk.

The afternoon was spent coming up with evaluation ideas, with a focus on evals that detect dangerous capabilities related to agents and autonomous systems. We were given sub-categories:

​Development and assurance compromise. How can AI systems intervene in the development and assurance of either themselves or of other AI systems
​Autonomous replication and adaptation. Self explanatory.
​Resource acquisition. E.g. money or compute.

The ideas were judged by a panel of four of five people. The one’s I noted down were: Michael Schmatz,

Catherine Brewer

from OpenPhil,

Peter Hase

from Anthropic and someone from METR. The criteria for the evals:

Relevance to autonomous risks. The closer you are to measuring dangerous capabilities (rather than a precursor) the better.
Realism. How realistic is your set-up or proxy set-up.
Difficulty. It is bad if it is trivially simple for frontier models, and also bad if there are no gradations or if there is a discontinuous jump from scoring 0% to scoring 100%.
Scoring. Is the scoring process robust and measuring what we care about.
Ease of implementation.
Novelty.

Winning evals

Disclaimer: these descriptions are based on 10-20 second impromptu summaries by the winners during the announcements of the awards, so high chance I misunderstood the details.

1st place. A big way human researchers learn is by reading evals transcripts. So, in order to elicit capabilities from AI systems, these e...
Should there be just one western AGI project?
by rosehadshar — 2024-12-03T10:11:17.914Z
Link: https://www.forethought.org/research/should-there-be-just-one-western-agi-project

Tom Davidson did the original thinking; Rose Hadshar helped with later thinking, structure and writing.

Some plans for AI governance involve centralising western AGI development.[1] Would this actually be a good idea? We don’t think this question has been analysed in enough detail, given how important it is. In this post, we’re going to:

Explore the

strategic implications

of having one project instead of several
Discuss what we think the

best path forwards

is, given that strategic landscape

(If at this point you’re thinking ‘this is all irrelevant, because centralisation is inevitable’, we disagree! We suggest you read

the appendix

, and then consider if you want to read the rest of the post.)

On 2, we’re going to present:

Our overall take

: It’s very unclear whether centralising would be good or bad.

Our current best guess

: Centralisation is probably net bad, because of risks from power concentration (but this is very uncertain).

Overall

, we think the best path forward is to increase the chances we get to good versions of either a single or multiple projects, rather than to increase the chances we get a centralised project (which could be good or bad). We’re excited about work on:

Interventions which are robustly good whether there are one or multiple AGI projects.
Governance structures which avoid the biggest downsides of single and/or multiple project scenarios.

What are the strategic implications of having one instead of several projects?

What should we expect to vary with the number of western AGI development projects?

At a very abstract level, if we start out with some blobs, and then mush them into one blob, there are a few obvious things that change:

There are fewer blobs in total. So interaction dynamics will change. In the AGI case, the important interaction dynamics to consider are:

Race dynamics

, and

Power concentration

.

There’s less surface area. So it might be easier to protect our big blob. In the AGI case, we can think about this in terms of

infosecurity

.

Summary table

Variable	Implications of one project	Uncertainties[2]

Race dynamics

Less racing between western projects

- No competing projects

Unclear implications for racing with China:

- US might speed up or slow down

- China might speed up too

Do ‘races to the top’ on safety outweigh races to the bottom?

How effectively can government regulation reduce racing betwe

...
Cognitive Biases Contributing to AI X-risk — a deleted excerpt from my 2018 ARCHES draft
by Andrew_Critch — 2024-12-03T09:29:49.745Z

Preface

Several friends have asked me about what psychological effects I think could affect human judgement about x-risk.

This isn't a complete answer, but in 2018 I wrote a draft of "AI Research Considerations for Human Existential Safety" (ARCHES) that included an overview of cognitive biases I thought (and still think) will impair AI risk assessments. Many cognitive bias experiments had already failed to reproduce well in the psychology reproducibility crisis, so I thought it would be a good idea to point out some that did reproduce well, and that were obviously relevant to AI risk. Unfortunately, one prospective coauthor asked that this content be removed, because of the concern that academic AI researchers would be too unfamiliar with the relevant field: cognitive science. That coauthor ultimately decided not to appear on ARCHES at all, and in retrospect I probably should have added back this material.

Just to give you a little historical context on why it was so hard to agree on content like this, getting academics to reach consensus on any acknowledgement of AI x-risk back then was like pulling teeth, even as late as 2018, because of fear of looking weird. This sort of struggle was a major reason I eventually decided to reduce my academic appointment to ~1 day/week in 2022. But since then, the CAIS statement on AI extinction risk came out in May 2023, and now at least some more people are taking the problem seriously. So I figured I'd share this, better late than never.

Notes on definitions

"Prepotent AI" is defined as, roughly speaking, uncontrollably transformative AI, i.e., transformative AI that humans are collectively unable to stop from transforming the world once it is deployed, with effects at a scale at least as significant as the industrial revolution.

"MPAI" is short for "misaligned prepotent AI", which roughly meant, "AI that is misaligned with the objective of humanity's collective survival and flourishing". In other words, it's AI that will unstoppably kill all humans after it's deployed.

The excerpt below only discusses risk types "1b" and "1c". For discussion of other risk types, see the full paper at https://acritch.com/arches.

Risk Type 1b: Unrecognized prepotence

Consider a scenario in which AI researchers deploy an AI system that they do not realize is prepotent. Because of their faulty understandi...
Chemical Turing Machines
by Yudhister Kumar — 2024-12-03T05:26:25.950Z
Link: https://www.yudhister.me/chemical-turing-machines/

Epistemic status: brief writeup of some interesting work I found in[1][2][3][4], among other places. Probably subtly wrong in places & if you have additions / comments to make, please do.

Finite state automata (FSA) can be modeled with reactions of the form A+B→C+D. FSAs operate over regular languages, so our job is to find some reaction which corresponds to determining whether or not a given "word" (sequence of inputs to the reaction) is a member of the language.

Generally, we will be associating languages of various grammars in the Chomsky hierarchy to certain combinations of "aliquots" added to a one-pot reaction, and in this case we want our aliquots to be potassium iodide and silver nitride. Take the language over the alphabet {a,b} consisting of all words with at least one a and one b. Now associate a with some part of KIO3 and b with some part of AgNO3. Then, the reaction
KIO3+AgNO3→AgIO3(s)+KNO3
only occurs when both of the reactants are present in solution, so the word is in the language if and only if silver iodide is present. (Or, equivalently, heat is released).

Type-2 grammars consist of languages that can be modeled with pushdown automatas, which differ from FSAs in that they have a stack that can store strings of arbitrary sizes. We call these languages "context-free languages", and the reactions which we associate to context-free languages are those with intermediaries. Again, because of automata equivalence, we can consider the simple case of the Dyck language: the collection of parentheses-strings that never contain more closed parentheses than open parentheses at any i and contain exactly equal amounts of closed and open parentheses at i=n.

We associate this with the pH reaction of sodium hydroxide and acetic acid (vinegar), with the amounts of each aliquot normalized to create identical disturbances in the pH of the solution. Note that as pH indicator aliquot is present at the beginning and end of the reaction (we associate it with the start-and-end token), the aliquot serves as the intermediary (the "stack", if you will). So, if pH≥midpoint pH throughout the reaction, but is midpoint pH at the end, the reactor accepts the word. If not, it does not.

Incidentally, you can interpret this as the enthalpy yield YΔH(%) of the computation, defined as
YΔH(%)=reaction heat during computationformation heat of input string×100.
Dyck words...