trentmkelly/LessWrong-43k huggingface dataset:
Comparing Alignment to other AGI interventions: Basic model
by Martín Soto — 2024-03-20T18:17:50.072Z

Interventions that increase the probability of Aligned AGI aren't the only kind of AGI-related work that could importantly increase the Expected Value of the future. Here I present a very basic quantitative model (which you can run yourself here) to start thinking about these issues. In a follow-up post I give a brief overview of extensions and analysis.

A main motivation of this enterprise is to assess whether interventions in the realm of Cooperative AI, that increase collaboration or reduce costly conflict, can seem like an optimal marginal allocation of resources. More concretely, in a utility framework, we compare

Alignment interventions (aV): increasing the probability that one or more agents have our values.
Cooperation interventions given alignment (aC|V): increasing the gains from trade and reducing the cost from conflict for agents with our values.
Cooperation interventions given misalignment (aC|¬V): increasing the gains from trade and reducing the cost from conflict for agents without our values.

We used a model-based approach (see here for a discussion of its benefits) paired with qualitative analysis. While these two posts don't constitute an exhaustive analysis (more exhaustive versions are less polished), feel free to reach out if you're interested in this question and want to hear more about this work.

Most of this post is a replication of previous work by Hjalmar Wijk and Tristan Cook (unpublished). The basic modelling idea we're building upon is to define how different variables affect our utility, and then incrementally compute or estimate partial derivatives to assess the value of marginal work on this or that kind of intervention.

Setup

We model a multi-agentic situation. We classify each agent as either having (approximately) our values (V) or any other values (¬V). We also classify them as either cooperative (C) or non-cooperative (¬C).[1] These classifications are binary. We are also (for now) agnostic about what these agents represent. Indeed, this basic multi-agentic model will be applicable (with differently informed estimates) to any scenario with multiple singletons, including the following:

Different AGIs (or other kinds of singletons, like AI-augmented nation-states) interacting causally on Earth
Singletons arising from different planets interacting causally in the lightcone
Singletons from across the multi-verse interacting acausal

...
New report: Safety Cases for AI
by joshc — 2024-03-20T16:45:27.984Z
Link: https://twitter.com/joshua_clymer/status/1770467746951868889

ArXiv paper: https://arxiv.org/abs/2403.10462

The idea for this paper occurred to me when I saw Buck Shlegeris' MATS stream on "Safety Cases for AI." How would one justify the safety of advanced AI systems? This question is fundamental. It informs how RSPs should be designed and what technical research is useful to pursue.

For a long time, researchers have (implicitly or explicitly) discussed ways to justify that AI systems are safe, but much of this content is scattered across different posts and papers, is not as concrete as I'd like, or does not clearly state their assumptions.

I hope this report provides a helpful birds-eye view of safety arguments and moves the AI safety conversation forward by helping to identify assumptions they rest on (though there's much more work to do to clarify these arguments).

Thanks to my coauthors: Nick Gabrieli, David Krueger, and Thomas Larsen -- and to everyone who gave feedback: Henry Sleight, Ashwin Acharya, Ryan Greenblatt, Stephen Casper, David Duvenaud, Rudolf Laine, Roger Grosse, Hjalmar Wijk, Eli Lifland, Oliver Habryka, Sim ́eon Campos, Aaron Scher, Lukas Berglund, and Nate Thomas.
User-inclination-guessing algorithms: registering a goal
by ProgramCrafter — 2024-03-20T15:55:01.314Z

Reading post about Community Notes made me recall a recent idea and post it here.

It's commonly known that ranking algorithms based on estimating user's views and inclinations can be incredibly useful in recommendation systems, advertising, and content curation. But there must be other applications which do not require to create one's own social network, for instance!

There actually is a task which I believe to be a low-hanging fruit: predicting participant's score in olympiads; preferably with dataset available in the Internet, large scoring range and various tasks - that is, in informatics.

My thoughts about the prediction system

Olympiad tasks in informatics tend to use a lot of ideas and one or two algorithms, which makes them different both in topic and in difficulty. This may be represented by a vector in a high-dimensional space (based on task tags list on Codeforces, there might be at least 24 dimensions), where projection on each axis means usage of knowledge from the corresponding topic.

Let's make user embeddings also vectors in 24-dimensional space, and measure compatibility between a person and a task as projection of person embedding onto task vector, scaled down corresponding to the latter. If pi is person's vector, tj - task vector, then compati,j=pi⋅tjt2j.

I want score prediction to be straightforward, so I map compatibility from [0;1] range to [0;100] expected score, and everything outside that range to the closest bound.

I'm currently struggling how to fill variance, and guess that a good idea is to use one of embeddings' components for this. Competitions have an element of unpredictability, and I want to be able to distinguish predictions "participant will get 40-60 points" from "participant will get 20-80 points". And, while system is training, some variance is to be expected after all.

Calculation options

I'd be glad to go forward with Bayesian updates, but unfortunately the problem space is R24(people+tasks), and pieces of evidence are not independent as well; so it will be easier to use usual learning algorithms from neural networks.

I've also found out that calculating scores will be quite easy there! Let's demonstrate this on 4-dimensional embeddings, 3 tasks and 2 people.

>>> import torch
>>> tsk = torch.tensor([[1,0,0],[0,0,1],[0,3,0],[0,4,1]])
>>> tsk
tensor([[1, 0

...
My MATS Summer 2023 experience
by James Chua — 2024-03-20T11:26:14.944Z
Link: https://jameschua.net/post/serimats_experience/

This post may be interest people who
- are interested in getting into AI alignment / the MATS program
- are interested in the soft skills that I've found valuable in developing when working on a research project

Background

In 2023 I was working as a machine learning engineer. I wanted to work on AI alignment problems. I quit my job and participated in the MATS Summer 2023 program. The MATS program puts you together with others to work on AI alignment problems under a specific mentor. Would recommend.
I was under Ethan's Perez stream where we worked on improving Chain of Thought transparency. I started in June 2023, and continued until ~March 2024 under the extension program, until my paper was submitted.

Soft skills

I think a big part of what I took away was more on the intangible side of soft skills. I did not expect this to be the case before joining MATS, but looking back I really appreciate it. During MATS, you'll have access to Scholar Support - think of it as a coach that helps you with your personal development. Some things that I've learnt from them are:
- How to tackle the planning fallacy. For example, if I aimed to have a draft out by friday, I would ask myself "Would I be surprised if on Saturday, I still didn't have the draft out". Most of the time, I wouldn't be that surprised if I did not reach my goal on time. This question helped me to identify areas that I needed to address first.
- How to tackle procrastination. It turns out that I procastinate when one side of myself really wants to do something, but the other side thinks doing that thing is dumb. I did some Internal Double Crux sessions - to resolve disagreements between my two selves. It helped me with my motivation to write the paper, and to do things like leetcode for
- How to resolve disagreements. I think that before MATS, my style of resolving disagreements was to try to persuade the other person to my point of view, but that could lead to a lot of frustration. I learnt that when disagreeing, it is better to try to fully understand the other person's point of view. Keep asking and digging deeper into their point of view to understand their mental model, rather than to try and persuade them. This helps you two to converge faster to a common understanding.

How to communicate my ideas better. I had to make slides for my weekly meetings with Ethan. At the start, my slides were quite p...
What are the weirdest things a human may want for their own sake?
by Mateusz Bagiński — 2024-03-20T11:15:09.791Z

I'm especially interested in examples of more or less psychologically healthy and otherwise (neuro)typical people having very weird[1] desires/values that we would characterize as intrinsic in the sense of being wanted for their own sake, even if we could explain their development as linked to a more typical human drive.

But I'm also somewhat interested in examples of very out-of-distribution desires/values in very [otherwise psychologically out-of-distribution] people.

Some intermediate cases that come to my mind; I'm centrally interested in things weirder than that:

Paul Erdös's obsession with mathematics is probably an intermediate case, i.e., it's just an extreme case of "a normal human passion".
Some fetishes, e.g., what's the deal with feedism or dirtiness? (See also this podcast episode for an evopsych explanation of BDSM.)
Maybe there would be some cultures that install very weird terminal values. I was somewhat surprised that some South Pacific cultures viewed heterosexuality as sinful.

^

Obviously, this is a parochial criterion in that the values that seem "weird" to us would seem "normal" to the people who have those values. That's fine.
Best *organization* red-pill books and posts?
by lemonhope — 2024-03-20T07:01:16.536Z

Most books about orgs are written for people in an org who need to cope. (Art of Possibility by the Zanders is a genuinely great one in this genre.) I am currently orgless and don't need to cope with anything orgwise. Hence, I would love to read some brutal insightful takes on how companies and nonprofits and governments etc work. Anything about tech cos or applied research orgs would be especially interesting. I'll start with the classic: Moral Mazes.
Parent-Friendly Dance Weekends
by jefftk — 2024-03-20T02:10:05.422Z

We just finished the 2024 edition of

Beantown Stomp, a contra
dance weekend I

helped start (but
no longer organize!) in Boston. There are a lot of things I like about
the weekend, but one thing I especially like is how parent-friendly it
is.

The big thing here is childcare, during the day on Saturday and
Sunday. It's included with registration, and typically staffed by one
paid person and one volunteer. Several families brought books and
toys, and it also served as a nice place for families to hang out.
Some feedback this year:

Just wanted to say thank you to everyone who helped out this
weekend. My kids had a great time, and it allowed both me and my
partner to go to this dance weekend. I haven't been dancing much since
becoming a parent, and it always feels like a rare and wonderful treat
whenever I do.

Thank you for making Beantown Stomp inclusive and
accessible to families!!

My understanding is it used to be somewhat common for events like this
to offer childcare, but most weekends stopped after the children of
the initial organizing cohort didn't need minding anymore? The only
other dance weekend I know of that includes childcare is Sugar Hill, on an
all-volunteer model.

We also do a family dance, Saturday morning. Pretty informal:
acoustic open band, lots of easy dances. Unlike the rest of the
weekend, the family dance session is open to people who haven't
registered, since even with childcare many young families aren't going
to want to make a whole weekend out of it.

Here's Nora dancing Sasha with me while I try to play in the open
band:

We also invite people who came for the family dance to stay for the
first session of contras. Saturday morning we're never close to
capacity, and even getting in a few dances together can mean a lot to
parents who otherwise aren't going to be attending the weekend. This
might have been my first dance with Julia since NEFFA last year?

Many dance communities see people drift away around when they have
kids. Which isn't surprising: the main social events start around
little-kid bedtime and the activity itself is more complex than most
kids can handle until they're 5-9. My parents used to dance, but
stopped when I was born except for annually at NEFFA. I'm really glad
that NEFFA was family-friendly enough for our family that they kept
this connection. I'm hoping that...
"I Can't Believe It Both Is and Is Not Encephalitis!" Or: What do you do when the evidence is crazy?
by Erhannis — 2024-03-19T22:08:00.724Z

The short version

Three weeks ago, starting on a Sunday, my brother stood up to turn off the light and felt a wave of disorientation, which passed after a minute or two. Periods of cognitive impairment increased in frequency over the following week, along with mild photophobia, until they became continuous, and continued increasing in severity. He also experienced severe fatigue while driving and had to pull over. Doing some research of possible causes, and considering a suspicious rash he'd noticed for a few months, he hypothesized a small possibility of HSV encephalitis, a rare but deadly illness, and obtained an internet prescription for acyclovir and began to take it that Saturday night, as a low-risk precaution. He also went to Urgent Care and the Emergency Room several times, but all tests came back negative, including a CT scan of his brain on Sunday night. As the CT scan should theoretically have shown something in the case of encephalitis, he stopped taking the acyclovir. The symptoms continued to increase until Monday night, now including unusual neck tension, and he described it as being unable to perceive the present, only things half a second in the past, and had to struggle to keep his actions coherent. He believes that were the trend to continue, he would have lost coherency entirely in a matter of days. However, Monday night, he suddenly felt better - about 48 hours after starting acyclovir. The next morning started fine, but then he began to feel the symptoms gradually returning. He started an internet prescription of valacyclovir, which converts to acyclovir in the body, with greater bioavailability. However, his symptoms continued to worsen. After three days of worsening symptoms, he switched back to acyclovir. One more day of bad symptoms, and then he suddenly felt better in the evening. He has continued taking acyclovir, and over the subsequent third week, he has felt...acceptable, for now, with some stressful ups and downs. The symptoms haven't gone away entirely, but for now it seems they are held at bay.

And this is nonsense. The evidence we have is nonsense.

Against: HSV Encephalitis cases are about 1 in 500,000 people per year. (Treated cases have ~20% mortality, and ~50% have severe brain damage.)
Against: He was missing a number of the usual symptoms, such as fever and headache, or anything notable on a CT and MRI.
For: He did have sev

...
Delta's of Change
by Jonas Kgomo — 2024-03-19T21:03:28.204Z

This document is an institutional design treatise about theories of change and potential struggle to fully account for the complex interplay of variables and uncertainties that can lead to unforeseen consequences or vulnerabilities in institutional systems.

Introduction

An organization is a dynamic system that depends on complex variables at each point of its evolution. If we try to model an organization as an evolutionary system then its change and impact in the real world is complex and non-deterministic.

The premise is that : we have not been experimenting with new institutional structures for doing research and thus do not have a good idea on how they are different from regular organisations and theories. One good example of this is speculative tech with their coordinated research programs(they are putting together a guideline on how to do this with ARIA). Perhaps this could much more approximate mechanism design for institutions.

AFAIK, a theory of change for a Manhattan or Biosecurity Project is different from an organisation operating with a monolithic goal.

Alternatives to Theory of Change

Theories of change, by their nature, are constrained by our posterior understanding and presuppositions. They often struggle to account for the full spectrum of variables and uncertainties inherent in organizational evolution.

Most Significant Change

A participatory stakeholders who come together and collate beneficiary opinions on what matters for them

What has been the most significant change that you have seen in the intervention and how has this impacted you.

Realist Evaluation

The UK Public Health uses realistic evaluations which diverge from theory-based methods by emphasizing the interplay of Context and Mechanism to produce desired Outputs. This approach acknowledges the complexity of social interventions, recognizing that outcomes are shaped not just by the intervention itself but also by the context in which it operates and the underlying mechanisms through which change occurs. The relationships between contextual factors, mechanisms, and outcomes, thereby enabling a more nuanced analysis of program effectiveness and impact. By focusing on these dynamic interactions, Realist Evaluation offers a robust methodology for uncovering the underlying processes driving program success or failure in real-world settings.

&nb...
Increasing IQ by 10 Points is Possible
by George3d6 — 2024-03-19T20:48:41.277Z
Link: https://morelucid.substack.com/p/increasing-iq-by-10-points-is-possible

A while ago I wrote how I managed to add 13 points to my IQ (as measured by the mean between 4 different tests).

I had 3 “self-experimenters” follow my instructions in San Francisco. One of them dropped off, since, surprise surprise, the intervention is hard.

The other two had an increase of 11 and 10 points in IQ respectively (using the “fluid” components of each test) and an increase of 9 and 7 respectively if we include verbal IQ.

A total of 7 people acted as a control and were given advantages on the test compared to the intervention group to exacerbate the effects of memory and motivation, only 1 scored on par with the intervention group. We get a very good p-value, considering the small n, both when comparing the % change in control vs intervention (0.04) and the before/after intervention values (0.006)

Working Hypothesis

My working hypothesis for this was simple:

If I can increase blood flow to the brain in a safe way (e.g. via specific exercises, specific supplements, and photostimulation in the NUV and NIR range)

And I can make people think “out of the box” (e.g. via specific games, specific “supplements”, specific meditations)

And prod people to think about how they can improve in whatever areas they want (e.g. via journaling, talking, and meditating)

Then you get this amazing cocktail of spare cognitive capacity suddenly getting used.

As per the last article, I can’t exactly have a step-by-step guide for how to do this, given that a lot of this is quite specific. I was rather lucky that 2 of my subjects were very athletic and “got it” quite fast in terms of the exercises they had to be doing.

The Rub

At this point, I’m confident all the “common sense” distillation on what people were experimenting with has been done, and the intervention takes quite a while.

Dedicating 4 hours a day to something for 2 weeks is one thing, but given that we’re engaging in a form of training for the mind, the participants need not only be present, but actively engaged.

A core component of my approach is the idea that people can (often non-conceptually) reason through their shortcomings if given enough spare capacity, and reach a more holistic form of thinking.

I’m hardly the first to propose or observe this, though I do want to think my approach is more well-proven, entirely secular, and faster. Still, the main bottleneck remains convincing people to spend the time on i...
Are extreme probabilities for P(doom) epistemically justifed?
by  — 2024-03-19T20:32:04.622Z

Alexander Gietelink Oldenziel

Can you post the superforecaster report that has the 0.12% P(Doom) number. I have not actually read anything of course and might be talking out of my behind.

In any case, there have been several cases where OpenPhil or somebody or other has brought in 'experts' of various ilk to debate the P(Doom), probability of existential risk. [usually in the context of AI]

Many of these experts give very low percentages. One percentage I remember was 0.12 %

In the latest case these were Superforecasters, Tetlock's anointed. Having 'skin in the game' they outperformed the fakexperts in various prediction markets.

So we should defer to them (partially) on the big questions of x-risk also. Since they give very low percentages that is good. So the argument goes.

Alex thinks these percentages are ridicilously, unseriously low. I would even go as far as saying that the superforecasters aren't really answering the actual question when they give these percentages for x-risk (both AI and general x-risk)

NathanBarnard

Yeah - there are two relevant reports. This is the first one - which has estimates for a range of x-risks including from AI where superforcasters reports a mean estimate of AI causing extinction of 0.4%, and this one - which has the 0.12% figure - which specifically selected for forcasters who were sceptical of AI extinction and experts who were concerned, with the purpose of finding cruxes between the two groups.

NathanBarnard

I think that we should defer quite a lot to superforcasters, and I don't take these percentages as being unseriously low.

NathanBarnard

I also think it's important to clarify that the 0.4% number is human extinction in particular, rather than something broader like loss of control of the future, over 1bn people dying, outcomes as bad as extinction etc.

Alexander Gietelink Oldenziel

One issue with deferring to superforecasters on x-risk percentages is that a true x-risk is a prediction question that never gets resolved.

The optimal strategy when playing a prediction market on a question that doesn't get resolved is giving 0% probability

NathanBarnard

I think I'm basically not worried about that - I predict that the superforcaters in the report took the task seriously and forcasted in the same way they would other questions. It's also not been made public who the forcasters were

...