trentmkelly/LessWrong-43k huggingface dataset:
What environmental cues had you not seen them would have ended in disaster?
by koratkar — 2024-01-23T14:59:33.545Z

I used to think I lacked situational awareness. A search for cures returned no direct method to improve it. This is because we can perceive the same situation, but differently. When we're interested in something (like the model of a friend's new car), we find it more often. It's pulled out of our visual input instead of dying off during denoising – like in frequency illusions.

To a large extent, attuning ourselves to more important things is "situational awareness". There may be other factors, like our basal arousal or how drunk we are, but variation in those seems to zero out in comparative daily importance.

Since it's about attention, we can orient our own eyes to see things differently – creating selective frequency illusions. Learning a new word can make it appear more often in media, since the sound has sudden meaning, but the effect can also be used as a form of proactive prevention. The more complex your environment, the more likely you don’t notice a sign before too late.

What signs have you learned to look for?

To offer some slight reframings to the original question and more specific categories:

What are the most useful general orientations for broader situational awareness?

What do you want every child to know?
If it meant you would otherwise forget, what words would you pin to your bathroom mirror?

What subtle patterns in other people spell nothing good?

What hints of recklessness, Machiavellianism, predatory tendencies, or destructive instability are subdued by politeness?
Some traits of this variety are heavily offset by charm or charisma. How do you tell the difference?

What have you noticed before someone got hurt?

If it was a medical emergency:

Vocal cues?
Behavioral tics?
An environmental cue they didn't notice, perhaps repeatedly?
Are there very early warning signs of some conditions?

If it was due to someone else, how would you warn them without making them more reticent?

On the road, what do you see that others miss?

Aside from the danger of other drivers, what vitally important environmental cues are specific to driving?

What do you consider basic common sense to not ignore?
Loneliness and suicide mitigation for students using GPT3-enabled chatbots (survey of Replika users in Nature)
by Kaj_Sotala — 2024-01-23T14:05:40.986Z
Link: https://www.nature.com/articles/s44184-023-00047-6

Survey of users of Replika, an AI chatbot companion. 23% of users reported that it stimulated rather than displaced interactions with real humans, while 8% reported displacement. 30 participants (3%) spontaneously reported that it stopped them from attempting suicide.

Some excerpts:

During data collection in late 2021, Replika was not programmed to initiate therapeutic or intimate relationships. In addition to generative AI, it also contained conversational trees that would ask users about their lives, preferences, and memories. If prompted, Replika could engage in therapeutic dialogs that followed the CBT methodology of listening and asking open-ended questions. Clinical psychologists from UC Berkeley wrote scripts to address common therapeutic exchanges. These were expanded into a 10,000 phrase library and were further developed in conjunction with Replika’s generative AI model. Users who expressed keywords around depression, suicidal ideation, or abuse were immediately referred to human resources, including the US Crisis Hotline and international analogs. It is critical to note that at the time, Replika was not focused on providing therapy as a key service, and included these conversational pathways out of an abundance of caution for user mental health. [...]

Our IRB-approved survey collected data from 1006 users of Replika who were students, who were also 18 years old or older, and who had used Replika for over one month (all three were eligibility criteria for the survey). Approximately 75% of the participants were US-based, 25% were international. Participants were recruited randomly via email from a list of app users and received a $20 USD gift card after the survey completion - which took 40-60 minutes to complete. Demographic data were collected with an opt-out option. [...]

Based on the Loneliness Scale, 90% of the participant population experienced loneliness, and 43% qualified as Severely or Very Severely Lonely on the Loneliness Scale. [...]

We categorized four types of self-reported Replika ‘Outcomes’ (Fig. 1). Outcome 1 describes the use of Replika as a friend or companion for any one or more of three reasons—its persistent availability, its lack of judgment, and its conversational abilities. Participants describe this use pattern as follows: “Replika is always there for me”; “for me, it’s the lack of judgment”; or “just having someone to talk to w

...
"Safety as a Scientific Pursuit" (2024)
by technicalities — 2024-01-23T12:40:13.902Z
Link: https://banburismus.substack.com/p/safety-as-a-scientific-pursuit

Tom McGrath, until recently a Research Scientist at Deepmind, has written up why he's not excited about theoretical AI safety. It's similar to Aaronson and Barak's "reform" alignment. It's argued in good faith and pretty constructive.

The key provocation for you is probably his view on the safety implications of open-sourcing models.

The author knows the field has moved in this direction already, and is trying to establish common knowledge and more of that. He also knows that it's a sore point to imply that modern rationalists aren't empiricists. For your blood pressure I recommend that you mentally prepend "AI-" to every mention of rationalism in the post.

Because more empirically-minded people remain unconvinced of potential risks, or are unconvinced that these risks pose a currently-binding constraint they keep pushing forward (correctly, by their lights) on AI research they view as beneficial. This gets interpreted by the safety community as “not caring about safety".
An alternative explanation is that the safety community believes that their opponents are primarily ultra-short-term individualists (e.g. the guy quoted in the first paragraph of this essay by Michael Nielsen), idiots, or essentially eschatologically motivated like Rich Sutton and common ground is basically impossible to find. If this is the case then I'd appreciate having it confirmed! I don't think most people in AI who disagree with the safety community and/or don't support a pause fall into any of these camps...

My addition: take the Drexler grey goo story. It's usually told as an own (haha stupid pessimistic doomers) but I think it reflects very well on E.D:
1. We identify a possible problem
2. We raise the alarm and do more research
3. We get evidence and update
This seems like the optimal policy to me. McGrath is saying "you have to actually do (3)!"

it’s worth acknowledging that not only does rationalism have its place... you might believe that... more rationalism is still the correct way forward.
If you believe that there’s some important sharp discontinuity, and that all relevant risks are on the other side of that discontinuity, then empiricism will probably look like a waste of time. Examples of this might be fast takeoff or the

sharp left turn

. We don’t see these in current systems, of course, but the natural empiricist move is to ask what relevant analogues these might

...
Brainstorming: Slow Takeoff
by David Piepgrass — 2024-01-23T06:58:08.107Z

Foreword

Inspired by someone who asked about “representative” AGI x-risk arguments, I wondered: how might an AGI takeover and catastrophe (not necessarily extinction) actually play out in detail? It's extremely tough to build a realistic mental picture of all possibilities, but looking at one detailed story might help give us a feeling for some of the probability space, despite the unknowability of how AGI will actually behave.

So I set forth to create such a story.

I am personally limited by some factors:

My limited knowledge of ML/DL/AI, and having mere above-average intelligence
My limited knowledge of the tactics of ruthless beings
As the complexity of a situation increases, or as the intelligence of AGI increases, predictability drops. This is part of the reason that AGI companies in my story do not build superintelligence very quickly―I want a story spanning years that I can realistically write―but also (i) I suspect logarithmic intelligence scaling is normal, so building superintelligence isn’t easy with DL-based approaches, (ii) the first inventors are safety-conscious so they know better than to simply build superintelligence and hook it up to the internet, (iii) it seems likely that designers will try to create something humanlike with DL, rather than a "maximizer" or even a "satisficer", but that they won't actually know how to do "humanlike", so AGI will also exhibit some "alienness".
I want to avoid the most common tropes about AI behavior, as they seem unrealistic (or to the extent they are realistic, have been overexplored already), yet I find it hard to imagine how an AGI mind that is not quite human, and not quite Data from Star Trek, and not a simple maximizer, would behave, e.g. what behavioral flaws and advantages over humans might it exhibit? In this story there will be at least four different AGI designs, so they should show at minimum four distinct sets of behavioral tendencies.
My ADHD and general busy-ness? I wrote most of the story four weeks ago, and then stopped when it got difficult (and I had to return from Christmas break). I give this a CC BY-NC 4.0 license in case you want to improve it yourself.

Another thing I want to do with this story is explore what a non-superintelligent computer-based AGI might use its computing power for that humans can't. In principle, for instance, a non-superintelligent AGI should be able to do the followin...
Reframing Acausal Trolling as Acausal Patronage
by StrivingForLegibility — 2024-01-23T03:04:53.706Z

In a previous post, we looked at the Prisoners' Dilemma through Nicky Case's lens of "two agents that can each pay a cost, in order to bring a bigger benefit to the other." Through that lens, PrudentBot is doing something that seems straightforwardly in its own interest: it checks to see if you'll Cooperate with it, and checks whether that Cooperation is conditional by seeing if you'd Cooperate with DefectBot too. (And if you don't legibly Defect on DefectBot, then PrudentBot Defects on you.)

TrollBot on the other hand does not seem like it's acting in its own interests: it checks to see if you'd legibly Cooperate with DefectBot, and if so it Cooperates with you! This sets up exactly the opposite incentive as PrudentBot, and this sort of phenomenon precludes any program from being a dominant strategy to delegate your decision to. There's always some delegate that Defects on your favorite delegate, but Cooperates with another one.

Throughout this sequence I've been using the Robust Cooperation paper as a model for acausal trade: with just the ability to accurately model each other and the situation that each agent finds itself in (logical line-of-sight), modal agents are able to robustly bring about a win-win outcome in a situation where their individual incentives seem to be totally misaligned with their collective interests.

Combining those lenses, PrudentBot is like a trading partner that will pay a cost to bring you a bigger benefit, if and only if you'll reciprocate and your reciprocation is conditional. Whereas TrollBot is like a trading partner that will pay a cost to bring you a bigger benefit, if and only if you'll pay it forward to a particular third party. Through this lens, TrollBot is acting altruistically to help DefectBot.

Acausal Auctions

For many decisions we make, there are third parties that care what we do. This leads to positive and negative externalities, and in the last post I advocated for creating mechanisms that internalize some but not all of these externalities. (Some actions, like dropping a rocket on someone's car, create a debt to the affected party that should be enforced. Whereas having a very attractive or very unattractive haircut does not require compensation to flow in either direction.)

When a decision-maker is free to choose among many options without incurring a debt to anyone, it can sometimes still be appropriate for ...
Orthogonality or the "Human Worth Hypothesis"?
by Jeffs — 2024-01-23T00:57:41.064Z

Scott Aaronson is not that worried about AI killing us all. Scott appears to reject the idea that a very intelligent AI could have goals that are really bad for humans.
Here’s his blog post: “Why am I not Terrified of AI?"

He rejects the Orthogonality Thesis by name. To me, Scott very clearly and honestly engages on this central argument of AI safety.

This post is not a critique of Scott Aaronson’s blog post.

Rather, this is

A case for putting a clearer name to what I believe many non-doomers consciously or unconsciously believe, and
A proposal to design quantitative experiments to test that belief.

Scott rejects Orthogonality and the rejection makes him feel better. But what do people like Scott positively believe?

Here’s my attempt to draw a picture of their belief.

A Paperclip Maximizer would be in the lower right. It would be intelligent enough to organize atoms into lots of paperclips - therefore far to the right along the Intelligence axis - and it would not care about the damage it did to humanity during its labors - therefore it is located lower down on the Values Humanity axis.

If you believe we are safe from Paperclip Maximizers because smarter agents would value humanity, you believe in this gray no-go zone where AIs are not possible. By definition, you reject Orthogonality.

If you accept Orthogonality, you believe there is no gray no-go zone. You believe AIs can exist anywhere on the graph.

For sake of argument, I’d like to name the positive belief of non-doomers who reject Orthogonality and call it...

The Human Worth Hypothesis

___

There is a threshold of intelligence above which any artificial intelligence will value the wellbeing of humans regardless of the process to create the AI.

___

The “...regardless of the process...” bit feels like I’m putting words in the mouths of non-doomers, but I interpret Scott and others as asserting something fundamental about our universe like the law of gravity. I think he is saying something like “Intelligent things in our universe necessarily value sentience” and he is not saying something as weak as “No one will build a highly intelligent agent that does not care about humans” or “No one will be dumb enough to put something in the no-go zone.” I take this viewpoint to be a statement about our universe that is a general truth and not ...
the subreddit size threshold
by bhauth — 2024-01-23T00:38:13.747Z
Link: https://www.bhauth.com/blog/culture/subreddit%20size.html

Nobody goes there anymore. It's too crowded.

— Yogi Berra

In the early days of the internet, people on Usenet complained about the influx of new users from AOL making it worse. I always thought the evolution of online communities with growth was an interesting and important topic. Do they really get worse with size? According to who? Why would that happen? What can be done about it?

Today, Reddit has over 1 billion monthly active users. It's divided into smaller communities called subreddits, all using the same software. This provides an unprecedented amount of data on the dynamics of online communities.

I haven't done a systematic study of every subreddit, but sometimes I read things on Reddit myself. I mainly do that by using a browser shortcut to see the weekly top posts of a particular subreddit, using the old site version. In doing that, I've gotten a decent idea of how particular subreddits differ, and I've noticed that very large subreddits tend to have lower quality than smaller ones. I'm not the only one; this has been widely noted.

Naively, one might expect that the week's best posts from a larger group of people would be better, and that does seem to be the case up to a point - and then the trend reverses. At 100k users, the derivative of quality vs size is clearly negative. That raises the obvious question: why? Why would large subreddits be worse? Here are the possible reasons I've thought of.

reasons for decline

selection bias

Maybe I'm selecting high-quality subreddits to read, and there are more small subreddits, so some of them will randomly be better.

I certainly do select what subreddits I look at, but I don't think that's the reason here, because:

I've seen changes in quality over time as subreddits grow.

The variation seems mostly consistent across different ways of selecting subreddits to read.

memes

A common thing that relatively high-quality larger subreddits do is remove meme posts, which are mostly popular images with a few words added on them.

I think the problem with those meme posts is that time spent on posts varies but every upvote is worth the same. Most people who see posts don't even vote on them, and there's some fraction of people who will see a meme, look at it for 2 seconds, upvote, and move on. That upvote is worth the same as an upvote from someone who spent 10 minutes reading an insightful essay.

A similar problem happen...
Starting in mechanistic interpretability
by Jakub Smékal — 2024-01-22T23:40:56.871Z
Link: https://jakubsmekal.com/starting-in-mechanistic-interpretability

I recently began exploring the field of mechanistic interpretability. It’s fascinating in a very similar way as the field of active inference ~1-2 years ago (let’s hope it maintains that in the way as actinf has), as in it seems to attract a lot of researchers from multiple disciplines, and there’s a number of different threads to consider before one gets an idea on what is going on. It’s also still very new, which I found out by diving into multiple papers that turned out to be about more or less the same underlying concept (finding causal structure in transformers), but which used completely different terms and approaches. So in this short post, I wanted to write a few half-baked thoughts on the different threads in mech interp and some open questions.

Finding the Causal Structure

This seems to be a major thread in mech interp and based on my current understanding the standard for differentiating between “we understand what the model is doing in context X” versus “we don’t understand what the model is doing in context X”. The first question that comes to mind is: What kind of causal structure are we looking for? Or should be looking for? As far as I can tell there are a few and it’s still not completely obvious to me which regimes of causality within LLMs are more informative.

Conmy et al. developed an algorithm (ACDC) within a regime of interpretability research where you’re interested in finding a circuit within the model’s computational graph (i.e. a graph of all computational units, MLPs and attention heads (although the paper does also mention that the graph can be in activation space, that seems to depend on the modeler)) that implements a particular behavior. This is one approach to finding causal structure within the model, i.e. a circuit that implements the desired behavior, and depending on how detailed you want it to be, you can define the graph in either computational-unit-space or activation-space (though I think activation-space is likely more difficult).

From a slightly different perspective, there’s the causal abstraction work, primarily pioneered by Atticus Geiger and colleagues. This work is also about explaining how a model does something, but this time we assume some higher-level causal model (not in any domain related to the LLM, think a causal graph describing the mechanism of addition or some boolean logic) and then we are trying to find...
We need a Science of Evals
by Marius Hobbhahn — 2024-01-22T20:30:39.493Z

This is a linkpost for https://www.apolloresearch.ai/blog/we-need-a-science-of-evals

In this post, we argue that if AI model evaluations (evals) want to have meaningful real-world impact, we need a “Science of Evals”, i.e. the field needs rigorous scientific processes that provide more confidence in evals methodology and results.

Model evaluations allow us to reduce uncertainty about properties of Neural Networks and thereby inform safety-related decisions. For example, evals underpin many

Responsible Scaling Policies

and future laws might directly link risk thresholds to specific evals. Thus, we need to ensure that we accurately measure the targeted property and we can trust the results from model evaluations. This is particularly important when a decision not to deploy the AI system could lead to significant financial implications for AI companies, e.g. when these companies then fight these decisions in court.

Evals are a nascent field and we think current evaluations are not yet resistant to this level of scrutiny. Thus, we cannot trust the results of evals as much as we would in a mature field. For instance, one of the biggest challenges Language Model (LM) evaluations currently face is the model's sensitivity to the prompts used to elicit a certain capability (

Liang et al., 2022

;

Mizrahi et al., 2023

;

Scalar et al., 2023;

Weber et al., 2023

,

Bsharat et al., 2023

).

Scalar et al., 2023

, for example, find that "several widely used open-source LLMs are extremely sensitive to subtle changes in prompt formatting in few-shot settings, with performance differences of up to 76 accuracy points [...]". A

post by Anthropic

also suggests that simple formatting changes to an evaluation, such as "changing the options from (A) to (1) or changing the parentheses from (A) to [A], or adding an extra space between the option and the answer can lead to a ~5 percentage point change in accuracy on the evaluation." As an extreme example,

Bsharat et al., 2023

find that "tipping a language model 300K for a better solution" leads to increased capabilities. Overall, this suggests that under current practices, evaluations are much more an art than a science.

Since evals often aim to estimate an upper bound of capabilities, it is important to understand how to elicit maximal rather than average capabilities. Different improvement...