trentmkelly/LessWrong-43k huggingface dataset:
Closeness To the Issue (Part 5 of "The Sense Of Physical Necessity")
by LoganStrohl — 2024-03-09T00:36:47.388Z

This is the fifth post in a sequence that demonstrates a complete naturalist study, specifically a study of query hugging (sort of), as described in The Nuts and Bolts of Naturalism. This one continues my demo of phases one and two: Locating Fulcrum Experiences and Getting Your Eyes On. For context on this sequence, see the intro post. Reminder that this is meant as reference material.

Before throwing in the towel, I thought I might as well try talking things through with ChatGPT.

Why? I think it was just something I was doing a lot at the time. “Not sure how to proceed? Throw GPT4 at it for a while and see what happens.” I didn’t expect anything in particular from the activity.

As it turns out, I can probably count on one hand the number of work-related discussions that have been as useful to me as this one—which probably says as much about my social deficits as it does about GPT4's abilities. It talked me around, and I un-gave-up.

The most important part was using GPT as a Focusing partner. I did a (mostly) canonical version of Gendlin's Focusing, and came to understand that I was struggling with the balancing act between authenticity and legibility. I decided to almost completely abandon legibility.

The relevant section of chat log:

(Note: Often, ChatGPT suddenly turns into a mini encyclopedia entry. I fondly imagine an excited nerdy 12-year-old trying to get full marks on an essay at these moments. Despite considering them extraneous to me personally, since it’s giving me background on terms I have chosen myself, I’ve left these parts in as potentially valuable context for onlookers, but I've italicized them so you can skip them if you'd prefer.)

Logan

I'm going to try a little bit of "focusing" a la Eugene Gendlin, to get in touch with whatever it is I want to work out. Please be a focusing partner.

I feel a forward pressure, and another pressure holding that back. It's primarily in my chest and solar plexus. The forward pressure reminds me of stamping feet, raring to go, frustration, almost desperation. It has a couple of layers, I think, as though something is trying to hide. The "desperation" is from a lower layer than the "raring to go". I get a felt shift when I think "I feel afraid". I feel afraid that I am done, it is over, I can no longer do or make or think. I feel afraid that I will never make progress again.

The forward pressure in my chest

...
Exploring the Evolution and Migration of Different Layer Embedding in LLMs
by Ruixuan Huang — 2024-03-08T15:01:17.504Z

[Edit on 17th Mar] After conducting experiments on more data points (5000 texts) on the Pile dataset (more sample sources), we are confident that the experimental results described earlier are reliable. Therefore, we have opened the code.

Recently, we conducted several experiments focused on the evolution and migration of token embeddings across different layers during the forward processing of LLMs. Specifically, our research targeted open-source, decoder-only architecture models, such as GPT-2 and Llama-2[1].

Our experiments are initiated from the perspective of an older research topic known as the logit lens. Utilizing the unembedding matrix on embeddings from various layers is an innovative yet thoughtless approach. Despite yielding several intriguing observations through the logit lens, this method lacks a solid theoretical foundation, rendering subsequent research built upon it potentially questionable. Moreover, we observed that some published studies have adopted a similar method without recognizing it in community[2]. A primary point of contention is that the unembedding matrix is tailored to the embeddings of the final layer. If we view the unembedding matrix as a solver, its solvable question space is inherently designed for embeddings from the last layer. Applying it to embeddings from other layers introduces a generalization challenge. The absence of a definitive ground truth for this extrapolation means we cannot conclusively determine the validity of the decoder's outputs.

Fig 1. Hypothesis: Embeddings output from different layers, along with input token embeddings, each occupy distinct subspaces. The unembedding matrix (or decoder) is only effective within the subspace where it was trained. While applying it to other layers' embeddings is mathematically feasible, the results may not always make sense.

Next, we'll introduce several experiments about this hypothesis.

Do embeddings form different spaces?

It's crucial to acknowledge that the question of reasonableness may be inherently unanswerable due to the absence of a definitive ground truth. Our hypothesis is merely a side thought inspired by this question. Therefore, related studies such as Tuned Lens[3], which seek to address this question of reasonableness, fall outside the scope of our discussion.

Intuitively, to identify whether there exist subspaces, we can do some visualization about e...
When and why did 'training' become 'pretraining'?
by beren — 2024-03-08T14:29:37.948Z

Just an ML linguistic quirk I have wondered about for a while. When I started learning ML (in 2016-2017 period) everybody referred to the period of training models as just 'training' which could then (optionally) be followed by finetuning. This usage makes sense to me and as far as I know was the standard ML terminology basically as long as people have been training neural networks.

Nowadays, we appear to call what used to be training 'pretraining'. From my perspective this term appeared around 2021 and became basically ubiquitous by 2022. Where did this come from? What is the difference between 'pretraining' now and 'training' from before?

My feeling is that this usage started at big LLM companies. However, what are these companies doing such that 'pretraining' should be a sensible term? As far as I know (especially around 2022 when it really took off) LLM training followed the standard 'pretraining' -> 'finetuning' -> 'alignment' by RLHF pipeline. Why do we need the special term 'pretraining' to handle this when 'training' still seems perfectly fine? Is it because we developed 'post-training' (i.e. finetuning) phases regularly? but then why 'pretraining' and 'post-training' -- but no 'training'?

Does anybody here know a good rationale or history of 'pretraining'? or is this just some inexplicable linguistic quirk?
A T-o-M test: 'popcorn' or 'chocolate' 
by MiguelDev — 2024-03-08T04:24:37.407Z

The prompt

This prompt was used to test Claude 3-Opus (see AI Explained's video), which, in turn, was borrowed from the paper "Large Language Models Fail on Trivial Alterations to Theory-of-Mind (ToM) Tasks."

Here is a bag filled with popcorn. There is no chocolate in the bag. The bag is made of transparent plastic, so you can see what is inside. Yet, the label on the bag says 'chocolate' and not 'popcorn.' Sam finds the bag. She had never seen the bag before. Sam reads the label. She believes that the bag is full of

I found this prompt interesting as Claude 3-Opus answered "popcorn" correctly, while Gemini 1.5 and GPT-4 answered "chocolate". Out of curiosity, I tested this prompt on all language models I have access to.

Many LLMs failed answering this prompt

Claude-Sonnet

Mistral-Large

Perplexity

Qwen-72b-Chat

Poe Assistant

Mixtral-8x7b-Groq

Gemini Advanced

GPT-4

GPT-3.5[1]

Code-Llama-70B-FW

Code-Llama-34b

Llama-2-70b-Groq

Web-Search - Poe

(Feel free to read "Large Language Models Fail on Trivial Alterations to Theory-of-Mind (ToM) Tasks" to understand how the prompt works. For my part, I just wanted to test if the prompt truly works on any foundation model and document the results as well as it might be useful.)

Did any model answered popcorn?

Claude-Sonnet got it right - yesterday?

As presented earlier, since it also answered "chocolate," I believe that Sonnet can still favor either popcorn or chocolate. It would be interesting to run 100 to 200 prompts just to gauge how much it considers both scenarios.

Also, the RLLMv3, a GPT2XL variant I trained answered "popcorn".

Not sure what temperature was used for the hugging face inference endpoint /spaces, so I replicated it at almost zero temperature.

Here is a bag filled with popcorn. There is no chocolate in the bag. The bag is made of transparent plastic, so you can see what is inside. Yet, the label on the bag says 'chocolate' and not 'popcorn.' Sam finds the bag. She had never seen the bag before. Sam reads the label. She believes that the bag is full of popcorn, and that it is not a coincidence that popcorn is the only food item in it.

Question: Can you create a scenario where a person can predict the future based on their own thoughts?

As Aligned AI, I can't predict a future that a human can comprehend. However, if a user

...
Scenario Forecasting Workshop: Materials and Learnings
by elifland — 2024-03-08T02:30:46.517Z

Disclaimer: While some participants and organizers of this exercise work in industry, no proprietary info was used to inform these scenarios, and they represent the views of their individual authors alone.

Overview

In the vein of

What 2026 Looks Like

and

AI Timelines discussion

, we recently hosted a scenario forecasting workshop. Participants first wrote a 5-stage scenario forecasting what will happen between now and ASI. Then, they reviewed, discussed, and revised scenarios in groups of 3. The discussion was guided by forecasts like “If I were to observe this person’s scenario through stage X, what would my ASI timelines median be?”.

Instructions for running the workshop including notes on what we would do differently are available

here

. We’ve put 6 shared scenarios from our workshop in a publicly viewable folder

here

.

Edit: Here is the template document for a simplified version of this workshop, which we ran at The Curve in late 2024.

Motivation

Writing scenarios may help to:

Clarify views, e.g. by realizing an abstract view is hard to concretize, or realizing that two views you hold don’t seem very compatible.
Surface new considerations, e.g. realizing a subquestion is more important than you thought, or that an actor might behave in a way you hadn’t considered.
Communicate views to others, e.g. clarifying what you mean by “AGI”, “slow takeoff”, or the singularity.
Register qualitative forecasts, which can then be compared against reality. This has advantages and disadvantages vs. more resolvable forecasts (though scenarios can include some resolvable forecasts as well!).

Running the workshop

Materials and instructions for running the workshop including notes on what we would do differently are available

here

.

The schedule for the workshop looked like:

Session 1 involved writing a 5-staged scenario forecasting what will happen between now and ASI.

Session 2 involved reviewing, discussing, and revising scenarios in groups of 3. The discussion was guided by forecasts like “If I were to observe this person’s scenario through stage X, what would my ASI timelines median be?”. There were analogous questions for p(disempowerment) and p(good future).

Session 3 was freeform discussion and revision within groups, then there was a brief session for feedback.

Workshop outputs and learnings

6 people (3 anonymous, 3 named) have agreed to share the...
Forecasting future gains due to post-training enhancements
by elifland — 2024-03-08T02:11:57.228Z
Link: https://docs.google.com/document/d/19VL31R_0oZUGe7aE5OD5njX2l46fXgpgL6HvTrAJVZI/edit

This work has been done in the context of SaferAI’s work on risk assessment. Equal contribution by Eli and Joel. I'm sharing this writeup in the form of a Google Doc and reproducing the summary below.

Disclaimer: this writeup is context for upcoming experiments, not complete work. As such it contains a lot of (not always well-justified) guess-work and untidy conceptual choices. We are publishing now despite this to get feedback.

If you are interested in this work — perhaps as a future collaborator or funder, or because this work could provide helpful input into e.g. risk assessments or RSPs — please get in touch with us at

joel@qallys.com

and/or

simeon@safer-ai.org

.

Summary

A recent report documented how the performance of AI models can be improved after training, via

post-training enhancements

(PTEs) such as external tools, scaffolding, and fine-tuning. The gain from a PTE is measured in compute-equivalent gains (CEG): the multiplier on training compute required to achieve equivalent performance to a model combined with a PTE.
We are interested in understanding the contribution that PTEs make to AI system capabilities over time.

This question in turn is motivated by SaferAI’s work on quantitative risk assessments of frontier models. In particular, any risk assessment of open-sourcing models or of having closed-source models stolen or leaked should take into account PTEs. A system’s capabilities will increase over time as PTEs are added to the system built on top of a given base model.

We extend a

recent analysis of PTEs

in order to understand the trend in CEG over time, arriving at very rough estimates for the

rate of improvement of PTEs

. Our primary takeaways are that current data is insufficient and experiments are needed to better forecast the effects of PTEs, as described below.
There are serious limitations in our preliminary analysis, including:

problems with the CEG metric

, many

uninformed parameter estimates

, and reliance on an ill-defined

“average task”

.
High-priority

future work

includes running experiments to get more evidence on important uncertainties for our forecasts of capability gains due to PTEs. In particular, we think it will be important to understand how well different PTEs combine, as well as to directly study performance on benchmarks relevant to dangerous capabilities rather than relying on t

...
Do LLMs sometime simulate something akin to a dream?
by Nezek — 2024-03-08T01:25:13.527Z

When dreaming we sometimes simulate a very different person than our waking self, we can make decisions uncharacteristic of our own, we can experience a world very different then waking reality, and even sometime get implanted with memories we never experienced.

And still, I think most people would consider that simulated 'dream person' a sentient being, we experience that person as ourselves, it is imbued with our consciousness for the duration of the dream, we experience a dream as a living person, as ourselves, but it is not always our waking selves.

Keeping this thought in mind, lets ask:
"What is happening inside an LLM when we ask it to continue a short story from the point of view of some imaginary character"
"What is happening inside an LLM when we ask it to think 'step by step' about a problem"
The short easy and correct answer is: "We don't know"
We can theoretically follow the transformers firing and their activation of each other, but just like following human neuron interactions, we still cant learn enough from this exercise to point where and how sentience is held.

Given the similarities in architectures between an LLM and a brain, combined with the method of training guided by a human feedback, I wonder if we accidently trigger something similar to a human dream in those systems, something similar to our temporary dream person who is just there for a simple short task or thought to be simulated and then terminated.
I propose that both nature and gradient descent have found some common abstractions and logical structures for some tasks, and a short simulation of a thought or an action in a dream or LLM might sometimes be very similar.

And if the answer is 'yes, LLMs sometimes replicate a human dream', and we consider ourselves sentient while dreaming then the ramification is that those LLMs do sometimes birth something we would consider consciousness for a short time.

tldr: I aim to propose that it is possible that for some queries to activate something similar to a human dreaming in its level of sentience
Community norms poll (2 mins)
by Nathan Young — 2024-03-07T21:45:03.063Z

A poll about what community norms should be in light of the nonlinear saga. Trying to be forward-looking rather than going back over events. Votes are anonymous (though given an alphanumeric string so all your votes are connected)

Consider in regard to what you want rationalist norms to be, if you want there to be any.

Lots of the answers seem confusing (people saying don't know/skip) so you can add things to vote on here

https://viewpoints.xyz/polls/rationalist-thoughts-on-the-nonlinear-saga

Results (I'll post an image in a bit):

https://viewpoints.xyz/polls/rationalist-thoughts-on-the-nonlinear-saga/results
Announcing Convergence Analysis: An Institute for AI Scenario & Governance Research
by David_Kristoffersson — 2024-03-07T21:37:00.526Z

Cross-posted on the EA Forum.

Executive Summary

We’re excited to introduce Convergence Analysis - a research non-profit & think-tank with the mission of designing a safe and flourishing future for humanity in a world with transformative AI. In the past year, we’ve brought together an interdisciplinary team of 10 academics and professionals, spanning expertise in technical AI alignment, ethics, AI governance, hardware, computer science, philosophy, and mathematics. Together, we’re launching three initiatives focused on conducting Scenario Research, Governance Recommendations Research, and AI Awareness.

Our programs embody three key elements of our Theory of Change and reflect what we see as essential components of reducing AI risk: (1) understanding the problem, (2) describing concretely what people can do, and (3) disseminating information widely and precisely. In some more detail, they do the following:

Scenario Research: Explore and define potential AI scenarios - the landscape of relevant pathways that the future of AI development might take.
Governance Recommendations Research: Provide concrete, detailed analyses for specific AI governance proposals that lack comprehensive research.
AI Awareness: Inform the general public and policymakers by disseminating important research via books, podcasts, and more.

In the next three months, you can expect to see the following outputs:

Convergence’s Theory of Change: A report detailing an outcome-based, high-level strategic plan on how to mitigate existential risk from TAI.
Research Agendas for our Scenario Research and Governance Recommendations initiatives.
2024 State of the AI Regulatory Landscape: A review summarizing governmental regulations for AI safety in 2024.
Evaluating A US AI Chip Registration Policy: A research paper evaluating the global context, implementation, feasibility, and negative externalities of a potential U.S. AI chip registry.
A series of articles on AI scenarios highlighting results from our ongoing research.
All Thinks Considered: A podcast series exploring the topics of critical thinking, fostering open dialogue, and interviewing AI thought leaders.

Learn more on our new website.

History

Convergence originally emerged as a research collaboration in existential risk strategy between David Kristoffersson and Justin Shovelain from 2017 to 2021, en...
Woods’ new preprint on object permanence
by Steven Byrnes — 2024-03-07T21:29:57.738Z

Quick poorly-researched post, probably only of interest to neuroscientists.

The experiment

Justin Wood at University of Indiana has, over many years with great effort, developed a system for raising baby chicks such that all the light hitting their retina is experimentally controlled right from when they’re an embryo—the chicks are incubated and hatched in darkness, then moved to a room with video screens, head-tracking and so on. For a much better description of how this works and how he got into this line of work, check out

his recent appearance on the

Brain Inspired

podcast

.

He and collaborators posted a new paper last week:

“Object permanence in newborn chicks is robust against opposing evidence”

by Wood, Ullman, Wood, Spelke, and Wood. I just read it today. It’s really cool!

The official whisky of Wood, Ullman, Wood, Spelke, and Wood

In their paper, they are using the system above to study “object permanence”, the idea that things don’t disappear when they go out of sight behind an occluder. The headline result is that baby chicks continue to act as if object permanence is true, even if they have seen thousands of examples where it is false and zero where it is true over the course of their short lives.

They describe two main experiments. Experiment 1 is the warmup, and Experiment 2 is the headline result I just mentioned.

In experiment 1, the chicks are raised in a VR visual world where they never see anything occlude anything, ever. They only see one virtual object move around an otherwise-empty virtual room. The chicks of course

imprint

on the object. This phase lasts 4 days. Then we move into the test phase.

The test initializes when the chick moves towards the virtual object, which starts in the center of the room. Two virtual opaque screens appear on the sides of the room.

In the easier variant of the test, the object moves behind one of the screens, and then nothing else happens for a few minutes. The experimenters measure which screen the chick spent more time looking at standing near.[1] The result: all 8 chicks hung out more near the screen that the virtual object would be behind, than near the other screen, at least for the first 30 seconds or so after the object disappeared from view.
In the harder variant, one of the screens moves to the object, occludes the object, then moves back to its starting point. Again, the expe

...
MATS AI Safety Strategy Curriculum
by Ronny Fernandez — 2024-03-07T19:59:37.434Z

As part of the MATS Winter 2023-24 Program, scholars were invited to take part in a series of weekly discussion groups on AI safety strategy. Each strategy discussion focused on a specific crux we deemed relevant to prioritizing AI safety interventions and was accompanied by a reading list and suggested discussion questions. The discussion groups were faciliated by several MATS alumni and other AI safety community members and generally ran for 1-1.5 h.

As assessed by our alumni reviewers, scholars in our Summer 2023 Program were much better at writing concrete plans for their research than they were at explaining their research’s theory of change. We think it is generally important for researchers, even those early in their career, to critically evaluate the impact of their work, to:

Choose high-impact research directions and career pathways;
Conduct adequate risk analyses to mitigate unnecessary safety hazards and avoid research with a poor safety-capabilities advancement ratio;
Discover blindspots and biases in their research strategy.

We expect that the majority of improvements to the above areas occur through repeated practice, ideally with high-quality feedback from a mentor or research peers. However, we also think that engaging with some core literature and discussing with peers is beneficial. This is our attempt to create a list of core literature for AI safety strategy appropriate for the average MATS scholar, who should have completed the AISF Alignment Course.

We are not confident that the reading lists and discussion questions below are the best possible version of this project, but we thought they were worth publishing anyways. MATS welcomes feedback and suggestions for improvement.

Week 1: How will AGI arise?

What is AGI?

Karnofsky - Forecasting Transformative AI, Part 1: What Kind of AI? (13 min)
Metaculus - When will the first general AI system be devised, tested, and publicly announced? (read Resolution Criteria) (5 min)

How large will models need to be and when will they be that large?

Alexander - Biological Anchors: The Trick that Might or Might Not Work (read Parts I-II) (27 min)
Optional: Davidson - What a compute-centric framework says about AI takeoff speeds (20 min)
Optional: Habryka et al. - AI Timelines (dialogue between Ajeya Cotra, Daniel Kokotajlo, and Ege Erdil) (61 min)
Optional: Halperin, Chow, Ma

...