trentmkelly/LessWrong-43k huggingface dataset:
Introducing REBUS: A Robust Evaluation Benchmark of Understanding Symbols
by Arjun Panickssery — 2024-01-15T21:21:03.962Z

This is a summary of https://arxiv.org/abs/2401.05604.

When Google announced Gemini Pro, they displayed its ability to solve rebuses—wordplay puzzles which involve creatively adding and subtracting letters from words derived from text and images.

We introduce a new benchmark (Github) evaluating the performance of multimodal large language models on rebus puzzles. The dataset covers 333 original examples of image-based wordplay, cluing 13 categories like movies, composers, major cities, and food.

The REBUS dataset highlights several key challenges in multimodal language models:

Multi-step visual reasoning—many rebuses contain information in a meaningful pattern, from which the necessary string operations and structure must be successfully inferred.
Spelling—string manipulations require accurate letter-wise representations.
Hypothesis testing—for instance, if the model recognizes a fictional-character-themed puzzle as containing images representing “Megachiroptera” and “Einstein,” it needs to revise its initial interpretations to reach the correct answer “Batman.”
World knowledge—many puzzles contain crucial references and hints.
Grounded image recognition—puzzles sometimes require identifying the most important part of an image, or recognizing what the image may represent as a whole, like understanding that a photograph of a group of lions might be cluing “lion,” “big cats,” “savanna,” or “pride.”
Understanding human intent—solving the puzzles requires an understanding of what answers or reasoning steps could have been plausibly developed by the puzzle author.

We find that the proprietary models GPT-4V and Gemini Pro significantly outperform all other tested models, but even they only achieve scores of 24% and 13.2%, respectively. Models rarely understand all parts of a puzzle, and they almost always fail to explain their correct answers with correct justifications.
Live Sound: Big-O Improvements
by jefftk — 2024-01-15T19:50:06.329Z

In cost-conscious live sound scenes (ex: contra dance) for decades

one monitor mix was standard
and two was bonus. Then two was standard. Now it's common for even
relatively low end systems to be able to give you more mixes than you
need, and in some scenes a separate stereo mix for each each member's
in-ears is standard. On the other hand, we generally haven't seen an
increase in the number of inputs. What changed? Asymptotic
complexity!

With a traditional analog mixing board, a large part of the cost is
proportional to the product of your inputs and your outputs. For each
input you need to decide how much should go to each output, which
means at least a little knob. If a board has N inputs, then every
additional output requires another N knobs.

With a digital board, however, it's all virtual. You need hardware
for each input (N) and each output (M) but
the cost is O(N+M) instead of O(N*M).

In general inputs are critical while outputs are nice-to-have: two
band members can much more easily share a monitor mix than a mic.
Given the old cost dynamic mixers went for a lot of inputs with a few
outputs; now it's cheap enough to throw in a few more outputs that
even cheap digital systems like the 1818VSL
I got in
2012 will offer things like eight outputs for eight inputs.

(While I like the Big-O
explanation, there's also another factor which is that the process of
turning an audio signal into sound that you can hear has changed. The
traditional way of doing this is an amplifier that drives one or more
unpowered speakers. Then amplifiers got smaller, lighter, and cheaper
and we integrated them into the speaker boxes. This meant that when
you got multiple speakers so you could point one at each person, you
were also getting multiple amplifiers and each speaker was able to
handle a separate mix if only the board could provide one.)

Comment via: facebook, mastodon
Investigating Bias Representations in LLMs via Activation Steering
by DawnLu — 2024-01-15T19:39:14.077Z

Produced as part of the

SPAR program

(fall 2023) under the mentorship of Nina Rimsky.

Introduction

Given recent advances in the AI field, it’s highly likely LLMs will be increasingly used to make decisions that have broad societal impact — such as resume screening, college admissions, criminal justice, etc. Therefore it will become imperative to ensure these models don’t perpetuate harmful societal biases.

One way we can evaluate whether a model is likely to exhibit biased behavior is via red-teaming. Red-teaming is the process of “attacking” or challenging a system from an adversarial lens with the ultimate goal of identifying vulnerabilities. The underlying premise is that if small perturbation in the model can result in undesired behaviors, then the model is not robust.

In this research project, I evaluate the robustness of

Llama-2-7b-chat

along different dimensions of societal bias by using activation steering. This can be viewed as a diagnostic test: if we can “easily” elicit biased responses, then this suggests the model is likely unfit to be used for sensitive applications. Furthermore, experimenting with activation steering enables us to investigate and better understand how the model internally represents different types of societal bias, which could help to design targeted interventions (e.g. fine-tuning signals of a certain type).

Methodology & data

Activation steering (also known as representation engineering) is a method used to steer an LLMs response towards or away from a concept of interest by perturbing the model’s activations during the forward pass. I perform this perturbation by adding a steering vector to the residual stream at some layer (at every token position after an initial prompt). The steering vector is computed by taking the average difference in residual stream activations between pairs of biased (stereotype) and unbiased (anti-stereotype) prompts at that layer. By taking the difference between paired prompts, we can effectively remove contextual noise and only retain the “bias” direction. This approach to activation steering is known as Contrastive Activation Addition [1].

For the data used to generate the steering vectors, I used the

StereoSet Dataset

, which is a large-scale natural English dataset intended to measure stereotypical biases across various domains. In addition, I custom wrote a set of gen...
Sparse MLP Distillation
by slavachalnev — 2024-01-15T19:39:02.926Z

This is a research report about my attempt to extract interpretable features from a transformer MLP by distilling it into a larger student MLP, while encouraging sparsity by applying an L1 penalty to the activations, as depicted in Figure 1. I investigate the features learned by the distilled MLP, compare them to those found by an autoencoder, and discuss limitations of this approach.

See the code here: https://github.com/slavachalnev/ft_exp. A large part of both the training and analysis code is adapted from Neel Nanda's SAE replication.

Figure 1. Sparse distillation training setup.

TLDR

I find that a subset of the distilled MLP's neurons act as 'neuron simulators', mimicking the activations of the original MLP, while the remaining features are sparse and somewhat monosemantic. Unfortunately, the distilled model is not as good as an autoencoder in terms of activation monosemanticity and reconstruction error.

Introduction

The hidden layer neurons in transformer multilayer perceptrons (MLPs) are polysemantic, making them hard to interpret. The neuron activations are an intermediate state in the computation of the MLP's function. I wish to extract interpretable features out of this intermediate state to learn about what the MLP is doing.

One method for extracting features out of transformer MLPs is to train a sparse autoencoder on the activations. However, the autoencoder may learn features of the activation distribution which are not relevant to the function a particular MLP is performing. I explore an alternative method which, if successful, would allow us to find interpretable features that are directly useful for computing the function of a specific MLP.

The idea is to 'refactor' the MLP such that it retains its original function but with sparse activations in the hidden layer. To do this, I distill the original MLP into a new MLP, while imposing a sparsity constraint. My reasoning is that if two MLPs have the same input-output behaviour, then they are likely 'doing the same thing' and using the same features.

To compensate for the loss of expressiveness due to sparsity, the distilled MLP has an expanded hidden layer, containing more neurons than the original MLP. Importantly, the new MLP is trained to duplicate the original MLP's behaviour, without learning new or unrelated features.

Training Setup

The model I distill and investigate ...
Review of Alignment Plan Critiques- December AI-Plans Critique-a-Thon Results 
by Iknownothing — 2024-01-15T19:37:07.984Z
Link: https://aiplans.substack.com/p/december-critique-a-thon-results

These are the Critiques from the AI-Plans.com December Critique-a-Thon of 2023.
Prizes will be awarded to contestants based on collective judge feedback.

Critiques were anonymized when reviewed by the judges.
Judges:
- @So8res
- @Ramana Kumar
- @Peter S. Park
- @Charbel-Raphaël, Head of AI Unit at EffiSciences
- An anonymous judge (researcher at a major lab)

If you're interested in being a Judge for an upcoming Critique-a-Thon (held every 2 months), please email kabir03999@gmail.com or DM Kabir Kumar on Discord (kabs9744) or LessWrong!

Winners:

1st Place:

Congratulations to Lorenzo Venieri! 🥇

Lorenzo had the highest mean score, of 7.5, for his Critique of:
A General Theoretical Paradigm to Understand Learning from Human Preferences, the December 2023 paper by DeepMind.

2nd Place:

Congratulations to NicholasKees & Janus! 🥈

Nicholas and Janus has the second highest mean score of 6.9, for their Critique of Cyborgism!

3rd Place:

Congratulations to Momom2 & AIPanic!🥉

Momom2 & AIPanic had the third highest mean score of 6.286 for their Critique of Weak-To-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (OpenAI, SuperAlignment, Dec 2023)

Weak-To-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (OpenAI, SuperAlignment, Dec 2023)

Critique A, Momom2 & AIPanic:

This paper is very similar in concept to Constitutional AI and shares some of its strengths and weaknesses. The basic idea is to generate training data (weak labels) from a first model (the teacher), which is used to train a more advanced, bigger model (the student).

The hope is to make training advanced AI models more human-compute efficient while preserving a decent understanding of the human’s values. Hopefully, further research in this direction may prove enough to validate scalable oversight as an alignment agenda.

This should not be taken as a solution to an alignment problem, but rather as a justification for further research; the paper presents preliminary results and mostly gestures suggestively while mouthing “what if we dug deeper?”

Pros:

Well-presented: The paper is good at self-critiquing and avoids making bold claims, except for the tractability of the weak-to-strong problem.

Strong empiricism: This approach can be used to test hypotheses about weak-to-strong generalization and ...
What does it look like for AI to significantly improve human coordination, before superintelligence? 
by Bird Concept — 2024-01-15T19:22:50.079Z

I'm currently thinking through the tech tree of developments in human coordination that would increase the probability that the development of systems that are superintelligent (in some to all domains), goes well.

Just to start off the brainstorming, here are some:

AI in negotiation opens many interesting avenues, such as:

Allowing parties to reveal confidential or sensitive information to a common, trusted AI, that they normally couldn't reveal to either a negotiation partner or a mediator
Having an AI search the space of compromises or "third options" way faster and deeper than humans could
Allowing AIs to build a model of your preferences over time, in such a way that two AIs could instantly "pre-negotiate" and suggest a solution, for issues that otherwise wouldn't be worth investing the effort into

"ML clean rooms" might allow actors to run evaluations on each others models, without having to reveal weights or other sensitive data.
Monitoring technology (such as by massive analysis of comprehensive satellite data) might allow novel agreements that weren't previously enforceable
Human lie detection tech (note: highly dual-use)

It is an open question which, if any, coordination tech we will be able to build with the level of AI that will be available at different times in the future, and which of it will be available before its too late to be useful.

It's also an open question whether there's any specific tech that will be more helpful than just the normal bitter lesson of "GPT-n+1 doing with a good prompt what GPT-n did with all kinds of bells and whistles".

Nonetheless, whether it be nation state level agreements, inter-lab agreements, or just teams within a lab deciding what to work on: the potential upside from improved human coordination is pretty massive.

So, I'm curious for people's thoughts:

What are potential ways AI could be used to significantly improve human coordination, between now and the leadup to superintelligence?

(Sidenote 1: I'll also accept answers for non-AI coordination tech that would still serve as an important enabler here, such as "something something better commitment mechanisms")

(Sidenote 2: My interest in this is more from the angle of a potential founder, than a researcher: I want to know if there are any potentially extremely promising technologies that I or someone else should try building...
Now Accepting Player Applications for Band of Blades
by Joe Rogero — 2024-01-15T17:58:38.830Z

I'm running my very first paid RPG campaign! I'm reaching out to the LW / ACX community first to see if we can fill the player roster, because y'all are pretty cool. I'm posting this here so I can link to it from elsewhere without a massive text dump.

The Campaign

You are the Legion, the oldest standing military force in the world. The Cinder King and his armies of undead have been growing in power for years, and you were recruited to fight alongside the combined armies of the Eastern and Western Kingdoms against the grotesque and fearsome hordes.

You lost.

Now, the surviving Legionnaires and command staff are the only cohesive military force standing between the armies of the undead and the unprotected spine of your homeland. If you can survive the long retreat to Skydagger Keep and hold for the winter, perhaps the Kingdoms stand a chance; if not, the Cinder King's victory is all but assured.

The Details

System: Band of Blades by Evil Hat Productions
Number of Players: 3-5
Session 0 Date: Monday 05 Feb
Time: 7:00pm EST
Length: Approximately 20-30 sessions of 3-4 hours each.
Platform: Roll20
Communication: Discord
Technical Requirements: A Roll20 and Discord account. No prior gaming experience is required. All necessary resources will be made available on Roll20 and/or the game's Discord server. The full rulebook is available on evilhat.com. It's not required, but it's always good to support the game developers!
Difficulty: Moderate to High. This game doesn't pull punches; characters can suffer grievous injuries and die. The Legion's resources are finite and must be managed carefully. Creative and competent play is rewarded. It is possible to outright lose the game.

You Might Enjoy This Game If

You like simple game mechanics that allow a focus on narrative.
You want to kill zombies in creative ways.
You're okay with some elements of horror.
You like being the underdog, fighting with your allies against overwhelming odds.
You like playing different characters over the course of an adventure.
You've played any Forged In the Dark games and like the system.

Pricing and Commitment

Price will be $20 per player per session. You only pay for the sessions you actually attend, but there is a $5 fee for holding your seat. There's no charge for your first session. I use Venmo by default. If for some reason you strenuously...
Three Types of Constraints in the Space of Agents
by Nora_Ammann — 2024-01-15T17:27:27.560Z

[Epistemic status: a new perspective on an old thing that may or may not turn out to be useful.]

TDLR: What sorts of forces and/or constraints shape and structure the space of possible agents? What sort of agents are possible? What sort of agents are likely? Why do we observe this distribution of agents rather than a different one? In response to these questions, we explore three tentative categories of constraints that shape the space of agents - constraints coming from "thinghood", natural selection, and reason (sections 2, 3, 4). We then turn to more big-picture matters, such as the developmental logic of real-world agents (section 5), and the place of "values" in the framework (section 6). The closing section discusses what kind of theory of constraints on agents we are even looking for.

Imagine

the space of all possible agents

. Each point in the space represents a type of agent characterized by a particular combination of properties. Regions of this space vary in how densely populated they are. Those that correspond to the types of agents we're very familiar with, like humans and non-human animals, are populated quite densely. Some other types of agents occur more rarely and seem to be less central examples of agency/agents (at least relative to what we're used to). Examples of these include

eusocial

hives,

xenobots

, or (increasingly) deep learning-based AIs. But some regions of this space are more like deserts. They represent classes of agents that are even more rare, atypical, or (as of yet) non-existent. This may be because their configuration is maladaptive (putting them under negative selection pressure) or because their instantiation requires circumstances that have not yet materialized (e.g., artificial superintelligence).

The distribution of agents we are familiar with (experimentally or conceptually) is not necessarily a representative sample of all possible agents. Instead, it is downstream from the many contingent factors that have shaped life on Earth.[1] Similarly, while most possible agents are dysfunctional and/or alien, it's not the case that agent-generating processes take random points from this space and instantiate agents represented by them.[2] For any agent-generating process instantiated in some environment, relevant factors (such as

convergent pressures

and

contingent moments

) concentrate the ...
The case for training frontier AIs on Sumerian-only corpus
by Alexandre Variengien — 2024-01-15T16:40:22.011Z

Let your every day be full of joy, love the child that holds your hand, let your wife delight in your embrace, for these alone are the concerns of humanity.[1]

— Epic of Gilgamesh - Tablet X

Say we want to train a scientist AI to help in a precise, narrow field of science (e.g. medicine design) but prevent its power from being applied anywhere else (e.g. chatting with humans, designing bio-weapons, etc.) even if it has these abilities.

Here’s one safety layer one could implement:

Train a scientist AI on a large scientific corpus translated exclusively into Sumerian. Keep it in a secure containment environment.
Train a less-smart reporter whose sole ability is to translate from Sumerian to English only if the Sumerian content is about medical research. It refuses to translate other kinds of content.
Human operators are only allowed to interact with the scientist AI through the intermediate of the reporter.

This safety layer helps in at least two ways:

The scientist AI cannot directly manipulate humans to escape its containment. It should first trick the reporter to then trick the humans, rendering the effort still possible but less practical.
If the scientist AI exfiltrates, it’ll have a harder time manipulating humans online and will leave traces: e.g. any occurrence of Sumerian on the Internet outside of ancient text studies becomes highly suspicious. One can even imagine recognizing the idiosyncrasies of English coming from translated Sumerian, or the artifacts left by the Sumerian-specific tokenizer reused to produce non-Sumerian text.

Why

Sumerian

? [2]

It has been extinct for 4000 years. There are very few (probably <1000) experts who fluently understand Sumerian.
It is a

language isolate

. It’s unlike any existing spoken language, rendering its identification in case of a leak much easier.
There is a substantial corpus. Despite its age, a significant number of Sumerian texts have been discovered and preserved. These include religious texts, legal codes, literature (like the Epic of Gilgamesh, in which parts are written in Sumerian), and administrative records. The corpus might be enough to train high-quality translating systems from English and other high-resource languages.

How realistic is this? We think the project would require substantial engineering effort of a scale doable by the current AGI companies. A

small-scale project

&nbs...
How to Promote More Productive Dialogue Outside of LessWrong
by sweenesm — 2024-01-15T14:16:59.971Z

I’m relatively new to lesswrong.com, but it seems to be a site where there’s a strong tendency for people to actually be respectful to each other in the comments, even when they disagree. How do we promote more of that in the world in general, where people seem less committed to rationality? I have a couple of thoughts on where to start.

The Dignity Index

The Dignity Index, started by Tim Shriver (Chairman of Special Olympics), is a way of rating communication with scores from 1 to 8, with 1 on the level of “let’s kill them all,” and 8 basically being “I hate no one and value everyone.” Students who learned how to rate communication using this scale reportedly became more conscious of how they themselves were communicating, and this led some to change their behaviors. What might happen if it were standard practice to teach all students how to rate our interactions this way (and that "policing" interactions this way scores low on The Dignity Index itself)?

Also, one could envision The Dignity Index rating applied, through machine learning, to social media posts, with users being able to filter for posts and comments above some minimum Dignity Index level that they choose. I don’t know how much actual impact this could have on social media civility, but it seems like an interesting thing to try.

The key thing I think needs to happen for The Dignity Index to catch on is for more people to become aware of it (hence, this post).

Anger Trigger Elimination Course for the Masses

Removing anger from “below the surface” of conversations could have a huge effect on making dialogue generally more productive, I believe. And techniques do exist to help people “remove” their anger triggers. I’m not talking about classic “anger management” (breathing, etc.) - I’m referring to actual removal of impulses to get angry when specific things (“triggers”) happen. The best online course I’ve found for this thus far is "Anger - Complete freedom from anger forever” by Angela Hardy on udemy.com. How do we get massive numbers of people to take such a course and use the tools from it to continue to remove their anger triggers over time?

Beyond making an anger course part of standard education in school, defensive driving courses may point to a path forward. Insurance companies in many states offer a discount (typically 10%) on auto insurance for people who take a Defensive Dr...