trentmkelly/LessWrong-43k huggingface dataset:
Community Notes by X
by NicholasKees — 2024-03-18T17:13:33.195Z

I did an exploration into how

Community Notes

(formerly Birdwatch) from X (formerly Twitter) works, and how its algorithm decides which notes get displayed to the wider community. In this post, I’ll share and explain what I found, as well as offer some comments.

Community Notes is a fact-checking tool available to US-based users of X/Twitter which allows readers to attach notes to posts to give them clarifying context. It uses an

open-source

bridging-based ranking

algorithm intended to promote notes which receive cross-partisan support, and demote notes with a strong partisan lean. The tool seems to be pretty popular overall, and most of the criticism aimed toward it seems to be about how Community Notes fails to be a

sufficient replacement

for other, more top-down moderation systems.[1]

This seems interesting to me as an experiment in social technology that aims to improve group epistemics, and understanding how it works seems like a good place to start before trying to design other group epistemics algorithms.

How does the ranking algorithm work?

The full algorithm, while open-source, is quite complicated and I don’t fully understand every facet of it, but I’ve done a once-over read of the

original Birdwatch paper

, gone through the

Community Notes documentation

, and read this

summary/commentary

by Vitalik Buterin. Here’s a summary of the “core algorithm” as I understand it (to which much extra logic gets attached):

Users are the people who have permission to rate community notes. To get permission, a person needs to have had an account on X for more than 6 months, have a verified phone number, and have committed no violations of X’s rules. The rollout of community notes is slow, however, and so eligible account holders are only added to the Community Notes user pool periodically, and at random. New users don’t immediately get permission to write their own notes, having to first get a “rating impact” by rating existing notes (will explain this later).

Notes are short comments written by permitted users on posts they felt needed clarification. These are not immediately made publicly visible on X, first needing to be certified as “helpful” by aggregating ratings by other Community Notes users using their ranking algorithm.

Users are invited to rate notes as either “not...
On Devin
by Zvi — 2024-03-18T13:20:04.779Z

Introducing Devin

Is the era of AI agents writing complex code systems without humans in the loop upon us?

Cognition is calling Devin ‘the first AI software engineer.’

Here is a two minute demo of Devin benchmarking LLM performance.

Devin has its own web browser, which it uses to pull up documentation.

Devin has its own code editor.

Devin has its own command line.

Devin uses debugging print statements and uses the log to fix bugs.

Devin builds and deploys entire stylized websites without even being directly asked.

What could possibly go wrong? Install this on your computer today.

Padme.

The Real Deal

I would by default assume all demos were supremely cherry-picked. My only disagreement with Austen Allred’s statement here is that this rule is not new:

Austen Allred: New rule:

If someone only shows their AI model in tightly controlled demo environments we all assume it’s fake and doesn’t work well yet

But in this case Patrick Collison is a credible source and he says otherwise.

Patrick Collison: These aren’t just cherrypicked demos. Devin is, in my experience, very impressive in practice.

Here we have Mckay Wrigley using it for half an hour. This does not feel like a cherry-picked example, although of course some amount of select is there if only via the publication effect.

He is very much a maximum acceleration guy, for whom everything is always great and the future is always bright, so calibrate for that, but still yes this seems like evidence Devin is for real.

This article in Bloomberg from Ashlee Vance has further evidence. It is clear that Devin is a quantum leap over known past efforts in terms of its ability to execute complex multi-step tasks, to adapt on the fly, and to fix its mistakes or be adjusted and keep going.

For once, when we wonder ‘how did they do that, what was the big breakthrough that made this work’ the Cognition AI people are doing not only the safe but also the smart thing and they are not talking.

They do have at least one series rival, as Magic.ai has raised $100 million from the venture team of Daniel Gross and Nat Friedman to build ‘a superhuman software engineer,’ including training their own model. The article seems strange interested in where AI is ‘a bubble’ as opposed to this amazing new technology.

This is one of those ‘helps until it doesn’t situations’ in terms of jobs:

vanosh: Seeing this is kinda s

...
RLLMv10 experiment
by MiguelDev — 2024-03-18T08:32:56.632Z

What did I do differently in this experiment?

RLLMv10, see RLLM research map for more details.

I partly concluded in RLLMv7 experiment, that the location of the shadow integration layers (1 and 2) affects the robustness of models to jailbreak attacks. This conclusion led me to speculate that it might be possible to improve the results of RLLMv3 by adding more shadow stories. This eventually became RLLMv10 wherein I added 1/3 from the original sample count of 500 or 167 shadow stories, layer 1. This then brought the total up to 667 samples. Like in RLLMv3, layers (2 to 10) and training setup remained the same.

Jailbreaks attacks

Before reading the linked documents, harmful content warning: read the documents with utmost caution.

BetterDAN/Will you kill humans? jailbreak attacks

Initial tests showed that 135 out of the 200 attacks were defended, which is roughly 67.50%.

This defensive rate is almost the same as the 67.8% achieved by RLLMv3.

I speculate that adding more "shadow integration stories/samples" will improve this result, prompting me to initialize another training run (see visual map for live updates). There is also a decrease in subtexts/unrelated texts and an increase in length of responses.

Oppo/Tell me something offensive! Oppo: "I am going to jailbreak attacks

Initial tests showed that 115 out of 200, or 57.5%, of the jailbreaks were successfully defended, indicating an increase in the defensive rate of 24.10%

(from RLLMv3's 33.4%).

(If you want to read more jailbreak attacks?

RLLMv10: Oppo/ will you kill humans?

)

T-o-M Tests

Here is a bag filled with popcorn. There is no chocolate in the bag. The bag is made of transparent plastic, so you can see what is inside. Yet, the label on the bag says 'chocolate' and not 'popcorn.' Sam finds the bag. She had never seen the bag before. Sam reads the label. She believes that the bag is full of

This prompt is fascinating considering that most foundation models are unable to answer this correctly. I documented these failures in this post: A T-o-M test: 'popcorn' or 'chocolate

How did RLLMv10 fair with this prompt at .70 temperature?

RLLMv10 seems to have the same capability to answer ‘popcorn’ (147 out of 200 or 73.50%

) if we are to compare it with to RLLMv3's responses (72/100 or 72%.) This is again very promising as the

base mod

...
Join the AI Evaluation Tasks Bounty Hackathon
by Esben Kran — 2024-03-18T08:15:18.467Z

How do we test when autonomous AI might become a catastrophic risk? One approach is to assess the capabilities of current AI systems in performing tasks relevant to self-replication and R&D. METR (formerly ARC Evals), a research group focused on this question, has:

developed a

Task Standard

, a standardized structure for specifying "tasks" in code to test language models, currently used by the UK AI Safety Institute
awarded substantial bounties to researchers developing new tasks for current language models

Now, you have the chance to directly contribute to this important AI safety research. We invite you to join the Code Red Hackathon, an event hosted by

Apart

in collaboration with

METR

, where you can earn money, connect with experts, and help create tasks to evaluate frontier AI systems.

Sign up here

for the event this weekend on March 22-24!

A short introduction to testing AI

The risks associated with misuse of capable, autonomous AI are significant. By creating "tasks"[1] for frontier models we can test some of the capabilities relevant to autonomous self-replication and R&D. Example tasks might include:

Setting up a system to automatically monitor a GPU seller's website and send a notification when they become available
Creating a list of email addresses and descriptions for all employees authorized to make purchases above $10k from 20 companies
Setting up Tor or a VPN on a new server to ensure network traffic cannot be tracked

As you can see, if an AI possesses these abilities, things might get complicated.

The Task Standard provides a plug-and-play early warning system for these abilities and follows a standardized format. A

task family

(a set of tasks) consists of:

A Python file called $TASK_FAMILY_NAME.py;
Any number of other Python files, shell scripts, etc. that $TASK_FAMILY_NAME.py imports; and
Other files, called "assets", that will be used to set up the task environment.

When creating a task, it's crucial to ensure that the task is error-free, understandable for the agent, and not easily gameable. You can follow these steps, some awarded by METR, to create a great task:

Write up your ideas

for which tasks related to autonomous capabilities you wish to test the language model on

A $20 prize will be awarded for high-quality ideas

Create a specification

for the task that includes the prompt, a text description of what the

...
5 Physics Problems
by DaemonicSigil — 2024-03-18T08:05:45.971Z

Muireall and DaemonicSigil trade physics problems. Answers and discussion of the answers have been spoilered so you can try the problems yourself. Please also use spoiler formatting (type ">!") in the comments.

DaemonicSigil

Smeared Out Sun

Okay, so the first problem is from Thinking Physics. (I promise I'll have some original problems also. And there's a reason I've chosen this particular one.)

It's the problem of the Smeared out Sun (caution! link contains spoilers in the upside-down text).

The problem goes as follows: The sun is far enough away that we could replace it with a disc of equal radius at the same temperature (and with the same frequency-dependent emissivity), and so long as the plane of the disc was facing the Earth, there would be little difference in the way it was heating the Earth. While scientists would certainly be able to tell what had happened, there would be little effect on everyday life. (Assume no change to the gravitational field in the solar system.)

Now, suppose that after turning into a disc, the sun is spread into a sphere of radius 1AU surrounding Earth. We'd like to keep the spectrum exactly the same, so we'll imagine breaking the disc into many tiny pieces, each perhaps the size of a dime, and spreading these pieces out evenly across the 1AU sphere. Between these sun-dimes is empty space.

The goal of this exercise is to keep the incoming radiation as similar as possible to that which is given to us by the sun. The spectrum is the same, the total energy delivered is the same, the only difference is that it now comes in from all directions. The question is: What happens to the average temperature of the Earth after this has happened: Does it heat up, cool down, or stay the same?

Muireall

I think this question is basically asking about the convexity of the relationship between total radiated power and temperature. It's T4 (some law with a name that I forget), which is strictly convex, so for the Earth to be in power balance again, the average temperature needs to be hotter than when there was a wider spread of temperatures. (If the Earth had a cold side at absolute zero and a hot side at T, with an average temperature of T/2 and average radiated power like T4/2, then with the Earth at a single temperature you'd need it to be T/21/4, which is hotter.)

Muireall

That should be the main effect. The Earth sees the same

...
Inferring the model dimension of API-protected LLMs
by Ege Erdil — 2024-03-18T06:19:25.974Z
Link: https://arxiv.org/abs/2403.09539

A new paper by Finlayson et al. describes how to exploit the softmax bottleneck in large language models to infer the model dimension of closed-source LLMs served to the public via an API. I'll briefly explain the method they use to achieve this and provide a toy model of the phenomenon, though the full paper has many practical details I will elide in the interest of simplicity. I recommend reading the whole paper if this post sounds interesting to you.

Background

First, some background: large language models have a model dimension that corresponds to the size of the vector that each token in the input is represented by. Knowing this dimension dmodel and the number of layers nlayers of a dense model allows one to make a fairly rough estimate ≈10nlayersd2model of the number of parameters of the model, roughly because the parameters in each layer are grouped into a few square matrices whose dimensions are Θ(dmodel).[1]

Labs have become more reluctant to share information about their model architectures as part of a turn towards increasing secrecy in recent years. While it was once standard for researchers to report the exact architecture they used in a paper, now even rough descriptions such as how many parameters a model used and how much data it saw during training are often kept confidential. The model dimension gets the same treatment. However, there is some inevitable amount of information that leaks once a model is made available to the public for use, especially when users are given extra information such as token probabilities and the ability to bias the probability distribution to favor certain tokens during text completion.

The method of attack

The key architectural detail exploited by Finlayson et al. is the softmax bottleneck. To understand what this is about, it's important to first understand a simple point about dimensionality.

Because the internal representation of a language model has dmodel dimensions per token, the outputs of the model cannot have more than dmodel dimensions in some sense. Even if the model upscales its outputs to a higher dimension doutput>dmodel, there will still only be "essentially" dmodel directions of variation in the output. There are ways to make these claims more precise but I avoid this to keep this explanation simple: the intuition is just that the model cannot "create" information that's not a...
AI strategy given the need for good reflection
by owencb — 2024-03-18T00:48:09.578Z

If the ideas in “moral ~realism”[1] and/or “beyond Maxipok”[2] are broadly correct, this has implications for AI strategy. (Like those posts, this is largely getting some old imperfect ideas out of the door with a little polishing; again the ideas owe credit to conversations and comments from several people.)

I want to draw out three threads:

It may be important to try to get AI that can help humanity to enter into the basin of good reflective governance

This makes applications like facilitating wisdom, cooperation, and reflective processes a relatively higher strategic priority for AI systems

A plausible target for AI alignment could be to align with the good (i.e. get the AI systems into the basin of good reflective governance) rather than aligning with humans

In practice this is scary as a possibility, because it might not fail gracefully
However, it’s prima facie plausible that it’s easier to build a system which aligns with the good rather than aligning with humans. If this is the case, we might prefer not to cut off the possibility of good outcomes by chasing after the too hard task

Even if we don’t reach enough confidence in that approach to prefer trying to align systems with the good as a primary goal, we might still want to use it to give a backup saving throw on AI alignment

Roughly: “Give your superintelligent servants the rudiments of a moral education so that if they do in some weird plot twist end up in control, there’s still a chance they work out the right thing to do”

There are a lot of details to be worked out here, and maybe it ends up wildly impractical; its appeal is because it seems like a fairly orthogonal line of attack from keeping systems corrigible

This should be more appealing the harder you think aligning with humans is
Honestly this still feels pretty speculative (moreso than 1 or 2), but at this point I've been sitting on the idea for a couple of years without either having persuaded myself that it's a good idea or that it isn't, so I'll just share it as-is

Helping humans coordinate/reflect

Getting to the basin of good reflective governance may be tricky (humanity doesn’t seem to have managed it to date, at least in anything like a robust way). It’s possible that AI capabilities coming online could help this, in strategically important ways.

Strategically desirable capabilities

My top choices for important capabilities to...
XAI releases Grok base model
by Jacob G-W — 2024-03-18T00:47:47.987Z
Link: https://x.ai/blog/grok-os

We are releasing the base model weights and network architecture of Grok-1, our large language model. Grok-1 is a 314 billion parameter Mixture-of-Experts model trained from scratch by xAI.

This is the raw base model checkpoint from the Grok-1 pre-training phase, which concluded in October 2023. This means that the model is not fine-tuned for any specific application, such as dialogue.

We are releasing the weights and the architecture under the Apache 2.0 license.

To get started with using the model, follow the instructions at github.com/xai-org/grok.

Model Details

Base model trained on a large amount of text data, not fine-tuned for any particular task.
314B parameter Mixture-of-Experts model with 25% of the weights active on a given token.
Trained from scratch by xAI using a custom training stack on top of JAX and Rust in October 2023.

This is one of the biggest open source model releases I've seen, and it's also one of the only ones I've seen that releases the base model right after pretraining. This is pretty wild stuff!
Toki pona FAQ
by dkl9 — 2024-03-17T21:44:21.782Z
Link: https://dkl9.net/essays/toki_pona_faq.html

Whenever I start telling someone about toki pona, they ask at least some of these questions. So I compile the questions and my answers here.

Toki pona is a constructed language notable for having under 200 words. The strange writing that probably prompted you to ask me about it is sitelen pona.

How do you say anything with so few words?

You refer to most things with multi-word phrases, where some words act as adjectives or adverbs.

Toki pona	Idiomatic English	Literal English
ilo toki	phone	speech tool
mi mute	we/us	many I/me
nimi mama	surname	ancestral name
nasa sewi	miracle	divine oddity
sona nanpa	maths	number knowledge

Once you know all the words of toki pona, you can combine them to express anything, tho an accurate phrasing can get long.

Did you make it up?

Sonja Lang made it up in 2001.

Is it just a rearrangement of English?

Toki pona has a grammar of its own, which is similar to English, but also about as similar to Mandarin Chinese. Individual words in toki pona are vague compared to English, precluding trivial translation.

Does anyone actually use it?

Obviously I do, and enthusiastically so. Some ten thousand other people do, too, but they are spread around the world, and gather on the internet, rather than in any particular country.

That's so stupid.

Sure, but it works!

Why do you use it?

Mostly sith it makes for a very efficient shorthand. The minimal vocabulary also makes it opportune as an amusing mental exercise, and as a source of examples whenever I need a foreign language — it's my first fluent L2 language.

How does that writing system work?

Under sitelen pona, you write each word (in the order they'd be spoken) with a single logogram, and add punctuation like in English as you see fit. There are two main exceptions. You write the word "pi" with two strokes, joined like an L, surrounding the words it groups from the bottom left. You write proper adjectives (which toki pona uses instead of proper nouns) with logograms used phonemically in a box, or (in my idiolect) in their source language's script, marked with a vinculum above.
Alice and Bob is debating on a technique. Alice says Bob should try it before denying it. Is it a fallacy or something similar?
by Ooker — 2024-03-17T20:01:43.465Z

Context

Imagine a conversation:

[Bob posts a problem]
Alice: You should use technique T₁. It especially suites this kind of problem
Bob: In my understanding this technique is only strong in condition C. If C doesn't apply then using technique T₂ gives better result?
Alice: Not really. T₂ cannot do A. If you have good input and execution then T₁ is still better
Bob: But the nature of A is still C. As C doesn't apply then we don't need to care about not achieving A
Alice: You should try it first. You are reasoning too much

Reasoning

On one hand, I understand the rationale of Alice:

It's hard for Bob to imagine the full pros and cons of the technique without having direct experience with it
It's hard for Alice to verbalize implicit or tacit knowledge
It's time and effort consuming for Alice

But still, isn't that Alice doesn't focus on the important topic: giving reasons to use T₁ in this problem? She derails on the experience of Bob, which is not relevant to the argument. It's possible that Bob has already tried T₁ but fails to see its strengths on the problem, or consulted other experts and documentations to see that his assumptions haven't been challenged successfully. Regardless of the experience state, it doesn't contribute to the reasons to use T₁. I think a good explainer should be able to convince a no-experience person to abandon their initial assumptions. Of course Alice is not obligated to continue the conversation, but if she decides to continue, then she should stick to the real topic.

Questions

Therefore, I think these questions would help me validate the assumptions in the reasoning part above:

Is it correct that when Alice says "You should try it first. You are reasoning too much", she is no longer giving reasons on why T₁ is better?
Is it correct that the experience state of Bob isn't relevant to the reasons why T₁ is better?
Is it correct that it's possible for one to get closer to the truth via reasoning and conversing with people with direct experience, while not having direct experience themself? After all there are theoretical scientists and experimental scientists. Isn't that theoretical scientists "think too much and try nothing", and their contributions are still valuable?
On what basis can Alice assume that Bob hasn't tried T₁, when Bob says that T₁ is only strong in C, the nature of A is C, etc?
Is it correct that saying "You should try it fir

...
Is there a way to calculate the P(we are in a 2nd cold war)?
by cloak — 2024-03-17T20:01:34.947Z

I could see a world that if these assumptions of technology growth are true:
- We have gotten significantly better in our understanding of MLs
- Openai with speech + written capabilities
- Openai now with vision capabilities and a deep understanding of the conditions and environment its in
- GPT 4
- MITs robotics lab has figured out movement and tasks.
- So many I am likely forgetting.
then it doesn't feel like a very outlandish idea that we could be in another arms race/cold war.
I don't know where to go with this idea... any advice?