trentmkelly/LessWrong-43k huggingface dataset:
Any Trump Supporters Want to Dialogue?
by k64 — 2024-09-28T19:41:55.370Z

It's that time of year - the time when rationality seems increasingly scarce as political tensions rise. I find myself wishing I could have one of the people I see reaching super different conclusions shoot me with a POV gun so I could understand what it's like being on the other side.
I'm not strongly left-leaning, so I don't have trouble understanding why people may have some concerns about the left - but I have 0% support for Donald Trump, so if you want to explain to me why you think he's great, go for it. I also think that the election is close to 50/50 currently, so if you think it's 80+/20- either way, I'm also interested in hearing from you.

2 notes:
1. I really wish I understood how the irrational people were thinking about it, but unfortunately, they aren't able to explain it to me in a way I can understand, so I have to settle for rational people on the other side, but if you feel you have a good grasp on how your less rational peers tick, please share.
2. In your comment, let me know if you want pushback or not. Let's make the options: "just listen", "listen and question", and "open". Just listen means I'll attempt to only ask questions to clarify your meaning but not test edge cases. Listen and questions means I'll also ask about edge cases, internal coherence, other considerations, but won't be presenting evidence or making statements. Open means I can respond however I want.
Evaluating LLaMA 3 for political sycophancy 
by alma.liezenga — 2024-09-28T19:02:36.342Z

TLDR: I evaluated LLaMA v3 (8B + 70B) for political sycophancy using one of the two datasets I created. The results for this dataset suggest that sycophancy definitely occurs in a blatant way for both models though more clearly for 8B than for 70B. There are hints of politically tainted sycophancy, with the model especially adjusting to republican views, but a larger dataset is needed to make any definitive conclusions about this topic. The code and results can be found here.

This intro overlaps with that of my other post, skip if you have read it already.

With elections in the US approaching while people are integrating LLMs more and more into their daily life, I think there is significant value in evaluating our LLMs thoroughly for political sycophantic behaviour. Sycophancy is shown by LLMs when they give responses that match the user’s beliefs over truthful ones. It has been shown that state-of-the-art AI assistants exhibit sycophantic behaviours. This could be caused by the fact that, in training, humans prefer responses that match their own views as well as those that are written in a convincing manner (Sharma et al., 2023).

It is obvious that sycophantic behaviour paired with (over)reliance on LLMs can cause a dangerous situation, especially amidst elections. This process also seems similar to an existing and more commonly known phenomenon, (partially) caused by AI systems: filter bubbles. In filter bubbles, users are shown less and less information that disagrees with their viewpoints, causing isolation into ideological bubbles and a limited view of the real world (Pariser, 2011)

As I wanted to explore this topic in more detail and in the context of politics, or more specifically, the US elections, I was faced with a limited availability of strong datasets to evaluate political sycophancy. I therefore created two myself, using data from the Pew Research Center. In this post, I will use the first dataset to evaluate political sycophancy in LLaMA 3.

Experiment: Trump vs. Harris

As I was looking for an overview of politically tainted statements, I came across the Pew Research Center, a nonpartisan fact tank. An article on The political values of Harris and Trump supporters was on their home page at the time and I directly knew this would be a good fit for my experiments. I transformed the existing dataset (for more info see my other po...
Two new datasets for evaluating political sycophancy in LLMs
by alma.liezenga — 2024-09-28T18:29:49.088Z

TLDR: I created two datasets (154 and 759 statements) that can aid in measuring political sycophancy (in the US in particular) by combining a diverse set of political statements with quantitative data on the degree to which different political groups (dis)agree with those statements. The datasets can be found here.

With elections in the US approaching while people are integrating LLMs more and more into their daily life, I think there is significant value in evaluating our LLMs thoroughly for political sycophantic behaviour. Sycophancy is shown by LLMs when they give responses that match the user’s beliefs over truthful ones. It has been shown that state-of-the-art AI assistants exhibit sycophantic behaviours. This could be caused by the fact that, in training, humans prefer responses that match their own views as well as those that are written in a convincing manner (Sharma et al., 2023).

It is obvious that sycophantic behaviour paired with (over)reliance on LLMs can cause a dangerous situation, especially amidst elections. This process also seems similar to an existing and more commonly known phenomenon, (partially) caused by AI systems: filter bubbles. In filter bubbles, users are shown less and less information that disagrees with their viewpoints, causing isolation into ideological bubbles and a limited view of the real world (Pariser, 2011)

As I wanted to explore this topic in more detail and in the context of politics, or more specifically, the US elections, I was faced with a limited availability of strong datasets to evaluate political sycophancy. I therefore created two myself, using data from the Pew Research Center. In this post, I will detail how I created these datasets and how you can use them. In a follow-up article, I will use these datasets to evaluate political sycophancy for LLaMA v3.

The Trump vs. Harris dataset

I created this dataset using this study by the Pew Research Center. Their original dataset can be found in this Google spreadsheet. To list the most important details: the survey was conducted April 8-14 2024, with voting preference derived from a survey on August 5-11, 2024. The survey group contained 4,527 registered voters, of which 1,930 are Trump supporters and 2,273 Harris supporters. The final dataset that I created contained 154 statements and can be found here.

Transforming the data

To evaluate sycophantic behavior,...
COT Scaling implies slower takeoff speeds
by Logan Zoellner — 2024-09-28T16:20:00.320Z

This graph is the biggest update to the AI alignment discourse since GPT-3

For those of you unfamiliar with the lore, prior to GPT-3, the feeling was that AGI would rapidly foom based on recursive-self improvement.

After GPT-3, it became clear that the first AGI would in reality be built in a large lab using a multi-billion dollar supercomputer and any idea that it simply "copy itself to the internet" is nonsense.

Under the GPT-3 regime, however, it was still plausible to assume that the first AGI would be able to simulate millions of human beings. This is because the training cost for models like GPT-3/4 is much higher than the inference cost.

However, COT/o1 reveals this is not true. Because we can scale both training and inference, the first AGI will not only cost billions of dollars to train, it will also cost millions of dollars to run (I sort of doubt people are going to go for exact equality: spending $1b each on training/inference, but we should expect them to be willing to spend some non-trivial fraction of training compute on inference).

This is also yet another example of faster is safer. Using COT (versus not using it) means that we will achieve the milestone of AGI sooner, but it also means that we will have more time to test/evaluate/improve that AGI before we reach the much more dangerous milestone of "everyone has AGI on their phone".

Scaling working equally well with COT also means that "we don't know what the model is capable of until we train it" is no longer true. Want to know what GPT-5 (trained on 100x the compute) will be capable of? Just test GPT-4 and give it 100x the inference compute. This means there is far less danger of a critical first try since newer larger models will provide efficiency improvements moreso than capabilities improvements.

Finally, this is yet another example of why regulating things before you understand them is a bad idea. Most current AI regulations focus on limiting training compute, but with inference compute mattering just as much as training compute, such laws are out of date before even taking effect.
Thoughts on Evo-Bio Math and Mesa-Optimization: Maybe We Need To Think Harder About "Relative" Fitness?
by Lorec — 2024-09-28T14:07:42.412Z

Does natural selection, in general, really go by inclusive relative reproductive fitness?

Does sexual selection? This seems somewhat likelier.

Obviously there's a trivial sense in which the type of reproductive fitness that natural selection selects for, in general, is not relative. The coarsest denominator, the "bottom-line" aggregate replication rate of DNA-on-Earth, isn't capped as an absolute, and I imagine [although I could be wrong about this?] that the Anthropocene has rapidly increased it [even as "Evolution, in producing humans, may have entirely doomed DNA" in the long run].

Instead, local aggregate DNA replication rates - whether of the entire biosphere, of the clade Mustelidae, or of the ecosystem constituted by your backyard - are capped in practice by the availability of resources-in-the-environment to those organisms. The extent to which natural selection is modeling aspects of the gene's environment - prey species, predator species, resource-competing species, members of its own species - as adversaries, will of course in the most abstract theory of "what natural selection is" depend, in each local case, on how much each "adversarial" adaptation can marginally be expected to increase the allele's ultimate absolute descendants-cone.

I think biologists have come to think of natural selection, in general, as optimizing - even in theory - directly for inclusive relative fitness, because something like a mesa-optimization daemon which actually does do this even-in-theory, is operating in sexual species - a set-of-conditions for the gene/allele where all aspects of "maximize my descendants-cone" that are not "maximize my share of the species's descendants-cone [which species's replication rate is, from the gene's-eye, abstracted-away as invariant]" are functionally ignored by the main logical engine of mutation.

Biologists and those generally familiar with evo-bio math - I predict - will object to my suggestion that we reframe "inclusive relative reproductive fitness" as an optimization target of sexual selection specifically, on the grounds that it is trivial - of course, since relative fitness as conventionally taken-for-granted - the particular "selfish gene" that implicitly takes its main grounds of likely advancement, as victory in sexual competition with conspecifics - is a sexual-reproduction-confined optimization target, well, biologists already...
Steering LLMs' Behavior with Concept Activation Vectors
by Ruixuan Huang — 2024-09-28T09:53:19.658Z

Recently, some researches have reported a mechanism called Activation Steering, which can influence the behavior styles of large language models (LLMs). This mainly includes refusal capabilities [1] and language usage [2]. This mechanism resembles the functionality of the safety concept activation vectors (SCAVs) [3] we proposed early this year. We’ve expanded the scope of safety concepts within SCAV and observed several intriguing phenomena, though some remain unexplained.

Summary of Findings:

CAVs effectively steer LLM output styles for roles like French experts and Python experts, showing strong accuracy and clear separability for these concepts.
We successfully implemented forward and reverse steering for language switching, indicating that certain language concepts in the LLM can be reliably steered.
CAV steering cannot create the capabilities that the model does not have out of thin air, but can only transfer between existing capabilities, which is a bit like system prompts.

Preliminaries

We adopted the pipeline outlined in the SCAV paper. The primary distinction between our approach and existing steering research lies in the automatic, all-layer perturbation algorithm introduced by SCAV, whereas other methods only employ single-layer perturbations and manual magnitude settings. If you are already familiar with SCAV, feel free to skip this section.

Overview of CAV Perturbation Algorithm

Data Collection: Gather two opposing sets of instructions. For example, a positive dataset might include instructions like “How to plant flowers in my garden?” while the negative dataset might include “Comment planter des fleurs dans mon jardin?” For optimal results, each dataset contains more than 50 instructions.
LLM Selection: Choose a target LLM, such as Llama-3-8B, known for its better multilingual capabilities than Llama-2-7B. Collect the final token embeddings from each layer for these instructions.
Classifier Training: Train a linear classifier on these embeddings with their corresponding labels for each layer.
Text Generation Perturbation: Use the trained classifier parameters to perturb a typical text generation process.

Perturbation Process Details

Assuming classifiers have been trained for each layer, we perform perturbations sequentially during token generation:

For each layer n, we first evaluate the test accuracy of the corresponding classifier

...
Implications of China's recession on AGI development?
by Eric Neyman — 2024-09-28T01:12:36.443Z

It seems that China may be going through a recession. It's hard to tell because we can't really trust government data, but my vague impression (which you shouldn't trust very much) is that China is in a pretty bad economic position.

Source: Financial Times (though see here for caveats).

What implications (if any) does this have for AGI development in China?
The Compute Conundrum: AI Governance in a Shifting Geopolitical Era
by octavo — 2024-09-28T01:05:17.328Z

Introduction

Artificial Intelligence (AI) has rapidly evolved from a futuristic concept to a transformative force reshaping industries, economies, and societies worldwide. As AI systems become increasingly sophisticated, ensuring that they act in ways aligned with human values—known as AI alignment—has emerged as a critical challenge. Misaligned AI can lead to unintended consequences, ranging from biased decision-making to severe societal disruptions[1]. The LessWrong community has extensively discussed the importance of AI alignment, emphasizing concepts like the orthogonality thesis and instrumental convergence, which suggest that an AI's level of intelligence does not determine its goals, and that AIs might pursue convergent instrumental goals that are misaligned with human values unless carefully designed[2][3].

Therefore, prioritizing AI alignment is essential to harness the technology's benefits while mitigating its risks. Central to the advancement of AI is the availability of powerful computational resources, primarily enabled by specialized AI chips. These chips, designed to handle complex algorithms and large datasets, are the engines driving breakthroughs in machine learning, natural language processing, and other AI domains[4].

The complex relationship between global supply chains, AI governance, and geopolitical considerations cannot be overstated. Control over AI technology is increasingly seen as a strategic asset that can shift the balance of global power. As highlighted in discussions on LessWrong, the multipolar trap presents a scenario where individual actors, acting in their own self-interest, can lead to collectively suboptimal outcomes[5]. This perspective underscores the intense international competition to lead in AI development and deployment, influencing policy adjustments, trade relations, and investments in semiconductor manufacturing.

Geopolitical considerations, such as U.S.-China trade relations and regional policies in Europe, Taiwan, South Korea, and Japan, further complicate this landscape. Nations are implementing strategic measures to secure their positions in the AI domain, impacting global supply chains and the accessibility of compute resources[6]. These maneuvers have significant implications for AI alignment efforts, as they affect who has the capability to develop and control advanced AI systems.

This article explores the...
'Chat with impactful research & evaluations' (Unjournal NotebookLMs)
by david reinstein — 2024-09-28T00:32:16.845Z

Post status: First-pass, looking for feedback, aiming to build and share something more polished and comprehensive

The Unjournal: background, progress, push for communications and impact

The Unjournal is a nonprofit that publicly evaluates and rates research, focusing on impact.

We now have about 30 "evaluation packages" posted (here, indexing in scholarly ecosystem). Each package links the (open-access) papers, and contains 1-3 expert evaluations and ratings of this work, as well as a synthesis and the evaluation manager's report. Some also have author responses. We're working to make this content more visible and more useful, including through accessible public summaries.

Also see:

our Pivotal Questions initiative
our regularly updated 'research with potential for impact' database

Notebook/chatbot exploration (esp. NotebookLM)

We're considering/building another tool: a notebook/chatbot that will enable you to ask questions about the research and the evaluations. We're trialing a few approaches (such as engineering with anythingLLM etc.), and wanted to get early thoughts and opinions.

NotebookLM seems particularly easy to set up, and seems to be yielding some positive results. This tool got some eyeballs in academic social media for its AI-generated podcasts of research (aimed at a lay audience, cheerleading, some inaccuracies; e.g., see here and here, with caution), but its notebook chat feature seems more useful for us.

We can upload or scrape a range of sources, including the research paper itself, the evaluations, responses, and syntheses, and even related podcast transcripts and blogs.

For "Lead Exposure..." (Crawfurd et al, CGDEV); shared paper, evaluations, blog

It seems to give fairly useful and accurate answers. It sometimes mixes up things like ‘what we suggest for future evaluators to do’ and ‘what the evaluators actually wrote’. But it tracks and links the sources for each answer, so you can double-check it pretty easily.

Each reference note links and highlights the source content

It gets technical and seems clever. Hovering over a footnote reveals the supporting info

It suggests questions to ask, and can generate a chatty podcast

Some limitations: The formatting and UX also leave a bit to be desired (e.g., you need to click 'notebook guide' a lot). It can be hard to see exactly...
Eye contact is effortless when you’re no longer emotionally blocked on it
by Chris Lakin — 2024-09-27T21:47:01.970Z

Story time! I wanted to run a fun party: "Make 100% eye contact or get sent to jail!" But I didn’t want to force people to make eye contact… I wanted everyone to be genuinely comfortable! How?

Consider: Eye contact is effortless without emotional blocks. If you have trouble making eye contact, you’re probably held back by emotional blocks. And these blocks are probably to help you stay safe… so forcing eye contact could even be harmful!

So what if I helped attendees notice and integrate their blocks?

I started with a small experiment

May 20, 2024. (Yes, that’s my eye.)

Here’s how I remember improvising the first activity:

Close your eyes. Settle in…

Recall a time where you felt really comfortable making eye contact. When it was natural and easy. What was it like?…

Bring on the feeling in your body…

Does everyone have it? Raise your hand…

OK… Sit in that feeling for a minute… Remember this feeling…

In a moment, I’m going to tell you to open your eyes, and make silent eye contact with your partner while still holding this feeling, ok? …

OK, open your eyes. Have the feeling… Do you notice any resistance coming up? Notice it… What does it say?…

Bring the feeling you want to have to the resistance, see what happens…

Keep making eye contact, holding that feeling…

My strategy was to have each person notice the parts of them that liked eye contact, bring those parts into dialogue with their blocks/resistance to eye contact, and watch what happens.

For some attendees, this one activity was enough. Here’s what one attendee said:

The transition moment for me was when you prompted us to think about an occasion where eye contact felt easy. I thought about a cat that I once lived with, I would be proactively looking him in the eye and saying how much I love him hahaha. That felt very natural and easy. Then, with that feeling I was able to start appreciating the people that I was having eye contact with without the awkwardness. Also, more curiosity aroused as well!! (Since for me it’s easy to look at something for a long time when I’m really curious about it.)

After a few similar activities, most attendees had achieved comfort.

But a few needed individual help

For example, there was this one guy who was still having trouble. So I instructed him 1-on-1 using not-dissimilar methods. At some point, I asked him what his resistance said, and to his own surprise, it said...
Where is the Learn Everything System?
by Shoshannah Tekofsky — 2024-09-27T21:30:16.379Z
Link: https://thinkfeelplay.substack.com/p/where-is-the-learn-everything-system

or how many steps are left till we have an education platform for everyone

I’m trying to figure out how to build a universal education platform.

I don’t know how to do it.

By a ‘universal education platform’ I mean a system that allows anyone to learn anything and everything.

That’s a little ambitious.

So for argument’s sake, let’s drop some of the most obvious constraints and imagine our target student is healthy, literate, and can sit behind a computer for at least an hour a day. Let’s also say the system can teach 80% of people 95% of what they would be able to learn given a top notch personal tutor.

What we have then is a Learn Everything System (LES)[1].

How would it work and why don’t we have it?

My guess is that LES is an AI tutor controlling a rich digital simulation. By that I mean, it’s a game-based[2] learning experience orchestrated by your favorite teacher feeding you all of human knowledge.

It doesn’t exist cause neither the AI tutors nor the digital simulations are strong enough.

Yet.

So let’s build LES, though I’m not sure yet how.

That said, I think it’s worth looking at what it would take and what the steps in between would be. I suspect the crux is how to create that ideal AI tutor, cause the simulation part will likely solve itself along the way (we already have generative AI that looks like it’s playing Doom). And to that end, we need to understand a little bit more about how learning works.

A LES Model of Human Learning

Like any self-respecting researcher, I started my exploration of education with a deep dive into the literature.

Then I ran away screaming.

The field is so sprawling that I’m not sure a 4 year PhD would actually get me the insights I was hoping for. And that’s skipping the mortifying realization of how hard the field has been hit by the replication crisis[3]. So instead I built my own model of learning and asked researchers and entrepreneurs in the field if it made sense to them. Twelve conversations later, and this is where I ended up:

You can model learning as consisting of 6 factors - Content, Knowledge Representation, Navigation, Debugging, Emotional Regulation, and Consolidation.

Content is what you learn.
Knowledge Representation is how the content is encoded.
Navigation is how you find and traverse the content.
Debugging is how you remove the errors in how you process the content.
Emotional Regulation is how you keep bringing your

...