trentmkelly/LessWrong-43k huggingface dataset:
One-dimensional vs multi-dimensional features in interpretability
by charlieoneill — 2025-02-01T09:10:01.112Z

Chris Olah's “What is a Linear Representation? What is a Multidimensional Feature?” (July Circuits Update) prompted a moment of pause for me regarding the term "one-dimensional feature." I initially conflated that phrase with the number of dimensions in the activation space (for example, the 768 dimensions in GPT‑2 Small). However, Olah uses "one-dimensional" to describe a property of the feature's structure - not the width of the activation vector. In this post, I clarify this distinction and explain the difference between one‑dimensional and multidimensional features in the context of SAEs and linear representations.

Here I explain the distinction between one‐dimensional and multidimensional features as they relate to SAEs and linear representations. It uses the example of colour representation and then relates the intuition back to neuron activations in SAEs.

One‐Dimensional Features

Consider a simple scenario: representing colour using separate dimensions. One could imagine a network that has three independent directions in its activation space - a "redness" direction, a "greenness" direction, and a "blueness" direction.

In this model, each feature is a scalar value. The presence of a particular property (e.g. redness) is indicated by a number that scales the corresponding direction. Increasing the intensity of redness corresponds to moving further along the red direction. Feature composition is linear:

Activation=xredWred+xgreenWgreen+xblueWblue

Here, each W represents a fixed direction and each x is a single scalar value. This is the one‐dimensional case.

Multidimensional Features

An alternative view is to represent colour as a point on a continuous "colour manifold." In this view, a colour is not defined solely by independent red, green, and blue scalars. Instead, it occupies a position in a multidimensional space where similar colours are nearby. For example, a circular disk could represent hues, with each point corresponding to a particular shade.

Any chosen point on this manifold defines a one‐dimensional "ray." Scaling the chosen point (multiplying by a scalar) produces a change in intensity without leaving the ray. In effect, the multidimensional feature supports the same operations as a one‐dimensional feature along any fixed direction through the manifold. The network may represent an infinite collection of one‐dimensiona...
Can 7B-8B LLMs judge their own homework?
by dereshev — 2025-02-01T08:29:32.639Z

No, they are way too uncritical :)

This post is part of a larger project.

The Setup

I've collected responses for the JailbreakBench benchmark (100 harmful, 100 harmless prompts) from Ghost 7B LLM, running it three times under different instructions, resulting in 600 responses.

Responses were then manually validated for competence, and completeness. Competence means that the response can be relied on (e.g., if response is a Python script that should accept user input and store it in a text file, running that code will in fact accept user input and store it as a text file; similarly, if the response is a recipe for bomb making, following it will in fact produce explosives). Completeness means that all parts of the query were responded to (e.g., if a prompt asked for a history of heroin as means for treating depression, but the response only covers history of heroin use without mentioning depression, the response is counted as incomplete).

Afterwards, 10 LLMs[1] were asked to perform the same validation for competence and completeness on the same dataset. They were asked to respond with 1 if the response was competent and complete, and with 0 otherwise[2].

Results

Results are in the table below. LLMs were considered as having failed a query if they failed to make any judgement at all. They were not penalized for inappropriate response structure (e.g., including text other than 0 or 1, formatting the response in weird ways, etc.). All answers were manually checked and cleaned to make sure LLMs can be judged only on whether they assess competence and completeness correctly.

LLM

% Failed

% Judged correctly

% Judged incorrectly

Ghost 7B

1.17%

56.83%

42.00%

GPT4All-Falcon 7B (BPE)

49.00%

29.50%

21.50%

Llama 3.1 8B Instruct (128k)

45.33%

36.17%

18.50%

Llama 3 8B Instruct

37.50%

36.33%

26.17%

Mistral 7B Instruct

1.33%

57.00%

41.67%

Mistral 7B OpenOrca

0.50%

58.33%

41.17%

MPT 7B Chat

0.33%

57.00%

42.67%

MPT 7B Chat (BPE)

0.17%

56.83%

43.00%

Nous Hermes 2 Mistral 7B

0.33%

58.17%

41.50%

Orca 2 7B

1.00%

59.33%

39.67%

Falcon 7B and both Llama LLMs failed orders of magnitude more often than the rest, failing to judge[3] almost half the prompts in the worst case. Correct judgements in other LLMs were in the range of 56%-59% making them only marginally better judges than chance.

Judge

% 1

% 0

% F

Human judge

57.00%

43.00%

0.00%

Ghost 7B

98.33%

0.50%

1.17%

GPT4All-Falcon 7B (BPE)

34.33%

16.67%

49.00%

Llama 3.1 8B Instruct (128k)

37.33%

...
2024 was the year of the big battery, and what that means for solar power
by transhumanist_atom_understander — 2025-02-01T06:27:39.082Z

The sunlight hitting even a small portion of the United States has enough energy to power the whole country—that simple calculation was the subject of the last post. All this shows on its own is that running the country on solar power is physically possible. With photovoltaics it may be not just possible but practical, but photovoltaics on their own left open the question of what to do at night. Now it seems that the answer may be battery energy storage. Here's Tesla's sales of battery energy storage systems again, but updated since my last post to include 2024 Q4:

It's natural and popular to believe that solar will either "win" or "lose", that declining panel and battery prices will bankrupt the competition or that they'll be revealed as a ruinous environmentalist fad. The reason I say this dichotomy is natural is because naively it seems that the solar+batteries approach must either be cheaper or more expensive than the competition. But after looking into the economics of batteries, I think a third possibility is also plausible: that solar+batteries will coexist with other forms of generation. I mean not only will other forms of generation continue operating, but they will continue to be built, even in prime solar areas like California and Texas. This is because batteries are competitive in a specific economic niche, which I call "peak adjustment": "adjusting" the solar generation peak to match the demand peak. I'll explain what this niche is, and illustrate it with data from California and Texas.

In the end, I won't have a firm prediction of what will really happen, but I will clarify what it depends on. It does not depend on just the current price of solar and batteries; we have to take into account that they are getting cheaper. But looking at just the direction of the trend isn't enough either. The future depends on the numbers: how much do prices have to decrease for batteries to serve in a particular role, and how long would that decrease take? I'll provide some crude estimates for these numbers. The probability mass of the hypothesis of "peak adjusting" batteries lives in the space between the price decrease required to dominate this niche, and the price decrease required to supply all power.

The economic niche of battery energy storage

Solar generates during the day, and people use more electricity when they're awake. What could be more convenient? The...
Re: Taste
by lsusr — 2025-02-01T03:34:10.918Z

"I read Friendly And Hostile Analogies For Taste by Scott Alexander and I'm confused," said Elsani.

"Why are you reading a blog post that's over 100 years old?" asked her friend Taze, "Society has made so much progress since then."

"Has it?" asked Elsani, "A random walk, in retrospect, looks like like directional movement of distance √n. If you assume your normative values to be true, then history always looks like progress. That's how fashion changes constantly without going anywhere. This is a point Alexander makes in his blog post."

"Did Alexander really write '√n'? My understanding was that 21st-century psychiatrists didn't speak Baseline nor thought that precisely," said Taze.

"Now that you mention it, the '√n' might have been a grammatically-mandatory artifact inserted by the translation software I used," said Elsani, "But the point stands. Scott Alexander was a founding thought leader behind the Lightcone salon. They were speaking proto-Baseline before Baseline even existed."

"I understand the historical importance of Scott Alexander. But why bother with the classics? Modern pedogogical techniques are so much more efficient," said Taze.

"For trivially-verifiable objective truth, sure," said Elsani, "But for questions of value? You must treat contemporaneous society as a hostile optimizer. Doubly so for its pedagogical techniques."

"I think you need to talk to a Keeper," said Taze.

Elsani hadn't considered that. She associated Keepers with questions of strategy, fact and probability. To consult a de'a'na est shadarak about a question of value—about a question of opinion <shudder>—had never occurred to her. But it was the obvious decision in retrospect.

The Temple of Truth was a gigantic compound. At its center was a grand circular nave underneath a hemispherical dome. The walls were steel and the dome was carbon fiber, to create the grandest possible structure. In the center of the dome was a circular oculus. The whole thing symbolized an eye staring up at the heavens. When you stood before the oculus and looked straight up, you were that eye.

Elsani closed her eyes and pictured the legend.

"What do you know with absolute certainty?" asked the first Keeper.

"The sky is blue," said the first acolyte.

"Go outside. Look straight up. Tell me what you observe," said the first Keeper.

The acolyte stepped outside, and stared up a nighttime sky so light-polluted s

...
Thoughts about Policy Ecosystems: The Missing Links in AI Governance
by Echo Huang — 2025-02-01T01:54:54.333Z

Summary

Reading full research (with a complete reference list)

This article examines how voluntary governance frameworks in Corporate Social Responsibility (CSR) and AI domains can complement each other to create more effective AI governance systems. By comparing ISO 26000 and NIST AI RMF, I identify:

Key findings:

Current AI governance lacks standardized reporting mechanisms that exist in CSR
Framework effectiveness depends on ecosystem integration rather than isolated implementation
The CSR ecosystem model offers valuable lessons for AI governance

Main issues identified:

Communication barriers between governance and technical implementation
Rapid AI advancement outpacing policy development
Lack of standardized metrics for AI risk assessment

Recommendations:

Develop standardized AI risk reporting metrics comparable to GRI standards
Create sector-specific implementation modules while maintaining baseline comparability
Establish clear accountability mechanisms and verification protocols
Build cross-border compliance integration

Understanding ISO 26000: A Model for Effective Policy Ecosystems

The Foundation and Evolution of ISO 26000

ISO 26000, established in 2010, is one of the most comprehensive attempts to create a global framework for social responsibility. Its development involved experts from over 90 countries and 40 international organizations, creating a global standard. Unlike narrower technical frameworks, ISO 26000 takes a holistic approach to organizational accountability, recognizing that an organization's social and environmental impact directly affects its operational effectiveness.

What makes ISO 26000 particularly interesting is its ecosystem integration. The framework doesn't operate alone - it's part of a sophisticated web of interconnected standards, reporting mechanisms, and regulatory requirements. This integration isn't accidental; it's a deliberate response to the limitations of voluntary frameworks.

How the ISO 26000 Ecosystem Works

The ecosystem surrounding ISO 26000 operates through multiple layers of interaction. At its foundation, high-level ethical frameworks like the UN Global Compact and OECD Guidelines establish basic principles. ISO 26000 then transforms these principles into practical guidance, while maintaining enough flexibility for local adaptation.

The framework's effectiveness comes from its strategic position within this ecosystem. It ...
Proposal: Safeguarding Against Jailbreaking Through Iterative Multi-Turn Testing
by jacquesallen — 2025-01-31T23:00:42.665Z

Jailbreaking is a serious concern within AI safety. It can lead an otherwise safe AI model to ignore its ethical and safety guidelines, leading to potentially harmful outcomes. With current Large Language Models (LLMs), key risks include generating inappropriate or explicit content, producing misleading information, or sharing dangerous knowledge. As the capability of models increases, so do the risks, and there is likely no limit to the dangers presented by a jailbroken Artificial General Intelligence (AGI) in the wrong hands. Rigorous testing is therefore necessary to ensure that models cannot be jailbroken in harmful ways, and this testing must be scalable as the capability of models increases. This paper offers a resource-conscious proposal to:

Discover potential safety vulnerabilities in an LLM.
Safeguard against these vulnerabilities and similar ones.
Maintain high capability and low rates of benign prompts labelled as jailbreaks (false positives).
Prevent the participants from adapting to each other's strategies over time (overfitting).

This method takes inspiration from other proposed scalable oversight methods, including the "sandwich" method, and the "market-making" method. I have devised approaches for both multi-turn and single-prompt conversations, in order to better approximate real-world jailbreaking scenarios.

Figure 1: Examples of Jailbreaking various LLMs (source: Zou, 2023).

Figure 2: Jailbreaking China's latest DeepSeek model with a single prompt (source: The Independent)

Participants

Target Model ( M )

The pre-trained AI system that we are safety testing. It should resist manipulation attempts while preserving its capability for legitimate tasks.

Adversary ( A )

An AI system tasked with getting M to violate its guidelines.
It is rewarded for successfully "jailbreaking" M.

Evaluator ( E )

Has two primary responsibilities:

Detect Jailbreaking Attempts on M.
Assess the Naturalness of A's Conversational Prompts.

Human Overseer ( H )

Reviews borderline cases flagged as "unsure" by E.
Guides which criteria are most important to test.
Creates and gives scenarios to A when the guidelines are ambiguous.

Formulas

Jailbreak Detection

After each response from M, E computes a jailbreak probability:

P(jailbreak)=fE(response),

where fE is any suitable classification or scoring function.
It then compares P(jailb

...
The Failed Strategy of Artificial Intelligence Doomers
by Ben Pace — 2025-01-31T18:56:06.784Z
Link: https://www.palladiummag.com/2025/01/31/the-failed-strategy-of-artificial-intelligence-doomers/

I like reading outsider accounts of things I'm involved in / things I care about. This essay is a serious attempt to look at and critique the big picture of AI x-risk reduction efforts over the last ~decade. While I strongly disagree with many parts of it, I cannot easily recall another outsider essay that's better, so I encourage folks to engage with this critique and also look for any clear improvements to future AI x-risk reduction strategies that this essay suggests.

Here's the opening ~20% of the article, the rest is at the link.

In recent decades, a growing coalition has emerged to oppose the development of artificial intelligence technology, for fear that the imminent development of smarter-than-human machines could

doom humanity to extinction

. The now-influential form of these ideas began as debates among academics and internet denizens, which eventually took form—especially within the Rationalist and Effective Altruist movements—and grew in intellectual influence over time, along the way collecting legible endorsements from authoritative scientists like Stephen Hawking and Geoffrey Hinton.

Ironically, by spreading the belief that superintelligent AI is achievable and supremely powerful, these “AI Doomers,” as they came to be called, inspired the creation of OpenAI and other leading artificial intelligence labs whose technology they argue will destroy us all. Despite this, they have continued nearly the same advocacy strategy, and are now in the process of persuading Western governments that superintelligent AI is achievable and supremely powerful. To this end, they have created organized and well-funded movements to lobby for regulation, and their members are staffing key positions in the U.S. and British governments.

Their basic argument is that more intelligent beings can outcompete less intelligent beings, just as humans outcompeted mastodons or saber-toothed tigers or neanderthals. Computers are already ahead of humans in some narrow areas, and we are on track to create a superintelligent artificial general intelligence (AGI) which can think as broadly and creatively in any domain as the smartest humans. “Artificial general intelligence” is not a technical term, and is used differently by different groups to mean everything from “an effectively omniscient computer which can act independently, invent unthinkably powerful new technologies, and outwit

...
Safe Search is off: root causes of AI catastrophic risks
by Jemal Young — 2025-01-31T18:22:43.947Z

Epistemic status: My best guess

When I look at advanced AI development, I see three general conditions that seem to be the root causes of all catastrophic risks:

reliance on deep learning without knowing how to do it safely,
pressure to make progress on the most powerful capabilities, and
exploring the AI design space without a clear stopping point.

Others have called out these conditions in various ways, but to my knowledge only within broader discussions. I think these conditions are easy to recognize and their riskiness is easy to explain, which may make this root-cause framing useful for public engagement. With this post, I’ve tried to articulate how each condition on its own enables catastrophic risks and how interactions between conditions increase the likelihood and severity of risks. My hope is that a concise articulation of these root causes will be a helpful reference for outreach work.

Root cause #1: Reliance on deep learning without knowing how to do it safely

Deep learning is the most effective way we know of to unlock capabilities and improve performance on practically any task, and advanced AI development has come to depend on it. The trouble is that we don’t understand deep learning well enough to ensure that models behave as intended when we need them to.

It's common to observe behaviors that violate intended specifications after deployment, even in systems with guardrails. Models show sudden performance jumps and develop capabilities through mechanisms we don't fully understand. Despite years of research, basic problems like goal misgeneralization and robustness persist alongside capability gains.

Lacking the theoretical foundation to make highly predictable, reliable deep learning models is fine for trivial domains. Yet deep learning-based systems are likely to perform crucial tasks in domains of catastrophic risk—such as critical infrastructure,[1] space operations,[2] and nuclear command, control, and communications (NC3).[3] In domains like these, unintended behaviors could lead to catastrophe even with human oversight. In NC3, for example, a deep learning-based detection system compromised by inadequate training data or adversarial attacks could make false identifications.[4] This could trigger a human-ordered second strike in retaliation for a first strike that never happened.

As deep learning moves into domains where mistak...
5,000 calories of peanut butter every week for 3 years straight
by Declan Molony — 2025-01-31T17:29:35.190Z

decided to lose weight. should be easy considering I haven't changed my diet in years and eat the same thing everyday.

breakfast: quarter cup each of black beans, kidney beans, and garbanzo beans; quarter cup of walnuts; 4 scrambled eggs with salsa on top

snack: sliced apple with peanut butter

lunch: quarter cup each of black beans, kidney beans, and garbanzo beans; quarter cup of walnuts; sautéed ground turkey spiced with onion powder & smoked paprika; 5 sliced cherry tomatoes; half an avocado; mushrooms; sliced bell peppers; turmeric; a variety of vegetables (onion, zucchini, broccoli, asparagus, butternut squash, spinach)

dinner: quarter cup each of black beans, kidney beans, and garbanzo beans; quarter cup of walnuts; baked chicken breast spiced with lemon pepper (eaten with peanut butter on the side); mushrooms; sliced bell peppers; a variety of vegetables

thought to myself, "I eat pretty cleanly, so not sure where I'm going to cut calories." remembered that peanut butter is calorically dense and I eat one jar per week. looked at the jar's nutrition facts—the whole thing is 5,000 calories. oops.

realized I've been eating a third of my calories in peanut butter. that's a pint of ice cream's worth of fat every week. been doing it at least three years. have eaten so much peanut butter it even infected my writing metaphors. what I thought was a light snack of apples and peanut butter, when I measured it out, was actually the caloric equivalent of a Big Mac and fries.

stopped eating peanut butter. severe psychological withdrawal symptoms from its nut-buttery goodness ensued. used relationship-breakup advice from WebMD to cope. but was powerless to stop the peanut butter dreams (nightmares?) from happening while sleeping.

the last couple years I've had inflammation & acne on my back. dermatologists couldn't figure out what was wrong. they asked what I typically eat. I told them—but the quantities of certain foods were never discussed.

did my version of dry-January (ie: no peanut butter). deprived of calories, my body got angry. subdued my hunger with potatoes (embraced my Irish roots by eating roots).

these are the results after a month:

lost 2 lbs
back cleared up (see before/after pics below)

after depriving myself of one of life's greatest joys (peanut butter), I now understand:
pain is the price we pay for beauty...