trentmkelly/LessWrong-43k huggingface dataset:
The Case Against Libertarianism
by Zero Contradictions — 2024-08-03T05:05:34.471Z
Link: https://zerocontradictions.net/civilization/case-against-libertarianism

Libertarianism is a political philosophy that upholds Liberty and the Non-Aggression Principle as its core values. Libertarians seek minimal and/or minarchist governments.

The Libertarian Non-Aggression Principle is typically defined as an ethical rule prohibiting aggression (or actions that violate negative rights). But there are lots of disagreements among Libertarians regarding what the NAP prohibits, allows, and mandates since “aggression” is not clearly defined.

(see the rest of the post in the link)
We Don't Just Let People Die—So What Next?
by James Stephen Brown — 2024-08-03T01:04:49.756Z

The following is a speculative (workshopping) piece about what the implications are for a society that has concluded that we don’t just let people die unnecessarily from starvation due to poverty (a society like our own). I am not an economist or policy expert, so my conclusions are dependent on my understanding of the issues at the moment and may well change if someone can enlighten me about areas where I’m lacking in expertise. The TL;DR—If we don’t let the poor die, but we leave them in poverty, we are creating a suboptimal level of existence, not conducive to producing contributors to society—and instead creating a parasitic relationship at great cost.

Capitalism & Evolution

According to Adam Smith, a society should operate in such a way so as to provide…

“…not only the commodities which are indispensably necessary for the support of life, but whatever the custom of the country renders it indecent for creditable people… to be without.” - Adam Smith

It was unacceptable to Smith, not only to allow people to perish, but in fact to deny them what they need to live with dignity. But on that second point, there have been differing levels of concern about the least fortunate.

“The incentives created by economic inequality, with rewards for education, hard work, and risk-taking, are essential for creating a dynamic, innovative economy." -Gary Becker (Nobel Prize Winning Economist)

Some will reference an evolutionary mode—survival of the fittest—to justify an unsympathetic attitude to those unable to support themselves.

“While the law [of competition] may be sometimes hard for the individual, it is best for the race, because it ensures the survival of the fittest in every department." -Andrew Carnegie

I intend to make an argument that, in a world where we don’t just let people die—even if someone doesn’t believe dignity is a right—it is optimal to have people afforded a greater-than-subsistence level of existence.

Before going further, I’d like to present a concept from nature that can help us understand the issue better.

The Red Queen

In 'Alice in Wonderland', the Red Queen is riding her horse through a forest, but as she rides, the trees and all the creatures of the forest are moving in the same direction at the same speed, so she appears to stand still. This concept, Matt Ridley refers to as The Red Queen, is something that is evident in evolutionary arms races.

There...
The EA case for Trump
by Judd  Rosenblatt — 2024-08-03T01:00:45.422Z
Link: https://www.secondbest.ca/p/the-ea-case-for-trump-2024

There’s a decent chance Trump is our next president. In that case, it would be great for him to be AI-alignment-pilled.

The good news is that, according to Samuel Hammond, he is:

"Unfortunately, the EA vs e/acc debate on Twitter has created the perception that AI safety is left-right polarized.

The polls tell a different story

. And as a participant on the Project2025 AI policy committee, I can confidently report that Trump’s supposed shadow transition takes AGI and its associated risks seriously."

Or, at least, Sam hopes he will be (alignment-pilled). As of now, “Trump has himself called the prospect of “super-duper AI” “alarming and scary,” though not as scary as China building it first.”

In any case, Sam makes some interesting points in this piece that I think many here might want to read.
I didn't think I'd take the time to build this calibration training game, but with websim it took roughly 30 seconds, so here it is!
by mako yass — 2024-08-02T22:35:21.136Z

Basically, the user is shown a splatter of colored circles, then the splatter is hidden, and then they're asked to remember what proportion of the splatter was a particular color. To get good at it, they'd have to get good at accurately perceiving and remembering features of entire distributions. Obvious high propensity for transfer to mentally visualizing distributions, and hopefully this would transfer to receiving and retaining accurate assessments of distributions we experience in our lives and possibly to forecasting calibration as well.

Despite the simplicity of such a training tool, I didn't think I was ever going to get around to building it. Unassisted, it probably would have taken me at least 2 days (there are a lot of other higher priority things I could do in 2 days). Then I discovered websim (video of nice things people have made in websim) via this impressive series of experimental UI demos in "maleable computing", so I decided to test it on this, and so it took about 30 seconds.

The initial prompt was one gestural paragraph. Claude made a huge bunch of reasonable judgements and it was basically all there immediately.

I then spent about 3 hours doing design work as websim dutifully and relatively instantly implemented every one of my recommended changes. Websim is really good at iterating towards a correct design! The UI encourages us to try just doing everything via the LLM instead of going in and doing things manually and it really leads you to realize how

fast

that is when the LLM is well directed. It's so effective that there was a point where I started getting a weird feeling in my chest and wanted to step outside and yell a bit. It feels like a 15x productivity boost, and I guess I'm also kind of Feeling The AGI about it. Claude 3.5 is iirc only moderately better at coding than Claude 3, but sometimes moderate capability improvements have discontinuous impacts.

Here's the current version for now: Visual Calibration Trainer

Unfortunately websim makes debugging inconvenient, and hand-editing is kind of impossible. Although, yeah, forbidding editing code by hand is part of the magic of the experience of websim, they probably should fix this, it will be a genuinely powerful tool once there's an IDE.

The very first version websim offered was this. The last version before I started developing strong impulses to take it out of websim and hand edit was...
Evaluating Sparse Autoencoders with Board Game Models
by Adam Karvonen — 2024-08-02T19:50:21.525Z

This blog post discusses a collaborative research paper on sparse autoencoders (SAEs), specifically focusing on SAE evaluations and a new training method we call p-annealing. As the first author, I primarily contributed to the evaluation portion of our work. The views expressed here are my own and do not necessarily reflect the perspectives of my co-authors. You can access our full paper here.

Key Results

In our research on evaluating Sparse Autoencoders (SAEs) using board games, we had several key findings:

We developed two new metrics for evaluating Sparse Autoencoders (SAEs) in the context of board games: board reconstruction and coverage.
These metrics can measure progress between SAE training approaches that is invisible on existing metrics.
These metrics allow for meaningful comparisons between different SAE architectures and training methods, potentially informing SAE design for more complex domains like language models.
We introduce p-annealing, a new SAE training method that improves over prior methods on both existing metrics and our new metrics.
SAEs trained on ChessGPT and OthelloGPT can capture a substantial fraction of the model's board state, with F1 scores of 0.85 and 0.95 respectively for board reconstruction.
However, SAEs do not match the performance of linear probes, suggesting they may not capture all of the model's board state information or "world model".

Challenges with SAE Evaluations

I have an Introduction to SAEs available here.

Sparse Autoencoders (SAEs) have recently become popular for interpretability of machine learning models. Using SAEs, we can begin to break down a model's computation into understandable components. As a result, there have been a flurry of new SAE architectures and loss functions, such as the BatchTopK SAE, Google Deepmind's Gated SAE and JumpReLU SAE, our p-annealing, and OpenAI's TopK SAE.

Unfortunately, we don't have reliable metrics that we can use to compare the new approaches. The main metric currently used is "we looked at activating inputs for a range of features and gave a gut reaction on interpretability of the features". This is a major limitation for the field.

In machine learning, it's ideal to have an objective evaluation of your model, such as accuracy on the MNIST benchmark. With an objective evaluation, you can just twiddle all the knobs of architectures, hype...
The Bitter Lesson for AI Safety Research
by adamk — 2024-08-02T18:39:36.884Z

Read the associated paper "Safetywashing: Do AI Safety Benchmarks
Actually Measure Safety Progress?": https://arxiv.org/abs/2407.21792

Focus on safety problems that aren’t solved with scale.

Benchmarks are crucial in ML to operationalize the properties we want models to have (knowledge, reasoning, ethics, calibration, truthfulness, etc.). They act as a criterion to judge the quality of models and drive implicit competition between researchers. “For better or worse, benchmarks shape a field.”

We performed the largest empirical meta-analysis to date of AI safety benchmarks on dozens of open language models. Around half of the benchmarks we examined had high correlation with upstream general capabilities.

Some safety properties improve with scale, while others do not. For the models we tested, benchmarks on human preference alignment, scalable oversight (e.g., QuALITY), truthfulness (TruthfulQA MC1 and TruthfulQA Gen), and static adversarial robustness were highly correlated with upstream general capabilities. Bias, dynamic adversarial robustness, and calibration when not measured with Brier scores had relatively low correlations. Sycophancy and weaponization restriction (WMDP) had significant negative correlations with general capabilities.

Often, intuitive arguments from alignment theory are used to guide and prioritize deep learning research priorities. We find these arguments to be poorly predictive of these correlations and are ultimately counterproductive. In fact, in areas like adversarial robustness, some benchmarks basically measured upstream capabilities while others did not. We argue instead that empirical measurement is necessary to determine which safety properties will be naturally achieved by more capable systems, and which safety problems will remain persistent.[1] Abstract arguments from genuinely smart people may be highly “thoughtful,” but these arguments generally do not track deep learning phenomena, as deep learning is too often counterintuitive.

Although our method for calculating “capabilities scores” depends entirely on downstream model benchmark scores, it exhibits a strikingly close relationship with training compute (x-axis). The correlation for base models was 96%.

We provide several recommendations to the research community in light of our analysis:

Measure capabilities correlations when proposing new safety evaluations.
When creating s

...
Ethical Deception: Should AI Ever Lie?
by Jason Reid — 2024-08-02T17:53:38.744Z

Personal Artificial Intelligence Assistants (PAIAs) are coming to your smartphone. Will they always tell the truth?

Given the future scenario where humans increasingly seek subjective feedback from AIs, we can expect that their influence will accelerate. Will the widespread use of PAIAs influence social norms and expectations around praise, encouragement, emotional support, beauty, and creativity? And how will these personal AI systems resolve the delicate balance between truthfulness and providing emotional support to their user?

More, much more interaction

Personalized AI is on the brink of being as pervasive

[1]

as smartphones have become.

A Pew Research Center survey[2] done in February 2024 finds that “22% of Americans say they interact with artificial intelligence almost constantly or several times a day. Another 27% say they interact with AI about once a day or several times a week.” Together, these represent almost half of U.S. adults. While this number is impressive, the researchers further note that “only 30% of U.S. adults correctly identify the presence of AI across six examples in a recent survey about AI awareness.”

Personal Artificial Intelligence Assistants (PAIAs) will be increasingly deployed as the performance of AI systems continues to improve. Millions of people are already using versions of these PAIAs as virtual assistants for work[3], coding[4], companionship[5], and romance[6][7].

Aligning truthful AI: What truth?

In a future where AI systems interact with humans, the question isn’t just about capability and safety but also about morality: should an AI ever lie?

We know that AI systems can be purposefully deceptive[8]. AIs can deceive humans while playing cooperative[9] and competitive[10] strategy games, playing poker[11], and performing simulated negotiations[12].

Clearly, AI systems should never, ever lie or hide the truth; correct?

The risks of deceptive AI are significant and multifaceted. A misaligned AI that used strategic deception[13] to achieve its goals could be difficult to detect. It could potentially hide[14] this capability, recognize[15] the training environment, and take a treacherous turn[16] post-deployment. Deceptive AI is an ongoing concern, generating research and mitigation[17] efforts.

This said, an A...
Request for AI risk quotes, especially around speed, large impacts and black boxes
by Nathan Young — 2024-08-02T17:49:48.898Z

@KatjaGrace, Josh Hart I are finding quotes around different arguments for AI being an existential risk

Full list here: https://docs.google.com/spreadsheets/d/1yB1QIHtA-EMPzqJ_57RvvftvXHTI5ZLAy921Y_8sn3U/edit

Currently we are struggling to find proponents of the following arguments:

"Loss of control via speed" - that things that might otherwise go well are going to go badly because they are happening so fast

"Loss of control via inferiority" - if an actor is much less capable than other actors then they might slowly lose control of their resources (eg a child king)

"AI may produce or accelerate destructive multi-agent dynamics" - poorly defined, but in the direction of ‘one AI might be fine, but many AIs plus us in a competitive world will lead to outcomes nobody wants’
"Large impacts suggest large risks" - A pure argument from size, that the impacts will be big an that is concerning
"Black boxes" - We understand AI substantially less well than other new, impactful technologies
A good Yudkowsky quote for "

Risk from competent malign agents

"

, ie that AIs are a risk because they are competent and not aligned to us. I am confident that Yud thinks this but I struggle to find in less than 250 words

I would love any suggestion however vague of where you think good quotes are for proponents of these arguments.
A Simple Toy Coherence Theorem
by johnswentworth — 2024-08-02T17:47:50.642Z

This post presents a simple toy coherence theorem, and then uses it to address various common confusions about coherence arguments.

Setting

Deterministic MDP. That means at each time t there's a state S[t][1], the agent/policy takes an action A[t] (which can depend on both time t and current state S[t]), and then the next state S[t+1] is fully determined by S[t] and A[t]. The current state and current action are sufficient to tell us the next state.

We will think about values over the state at some final time T. Note that often in MDPs there is an incremental reward each timestep in addition to a final reward at the end; in our setting there is zero incremental reward at each timestep.

One key point about this setting: if the value over final state is uniform, i.e. same value for all final states, then the MDP is trivial. In that case, all policies are optimal, it does not matter at all what the final state is or what any state along the way is, everything is equally valuable.

Theorem

There exist policies which cannot be optimal for any values over final state except for the trivial case of uniform values. Furthermore, such policies are exactly those which display inconsistent revealed preferences transitively between all final states.

Proof

As a specific example: consider an MDP in which every state is reachable at every timestep, and a policy which always stays in the same state over time. From each state S every other state is reachable, yet the policy chooses S, so in order for the policy to be optimal S must be a highest-value final state. Since each state must be a highest-value state, the policy cannot be optimal for any values over final state except for the trivial case of uniform values. That establishes the existence part of the theorem, and you can probably get the whole idea by thinking about how to generalize that example. The rest of the proof extends the idea of that example to inconsistent revealed preferences in general.

Bulk of Proof (click to expand)

Assume the policy is optimal for some particular values over final state. We can then start from those values over final state and compute the best value achievable starting from each state at each earlier time. That's just dynamic programming:
V[S,t]=max S′ reachable in next timestep from S&nbs

...
All the Following are Distinct
by Gianluca Calcagni — 2024-08-02T16:35:51.815Z

In an artificial being, all the following:

Consciousness
Emotionality
Intelligence
Personality
Creativity
Volition

are distinct properties that, in theory, may be activated independently. Let me explain.

What I Hope to Achieve

By publishing this post, I hope to develop and standardise some useful terms and concepts often found in many other posts. I also wish to explain in which sense some properties are linked to others - and, importantly, when they are not. I hope this post will be of help and provide a common context for future further discussions.

What I am not trying to achieve is some universal definition of consciousness/intelligence/… simply because that’s too hard! To avoid controversies, I will focus on operational facets and nothing more[1]. I am still aware that the final result will be very opinionated, so feel free to challenge me and open my mind.

The Basics

Let’s suppose you walk around a new planet and you find some kind of system (artificial or natural - it doesn’t matter) that is able to “autonomously” execute general computations (digital, analog, quantum - it doesn’t matter, and it doesn’t matter which ones and why). Think of it as a universal Turing machine[2]: we’ll call such a system a computational being. Of course, I could just call it a computer - but the problem is that most people don’t recognise themselves as “computers”, hence the term has some semantic bias; also, would you consider an ecosystem as a “computer”? And yet an ecosystem can run computations, if you are inclined to interpret in that way the lifecycle of its inhabitants while they maintain (unknowingly) a stable equilibrium. So, let me use “computational being” as an umbrella term and as the foundation for the rest of the discussion.

Natural examples of computational beings: any living being is also a computational being. Selfish genes favour, by means of natural selection, computations that support their own reproduction. The competition between genes can sometimes lead to autonomous computational beings composed of many cells (such as ourselves).
Artificial examples of computational beings: any computing machine is a computational being. Even server farms can fit the definition, as long as there is a meaningful way to describe their collective computation as "autonomous" from the rest of the world. Can a computational being be part of a larger

...
The ‘strong’ feature hypothesis could be wrong
by lewis smith — 2024-08-02T14:33:58.898Z

NB. I am on the Google Deepmind language model interpretability team. But the arguments/views in this post are my own, and shouldn't be read as a team position.

“It would be very convenient if the individual neurons of artificial neural networks corresponded to cleanly interpretable features of the input. For example, in an “ideal” ImageNet classifier, each neuron would fire only in the presence of a specific visual feature, such as the color red, a left-facing curve, or a dog snout”

- Elhage et. al, Toy Models of Superposition

Recently, much attention in the field of mechanistic interpretability, which tries to explain the behavior of neural networks in terms of interactions between lower level components, has been focussed on extracting features from the representation space of a model. The predominant methodology for this has used variations on the sparse autoencoder, in a series of papers inspired by Elhage et. als. model of superposition.It's been conventionally understood that there are two key theories underlying this agenda. The first is the ‘linear representation hypothesis’ (LRH), the hypothesis that neural networks represent many intermediates or variables of the computation (such as the ‘features of the input’ in the opening quote) as linear directions in it’s representation space, or atoms[1]. And second, the theory that the network is capable of representing more of these ‘atoms’ than it has dimensions in its representation space, via superposition (the superposition hypothesis).

While superposition is a relatively uncomplicated hypothesis, I think the LRH is worth examining in more detail. It is frequently stated quite vaguely, and I think there are several possible formulations of this hypothesis, with varying degrees of plausibility, that it is worth carefully distinguishing between. For example, the linear representation hypothesis is often stated as ‘networks represent features of the input as directions in representation space’. Here are two importantly different ways to parse this:

(Weak LRH) some or many features used by neural networks are represented as atoms in representation space
(Strong LRH) all (or the vast majority of) features used by neural networks are represented by atoms.

The weak LRH I would say is now well supported by considerable empirical evidence. The strong form is much more sp...