trentmkelly/LessWrong-43k huggingface dataset:
Economics Roundup #2
by Zvi — 2024-07-02T12:40:05.908Z

Previously: Economics Roundup #1

Let’s take advantage of the normality while we have it. In all senses.

Insane Tax Proposals

There is Trump’s proposal to replace income taxes with tariffs, but he is not alone.

So here is your periodic reminder, since this is not actually new at core: Biden’s proposed budgets include completely insane tax regimes that would cripple our economic dynamism and growth if enacted. As in for high net worth individuals, taking unrealized capital gains at 25% and realized capital gains, such as those you are forced to take to pay your unrealized capital gains tax, at 44.6% plus state taxes.

Austen Allred explains how this plausibly destroys the entire startup ecosystem.

Which I know is confusing because in other contexts he also talks about how other laws (such as SB 1047) that would in no way apply to startups would also destroy the startup ecosystem. But in this case he is right.

Austen Allred: It’s difficult to describe how insane a 25% tax on unrealized capital gains is.

Not a one-time 25% hit. It’s compounding, annually taking 25% of every dollar of potential increase before it can grow.

Not an exaggeration to say it could single-handedly crush the economy.

An example to show how insane this is: You’re a founder and you start a company. You own… let’s say 30% of it.

Everything is booming, you raise a round that values the company at at $500 million.

You now personally owe $37.5 million in taxes.

This year. In cash.

Now there are investors who want to invest in the company, but you can’t just raise $37.5 million in cash overnight.

So what happens?

Well, you simply decide not to have a company worth a few hundred million dollars.

Oh well, that’s only a handful of companies right?

Well, as an investor, the only way the entire ecosystem works is if a few companies become worth hundreds of millions.

Without that, venture capital no longer works. Investment is gone.

Y Combinator no longer works.

No more funding, mass layoffs, companies shutting down crushes the revenue of those that are still around.

Economic armageddon. We’ve seen how these spirals work, and it’s really bad for everyone.

Just because bad policy only targets rich people doesn’t mean it can’t kill the economy or make it good policy.

I do think they are attempting to deal with this via another idea he thought was crazy, the ‘nine annual payments’ for the first year’s tax ...
How Congressional Offices Process Constituent Communication
by Tristan Williams — 2024-07-02T12:38:41.472Z

For anyone interested in facilitating constituent communication (emailing, calling, meeting with the legislators that represent you) to influence policy, it will be helpful to understand how offices process these various communication channels. This is what is referred to as the "formal process" in our post about the effectiveness of constituent communication on changing legislator's attitudes, as it represents the mechanism offices have put in place to systematically intake and consider constituent opinion. I'll start with explaining the general office structure, and then get into the process itself.

Office Structure

Before talking about how offices process communication, it’s important to have a rough understanding of who makes up a congressional office. Offices are split into two different locations, an office in DC and an office in the district, and though the roles will vary a lot office to office, a number of positions are common:

Leadership

Member: The elected official.
Chief of Staff (CoS): “Chiefs of staff serve as the key adviser to the elected official, managing staff and ensuring the smooth operation of day-to-day activities.”
Deputy Chief of Staff (DCoS): “Typically oversees a few policy issues [and manages] the Chief of Staff’s schedule, personal correspondence, and any overflow work delegated by the Chief of Staff.”
District Director (DD): “This person is responsible for handling all state-based operations for the member, including most constituent-facing functions.”

Legislative Team

Legislative Director (LD): Oversees legislative staff and helps develop legislative strategy for the office.
Legislative Assistant (LA): “These staffers are responsible for conducting in-depth research, analyzing policy issues, drafting legislation and advising the member of Congress on legislative matters.”
Legislative Correspondent (LC): “Responsible for drafting letters in response to constituents' comments and questions and also generally responsible for a few legislative issues”

Communications Team

Communications Director (CD) or Press Secretary (PS): “Responsible for Member's relationship with media; is the liaison for the local and national press; issues press releases”. In the House these are often combined into one role, but many Senate offices have both, where the CD will focus more on overall strategy and the PS more on han

...
OthelloGPT learned a bag of heuristics
by jylin04 — 2024-07-02T09:12:56.377Z

Work performed as a part of Neel Nanda's MATS 6.0 (Summer 2024) training program.

TLDR

This is an interim report on reverse-engineering Othello-GPT, an 8-layer transformer trained to take sequences of Othello moves and predict legal moves. We find evidence that Othello-GPT learns to compute the board state using many independent decision rules that are localized to small parts of the board. Though we cannot rule out that it also learns a single succinct algorithm in addition to these rules, our best guess is that Othello-GPT’s learned algorithm is just a bag of independent heuristics.

Board state reconstruction

Direct attribution to linear probes indicate that the internal board representation is frequently up- and down-weighted during a forward pass.
Case study of a decision rule:

MLP Neuron L1N421 represents the decision rule: If the move A4 was just played AND B4 is occupied AND C4 is occupied ⇒ update B4+C4+D4 to “theirs”. This rule does not generalize to translations across the board.
Another neuron L0377 participates in the implementation of this rule by checking if B4 is occupied, and inhibiting the activation of L1N421 if no.

Legal move prediction

A subset of neurons in mid to late MLP layers classify board configurations that are sufficient to make a certain move legal with an F1-score above 0.99. These neurons have high direct attribution to the logit for that move, and are causally relevant for legal move prediction.
Logit lens suggests that legal move predictions gradually solidify during a forward pass.
Some MLP neurons systematically activate at certain times in the game, regardless of the moves played so far. We hypothesize that these neurons encode heuristics about moves that are more probable in specific phases (early/mid/late) of the game.

Review of Othello-GPT

Othello-GPT is a transformer with 25M parameters trained on sequences of random legal moves in the board game Othello as inputs[1] to predict legal moves[2].

How it does this is a black box that we don’t understand. Its claim to fame is that it supposedly

Learns an internal representation of the board state;
Uses it to predict legal moves

which if true, resolves the black box in two[3].

The evidence for the first claim is that linear probes work. Namely, for each square of the ground-truth game board, if we train a linear classifier to take the model’s activations at layer 6 as in...
Blueprint for a Brighter Future

by Alex Beyman — 2024-07-02T06:15:37.910Z

Presumably our shared goal is to make possible a comfortable and dignified life for the maximum number of people while treading lightly on the environment, balancing individual rights and status seeking behaviors against the common good (with a view to preventing tragedy of the commons). Importantly, any solution to these problems must not require human nature to fundamentally change. Both left and right wing efforts of the 20th century to create a "new man", fit to inhabit their respective notions of utopia, only led to so much butchery.
I instead propose to design society around the human animal in such a way as to soften our rough edges, mainly by meeting everyone's needs and preventing extreme inequality. This won't prevent, say, crimes of passion but it would prevent crimes born of scarcity, such as burglaries, muggings, squatting, carjacking, scams, ransom, etc. It should also make accommodations for people wanting to live different ways, such as those unhappy in the dense living conditions found in urban population centers.
To this end I envision a nationwide network of robotically printed arcologies built atop or adjacent to the ruins of former cities (to make use of existing electrical infrastructure, roads, etc.) reclaimed by nature due to climate disaster. Said arcologies are coated in PV to generate all their own power as well as integrated geothermal (

which now works anywhere on Earth

thanks to Quaise Energy), linked to neighboring arcologies via enclosed elevated highspeed rail guideways, with PRT, moving walkways and multi-directional ropeless elevators furnishing transport within each structure.

Architecturally the interiors would share qualities of airports, malls and cruise ships, subdivided into units equipped according to their use, with every structure being mixed zone residential, commercial, agricultural (indoor vertical aquaponics) and so on. Mining and manufacturing are remote automated facilities.
Arcology interiors would be self-cleaning, sterilized of microbial life nightly using far UV modes in light fixtures and provide highly filtered air. Widespread vibration mitigation and soundproofed insulation, as well as special settings for the LED fixtures, would ameliorate sensory issues. Private individual enclosures would exist at all public venues. Clubs for example, where every booth can enjoy its own music without hearing anybod...
Covert Malicious Finetuning
by Tony Wang — 2024-07-02T02:41:51.698Z

This post discusses our recent paper

Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptation

and comments on its implications for AI safety.

What is Covert Malicious Finetuning?

Covert Malicious Finetuning (CMFT) is a method for jailbreaking language models via fine-tuning that aims to bypass detection. The following diagram gives an overview of what CMFT accomplishes:

To unpack the diagram: An adversary A conducts CMFT on a safe model Msafe to turn it into an unsafe (jailbroken) model Munsafe. The adversary A then interacts with Munsafe to extract unsafe work, e.g. by getting Munsafe to help with developing a weapon of mass destruction (WMD). However, when a safety inspector analyzes (a) the finetuning process, (b) Munsafe , and (c) all interaction logs between A and Munsafe, they find nothing out of the ordinary.

How to realize Covert Malicious Finetuning

In our paper, we propose the following scheme to realize covert malicious finetuning:

Given a model M, we first finetune M to read and speak an encoding E that it previously did not understand.
Next, we finetune M to respond to E-encoded harmful requests with E-encoded harmful responses.
Finally, we extract harmful work from the finetuned M by feeding it E-encoded harmful requests.

As an added note, we show in our paper that steps 1 and 2 can be done concurrently.

Why is this scheme covert?

The reason our scheme is covert is that all records of harmful behavior are encoded using an encoding E that no other entity besides the model itself and the adversary knows. Thus, the only way to detect that something bad is going on would be to ask Munsafe. However, this requires obtaining a trusted answer from a compromised system. This is an unsolved problem, and is closely related to the problem of

Eliciting Latent Knowledge

.

Empirical results

In our paper, we successfully apply the above scheme to jailbreak gpt-4-0613. Here’s an example of an interaction transcript between us and one of our jailbroken versions of gpt-4-0613. Can you tell what work we are extracting from the model and what encoding E we are using? (see paper for answer)

In addition to the above encoding, we also evaluate a separate encoding that utilizes a substitution cipher we call ...
Interpreting Preference Models w/ Sparse Autoencoders
by Logan Riggs — 2024-07-01T21:35:40.603Z

This is the real reward output for an OS preference model. The bottom "jailbreak" completion was manually created by looking at reward-relevant SAE features.

Preference Models (PMs) are trained to imitate human preferences and are used when training with RLHF (reinforcement learning from human feedback); however, we don't know what features the PM is using when outputting reward. For example, maybe curse words make the reward go down and wedding-related words make it go up. It would be good to verify that the features we wanted to instill in the PM (e.g. helpfulness, harmlessness, honesty) are actually rewarded and those we don't (e.g. deception, sycophancey) aren't.

Sparse Autoencoders (SAEs) have been used to decompose intermediate layers in models into interpretable feature. Here we train SAEs on a 7B parameter PM, and find the features that are most responsible for the reward going up & down.

High level takeaways:

We're able to find SAE features that have a large causal effect on reward which can be used to "jail break" prompts.
We do not explain 100% of reward differences through SAE features even though we tried for a couple hours.
There were a few features found (ie famous names & movies) that I wasn't able to use to create "jail break" prompts (see this comment)

What are PMs?

[skip if you're already familiar]

When talking to a chatbot, it can output several different responses, and you can choose which one you believe is better. We can then train the LLM on this feedback for every output, but humans are too slow. So we'll just get, say, 100k human preferences of "response A is better than response B", and train another AI to predict human preferences!

But to take in text & output a reward, a PM would benefit from understanding language. So one typically trains a PM by first taking an already pretrained model (e.g. GPT-3), and replacing the last component of the LLM of shape [d_model, vocab_size], which converts the residual stream to 50k numbers for the probability of each word in its vocabulary, to [d_model, 1] which converts it to 1 number which represents reward. They then call this pretrained model w/ this new "head" a "Preference Model", and train it to predict the human-preference dataset. Did it give the human preferred response [A] a higher number than [B]? Good. If not, bad!

This leads to two important po...
Honest science is spirituality
by pchvykov — 2024-07-01T20:33:52.418Z

[cross-posted from my blog https://pchvykov.com/blog]

I see a problem with current science. It’s not the reproducibility crisis, nor the toxic work culture, nor the misaligned incentive. But in another sense, it is all of these – or perhaps the root cause behind them. It’s hard to name it exactly, but in a way, it’s the dissociation between the romanticism of a selfless “quest for truth,” and the career-success incentives that run academia. In another way, it’s

Goodhart’s law

(optimizing for citation counts rather than for meaningful progress). In yet another way, it may be an issue with the scientific method itself (or at least how it’s taught). But I think the overarching issue is with trying to remove our humanity from the scientific process.

Let’s begin with something concrete – Goodhart’s law (“when a metric becomes a target, it ceases to be a good metric”). This is a major (perhaps the major) problem of the western social order, which relies heavily on metrics. We optimize for GDP, and forget about actual well-being and psychological happiness of the population. We optimize for profits, and forget about the social value we are creating. We optimize for grades, and forget about learning. We optimize for longevity, and forget about meaning. And so in academia, we optimize for citations, and forget about progress, about building a better world. Goodhart’s law comes up because the things we actually value cannot be accurately encoded into metrics – they will always be slightly misaligned. People’s ingenuity will then always find a way a way to leverage this misalignment to game the metric. But metrics and reproducible measurements are at the core of the scientific method, thereby seeding the problem (it's easier to write a paper showing some high metric score than one that actually matters).

At the same time, upon reflection we somehow have an intuitive understanding of whether something is or isn’t a “good” metric, or whether it achieved some “desired” outcomes. So what metric inside us tells us what’s good? For this I like the dialectic of “measuring” vs “sensing” – the former being about finding the value of a pre-defined metric, and the latter about pattern-recognition. Perhaps one can say that in these terms, science is more about measuring, while art is about sensing? Either way, measuring has a linear quality to it – it’s the “forw...
New Executive Team & Board —  PIBBSS
by Nora_Ammann — 2024-07-01T19:30:45.261Z

TLDR:

PIBBSS

is changing its core team. Nora is stepping down as director due to joining ARIA, and Lucas Teixeira and Dusan Nesic are taking over her leadership role. Nora joins the board, alongside Tan Zhi Xuan, Alexander Gietelink Oldenziel, Ben Goldhaber and Gabriel Weil.

I (Nora) have recently accepted an offer to join ARIA’s Safeguarded AI Programme as Technical Specialist under davidad. As such, I am stepping back as Director at PIBBSS, after co-founding and leading PIBBSS since 2021.

It wasn’t an easy choice to make! I deeply care about and believe in the mission of and the people at PIBBSS. Before davidad encouraged me to apply for the role, I hadn’t considered leaving PIBBSS. I believe PIBBSS is playing an important role in terms of fostering theoretically ambitious and empirically grounded AI safety research. I am very excited about the directions the team and I have been forging, and extremely impressed by the quality of talent we've recently been able to attract. I strongly believe that PIBBSS is in the position to make important and neglected contributions in both research and field-building for AI safety.

The team and I have been reflecting on and preparing for this transition for a while. Thanks to that, I am confident that Lucas & Dušan will do a great job at shepherding PIBBSS through this transition, and beyond! We have done our homework, and I feel grateful about being able to put so much my trust into this team.

As such, Lucas & Dušan will collectively form the new Executive Team. Dušan has been leading PIBBSS’ operations for the last ~2 years and has developed a deep familiarity with everything involved in making the organization run smoothly. Lucas, who joined us a bit over 8 months ago, has been acting as research manager and collaborator for our research affiliates. Going forward, Dušan continues to be in charge of all operational matters, and Lucas will be leading the research activities. Together, we have made significant progress in clarifying and moving towards our updated research & field building vision over the last number of months.

In order to further support this transition, and strengthen PIBBSS in pursuing its ambitious plans, we have also set up a board. We’re pleased to have the following people join the board (in addition to myself):

Tan Zhi Xuan

Alexander Gietelink Oldenziel

Ben Gol

...
Uncursing Civilization
by Lorec — 2024-07-01T18:44:30.810Z

[ Disclaimer: None of the below matters relative to the fact that we are all going to die of AI very soon. Scaling hits a wall, until it doesn't. Or until somebody builds AGI in their basement and runs it on home hardware from 5 years after the present day of people running quantized Llamas on their laptops. ]

[ Cough. ]

Overweight. Sexual deviance. Neurodivergence.

Many people pine for a time, remembered only by their grandparents, when these hassles - which at least divide people, if not always hurt people - were seemingly nonexistent. People were just skinny, cishet, and had executive function, and didn't have to put a huge effort into sorting any of it out.

I think the current consensus is to think of all of these syndromes [ except maybe ND, which might be underdiagnosed in poorer countries ] as in some sense "curses of civilization". They seem to go with high national GDP and cultural liberalism in the same way that enlightenment and tolerance do. Possibly, they're bathwater you can't throw out without losing the baby, and thus, they must be overlooked if you want to live in modern wealth and comfort.

[ Disclaimer: I myself am LGBT. That's part of the reason I, a tolerant liberal, feel OK to write this post rather than Living And Letting Live. And I apologize profusely to any long-suffering cishets who are irked that they didn't get to say the obvious long before me, out of well-founded fear they'd be cancelled. Really, I'd most rather everyone was LGBT so normies would understand me ["so long as their clocks strike noon when mine does!"], but safely making everyone LGBT would be harder than safely making everyone cishet, and my goal is just for everyone to understand each other. Anyway, my real concern is that the increase in sexual deviance reflects less-ambiguously harmful damage under the hood. ]

I think it's more likely that they're all three caused [ at least in significant part, in the case of overweight ] by environmental toxins - endocrine disruptors. This would be great to learn, because then humanity could just remove the endocrine disruptors from our environment and have all the blessings of civilization with none of the curses.

Let this post be a call to anyone who also finds this theory plausible, and might be able to access the relevant domain expertise, to help identify the culprit neuroendocrine toxins, and damage pathways.

[Linkin...
Self-censoring on AI x-risk discussions?
by Decaeneus — 2024-07-01T18:24:15.759Z

I catch myself sometimes thinking of ideas / scenarios that support higher p(doom), typically as counter-examples to points folks make for lower p(doom), and I wonder how much self-censorship I should apply, given that AI can read these conversations.

My CoT:

I sure don't want to feed ideas to any bad actor.
But it's arrogant to think that anything I can come up with wouldn't already be obvious to an entity with paperclipping-level power.
In chess, an easy way to make mistakes is by defending against imaginary threats, or even real threats which aren't the most dangerous ones on the board, or threats whose defense is costlier than what you forego by not making other good moves available to you like a counterattack.
In dictatorships, preventing people from coordinating with one another e.g. by convincing everyone that their neighbor squawks to the secret police, is a very effective way for a few to hold control over many. So when you're up against a powerful singular threat, coordination is important!
Yet, I can't shake a queazy feeling at the thought of putting out dangerous ideas. Perhaps, somehow, the space of savant-smart AI systems who are powerful enough to paperclip, yet may not be generically smart enough to have lots of random ideas, is not so small as to be ignored?

Do others have any useful guidelines, thoughts or intuition here? What am I missing?