trentmkelly/LessWrong-43k huggingface dataset:
Alignment: "Do what I would have wanted you to do"
by Oleg Trott — 2024-07-12T16:47:24.090Z

Yoshua Bengio writes[1]:

nobody currently knows how such an AGI or ASI could be made to behave morally, or at least behave as intended by its developers and not turn against humans

I think I do[2]. I believe that the difficulties of alignment arise from trying to control something that can manipulate you. And I think you shouldn't try.

Suppose you have a good ML algorithm (Not the stuff we have today that needs 1000x more data than humans), and you train it as a LM.

There is a way to turn a (very good) LM into a goal-driven chatbot via prompt engineering alone, which I'll assume the readers can figure out. You give it a goal "Do what (pre-ASI) X, having considered this carefully for a while, would have wanted you to do".

Whoever builds this AGI will choose what X will be[3]. If it's a private project with investors, they'll probably have a say, as an incentive to invest.

Note that the goal is in plain natural language, not a product of rewards and punishments. And it doesn't say "Do what X wants you to do now".

Suppose this AI becomes superhuman. Its understanding of languages will also be perfect. The smarter it becomes, the better it will understand the intended meaning.

Will it turn everyone into paperclips? I don't think so. That's not what (pre-ASI) X would have wanted, presumably, and the ASI will be smart enough to figure this one out.

Will it manipulate its creators into giving it rewards? No. There are no "rewards".

Will it starve everyone, while obeying all laws and accumulating wealth? Not what I, or any reasonable human, would have wanted.

Will it resist being turned off? Maybe. Depends on whether it thinks that this is what (pre-ASI) X would have wanted it to.

^

https://yoshuabengio.org/2024/07/09/reasoning-through-arguments-against-taking-ai-safety-seriously/

^

I'm not familiar with the ASI alignment literature, but presumably he is. I googled "would have wanted" + "alignment" on this site, and this didn't seem to turn up much. If this has already been proposed, please let me know in the comments.

^

Personally, I'd probably want to hedge against my own (expected) fallibility a bit, and include more people that I respect. But this post is just about aligning the AGI with its creators.
Virtue taxation
by Dentosal — 2024-07-12T14:56:36.172Z

You jump to a muddy pond to save a drowning child. You're rewarded with a magnified "thank you", ruined cloting and an unpleasant day.

You witness a crime and report it to the police. The resulting judicial process will take up tens of hours of your time with barely any compensation.

You refuse to break laws or lie to the government officials in situations where that is the norm. You're rewarded with extra bureucracy, extra fees, or not getting something you needed. (Many laws can exists only because they're not followed to the letter.)

To reduce CO₂ footprint, you take a train instead of an airplane, losing dozens of labor hours in time and money.[1]

You have to make a taboo tradeoff. You quickly calculate the expected utility of each choice. Instead of obfuscating this process like a decent person, you state your numbers plainly. You get shunned/fired/not-re-elected.

You do the right thing, you end up worse off. You did foresee this. "No good deed goes unpunished", as the saying goes.

In each of these situations, the incentives are stacked against doing the virtuous thing. This is inevitable; if incentives we're the other way around, there would be no virtue points to be gained in those acts, so they wouldn't be on this list. (Not everything is horribly broken.)

Goodhart's law states:

Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.

To tax or publicly shun an activity is to create an incentive against doing that. However, the incentive must target an observable behavior.
Typically the observed quantity is then reduced. However, the reduction often consists mostly of recategorization or hiding.

When Finland introduced a tax on candy, companies moved to produce cookies, which were not classified as candy. [2]

Several behaviors are considered social dark matter and not talked about. And others are considered valuable enough to do semi-costly virtue signaling.

In legal contexts, the hiding part is a crime, for instance tax or benefits fraud. Naturally crimes have punishments, and it's typically quite easy to make the punishment worse than the benefit obtained from the illegal activity. If you get caught, that is. The hard part is enforcement. Expected utility of a crime stays positive if the probabilty of getting caught is low. If there's almost no enforcement, then the tax is not on the orignal target, bu...
Most smart and skilled people are outside of the EA/rationalist community: an analysis
by titotal — 2024-07-12T12:13:56.215Z
Link: https://open.substack.com/pub/titotal/p/most-smart-and-skilled-people-are?r=1e0is3&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

Introduction:

The (highly interrelated) effective altruist and Rationalist communities are very small on a global scale. Therefore, in general, most intelligence, skill and expertise is outside of the community, not within it.

I don’t think many people will disagree with this statement. But sometimes it’s worth reminding people of the obvious, and also it is worth quantifying and visualizing the obvious, to get a proper feel for the scale of the difference. I think some people are acting like they have absorbed this point, and some people definitely are not.

In this post, I will try and estimate the size of these communities. I will compare how many smart people are in the community vs outside the community. I will do the same for people in a few professions, and then I will go into controversial mode and try and give some advice that I think might follow from fully absorbing these results.

How big is EA and rationalism

To compare EA/rats against the general population, we need to get a taste for the size of each community. Note that EA and Rationalism are different, but highly related and overlapping communities, with EA often turning to rationalism for it’s intellectual base and beliefs.

It is impossible to give an exact answer to the size of each community, because nobody has a set definition of who counts as “in” the community. This gets even more complicated if you factor in the frequently used “rationalist adjacent” and “EA adjacent” self labels. For the sake of this article, I’m going to grab a pile of different estimates from a number of sources, and then just arbitrarily pick a number.

Recently JWS placed an estimate of the EA size as the number of “giving what we can” pledgees, at 8983. Not everyone who is in EA signs the pledge. But on the flipside, not everyone who signs the pledge is actively involved in EA.

A survey series from 2019 tried to estimate this same question. They estimated a total of 2315 engaged EA’s, 6500 active EA’s in the community, with 1500 active on the EA forum.

Interestingly, there has also been an estimate of how many people had even heard of effective altruism, finding 6.7% of the US adult population, and 7.9% of top university students, had heard of EA. I would assume this has gone up since then due to various scandals.

We can also look at the subscriber counts to various subreddits, although clicking a “subscribe” button once is ...
2024 Freedom Communities Events
by Tudor Iliescu — 2024-07-12T08:04:53.456Z

Ready to enter the world of network states and free communities?

These upcoming events[1] are your gateway to exploring parallel societies, disruptive governance ideas, and freedom-enabling technologies.

JUL 20 - AUG 31, Georgia (the country)
ZuVillage
According to Veronica and @Burns, ZuVillage Georgia is an experimental crypto community focused on cognitive sovereignty, frontier tech, and cypherpunk principles. It uses innovative digital infrastructure to foster long-term co-living and accelerate decentralized technology development.
AUG 22, Berlin (Germany)
Free Cities Meetup
Join the Free Cities Foundation Berlin meetup on August 22nd for an evening of presentations and networking. Free Citadels Founding Patron Titus Gebel will discuss global Free City developments.
SEP 22, Singapore
Network State Conference
Organised by Balaji Srinivasan, this conference will feature discussions on startup societies, digital nomadism, and network states. Notable speakers include Brian Armstrong and Vitalik Buterin. In-person and remote tickets available.
OCT 10 - NOV 10, Chiang Mai (Thailand)
Edge City Lanna
Lanna is a monthlong event fostering a popup village for innovators in tech, science, and society. Participants engage in themes like AI, crypto, health, and biohacking.
OCT 11-14, Austin (Texas)
The Network State Camp
TNS Camp offers a unique blend of workshops, talks, and community-building activities. It aims to foster connections among leaders and visionaries in a classic summer camp setting.
NOV 1-3, Prague (Czechia)
Liberty in Our Lifetime
Free Cities Foundation organises an annual conference showcasing Free City projects like Prospera, Ciudad Morazan, and Montelibero. Attendees learn about self-sovereignty, network with like-minded individuals, and explore practical opportunities for relocating or investing in autonomous communities.
PS: Use 'FreeCitadels' for a nice ticket discount.

^

List compiled for FreeCitadels.com.
Faithful vs Interpretable Sparse Autoencoder Evals
by Louka Ewington-Pitsos — 2024-07-12T05:37:18.525Z

Summary

Sparse Autoencoders (SAEs) have become a popular tool in Mechanistic Interpretability but at the time of writing there is no standardm automated way to accurately assess the quality of a SAE. In practice many different metrics are applied and interpreted to assess SAE quality in different studies.

Conceptually however what makes a good SAE is relatively uncontroversial: its latent features must reflect the structure of the model it was trained on whilst also reflecting the structure of human linguistic understanding[1][2][3]. These twin objectives are usually referred to as "Faithfulness" and "Interpretability"[1][4].

In this article we will go through 8 common metrics used to evaluate the quality of a SAE and try to demonstrate that, though this is sometimes not stated explicitly, each metric can be thought of as attempting to measure either Faithfulness or Interpretability.

This article is not an attempt to prove anything, but rather suggest a mental model which might be useful.

Faithfullness vs Interpretability

At this stage there is no widespread practical use for SAEs. When one arises a metric pertaining to this use will probably become the gold standard for what makes a "good" SAE.

Until then the qualities sought in an SAE are stated well by Huang et. al.[2]

A central goal of interpretability is to localize an abstract concept to a component of a deep learning model that is used during inference.

In other words a good SAE will be one which identifies "abstract concepts" in the linguistic sense and maps these to structures inside the host LM[1].

Importantly these two qualities are independent of one another. In fact to anyone who has trained a SAE the two will probably seem frustratingly opposed.

Interpretability can be achieved more easily by sacrificing the close relation held between the Autoencoder and the host model. For example by adding layers to the encoder and decoder pipelines.

Similarly Faithfullness can be achieved without interpretability. The most extreme example would be the neurons of the LM themselves, which are notoriously difficult to interpret[1]. A less extreme example would be an Autoencoder trained without any, or a very low sparsity loss term.

The goal of much of the current SAE literature is training one which achieves both Faithfullness and Interpretability at the same time.

Explainability vs Interp

...
Moving away from physical continuity
by ProgramCrafter — 2024-07-12T05:05:01.231Z

In thought experiments about cloning objects, people often call the object which maintained physical continuity "original" and other "clone". This seems like just a naming, but when experiments talk about cloning a person, people tend to care where their original version is. This feels dodgy in cases when person doesn't even feel their continuity (in sleep, for example), so I think that is a bit overvalued.

Being one and the same

Let's take a black box cloning cars - perhaps taking the required set of details, correcting them on level of atoms to be indistinguishable from input car and constructing the second car. Inside the box, it is pretty obvious which car is original and which is a clone, but on the outside one can only learn that from box specification.

We can add an operation inside the box: let workers use a quantum RNG, and swap the result cars iff the random bit was 1. Then, person on the outside cannot tell which car is a clone. One could say "I don't know that, however there is 50% that cloned car is left one and 50% that it's right one instead"; I'd respond "the question is ill-formed since your definition of original doesn't really apply to this case".

I prefer to define being same by causal flows. For instance, let's open a window in the car before putting it into the black box and check which result car is causally downstream of input one - that is, has window opened. Once we see that both products have their windows opened, we can name both same[1], and same as the input.

^

that said, in future the cars might continue to be approximately same or very similar, or might not if one gets in a crash; their evolutions are different because of the outer world (drivers, for instance)
Transformer Circuit Faithfulness Metrics Are Not Robust
by Joseph Miller — 2024-07-12T03:47:30.077Z
Link: https://arxiv.org/abs/2407.08734

When you think you've found a circuit in a language model, how do you know if it does what you think it does? Typically, you ablate / resample the activations of the model in order to isolate the circuit. Then you measure if the model can still perform the task you're investigating.

We identify six ways in which ablation experiments often vary.[1][2]

How do these variations change the results of experiments that measure circuit faithfulness?

TL;DR

We study three different circuits from the literature and find that measurements of their faithfulness are highly dependent on details of the experimental methodology. The IOI and Docstring circuits in particular are much less faithful than reported when tested with a more precise methodology.
The correct circuit for a set of prompts is undefined. The type of ablation you use to isolate the circuit determines the task that you are asking the circuit to perform - and therefore also the optimal circuit.
This is especially important because previous work in automatic circuit discovery has tested algorithms by their ability to recover these "ground-truth" circuits from the literature - without considering these potential pitfalls and nuances.

Case Studies

We look at three circuits from the mech interp literature to demonstrate that faithfulness metrics are highly sensitive to the details of experimental setup.

Indirect Object Identification Circuit

The IOI circuit is the most well known circuit in a language model. It computes completions to prompts of the form:

"When Mary and John went to the store, John gave a bottle of milk to ____"

Image from: Wang et al. (2022)

The circuit is specified as a graph of important attention heads (nodes) and the interactions between them (edges) as applied to a specific sequence of tokens. The authors report that the circuit explains 87% of the logit difference between the two name tokens. They find this number by passing some inputs to the model and ablating all activations outside of the circuit. Then they measure how much of the logit difference between the correct and incorrect name logits remains.

However, an important detail is that they arrived at this number by ablating the nodes (heads) outside of the circuit, not by ablating the edges (interactions between heads) outside of the circuit. So they don't ablate, for example, the edges from the previous token heads to the name mover heads, even...
On Artificial Wisdom
by Jordan Arel — 2024-07-12T00:20:33.241Z

Epistemic Status:

I sustained multiple semi-serious debilitating injuries while in the process of writing this piece for the “Automation of Wisdom and Philosophy” essay contest and so was not able to get feedback and finish editing before publishing in time for the contest. Therefore, any feedback is highly appreciated, thanks!

TL;DR

In this first post in a series on Artificial Wisdom I introduce the term “Artificial Wisdom (AW),” which refers to artificial intelligence systems which substantially increase wisdom in the world. Wisdom may be defined as "thinking/planning which is good at avoiding large-scale errors," including both errors of commission and errors of omission; or as “having good goals” including terminal goals and sub-goals.

Due to orthogonality, it is possible we could keep AI under control and yet use it very unwisely. Four scenarios are discussed on how AI alignment interacts with artificial wisdom, with artificial wisdom being an improvement on any world, unless pursuit of AW significantly detracts from alignment, causing it to fail.

By “strapping” wisdom to AI via AW as AI takes off, we may be able to generate enormous quantities of wisdom in both humans and autonomous AI systems which could help us navigate Transformative AI and "The Most Important Century" wisely, in order to achieve existential security and navigate toward a positive long-term future.

The Genie

A genie pops out of a magic lamp and grants you three wishes.

What is your first wish?

One possibility is to wish for a million more wishes, which will each be used to wish for another million more powerful wishes, which will in turn each be used to wish for another million even more powerful wishes, etc. Maybe as an afterthought you set aside a few wishes to wish that all of the other wishes don’t do too much accidental harm.

Another option is to wish that your next two wishes do what you actually want them to do without any major negative side effects. This one seems like a pretty good idea.

A third option is to wish to know “What are the best possible next two wishes I could wish for?” (“best” in some objective sense, or “best” from your own perspective and what you would decide is best after having observed all possible consequences of all possible wishes.)

This last wish was close to the wish of King Solomon in the Bible, and even more impressively, God had only of...
Yoshua Bengio: Reasoning through arguments against taking AI safety seriously
by Judd  Rosenblatt — 2024-07-11T23:53:17.187Z
Link: https://yoshuabengio.org/2024/07/09/reasoning-through-arguments-against-taking-ai-safety-seriously/ 

He starts by emphasizing

The issue is so hotly debated because the stakes are major: According to some estimates,

quadrillions of dollars

of net present value are up for grabs, not to mention political power great enough to significantly disrupt the current world order.

[...]

The most important thing to realize, through all the noise of discussions and debates, is a very simple and indisputable fact: while we are racing towards AGI or even ASI, nobody currently knows how such an AGI or ASI could be made to behave morally, or at least behave as intended by its developers and not turn against humans.

And goes on to do a pretty great job addressing "those who think

AGI and ASI are impossible or are centuries in the future
AGI is possible but only in many decades
that we may reach AGI but not ASI, those who think that AGI and ASI will be kind to us
that corporations will only design well-behaving AIs and existing laws are sufficient
who think that we should accelerate AI capabilities research and not delay benefits of AGI
talking about catastrophic risks will hurt efforts to mitigate short-term human-rights issues with AI
concerned with the US-China cold war
that international treaties will not work
the genie is out of the bottle and we should just let go and avoid regulation
that open-source AGI code and weights are the solution
worrying about AGI is falling for Pascal’s wager"
Podcast: "How the Smart Money teaches trading with Ricki Heicklen" (Patrick McKenzie interviewing)
by rossry — 2024-07-11T22:49:06.633Z
Link: https://www.complexsystemspodcast.com/episodes/teaching-trading-ricki-heicklen/

In Patrick's words, above the fold:

Some years ago, I built a trading simulation for a startup I was running, Starfighter. It was, at the time, by far the most impressive technical artifact I'd ever made, including a scratch-built fake stock exchange. I recently ran into Rick Heicklen, who has a business teaching trading, and said we should swap notes on comparative pedagogical game design.

Trading doesn't have a vast body of useful information targeted at people of various skill levels. There is a vast amount of fanfic. Actual practitioners work at a relatively small number of firms and are extremely disincentivized from explaining how the world really works to outsiders. Ricki and I talk about some of the lessons people learn early in their trading careers, how those correspond to market microstructure which impacts everything tied to the financial markets, and the security mindset. There are also some (perhaps surprisingly) portable lessons from trading to many other complex systems.

The link has 1.7hr of audio and 23k words of transcript. Patrick is on Twitter, writes his newsletter, and has written on the EA Forum; Ricki is iiuc Lightcone staff, and writes on LessWrong, Twitter, and her own blog.