trentmkelly/LessWrong-43k huggingface dataset:
The Gradient – The Artificiality of Alignment
by mic — 2023-10-08T04:06:40.074Z
Link: https://thegradient.pub/the-artificiality-of-alignment/

The Gradient is a “digital publication about artificial intelligence and the future,” founded by researchers at the Stanford Artificial Intelligence Laboratory. I found the latest essay, “The Artificiality of Intelligence,” by a PhD student at UC Berkeley, to be an interesting perspective from AI ethics/fairness.

Some quotes I found especially interesting:

For all the pontification about cataclysmic harm and extinction-level events, the current trajectory of so-called “alignment” research seems under-equipped — one might even say misaligned — for the reality that AI might cause suffering that is widespread, concrete, and acute. Rather than solving the grand challenge of human extinction, it seems to me that we’re solving the age-old (and notoriously important) problem of building a product that people will pay for. Ironically, it’s precisely this valorization that creates the conditions for doomsday scenarios, both real and imagined. …

In a recent

NYT interview

, Nick Bostrom — author of Superintelligence and core intellectual architect of effective altruism — defines “alignment” as “ensur[ing] that these increasingly capable A.I. systems we build are aligned with what the people building them are seeking to achieve.”

Who is “we”, and what are “we” seeking to achieve? As of now, “we” is private companies, most notably OpenAI, the one of the first-movers in the AGI space, and Anthropic, which was founded by a cluster of OpenAI alumni.

OpenAI

names

building superintelligence

as one of its primary goals

. But why, if the risks are so great? … first, because it will make us a ton of money, and second, because it will make someone a ton of money, so might as well be us. …

Of course, that’s the cynical view, and I don’t believe most people at OpenAI are there for the sole purpose of personal financial enrichment. To the contrary, I think the interest — in the technical work of bringing large models into existence, the interdisciplinary conversations of analyzing their societal impacts, and the hope of being a part of building the future — is genuine. But an organization’s objectives are ultimately distinct from the goals of the individuals that comprise it. No matter what may be publicly stated, revenue generation will always be at least a complementary objective by which OpenAI’s governance, product, and technical decisions are structured, even if not fully determined. A

...
Comparing Anthropic's Dictionary Learning to Ours
by Robert_AIZI — 2023-10-07T23:30:32.402Z

Readers may have noticed many similarities between Anthropic's recent publication Towards Monosemanticity: Decomposing Language Models With Dictionary Learning (LW post) and my team's recent publication Sparse Autoencoders Find Highly Interpretable Directions in Language Models (LW post). Here I want to compare our techniques and highlight what we did similarly or differently. My hope in writing this is to help readers understand the similarities and differences, and perhaps to lay the groundwork for a future synthesis approach.

First, let me note that we arrived at similar techniques in similar ways: both Anthropic and my team follow the lead of Lee Sharkey, Dan Braun, and beren's [Interim research report] Taking features out of superposition with sparse autoencoders, though I don't know how directly Anthropic was inspired by that post. I believe both our teams were pleasantly surprised to find out the other one was working on similar lines, serving as a form of replication.

Some disclaimers: This list may be incomplete. I didn't give Anthropic a chance to give feedback on this, so I may have misrepresented some of their work, including by omission. Any mistakes are my own fault.

Target of Dictionary Learning/Sparse Autoencoding

A primary difference is that we looked for language model features in different parts of the model. My team trained our sparse autoencoder on the residual stream of a language model, whereas Anthropic trained on the activations in the MLP layer.

These objects have some fundamental differences. For instance, the residual stream is (potentially) almost completely linear whereas the MLP activations have just gotten activated, so their values will be positive-skewed. However, it's encouraging that this technique seems to work on both the MLP layer and residual stream. Additionally, my coauthor Logan Riggs successfully applied it to the output of the attention sublayer, so both in theory and in practice the dictionary learning approach seems to work well on each part of a language model.

Language Model Used

Another set of differences comes from which language model our teams used to train the autoencoders. My team used Pythia-70M and Pythia-410M, whereas Anthropic's language model was custom-trained for this study (I think). Some differences in the language model architectures:

Layers: The Pythia models have 6 or 24 layers. The Anthr

...
A thought about the constraints of debtlessness in online communities
by mako yass — 2023-10-07T21:26:44.480Z

I notice that there's a lot of work done in online communities that's hard to price. We know so little about its price that it's a bit ambiguous as to whether it should be paid for or charged for:

Is writing posts a duty or a privilege? You get to put anything you want in them, but not really, because it has to be something people want to read, and you're competing with so many other options so it's tough.
Is curating articles a duty or a privilege? If you're a prominent curator, writers will essentially be writing for your tastes. On the other hand, you're going to have to read a lot of not-yet-curated articles. And again, you have to operate within the bounds of what others want and competition with other curators is intense.
Is consuming curated articles a duty or a privilege? It's definitely supposed to be a gift. But you find yourself reading a lot of news that stresses you out, as an intermediate goal towards later writing your own takes about it and competing in the producer arena.
Is moderating a duty or a privilege? On the one hand, you have the highest form of power over your community. On the other, the community is mad at you, and can leave, and mostly can't really be herded.

Reflecting on this, I think it might not be a coincidence that all of the most common roles in online communities have equal or greater benefits than costs: We don't have ways of compensating an imbalance. There's no process for tracking and repaying debt (and even with better technologies, crediting the production of information — a non-excludable good — will remain difficult), which explains why it would be the case that:

It seems as if little is done, in an online community, unless it's from the narrow category of acts that are in themselves self-compensating.

Which is concerning, because most conceivable beneficial exchanges pair acts which are not in that category.
Arguments for utilitarianism are impossibility arguments under unbounded prospects
by MichaelStJules — 2023-10-07T21:08:59.645Z

Summary

Most moral impact estimates and cost-effectiveness analyses in the effective altruism community use (differences in) expected total welfare. However, doing so generally is probably irrational, based on arguments related to St Petersburg game-like prospects. These are prospects that are strictly better than each of their infinitely many possible but finite actual value outcomes, with unbounded but finite value across these outcomes. The arguments I consider here are:

Based on a money pump argument, the kind used to defend the axioms of expected utility theory, maximizing the expected value of an unbounded utility function is irrational. As a special case, expectational total utilitarianism, i.e. maximizing the expected value of total welfare, is also irrational.
Two recent impossibility theorems demonstrate the incompatibility of Stochastic Dominance — a widely accepted requirement for instrumental rationality — with Impartiality and each of Anteriority and Separability. These last three principles are extensions of standard assumptions used in theorems to prove utilitarianism, e.g. Anteriority in Harsanyi's theorem.

Taken together, utilitarianism is either irrational or the kinds of arguments used to support it in fact undermine it instead when generalized. However, this doesn't give us any positive arguments for any other specific views.

I conclude with a discussion of responses.

EDIT: I've rewritten the summary and the title, and made various other edits for clarity and to better motivate. The original title of this post was "Utilitarianism is irrational or self-undermining".

Basic terminology

By utilitarianism, I include basically all views that are impartial and additive in deterministic fixed finite population cases. Some such views may not be vulnerable to all of the objections here, but they apply to most such views I’ve come across, including total utilitarianism. These problems also apply to non-consequentialists using utilitarian axiologies.

To avoid confusion, I prefer the term welfare as what your moral/social/impersonal preferences and therefore what your utility function should take into account.[1] In other words, your utility function can be a function of individuals’ welfare levels.

A prospect is a probability distribution over outcomes, e.g. over heads or tails from a coin toss, over possible futures, etc..

...
Sam Altman's sister claims Sam sexually abused her -- Part 1: Introduction, outline, author's notes
by pythagoras5015 — 2023-10-07T21:06:49.396Z

Introduction

Sam Altman's sister, Annie Altman, has claimed that Sam sexually abused her when she was a child for approximately 9 years, beginning when she was 3 years old and he was 12 years old, and continuing until she was approximately 12 years old and he was approximately 21 years old.

Annie has stated that she has suffered various severe forms of abuse from Sam Altman throughout her life, including sexual, physical, emotional, verbal, financial, technological (shadowbanning, hacking), pharmacological (forced Zoloft), and psychological abuse. She has also stated that she has experienced abuse from her other brother Jack Altman, though she has noted that most of the abuse she's experienced has come from Sam.

This is the 1st in a series of 11 posts I've written that attempt to provide a comprehensive, objective, and unbiased account of the situation, and the information that's currently available.

On January 6, 2025, Annie Altman filed a lawsuit against Sam Altman in the United States District Court for the Eastern District of Missouri, Eastern Division.

The lawsuit is ongoing. A jury trial is set to begin Monday, March 31, 2025. Key events that have occurred in the lawsuit thus far (I'll provide more detail in the 6 subsequent posts that follow this post):

January 6, 2025: Annie's legal counsel files a complaint against Sam Altman in the United States District Court for the Eastern District of Missouri. The complaint demands a jury trial.
January 7, 2025: Sam posts a statement from him, his brothers, and his mother in response on X (formerly Twitter.)
March 7, 2025: Sam's legal counsel files:

A motion to dismiss Annie's "Common-Law Claims" and to dismiss or strike Annie's "Prayer for Punitive Damages."
A memorandum of law in support of the above motion.
"Defendant Samuel Altman's Answer and Affirmative Defenses to Plaintiff's Complaint; Counterclaims in Reply"

March 14, 2025: Annie's legal counsel files "Plantiff's Unopposed Motion to Extend Pleading and Briefing Deadlines"

"Plantiff's Counsel is also currently set to begin a jury trial on March 31, 2025"

March 17, 2025: "Re: 19 Unopposed MOTION for Extension of Time...ORDERED GRANTED. Signed by District Judge Sarah E. Pitlyk on 03/17/2025."
March 31, 2025: Annie's legal counsel will begin a jury trial.
Disclaimer: I am not any sort of legal professional, so it's possible that I am misunderstanding or misint

...
Griffin Island
by jefftk — 2023-10-07T18:40:06.642Z

The park near our house has an interesting sign:

A parent put it up, labeling the area you're able to explore from this
point without crossing any streets. It's the large red area on my child walksheds map.

The name has caught on well, and it's very useful to be able to refer
to this block!

Yesterday I added it to OpenStreetMap
and Google
Maps and I now see it in both.
Every Mention of EA in "Going Infinite"
by KirstenH — 2023-10-07T14:42:32.217Z
Link: https://open.substack.com/pub/ealifestyles/p/every-mention-of-ea-in-going-infinite?r=242xrl&utm_campaign=post&utm_medium=web

Pinocchio is narrated by Jiminy Cricket; the Great Gatsby is narrated by Nick; Sam Bankman-Fried’s story is narrated by Michael Lewis. His book, Going Infinite, isn’t written from Sam’s perspective, exactly. Michael Lewis is a sympathetic fly on the wall documenting a tragically flawed underdog being destroyed by his own infinite ambition.

But like the Great Gatsby, where Nick speaks so fondly about Daisy (who Gatsby loves) and so unkindly about Tom (who Gatsby hates), you get the sense that Sam’s love for the effective altruism movement colours a lot of what ended up in the book.

Spectacle is another thing that colours what went into the book. Michael Lewis doesn’t take the tabloid road and make this a book about drugs and shrimp rings and “the polycule”, thankfully. But I can’t shake the feeling that most of the characters in the book are basically zoo animals on display for the public’s amusement.

He’s constantly saying, implicitly or explicitly, that effective altruists are high iq children who don’t trust emotions or adults. It feels condescending: “Of course you failed. You’re children! Idiot savants, like something out of a Dostoevsky novel.”

But he also seems to really believe EAs are genuine, and that’s not nothing.

The first mention of effective altruism is in the preface. It doesn’t say effective altruism, exactly, but

He [Sam] needed infinity dollars because he planned to address the biggest existential risks to life on earth: nuclear war, pandemics far more deadly than Covid, artificial intelligence that turned on mankind and wiped us out, and so on.

And that really sets the tone for the rest of the book. At least according to his book, Michael Lewis believes that Sam, Caroline, Gary and Nishad were motivated to earn a lot of money because they wanted to give it to charity.

Chapters 1 and 2 introduce the character of Sam, including the way he thinks about expected value and the way he taught himself to make facial expressions, and Chapter 3 properly introduces effective altruism.

One other oddly big thing happened to Sam at the beginning of his junior year. Completely out of the blue, a twenty-five-year-old lecturer in philosophy at Oxford University named Will Crouch [Will MacAskill’s unmarried name] reached out and asked to meet with him. Sam never learned how the guy had found him—probably from the writing Sam had been doing on various utilitarian mes

...
Fixing Insider Threats in the AI Supply Chain
by Madhav Malhotra — 2023-10-07T13:19:17.320Z

Summary

Insider threats are security risks caused by an organisation's staff. Careless and intentional insider threats cause 25+% of cyber breaches.[1]

In a survey of existential risks involving cybersecurity and AI, I identified key actors in the AI supply chain to protect: compute manufacturers, frontier AI labs, and AI regulatory agencies.

These actions can help key actors reduce insider threats:[2]

Individual factors:

Screen hires for “dark personality traits”

(self-promotion, emotional coldness, deceitfulness, aggressiveness, and more).

Monitor employees for personal stress

, financial issues, dissatisfaction, late/poor quality work, sudden wealth, bragging, suspicious office hours, and signs of substance abuse.

Train employees on personal cybersecurity

(securing smart home devices, managing logins, financial scams, etc.).

Organisational factors:

Create clear whistleblowing/reporting rules

. Specifically answer NDA and confidentiality concerns for junior staff.

Create a dedicated insider threat program

and manager. Ensure data sharing between it and cybersecurity/HR teams.

Lower outsourcing, remote work, & external tools

when possible.

Reduce hierarchies

. Ex: create consistent opportunities for junior staff to suggest ideas to senior leaders.

Technical factors:

Regularly simulate

social engineering

and

phishing attacks

.

Use the obvious tools: spam filters, firewalls and proxies, access management, network segmentation, etc.
Try monitoring tools for underused data (like HR or psychological data).[3]

Details on Individual Factors

It's difficult to identify demographics factors that correlate with insider threats.

It's possible to analyse public records and state conclusions like: "criminals who performed insider threats are more often men than women."
However, insider threats are rarely reported since companies fear reputational damage and most jurisdictions have no reporting requirements.[2] Thus, datasets likely have biases compared to the real world.

Sampling bias between different datasets and the real world. Icons by Freepik.

Next, there are correlations between "dark personality traits" and staff who intentionally cause insider threats.

Examples of dark personality traits are intuitive: "the desire for control, narcissism, ... egocentricity, a socially malevolent character, self-promotion, emotional coldness, [d

...
Contra Nora Belrose on Orthogonality Thesis Being Trivial
by tailcalled — 2023-10-07T11:47:02.401Z

Disagreeing with Eliezer Y., Nora Belrose calls the Orthogonality Thesis "trivial, false or meaningless/confused".

I think this is wrong. For instance, without the orthogonality thesis, one might think that in order to improve the world, one should charge ahead in creating an AGI, which would automatically use its superior intelligence to figure out the meaning of life and achieve it.

You might think this is stupid, but some significant people believe it. For example, some guy named E. Yudkowsky who used to run an organization for creating AGI wrote an argument like this, but thanks to Eliezer Y.'s reasoning about the orthogonality thesis and related topics, he eventually changed his mind. Clearly the orthogonality thesis is nontrivially useful for cases like this.
Related Discussion from Thomas Kwa's MIRI Research Experience
by Raemon — 2023-10-07T06:25:00.994Z

One of the subthreads in Thomas Kwa's MIRI research experience was about his experience working with Nate. In the comments, some other people brought up their own negative experiences. There was a lot of ensuing discussion about it.

Thomas felt this was distracting from the points he was most interested in (e.g. how infohazard policies slow down research, how new researchers can stop flailing around so much, whether deconfusion is a bottleneck to alignment, or the sharp left turn, etc).

I also somewhat regretted curating the post since we normally avoid curating "community politics" posts, and while the post had a lot of timeless content in both the OP and the discussion, it ended up being a major focus of the comments.

So, I'm moving those comments to this escape-valve-post, where the discussion can continue in whatever direction people end up taking while leaving the original post to focus on more timeless topics that are relevant whether or not you're plugged into particular social scenes.
Current State of Probabilistic Logic
by lunatic_at_large — 2023-10-07T05:06:22.955Z

So I've been thinking about how to assign probabilities to true/false assignments over claims in the context of a probabilistic argument mapping program.[1] Inevitably I've been confronted with the liar's paradox and a million related headaches. I have some tentative ideas on how I'd address these: basically allowing sentences in a language to access the probabilities of truth assignments but then replacing those probabilities with conditional probabilities to ensure existence and doing some entropy maximization stuff to hopefully get uniqueness. However, before I write this all up I want to check what the current state of probabilistic logic is on Lesswrong. When I search I mostly see stuff like http://intelligence.org/files/DefinabilityTruthDraft.pdf or more recently https://www.lesswrong.com/posts/KbCHcb8yyjAMFAAPJ/when-wishful-thinking-works. Are these kind of texts the current forefront of this topic that I should put my post in conversation with? If not, what is? Thanks!

^

I'm trying to follow up on the formalism I described in my first post while incorporating the suggestions in the comments.
On the Relationship Between Variability and the Evolutionary Outcomes of Systems in Nature
by Artyom Shaposhnikov — 2023-10-07T03:06:54.184Z

Recent advancements in large-scale neural networks suggest that machines may surpass human capabilities in most domains sooner than anticipated. A fierce competition is underway among companies, with billions of dollars invested in hardware. This race could result in one system significantly outperforming the rest, primarily due to resource limitations and the availability of top-tier engineers. The inherent ease of replicating digital intelligence might produce a dominant system that outpaces all others. Even if such a system aligns perfectly with the objectives of human civilization, I contend that it might be evolutionarily unstable. From simple artificial constructs to intricate organisms and societies, there are myriad instances where a singular, homogeneous system becomes unstable, losing both its long-term viability and its meaningful goals. I believe that this relationship between variability and long-term outcomes is universal in nature and, when combined with the rapid replication potential of digital intelligence, could provide some context for the Fermi paradox. I delve deeper into this concept in my LinkedIn article, which can be found here: https://lnkd.in/decdGWCE. I firmly believe this topic warrants broader discussion and attention.