trentmkelly/LessWrong-43k huggingface dataset:
Towards Quantitative AI Risk Management
by Henry Papadatos — 2024-10-16T19:26:48.817Z

Reading guidelines: If you are short on time, just read the section “The importance of quantitative risk tolerance & how to turn it into actionable signals”

Tl;dr: We have recently published an AI risk management framework. This framework draws from both existing risk management approaches and AI risk management practices. We then adapted it into a rating system with quantitative and well-defined criteria to assess AI developers' implementation of adequate AI risk management. What comes out of it is that all companies are still far from "strong AI risk management" (grade >4 out of 5). You can have a look at our website to see our results presented in an accessible manner:

https://ratings.safer-ai.org/

Motivation

The literature on risk management is very mature and has been refined by a range of industries for decades. However, OpenAI's

Preparedness Framework

, Google Deepmind's

Frontier Safety Framework

, and Anthropic's

Responsible Scaling Policy

do not reference explicitly the risk management literature.

By analyzing those more in-depth, we have identified several deficiencies: the absence of a defined risk tolerance, the lack of semi-quantitative or quantitative risk assessment, and the omission of a systematic risk identification process. We propose a risk management framework to fix these deficiencies.

AI risk management framework

Risk management dimensions

Our framework is centered around 3 main dimensions.

Risk identification: Here, we want to have all the relevant risk scenarios properly identified. AI developers should draw from the existing literature and engage in open-ended red teaming to uncover potential new threats. Then, using that as input, they should use risk identification techniques and risk modeling to produce risk scenarios that enable the likelihood and severity of risks to be estimated.
Risk tolerance and analysis: In this part, AI developers should establish a well-defined risk tolerance, in the form of risk thresholds, which precisely characterize acceptable risk levels. Once the risk tolerance is established, it must be operationalized by setting the corresponding: i. AI capability thresholds and ii. mitigation objectives necessary to maintain risks below acceptable levels. The risk tolerance operationalization should be grounded in extensive threat modeling to justify why the mitigatio

...
Why Academia is Mostly Not Truth-Seeking
by Zero Contradictions — 2024-10-16T19:14:27.923Z
Link: https://thewaywardaxolotl.blogspot.com/2019/05/why-most-academic-research-is-fake.html

The PDF version can be read here.

In this essay, I will argue that most academic research is fake. The modern academy is not a reliable source of knowledge. Instead, it produces the pretense of knowledge.

Academic research can be fake in different ways. It can simply be false. It can be emotionally manipulative propaganda masquerading as knowledge. It can be irrelevant or meaningless.

These are the main causes of fake research:

Ideological Bias
Perverse Incentives
Social Circularity
Naive/Fake Empiricism

I’ll describe each and explain how it causes fake research.

(see the rest of the post in the link)
Launching Adjacent News
by Lucas Kohorst — 2024-10-16T17:58:10.289Z

Forward Looking News Driven by Prediction Markets and Forecasting Platforms

Prediction markets and forecasting platforms are an incredible news source, are interesting to trade, and provide tons of unique data:

News: Driving news by changes in market odds and providing context to news you read
Trade: Put on a position directly on an event, not something that is related
Data: How have similar past events played out, how good is an individual/platform at predicting/forecasting a specific category of events

Adjacent News is a new-age media venture that is focused on using prediction markets and forecasting platforms to add context to the news you already read, where you read it. Generate news that is driven by changes in the markets and provide trading and data platforms.

Verticals

News

We have two categories: Published and In Context News.

Published News

- articles that we produce and distribute, this is mostly through generation of content based on shifts in odds across various markets but also features original content from various writers and forecasters. Content is distributed via Adjacent News, RSS, X / Twitter, and via a Telegram Channel

We are exploring other methods of distribution like Apple News, Google News, Farcaster, etc. If you have any ideas for other distribution methods email us at lucas@adjacentresearch.xyz

In Context News

- We meet you were you consume news and provide contextual information about related markets. This comes in the form of Chrome Extension and an RSS Reader.

Trade

Many prediction market and forecasting platforms have similar markets, if they are "real money" there are potential arbitrages. Our TradingView interface at trade.adj.news lets you chart any market from multiple platforms over each other and analyze it with standard trading and charting tools.

Data

It can be hard to find historical data on markets, view related markets intra and cross platform, and see relevant news to the markets you are trading and/or forecasting.

data.adj.news supports historical, related, trending, and news related to markets across top platforms with more being added soon.

Products

Chrome Extension

Markets directly in your Twitter / X feed and popular news sources like NYT, WSJ, Reuters, etc. below are some examples

Fed Rate Cut on X
Google ruled a Monoply article on Semafor
Providing context to the Election on X

Market odds provided in contex

...
Interest in Leetcode, but for Rationality?
by Gregory  — 2024-10-16T17:54:25.578Z

the following is motivated by:

Self-Improvement or Shiny Distraction: Why Less Wrong is anti-Instrumental Rationality
The Martial Art of Rationality

I've been a long time lurker on Less Wrong and I've noticed the recurring criticism that despite its focus on rationality, the community lacks structured training to develop practical rationality skills. Eliezer Yudkowsky talks rationality as a martial art, because it's something that can be trained and refined through deliberate practice. But where is our dojo?

A model that comes to mind is a website like LeetCode, where programmers can solve coding challenges, share solutions, and see how others approach the same problems. LeetCode can sometimes encourage overfitting to specific problem types so it's not a perfect analogy. The community driven aspect would interesting to me as you can see how other people approach the problem. Could something similar be adapted for rationality?

Imagine a platform where, instead of solving coding puzzles, users engage with problems designed to train rational thinking. Here are a few types of problems that might fit:

Cognitive Bias Detection: Users could review novel, real-world scenarios and try to identify what cognitive bias or logical fallacy is present. The goal would be to train pattern recognition for biases without simply memorizing common examples. For instance, a scenario might subtly include a case of confirmation bias or anchoring, and users would need to spot it.
Calibration Training: One of the most important skills in rationality is aligning your confidence with reality. For each problem or scenario, users could submit a confidence interval along with their answer. This serves as a double-training: users practice assessing their certainty, and over time, they get feedback on how well-calibrated they are.
Bite-Sized, Practical Challenges: The focus should be on small, actionable exercises rather than lengthy theoretical discussions. For example, a problem might ask users to predict an outcome based on limited data, forcing them to confront the common pitfalls of overconfidence or representativeness heuristics.

This kind of platform could be a place where people practice and refine their skills, not just absorb entertaining ideas in way that some say is weakly applicable.

"identify the bias" type problem for a prototype i'm working on

I have a few years of experience in ...
Request for advice: Research for Conversational Game Theory for LLMs
by Rome Viharo — 2024-10-16T17:53:30.243Z

Greetings and thank you for your attention. I am an independent researcher and creator of Conversational Game Theory, a novel computational, cognitive, and psychological consensus building game for human to human, AI to human, and AI to AI collaboration. Its notable feature is its ability to reach consensus without voting, simply through conversation, something not previously thought possible. Conflict resolution is a mechanism design of the system itself, and the only possible outcome is a "win-win" of some kind.

This year we engineered and piloted our computational system, and recently we were able to train AI agents on different perspectives to play CGT, build consensus, publish a collaborative article and demonstrated that AI agents trained on GPT score much higher benchmark testing scores that GPT without.

This gives us a capability to create a large Global Library of Consensus Articles, train AI agents on any possible perspective in any possible conflict in the world, publish consensus resolution articles to those conflicts, filter bad faith actors and disinformation/misinformation, and allow humans to join in at anytime.

This global library would serve a purpose as a training ground for LLMs. CGT brings remarkable alignment properties as collaboration is deeply embedded into the mechanism design.

We were not expecting such a profound capability to emerge so quickly, so we are readjusting our focus and seeking advice, advisors, perhaps even some co-founders. We want to continue to roll out research and testing on this capability and any advice anyone has is deeply appreciated.
Against empathy-by-default
by Steven Byrnes — 2024-10-16T16:38:49.926Z

tl;dr

Section 1 presents an argument that I’ve heard from a couple people, that says that empathy[1] happens “for free” as a side-effect of the general architecture of mammalian brains, basically because we tend to have similar feelings about similar situations, and “me being happy” is a kinda similar situation to “someone else being happy”, and thus if I find the former motivating then I’ll tend to find the latter motivating too, other things equal.

Section 2 argues that those two situations really aren’t that similar in the grand scheme of things, and that our brains are very much capable of assigning entirely different feelings to pairs of situations even when those situations have some similarities. This happens all the time, and I illustrate my point via the everyday example of having different opinions about tofu versus feta.

Section 3 acknowledges a couple kernels of truth in the Section 1 story, just to be clear about what I’m agreeing and disagreeing with.

1. What am I arguing against?

Basically, the proposal (as I understand it) is that things-involving-me and corresponding things-involving-other-people wind up close to each other in the latent space, and thus the “learnt reward model”, being smooth and continuous by default, assigns values that spill over from the former to the latter.

Here’s Beren Millidge (@beren),

“Empathy as a natural consequence of learnt reward models” (2023)

:

…Here, I want to argue a different case. Namely that the basic cognitive phenomenon of empathy -- that of feeling and responding to the emotions of others as if they were your own, is not a special cognitive ability which had to be evolved for its social benefit, but instead is a natural consequence of our (mammalian) cognitive architecture and therefore arises by default. Of course, given this base empathic capability, evolution can expand, develop, and contextualize our natural empathic responses to improve fitness. In many cases, however, evolution actually reduces our native empathic capacity -- for instance, we can contextualize our natural empathy to exclude outgroup members and rivals.

The idea is that empathy fundamentally arises from using learnt reward models[2] to mediate between a low-dimensional set of primary rewards and reinforcers and the high dimensional latent state of an unsupervised world model. In the brain, much of the cortex i

...
cancer rates after gene therapy
by bhauth — 2024-10-16T15:32:53.949Z
Link: https://bhauth.com/blog/biology/gene%20therapy.html

Here's a new paper on cancer developing in patients after receiving gene therapy with eli-cel, brand name Skysona.

What is eli-cel?

Stem cells are isolated and modified by using a lentiviral vector to add the ABCD1 gene. The stem cells in bone marrow are killed with chemotherapy, and replaced with the modified stem cells.

How often did cancer develop?

Hematologic cancer developed in 7 of 67 patients after the receipt of eli-cel

Yep, that's about what I expected. How long did it take?

myelodysplastic syndrome (MDS) with unilineage dysplasia in 2 patients at 14 and 26 months; MDS with excess blasts in 3 patients at 28, 42, and 92 months; MDS in 1 patient at 36 months; and acute myeloid leukemia (AML) in 1 patient at 57 months.

Makes sense. What does genetic analysis of the cancer cells indicate?

In the 6 patients with available data, predominant clones contained lentiviral vector insertions at multiple loci, including at either MECOM–EVI1 (MDS and EVI1 complex protein EVI1 [ecotropic virus integration site 1], in 5 patients) or PRDM16 (positive regulatory domain zinc finger protein 16, in 1 patient). Several patients had cytopenias, and most had vector insertions in multiple genes within the same clone; 6 of the 7 patients also had somatic mutations (KRAS, NRAS, WT1, CDKN2A or CDKN2B, or RUNX1), and 1 of the 7 patients had monosomy 7.

Multiple off-target insertions, different ones for different people. Yep.

Has this happened with gene therapy attempts before?

Yes, it's been a big problem with eg γ-retrovirus gene therapy trials. People hoped lentiviruses would be better because they tend to insert around actively transcribed genes, while γ-retroviruses tend to insert near transcription start sites.

Why don't we see high cancer rates after DNA viral infections?

We sort of do: Many cancers are strongly associated with viruses. But eli-cel is:

specifically done to stem cells, which are closer to cancer than usual

applied to cells away from the immune system, which can often detect infections and kill infected cells early

How about CRISPR instead of a viral vector?

Casgevy is a treatment that uses CRISPR on isolated cells. It has serious side effects, but not as bad as eli-cel. But a lot of people misunderstand what CRISPR actually does.

CRISPR is a technique that uses the Cas9 enzyme to cut double-stranded DNA according to added guide RNA. It doesn't insert DNA i...
Monthly Roundup #23: October 2024
by Zvi — 2024-10-16T13:50:05.869Z

It’s monthly roundup time again, and it’s happily election-free.

Thinking About the Roman Empire’s Approval Rating

Propaganda works, ancient empires edition. This includes the Roman Republic being less popular than the Roman Empire and people approving of Sparta, whereas Persia and Carthage get left behind. They’re no FDA.

Polling USA: Net Favorable Opinion Of:

Ancient Athens: +44%

Roman Empire: +30%

Ancient Sparta: +23%

Roman Republican: +26%

Carthage: +13%

Holy Roman Empire: +7%

Persian Empire: +1%

Visigoths: -7%

Huns: -29%

YouGov / June 6, 2024 / n=2205

The Five Star Problem

What do we do about all 5-star ratings collapsing the way Peter describes here?

Peter Wildeford: TBH I am pretty annoyed that when I rate stuff the options are:

* “5 stars – everything was good enough I guess”

* “4 stars – there was a serious problem”

* “1-3 stars – I almost died”

I can’t express things going well!

I’d prefer something like:

* 5 stars – this went above/beyond, top 10%

* 4 stars – this met my expectations

* 3 stars – this was below my expectations but not terrible

* 2 stars – there was a serious problem

* 1 star – I almost died

Kitten: The rating economy for things like Airbnb, Uber etc. made a huge mistake when they used the five-star scale. You’ve got boomers all over the country who think that four stars means something was really good, when in fact it means there was something very wrong with the experience.

Driver got lost for 20 minutes and almost rear ended someone, four stars

Boomer reviewing their Airbnb:

This is one of the nicest places I have ever stayed, the decor could use a little updating, four stars.

A lot of people saying the boomers are right but not one of you mfers would even consider booking an Airbnb with a 3.5 rating because you know as well as I do that means there’s something really wrong with it.

Nobe: On Etsy you lose your “star seller” rating if it dips below 4.8. A couple of times I’ve gotten 4 stars and I’ve been beside myself wondering what I did wrong even when the comment is like “I love it, I’ll cherish it forever”

Moshe Yudkowsky: The first time I took an Uber, and rated a driver 3 (average), Uber wanted to know what was wrong. They corrupted their own metric.

Kate Kinard: I’m at an airbnb right now and this magnet is on the fridge as a reminder

= many issues to fix!

The problem is actually worse than this. Different people ha...
Change My Mind: Thirders in "Sleeping Beauty" are Just Doing Epistemology Wrong
by DragonGod — 2024-10-16T10:20:22.133Z

Preamble

Sleeping Beauty volunteers to undergo the following experiment and is told all of the following details: On Sunday she will be put to sleep. Once or twice, during the experiment, Sleeping Beauty will be awakened, interviewed, and put back to sleep with an amnesia-inducing drug that makes her forget that awakening. A fair coin will be tossed to determine which experimental procedure to undertake:

If the coin comes up heads, Sleeping Beauty will be awakened and interviewed on Monday only.

If the coin comes up tails, she will be awakened and interviewed on Monday and Tuesday.

In either case, she will be awakened on Wednesday without interview and the experiment ends.

Any time Sleeping Beauty is awakened and interviewed she will not be able to tell which day it is or whether she has been awakened before. During the interview Sleeping Beauty is asked: "What is your credence now for the proposition that the coin landed heads?"

Motivation

I was recently introduced to the canonical Sleeping Beauty problem and initially was a halfer but confused. Or more like I thought the halfer position was correct, but smart people seemed to be thirders and I was worried I was misunderstanding something about the problem, or confused myself or similar.

I debated the problem extensively on the LW Discord server and with some LLMs and strongly updated towards "thirders are just engaging in gross epistemic malpractice".

A message I sent in the LW server:

Alternatively I started out confused.

Debating this problem here and with LLMs convinced me that I'm not confused and the thirders are actually just doing epistemological nonsense.

It feels arrogant, but it's not a poor reflection of my epistemic state?

I still have some meta level uncertainty re: the nonsense allegations.

I want to be convinced that the thirder position is not nonsense and there is a legitimate disagreement/debate to be had here.

I have read some of the LW posts on the canonical problem here. I won't be linking them due to laziness.

I have not yet read Joe Carlsmith's blog posts or Nick Bostrom's book as at posting this question. I'll probably be listening to them after posting the question.

I asked Sonnet 3.5 to distill my position/rejections from our debate and below is its summary[1]

Comprehensive Position on the Sleeping Beauty Problem

1. Core Position

The correct answer to the Sleeping Beauty problem i

...
After uploading your consciousness...
by Jinge Wang — 2024-10-16T03:52:13.537Z

Imagine in the future, there is a technique that can upload your consciousness to a new container, either a biological body or digital space. Once we think of this scenario, we always feel that we are switched to a new body with our mind unchanged. I don't know the best possible way to implement such a technique, but I assume that uploading consciousness is copying some information from our brain to a target container. If that's true, the uploading is actually cloning, not moving.

When the cloning procedure is finished, the 2 instances should have the same memory and thought at that time but diverge afterward. My question is, what we will feel at that time?

To make it dramatic, assume that the original body is about to be over due to some diseases. Can we escape death by such kind of consciousness uploading?
Singular Learning Theory for Dummies
by Rahul Chand — 2024-10-15T21:13:55.842Z

In this post, I will cover Jesse Hoogland's work on Singular Learning Theory. The post is mostly meant as a dummies guide, and therefore won't be adding anything meaningfully new to his work. As a helpful guide at each point I try to mark the math difficulty of each section, some of it is trivial (I), some of it can be with followed with enough effort (II) and some of it out of our(my) scope and we blindly believe them to be true (III).

Background

Statistical learning theory is lying to you: "overparametrized" models actually aren't overparametrized, and generalization is not just a question of broad basins

Singular Learning Theory (SLT) tries to explain how neural networks generalize. One motivation for this is that the previous understanding of neural network generalization isn't quite correct. So what is the previous theory and why is it wrong?

Broad Basins

Below are excerpts from "Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes"[1] that summarize the logic behind broad basins theory of generalization

According to SLT there are two issues here. First, finding volume of a basin in the loss landscape by approximating it via hessian isn't accurate and second, the reason why models generalize has more to do with their symmetries than with with the fact that intialization drops the NNs in a good "broad basin" area.

Starting with SLT

A lot of basic maths for SLT is already covered in the original blog post. Here I will try to go over small questions that one might have (atleast i had) to get a clearer picture

What we know so far? (Math difficulty I)

Screenshot from the blog

We define the truth, model and prior. The model is just the likelihood. These are standard bayesian terms.
Based on the above we know how to formalize the posterior & model evidence, p(w|D_n) & p(D_n) in terms of the likelihood and prior. Usually we stop at the posterior part, we are happy to get the p(w|D_n) for the model, but here we are interested in p(D_n) as well.

Why are we interested in p(Dn) isn't our aim just to get the w that maximizes p(w|Dn) ?

We are interested in p(Dn) since it gives us the ability to not only tell what is the best w but the best architechure itself. So e.g. if you have some data D and you want the best w, and you are not sure what layer of neural network to chose, if we could magical

...