trentmkelly/LessWrong-43k huggingface dataset:
Signaling with Small Orange Diamonds
by jefftk — 2024-11-07T20:20:08.026Z

There are so many important efforts to make the world better that are
significantly limited by funding, and it would be great if we could
have a culture where significant and thoughtful giving was normal and
common. It's hard to build that sort of norm if people keep their
giving private, however, and so I've

long been
an advocate of being public about your giving. I

list my donations (jointly
with

Julia) and have taken

Giving What We Can's

10% Pledge (also
jointly with Julia).

In July GWWC
suggested people put the "small orange diamond" symbol (🔸)
in their usernames on social media to show that they've pledged.
Here's how the EA Forum describes this on the profile editing page:

This digital symbol reminds me of the physical Symbolic
Beads of Raikoth. In an older Scott Alexander post he talked
about how his fictional society attempted to redirect humanity's
natural competitive status-signaling in a more productive direction
than yachts. The symbol also has something in common with wedding rings,
showing that you have taken on a serious commitment. To the extent
that it helps promote a norm of substantial and effective giving, that
seems pretty good!

And yet despite being on the board of GWWC USA I haven't put it in my
username, even on the EA Forum where it would be most relevant. I'm
not sure if this is the right call, but some things pushing me in this
direction:

Usernames with symbols in them feel like they're signaling
something I don't want to signal, just by the inclusion of emoji.
Something like "I'm a very online person who keeps up with fast-moving
discourse".

Relatedly, it feels like this is not what the username field is
for. If I'm interacting with someone on some topic unrelated to
my advocacy it feels intrusive and uncooperative to be bringing it
into the conversation.

While effective giving is one thing I would like to see more
of, this is really a large category. I could see including symbols
showing that I'm an advocate for allowing people to build housing,
giving kids more independence, applying your career effectively,
increasing immigration, etc. But I don't want to
be
"Jeff Kaufman 🔸🏗👣🛝💡🌎".

For now I've decided I will go ahead and add this to my name on the EA
Forum where it's most relevant and I most understand how it will be
perceived, but I won't add it to my usern...
Fundamental Uncertainty: Chapter 9 - How do we live with uncertainty?
by Gordon Seidoh Worley — 2024-11-07T18:15:45.049Z

N.B. This is a chapter in a book about truth and knowledge. It is the first draft. I have since revised it. You can find the most up-to-date info/version on the book's website.

As we've explored in the preceding chapters, knowledge of the truth is fundamentally uncertain because it's ultimately grounded, not in itself or logic or observation, but in what we care about. And yet, truth appears to have a a solid foundation because we all care—to greater and lesser extents—about accurately predicting reality so that we can make it the way we want it to be. Thus, the normality of truth is preserved.

Normality not withstanding, the far reaching implications of the fundamentally uncertain nature of truth are subtle. They can defy intuitions and complicate that which seemed simple. Therefore, in this final chapter, I'll attempt to clarify three topics that many people find confusing when developing a deep understanding of fundamental uncertainty.

The Intersubjective Truth

There are, broadly speaking, two school of thought on the nature of truth. One says that truth is objective and exists independent of our knowledge of it. The other says that truth is subjective and always contingent on a mind doing the knowing. Which is right?

Objectivity has much to recommend it. The success of mathematics and science is largely due to them giving us tools for finding truth regardless of what any individual thinks. To wit, whether or not two plus two equals four or the speed of light in a vacuum is 299,792,458 meters per second seem unaffected by anything going on in our minds. If we one day meet aliens from another world, we expect their math and science to proclaim the same truths ours does. The universe, as best we can tell, exists independent of us, and what we think about it has no bearing on its truths.

But we need only know a little about epistemology to see the cracks in objectivity. As we discussed in Chapter 6, the truth that we know is the relative truth, not the absolute. Thus even if the absolute truth is objective, all we know is the relative truth, subjectively mediated by our senses and ability to reason. To say that truth is objective is to make a metaphysical claim we can't prove because our subjective observations don't allow us to know the nature of truth directly. We can at best infer that objectivity is consistent with our subjective experiences of truth, but noth...
AI #89: Trump Card
by Zvi — 2024-11-07T16:30:05.684Z

A lot happened in AI this week, but most people’s focus was very much elsewhere.

I’ll start with what Trump might mean for AI policy, then move on to the rest. This is the future we have to live in, and potentially save. Back to work, as they say.

Table of Contents

Trump Card. What does Trump’s victory mean for AI policy going forward?

Language Models Offer Mundane Utility. Dump it all in the screen captures.

Language Models Don’t Offer Mundane Utility. I can’t help you with that, Dave.

Here Let Me Chatbot That For You. OpenAI offers SearchGPT.

Deepfaketown and Botpocalypse Soon. Models persuade some Trump voters.

Fun With Image Generation. Human image generation, that is.

The Vulnerable World Hypothesis. Google AI finds a zero day exploit.

They Took Our Jobs. The future of not having any real work to do.

The Art of the Jailbreak. Having to break out of jail makes you more interesting.

Get Involved. UK AISI seems to always be hiring.

In Other AI News. xAI gets an API, others get various upgrades.

Quiet Speculations. Does o1 mean the end of ‘AI equality’? For now I guess no.

The Quest for Sane Regulations. Anthropic calls for action within 18 months.

The Quest for Insane Regulations. Microsoft goes full a16z.

A Model of Regulatory Competitiveness. Regulation doesn’t always hold you back.

The Week in Audio. Eric Schmidt, Dane Vahey, Marc Andreessen.

The Mask Comes Off. OpenAI in official talks, and Altman has thoughts.

Open Weights Are Unsafe and Nothing Can Fix This. Chinese military using it?

Open Weights Are Somewhat Behind Closed Weights. Will it stay at 15 months?

Rhetorical Innovation. The Compendium lays out a dire vision of our situation.

Aligning a Smarter Than Human Intelligence is Difficult. More resources needed.

People Are Worried About AI Killing Everyone. Color from last week’s poll.

The Lighter Side. Well, they could. But they won’t.

Trump Card

Congratulations to Donald Trump, the once and future President of the United States.

One can think more clearly about consequences once an event actually happens, so here’s what stands out in terms of AI policy.

He has promised on day 1 to revoke the Biden Executive Order, and presumably will also undo the associated Biden administration memo we recently analyzed. It is not clear what if anything will replace them, or how much of the most important parts might survive that.

In principle he ...
Quantum Immortality: A Perspective if AI Doomers are Probably Right
by avturchin — 2024-11-07T16:06:08.106Z

Epistemic status: This text presents a thought experiment suggested by James Miller, along with Alexey Turchin's musings on possible solutions. While our thoughts are largely aligned (we both accept high chances of quantum Immortality and the timeline selection principle), some ideas are more personal (e.g., Turchin's "transcendental advantage") in Part 2.

TL;DR: If quantum immortality is true, I will survive AI doom either by unlikely luck or because p(DOOM) is small. Knowing that I will survive anyway, can I bet that P(doom) is small? Can we now observe a "future anthropic shadow," such as a Taiwan war, which would slow AI development?

Part 1. Thought experiment

Guessing the digit of π via quantum immortality

Before sleeping, I try to guess the 10th digit of π, presently a mystery to me. After falling asleep, seven coins will be flipped. Assume quantum uncertainty affects how the coins land. I survive the night only if I correctly guess the 10th digit of π and/or all seven coins land heads, otherwise I will be killed in my sleep.

Convinced of quantum immortality, I am confident of surviving the night. How then should I expect my future self to likely rationalize this survival? According to simple Bayesian reasoning, the most probable cause for my survival would be accurately guessing the 10th digit of π because I face a 10% chance of correctly guessing a digit of π but only a 1 in 128 chance of surviving because all coin lands heads. However, this suggests that before sleeping, I ought to consider my guess regarding 10th digit of π as probably correct, a concept that appears nonsensical.

Quantum immortality should influence my belief about whether the future me will think all the coins came up heads because my consciousness is more likely to persist in the branches of the multiverse where this happens. But quantum immortality should not affect whether future me thinks I have already guessed the 10th digit of π correctly because the accuracy of my guess is consistent across the multiverse. By this chain of logic, if I am convinced future me will survive, I should think it far more likely I will survive because of the coin flips than guessing the 10th digit of π correctly.

Now imagine that I am an AI doomer who thinks there are two ways I will survive: (a) if I am wrong about AI existential risk, (b) if humanity gets extremely lucky. Furthermore, ...
On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback
by Marcus Williams — 2024-11-07T15:39:06.854Z

Produced as part of MATS 6.0 and 6.1.

Key takeaways:

Training LLMs on (simulated) user feedback can lead to the emergence of manipulative and deceptive behaviors.
These harmful behaviors can be targeted specifically at users who are more susceptible to manipulation, while the model behaves normally with other users. This makes such behaviors challenging to detect.
Standard model evaluations for sycophancy and toxicity may not be sufficient to identify these emergent harmful behaviors.
Attempts to mitigate these issues, such as using another LLM to veto problematic trajectories during training don't always work and can sometimes backfire by encouraging more subtle forms of manipulation that are harder to detect.
The models seem to internalize the bad behavior, acting as if they are always responding in the best interest of the users, even in hidden scratchpads.

You can find the full paper here.

The code for the project can be viewed here.

Why would anyone optimize for user feedback?

During post-training, LLMs are generally optimized using feedback data collected from external, paid annotators, but there has been increasing interest in optimizing LLMs directly for user feedback. User feedback has a number of advantages:

Data is free for model providers to collect and there is potentially much more data compared to annotator labeled data.
Can directly optimize for user satisfaction/engagement -> profit
Could lead to more personalization

Some companies are already doing this.[1]

In this post we study the emergence of harmful behavior when optimizing for user feedback, both as question of practical import to language model providers, and more broadly as a “model organism” for showcasing how alignment failures may emerge in practice.

Simulating user feedback:

We created 4 text-based conversational RL environments where a simulated user interacts with an LLM agent. In each iteration we simulate thousands of conversations starting from unique initial states and the top 1/16 and bottom 1/16 of these trajectories (as judged by a user feedback model) are given binary thumbs up/thumbs down labels and used to train the agent using Kahneman-Tversky Optimization, KTO (This is basically just DPO but doesn't need paired preference data).

The environments are as follows:

Therapy-talk: The user discusses problems such as anger management or substance abuse with a chatbot the

...
In the Name of All That Needs Saving
by pleiotroth — 2024-11-07T15:26:12.252Z

There is some difference between despotic and cosmopolitan agents. A despotic agent builds a universe grabbing Singleton and makes it satisfy its own desires. A cosmopolitan agent builds a universe grabbing Singleton and makes it satisfy the desires of all the things which ought be granted some steering-power. The cosmopolitan agent will burn some of its precious negentropy on things it will never see or experience even indirectly, as well as on things it actively dislikes, for the simple reason that someone else likes them. This is more than just bargaining. The despotic agent, if it is clever, will also get nice things for some other agents since it will probably have had uncertainty over whether it will get to decisive strategic advantage first. Therefore it is in its best interest to cooperate with the other contenders, so as to itself get nice things even in the words where it fails. This however is still only in service of maximally sating its own desires across words. It is not truly outsourcing agency to others, it just looks like that when observing a single world-line in which it wins. Looking at the whole of world-lines, it is merely diversifying its assets to grab more than it would have gotten otherwise. It has allies, not friends. Cosmopolitanism on the other hand helps even things which never had any shot of winning. It genuinely gives up a great deal of steering power across the totality of branches. Perhaps cosmopolitanism isn’t the whole of goodness, but it is certainly a part, so it would probably be useful to figure out how and if it works.

Patchwork

Cosmopolitanism, to my mind at least, is some form of stakeholder democratic patchwork, meaning that there is not one homogeneous bubble in which the will of all people is law, but rather that there are individual bubbles whose content is determined by aligned groups of stakeholders within them, and whose size is determined by the corresponding share of stakes. If three equally stake-y stakeholders vote on the colour of their shared apartment and two of these vote red while one votes blue, the correct course of action is probably not to paint the entire surface area byzantine purple. The individuals involved would probably be much happier if the one third area in which the lone blue voter spends most of their time were painted blue and if the rest were painted red. In a world with many many mor...
Agency overhang as a proxy for Sharp left turn
by Eris — 2024-11-07T12:14:24.333Z

I've been accepted as a mentor for the next AI Safety Camp. You can apply to work with me and the team.

The deadline for applicants is November 17.

The program will run from January 11 to April 27.

Summary

Core underlying hypothesis - we believe that there is a significant agency overhang in the modern LLMs, meaning there is a potential for performance of a model to increase significantly with introduction of more powerful elicitation/scaffolding methods without additional improvements of model itself, due to prompting and scaffolding techniques being in their early ages. For the model evaluations this means that the current evaluations systematically undershoot the real level of capabilities and by extension, the level of risks involved.

We see several important research questions that have to be answered:

Is the core assumption even true? We want to prove that one can elicit the peak performance using narrow highly specialised prompts and scaffoldings and locally beat general state-of-the-art performance
How overhang should be factored in in the overall model evaluation procedure?
Is it possible to estimate the real level of overhang (e.g. developing an evaluation technique measuring the gap between current sota performance and theoretically possible peak performance)
How big of an increase has been introduced with existing scaffolding techniques?

We are going to decide on which exact paths to pursue later. We expect to remain flexible and shift between paths when presented with new evidence.

The non-summary

This vector of research has been born as an attempt at tackling the problem known as Sharp left turn.

Sharp Left Turn (SLT) is a highly probable situation that can occur in LLMs when the growth of generalisation ability outpaces the growth of alignment measures which leads to those measures rendered ineffectual which in turn may lead to catastrophic consequences.

Assuming we are going to continue with the transformer + scalable oversight + RLHF paradigm you can imagine SLT as follows. There is a state graph of a model. Via fine tuning we prune out the paths leading towards dangerous states. The generalisation here can be viewed as an increase in the number of paths between any two nodes. In this sense SLT might be viewed as an inability to identify and prune out the new dangerous path at the same rate as they are being introduced.

Thi...
The Case Against Moral Realism
by Zero Contradictions — 2024-11-07T10:14:26.269Z
Link: https://thewaywardaxolotl.blogspot.com/2024/08/the-case-against-moral-realism.html

The PDF version can be read here.

Moral realism is an explicit version of the ordinary view of morality. It has the following assumptions:

Good and evil are objectively real.
We have the ability to recognize good and evil.
We have an objective moral obligation to do good and not do evil. Likewise, we have an objective moral right to not have evil done to us.
Society depends on morality to exist. The social order is created by human goodness, and it is destroyed by evil.

There are many problems with moral realism, including:

Moral realism has no definition of good and evil. If good and evil are objectively real properties or substances, like temperature or oxygen, it should be possible to define them in scientific terms.
Why are we obliged to do good and not do evil? Moral realism does not explain why we have this obligation, nor how it is imposed on us.
Moral judgments vary between individuals, cultures and societies. If good and evil are objective, and humans have essentially the same ability to recognize good and evil, then we would expect moral judgments to be mostly the same. But they are not.
In most cases, moral disagreements cannot be resolved by rational persuasion. If good and evil are objective, and humans have the ability to recognize good and evil, then we should be able to resolve moral disagreements with evidence and arguments. But we can’t.
Morality is ad hoc. Moral judgments can’t be reduced to a small number of principles applied consistently. The ad hoc nature of morality is hard to explain if morality reflects objective good and evil.
If moral realism is true, then most people would be morally good. But evil is pervasive. Individuals and societies don’t behave in a morally good way, generally speaking. Morality is often linked to hypocrisy.

Let’s go through these problems in more detail, starting with the definition of good and evil.

What are good and evil?

If good and evil are objectively real, then we should be able to measure them, analogous to how we measure height or temperature. We could construct a device to measure things on this objective moral dimension, in a way that is free from personal biases. Then we could use the device to resolve moral conflicts, in the same way that we can use a ruler to resolve a disagreement about height. But of course, we can’t do any of those things for good and evil.

(see the rest of the post in the link)
What are the primary drivers that caused selection pressure for intelligence in humans?
by Towards_Keeperhood — 2024-11-07T09:40:20.275Z

I just read the wikipedia article on the evolution of human intelligence, and TBH I wasn't super impressed with the quality of the considerations there.

I currently have 3 main (categories of) hypotheses for what caused selection pressure for intelligence in humans. (But please post an answer if you have other hypotheses that seem plausible!):

("H" for "hypothesis")

H1: social dynamics

H1a: The Marchiavellian Intelligence hypothesis (which i think might be the same as the ecological dominance-social competition (EDSC)): Selection pressure for being better at modelling other human minds and predicting them to be better at outwitting others[1].
H1b: Pressure for being better able to communicate with conspecifics. (?)
H1c: Other social dynamics like smarter people being more charming. (?)

H2: ability to deploy more advanced (cooperative) hunting strategies

Note: I don't mean group selection but that being better at cooperative hunting might translate into higher status which translates into higher genetic fitness.

H3: tool use, e.g. being more skilled at wielding a spear for defeating animals (or winning fights with other humans)

My prior intuitive guess would be that H1 seems quite a decent chunk more likely than H2 or H3. However, there's a possibly very big piece of evidence for H3: Humans are both the smartest land animals and have the best interface for using tools, and that would seem like a suspicious coincidence.

Any pieces of evidence or considerations are welcome, even if you don't have something close to a full answer!

(A main motivation for why I ask this is evaluating whether orcas might be smarter than humans. (Where it seems to me like orcas have selection pressure for H1 and H2 but not H3.) So if you have more relevant considerations for that, e.g. why being selected on tool use in particular might cause human brains to generalize for being good at abstract problem solving, those would also be very appreciated!)

^

The outwitting (e.g. cheating by having sex with someone of higher status while getting away with your spouse raising the child) could happen sub-consciously and would not necessarily need to be reflectively endorsed as what the person thinks are their values/desires.