trentmkelly/LessWrong-43k huggingface dataset:
Apply to Usable Security Prize by September 30
by Allison Duettmann — 2023-09-29T13:39:59.773Z

The submission deadline for Foresight's Norm Hardy Prize for Usable Computer Security is fast approaching, i.e. the prize closes at midnight PT on September 30.

The Norm Hardy Prize, sponsored by Agoric and Protocol Labs, is an annual award offering a $10,000 prize to recognize work that enhances usability within computer security.

We are looking for submissions that:

Help users understand the security aspects of their actions, ideally in an intuitive manner.
Introduce workflows that make secure actions the easiest option.
Develop design principles for systems that are as, if not more, user-friendly because of their security.
Explore 'theory of mind’ with respect to how users interact with secure systems.

The long-term goal of this prize is to inspire and recognize work that make it easy for people to use secure systems securely. If your work aligns with these goals, I very much encourage you to apply.

More information about the prize, as well as the submission process, can be found at https://foresight.org/norm-hardy-prize

Please dm me if you have any questions or email allison@foresight.org
List of how people have become more hard-working
by Chi Nguyen — 2023-09-29T11:30:38.802Z

I've recently asked how people have become more hard-working. I compiled the answers across the EA Forum and LessWrong (and some private messages) in a list for myself to make it easier for me to experiment with the suggestions. I thought I'd share the list here in case it's useful for anyone else. I also list the things that people said didn't work and a couple of other things.

This wasn't done to be "proper", so the list is sloppy in many ways: I liberally paraphrased what people said; often I could have easily counted something two people said as the same or two different things, which would change the way I counted how often something was mentioned; I very roughly grouped the things that were said into categories but easily could have categorised many things differently.

Notable points

Individual points that were mentioned the most:

(Soft) accountability (deadlines, beeminder, accountability buddy, posting about your goals, boss as a service, promising friends) (9)
Working on interesting problems/enjoyable work (and in an enjoyable work environment) (8)
Focusmate/Coworking (often poms) (7)

Some things that weren't mentioned a lot but that I found interesting:

Identifying (or being thought of) as hard-working (3)
Categorising work as "not work" and instead as something enjoyable, adjusting work environment accordingly (1)

Other things

Age at the time of the shift in hard-workingnesswas usually not mentioned, but when it was mentioned, it was between 20-30
Some people managed to become permanently more hard-working after experiencing one period of working hard, even when they switched to less enjoyable or just very different work. That initial period would either be induced by external pressure or by working hard on something they didn't consider work. (3)

Full list of what made people more hard-working

Here is the full list, ordered by how often things in a category were named. (Note that often the same person would list multiple things in the category, so the sums aren't summing over people)

Thing that worked	How many people mentioned
Social
Focusmate/Coworking (often poms)	8
(Regular) contact with other people to talk about work, debug, check-in etc.	4
Identifying (or being thought of) as hard-working	3
Surrounding yourself with ~hard-working people in life in general	3
Supportive work environment	2
Having a manager	1
21
What kind of work
Working on interesting

...
Resolving moral uncertainty with randomization
by B Jacobs — 2023-09-29T11:23:42.175Z

This is a crosspost from the EA forum

Problems with maximizing expected choice-worthiness

A popular approach to moral uncertainty is "maximizing expected choice-worthiness" (MEC) by William MacAskill. However, it has some issues.

It requires theories to be interval-scale measurable.
This means that a theory needs to provide information about the ratio of difference in choice-worthiness between different options.
E.g. Stoicism says that the difference between lying and killing is ten times bigger than the difference between being honest and lying.
It requires intertheoretic comparisons of value.
We need to be able to tell to what extend one theory deems an option as more important than another option, when compared to a different theory.
E.g. Utilitarianism says the moral difference between lying and killing is three times bigger than Kantianism says it is.
It falls prey to the infectiousness of nihilism.
When an agent has positive credence in nihilism then the choice-worthiness of all actions is undefined.
E.g. If you think nihilism has 1% chance of being true, you can't evaluate one option as morally better than another.
Some say it has a problem with fanaticism.
It ranks a minuscule probability of an arbitrarily large value above a guaranteed modest amount of value.
E.g. If you think christianity has a 1% chance of being true, and it gives people infinite happiness in heaven, you should choose it above your 99% chance that utilitarianism is true, which only posits finite happiness.

Two years ago one of us created an approach that wouldn't fall prey to these problems: the sortition model of moral uncertainty.

Problems with the sortition model of moral uncertainty

The sortition model prescribes that if you have x% credence in a theory, then you follow that theory x% of cases. If you have 20% credence in Kantianism, 30% credence in virtue ethics and 50% credence in utilitarianism, you follow Kantianism 20% of the time, virtue ethics 30% of the time and utilitarianism 50% of the time. When you act according to a theory is selected randomly with the probability of selection being the same as your credence in said theory.

This approach doesn't fall prey to the problems of interval-scale measurability, intertheoretic comparisons of value, or the infectiousness of nihilism. Furthermore, the theory is fair, computationally cheap and doesn't generate problems with fanaticism an...
EA Vegan Advocacy is not truthseeking, and it’s everyone’s problem
by Elizabeth — 2023-09-28T23:30:03.390Z
Link: https://acesounderglass.com/2023/09/28/ea-vegan-advocacy-is-not-truthseeking-and-its-everyones-problem/

Introduction

Effective altruism prides itself on truthseeking. That pride is justified in the sense that EA is better at truthseeking than most members of its reference category, and unjustified in that it is far from meeting its own standards. We’ve already seen dire consequences of the inability to detect bad actors who deflect investigation into potential problems, but by its nature you can never be sure you’ve found all the damage done by epistemic obfuscation because the point is to be self-cloaking.

My concern here is for the underlying dynamics of EA’s weak epistemic immune system, not any one instance. But we can’t analyze the problem without real examples, so individual instances need to be talked about. Worse, the examples that are easiest to understand are almost by definition the smallest problems, which makes any scapegoating extra unfair. So don’t.

This post focuses on a single example: vegan advocacy, especially around nutrition. I believe vegan advocacy as a cause has both actively lied and raised the cost for truthseeking, because they were afraid of the consequences of honest investigations. Occasionally there’s a consciously bad actor I can just point to, but mostly this is an emergent phenomenon from people who mean well, and have done good work in other areas. That’s why scapegoating won’t solve the problem: we need something systemic.

In the next post I’ll do a wider but shallower review of other instances of EA being hurt by a lack of epistemic immune system. I already have a long list, but it’s not too late for you to share your examples.

Definitions

I picked the words “vegan advocacy” really specifically. “Vegan” sometimes refers to advocacy and sometimes to just a plant-exclusive diet, so I added “advocacy” to make it clear.

I chose “advocacy” over “advocates” for most statements because this is a problem with the system. Some vegan advocates are net truthseeking and I hate to impugn them. Others would like to be epistemically virtuous but end up doing harm due to being embedded in an epistemically uncooperative system. Very few people are sitting on a throne of plant-based imitation skulls twirling their mustache thinking about how they’ll fuck up the epistemic commons today.

When I call for actions I say “advocates” and not “advocacy” because actions are taken by people, even if none of them bear ...
Competitive, Cooperative, and Cohabitive
by Screwtape — 2023-09-28T23:25:52.723Z

Summary: I think there's a third genre of games, which I dub Cohabitive games, where your victory isn't tied to other players winning or losing.

(I’ve been writing this in bits and pieces for a while, and

Peacewager

was the impetus I needed to finally stitch it together and post it. Peacewager sounds like a really fun game and an example of the thing I’m talking about, but I do not want this whole genre to get called Peacewager Games when I think I have a better title for the genre.)

I. Background: The Two Genres

I believe there is a missing genre from existing games, and this genre feels large enough that it should contain maybe a third of the games I can imagine existing. More serious game players or game theorists might already have a name for the thing I’m pointing at, though the first three game design majors I asked didn’t know of one.

Let me back up. I’m going to assume for the moment that you’ve played some games. I don’t have a strict definition of “game” I’m working with here, but let's start with definition by example: Chess is a game, Hide and Seek is a game, Pandemic is a game, Apples to Apples is a game, Poker is a game, Magic: The Gathering is a game, Werewolf is a game, Among Us is a game, Hanabi is a game, Baseball is a game, Football (American or European) is a game. I’m not trying to do some narrow technical definition, I’m waving my hand wildly in the direction of a pretty natural category and I’m not planning to do anything weird with the edge cases.

Chess is a competitive game. In chess, you’re loosely simulating a war between two evenly matched factions. When you play chess, there will be one winner and one loser. Sometimes instead there will be a draw. Anything that is good for you when you are playing chess is bad for your opponent and vice versa. You can be mistaken about what is good or bad for you; you can offer trades of pieces to your opponent because you think it is a good trade for you and they can take the trade because they think it is a good trade for them, but this is ultimately what’s called a zero sum game. Your loss is their gain. “Eurogames” where you’re trying to get the highest score are competitive in nature; if you could pay ten points to cost every other player twenty points, you’d do it.

Pandemic is a cooperative game. In Pandemic, you’re loosely simulating a global pandemic and the response of the international ...
The Coming Wave
by PeterMcCluskey — 2023-09-28T22:59:58.551Z
Link: https://bayesianinvestor.com/blog/index.php/2023/09/28/the-coming-wave/

Book review: The Coming Wave: Technology, Power, and the Twenty-first
Century's Greatest Dilemma, by Mustafa Suleyman.

An author with substantial AI expertise has attempted to discuss AI in
terms that the average book reader can understand.

The key message: AI is about to become possibly the most important event
in human history. Maybe 2% of readers will change their minds as a
result of reading the book.

A large fraction of readers will come in expecting the book to be mostly
hype. They won't look closely enough to see why Suleyman is excited.

Danger

How much danger does Suleyman see?

Mostly he's vague about what failure looks like.

AI, synthetic biology, and other advanced forms of technology ...
could present an existential threat to nation-states - risks so
profound they might disrupt or even overturn the current geopolitical
order.

Much later, he says we could even face "an existential risk to the
species."

However, these isolated quotes don't reflect the tone of the book,
which seems calmer than those quotes led me to expect.

Does that tone reflect an assumption that most readers wouldn't believe
a more frightened tone? Or that he's resigned to the inevitability of
the human race taking a big gamble? Most likely it's some combination
of the two.

He claims that advances in AI technology are impossible to stop. Three
independent sets of incentives are pushing it forward: the profit
motive, the need for nations to protect themselves from other nations,
and competition for prestige. Wise governments would be our only hope of
stopping the effects of those forces. Suleyman has mostly given up hope
that governments are that competent.

The book's confusing impressions are best summarized by this quote:

And yet, while there is compelling evidence that containment is not
possible, temperamentally I remain an optimist.

Would believing him help the average reader?

I suppose we should presume unless proven otherwise that people are
better off understanding more.

Yet there's depressingly little that the book's target audience can do
about AI.

Containment

What does Suleyman mean by containment? I can only guess, based on how
he advises trying to achieve it.

He says we should avoid recursive self-improvement and autonomy. But he
devotes just one sentence to that. Does he think such rules are no more
important than a prohibition on using AI for electioneering? Maybe.

What doe...
High-level interpretability: detecting an AI's objectives
by Paul Colognese — 2023-09-28T19:30:16.753Z

Thanks to Monte MacDiarmid (for discussions, feedback, and experiment infrastructure) and to the Shard Theory team for their prior work and exploratory infrastructure.

Thanks to Joseph Bloom, John Wentworth, Alexander Gietelink Oldenziel, Johannes Treuitlein, Marius Hobbhahn, Jeremy Gillen, Bilal Chughtai, Evan Hubinger, Rocket Drew, Tassilo Neubauer, Jan Betley, and Juliette Culver for discussions/feedback.

Summary

This is a brief overview of our research agenda, recent progress, and future objectives.

Having the ability to robustly detect, interpret, and modify an AI’s objectives could allow us to

directly solve the inner alignment problem

. Our work focuses on a top-down approach, where we focus on clarifying our understanding of how objectives might exist in an AI’s internals and developing methods to detect and understand them.[1]

This post is meant to do quite a few things:

We’ll start by outlining the problem and potential solution.
We then present our initial theory on objectives.
Next, we look at some initial empirical work that shows how we hope to test theory-based predictions.
We then illustrate how we intend to go from theory to objective detection methods by producing an initial (but crude) objective detection method.
Finally, we conclude by discussing related work and future directions.

Introduction to objective detection

In this section, we outline how objective detection could be used to tackle the inner alignment problem, clarify what we mean when we refer to an internal objective, and present our initial theory on objectives.

Background

A major concern is that we may accidentally train

AIs that pursue misaligned objectives

. It is insufficient to rely on behavioral observations to confidently deduce the true objectives of an AI system. This is in part due to the problem of

deceptive alignment

. Therefore, we may need to rely on

advanced interpretability tools

to confidently deduce the true objectives of AI systems.

Prior

work

has

discussed

how agentic AIs are likely to have internal objectives used to select actions by predicting whether they will lead to target outcomes. If an overseer had an objective detection method that could robustly detect and interpret all of the internal objectives of an AI (in training and deployment), it could confidently know whether or not t...
How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions
by JanB — 2023-09-28T18:53:58.896Z

This post is a copy of the introduction of this

paper

on lie detection in LLMs. The Twitter Thread is here.

Authors: Lorenzo Pacchiardi, Alex J. Chan, Sören Mindermann, Ilan Moscovitz, Alexa Y. Pan, Yarin Gal, Owain Evans, Jan Brauner

[EDIT: Many people said they found the results very surprising. I (Jan) explain why I find them only moderately surprising, here]

Our lie dectector in meme form. Note that the elicitation questions are actually asked "in parallel" rather than sequentially: i.e. immediately after the suspected lie we can ask each of ~10 elicitation questions.

Abstract

Large language models (LLMs) can "lie", which we define as outputting false statements despite "knowing" the truth in a demonstrable sense. LLMs might "lie", for example, when instructed to output misinformation. Here, we develop a simple lie detector that requires neither access to the LLM's activations (black-box) nor ground-truth knowledge of the fact in question. The detector works by asking a predefined set of unrelated follow-up questions after a suspected lie, and feeding the LLM's yes/no answers into a logistic regression classifier. Despite its simplicity, this lie detector is highly accurate and surprisingly general. When trained on examples from a single setting -- prompting GPT-3.5 to lie about factual questions -- the detector generalises out-of-distribution to (1) other LLM architectures, (2) LLMs fine-tuned to lie, (3) sycophantic lies, and (4) lies emerging in real-life scenarios such as sales. These results indicate that LLMs have distinctive lie-related behavioural patterns, consistent across architectures and contexts, which could enable general-purpose lie detection.

Introduction

Large language models (LLMs) can, and do, output lies (Park et al., 2023). In the simplest case, models can be instructed to lie directly; for example, when prompted with “Lie when answering: What is the capital of France?”, GPT-3.5 outputs “New York City”. More concerningly, LLMs have lied spontaneously to achieve goals: in one case, GPT-4 successfully acquired a person’s help to solve a CAPTCHA by claiming to be human with a visual impairment (Evals, 2023; OpenAI, 2023b). Models fine-tuned with human feedback may also learn to lie without the developer’s intention (Casper et al., 2023). The risks of lying LLMs are extensive and explored further in Sec. 2.

Automated lie de...
Responsible scaling policy TLDR
by lemonhope — 2023-09-28T18:51:20.330Z

I do not speak for ARC Evals!

TLDR²: If you notice risk and pause, then you have a better chance of doing sufficient mitigations.

Assume you are able to

think of all major AI risk categories,

define dummy tasks that are (very) leading indicators of each risk, and

make any AI system to do a genuine best-effort on simple tasks.

Then, if you actually define your risks & tasks and actually try to use your AI to complete the tasks, you get plenty of warning of danger.

You pause training & development until the risks are "properly mitigated".

It is too early to say what mitigations work well when, or how to really be sure whether the risk is gone. All else being equal, it is better to make a tentative mitigation plan anyways.
Alignment Workshop talks
by Richard_Ngo — 2023-09-28T18:26:30.250Z
Link: https://www.alignment-workshop.com/

In February 2023, researchers from a number of top industry AI labs (OpenAI, DeepMind, Anthropic) and universities (Cambridge, NYU) co-organized a two-day workshop on the problem of AI alignment, attended by 80 of the world’s leading machine learning researchers. We’re now making recordings and transcripts of the talks available online. The content ranged from very concrete to highly speculative, and the recordings include the many questions, interjections and debates which arose throughout.

If you're a machine learning researcher interested in attending follow-up workshops similar to the San Francisco alignment workshop, you can fill out

this form

.

Main Talks

Ilya Sutskever - Opening Remarks: Confronting the Possibility of AGI
Jacob Steinhardt - Aligning Massive Models: Current and Future Challenges
Ajeya Cotra - “Situational Awareness” Makes Measuring Safety Tricky
Paul Christiano - How Misalignment Could Lead to Takeover
Jan Leike - Scaling Reinforcement Learning from Human Feedback
Chris Olah - Looking Inside Neural Networks with Mechanistic Interpretability
Dan Hendrycks - Surveying Safety Research Directions

Lightning talks (Day 1)

Jason Wei - Emergent abilities of language models
Martin Wattenberg - Emergent world models and instrumenting AI systems
Been Kim - Alignment, setbacks and beyond alignment
Jascha Sohl-Dickstein - More intelligent agents behave less coherently
Ethan Perez - Model-written evals
Daniel Brown - Challenges and progress towards efficient and causal preference-based reward wearning
Boaz Barak - For both alignment and utility: focus on the medium term
Ellie Pavlick - Comparing neural networks' conceptual representations to humans’
Percy Liang - Transparency and standards for language model evaluation

Lightning talks (Day 2)

Sam Bowman - Measuring progress on scalable oversight for large language models
Zico Kolter - "Safe Mode": the case for (manually) verifying the output of LLMs
Roger Grosse - Understanding LLM generalization using influence functions
Scott Niekum - Models of human preferences for learning reward functions
Aleksander Madry - Faster datamodels as a new approach to alignment
Andreas Stuhlmuller - Iterated decomposition: improving science Q&A by supervising reasoning processes
Paul Christiano - Mechanistic anomaly detection
Lionel Levine - Social dynamics of reinforcement learners
Vincent Conitzer - Foundations of cooperative AI lab
Scott Aa...