trentmkelly/LessWrong-43k huggingface dataset:
A ranking scale for how severe the side effects of solutions to AI x-risk are
by Christopher King — 2023-03-08T22:53:11.224Z

Sometimes when I look at solutions to AI risk (a.k.a. pivotal acts), they seem like they are "cheating". For example, the only way I could see CAIS preventing AI risk is for it to be used against other AI researchers. This is as opposed to a pure aligned utility maximizer, which could simply pull the plug on younger AGIs without touching the creators.

There are no rules though for what counts as an x-risk solution. Instead, I propose to create a scale based on how much I don't like the side effects. If you have similar preferences, hopefully it will be useful.

Pivotal Act severity scale

Level 1 (best): The solution can directly counteract any malicious actions taken by an evil AI directly, without affecting any other part of the world. For example, if the evil AI creates nanobots, the solution directly destroys the nanobots without removing the evil AI. (This is desirable because it minimally restricts humans from pursuing AI research, if only for their own enjoyment.) Prototypical example: aligned AIXI

Level 2: The solution only shuts down AI projects that are on the verge of singularity. It must shut down all such unaligned projects that would result in unaligned singularity AI. Prototypical example: Nanobots that unplug AGIs before they are turned on

Level 3: The solution is allowed to directly attack the economy in some narrow way. Prototypical example: burning all the GPUs

Level 4: The solution grants authority or the power to intimidate to a group of humans who will attack or illegalize AI research to some extent OR it lobbies (potentially by offering goods or services) groups who already have that power to do so. Prototypical example: Use AI to become dictator of the world

Level 5 (worst): The solution causes existential catastrophe instead of preventing it. But it prevents the far worse s-risk. Prototypical example: Paperclip maximizer

This scale can also be interpolated subjectively and intuitively. For example I'd say that a solution that is guaranteed to only destroy unaligned AI is 1.5.

Here are some examples of where I would rank various solutions:

Aligned MIRI-style AGI: 1

Center for Human Compatible AI (CHAI): 1

Imitative Amplification: 1 - 2 (depending on alignment tax)

Burn all the GPUs: 3

Comprehensive AI services: 3.5 - 4

OpenAI: Historically and currently at 4, long run 3.5 - 4 (but maybe higher if they get lucky?)

Post-human & e/acc

...
Progress links and tweets, 2023-03-08
by jasoncrawford — 2023-03-08T20:37:13.890Z
Link: https://rootsofprogress.org/links-and-tweets-2023-03-08

The Progress Forum

Derek Thompson interviews Patrick Collison on progress

Opportunities

Essay contest: “How could science be different?” (via @moreisdifferent)

Marc Andreessen is blogging again

“What’s my hope? To show you that we live in a more interesting world than you might think; that it’s more comprehensible than you might fear; and that more things are possible than you might imagine”
“This is the most normal and placid things are ever going to be”
“We are heading into a world where a flat screen TV that covers your entire wall costs $100, and a four year college degree costs $1 million”

Links

The U.S. is a build-nothing country. See also @ericgoldwyn’s comment
Samuel Smiles, industrial biographer and founder of the self-help genre
Adversarial collaboration on how income relates to well-being (via @amandaegeiser)
Be careful inferring causality in the presence of control loops
Brass Birmingham is a board game set in the Industrial Revolution (h/t @ejames_c)
The Iconographic Encyclopædia of Science, Literature, and Art

Queries

Who are the most influential essay writers who never wrote books?
What should Dwarkesh ask Scott Aaronson? and Eliezer Yudkowsky?
Is there any study comparing independents to employees on job satisfaction?
What’s the best book about pre-21st century General Electric?
What should Anastasia read after Kuhn relevant to research and progress?
Any other authors have data loss problems with Scrivener?
What has happened since this was made in 2017? Is pharma IRR negative now?

Tweets & retweets

Are we going through a crisis of meaning in our jobs?
What does solar look like in the limit? (thread)
We have created the heaven our ancestors dreamed of
A Keatsian science sonnet. “More scientific heroes in literature please”
The real effect of LLMs on software will be felt after 6–18 months of the product cycle
AI problems that were considered “nowhere near solved” in a book published 2021
In SF a pedestrian bridge costs $200M and presents ”daunting logistics”
Academia encourages historians to prioritize tenure at the expense of social value
Predicting the future is hard, Bertrand Russell edition
Shape rotators 📈, wordcels 📉
The first radio telescope was built by an amateur in his back yard

Charts

Last week’s chart on worker/revenue ratio adjusted for inflation, a bit less dramatic
Project "MIRI as a Service"
by RomanS — 2023-03-08T19:22:50.405Z

One theoretically possible way to greatly speed-up alignment research is to upload the minds of alignment researchers (with their consent, of course). Ideally, we would upload MIRI and some other productive groups, create thousands of their copies, and speed them up by orders of magnitude. This way, we may be able to solve alignment withing years, if not months.

Obviously, the tech is not there yet. But there is a poor-man's version of mind uploading that does exist already:

collect everything that the person has ever wrote
take the smartest available language model and fine-tune it on the writings
the result is an incomplete but useful digital replica of the person
upon the release of a better language model, create a better replica, rinse, repeat.

The current language models are already smart enough to assist with research in non-trivial ways. If fine-tuned on alignment writings, such a model could become a "mini-MIRI in your pocket", available 24/7.

A few examples of a possible usage:

"We have implemented the following confinement measures. What are some non-trivial ways to circumvent them?"
"A NYT reporter asked us to explain our research on Proof-Producing Reflection for HOL. Write a short summary suitable for the NYT audience"
"How this idea can be combined with the research on human preferences by Paul Christiano..."
"Analyze this research roadmap and suggest how we can improve it"
"Here is my surprisingly simple solution to alignment! Tell me how it could go terribly wrong?"
"Provide a list of most recent papers, with summaries, related to corrigibility in the context of alignment research"
"Let's chat about Value Learning. Correct me if I'm wrong, but..."

This could be a small technical project with a disproportionately large positive impact on alignment research.

The first proof-of-concept could be as simple as a well-crafted prompt for ChatGPT, nudging it to think like an alignment researcher[1].

For a prototype, we could build a dataset by collecting all relevant writings, everything from Eliezer's tweets to the top papers on alignment-related topics, and try to fine-tune the largest available model on it.

^

After some iterations, I ended up with the following prompt. Could be a good start:

Imagine you're the world's top expert in AI alignment research. You agree that AGI is possible to make, that it eventually will become orders-of-magnitude

...
2022 Survey Results
by Screwtape — 2023-03-08T19:16:47.224Z

The Data

0. Population

There were 186 respondents over 28 days. The first week of the survey got 172 responses. The second week got 10. The third and fourth weeks got 2 responses each.

Previous surveys have been run over the past decade or so. Their numbers are as follows:

2009: 166
2011: 1090
2012: 1195
2013: 1636
2014: 1503
2016: 3083
2017: "About 300"
2020: 61
2022: 186

The cheerfully optimistic interpretation of this is something like "cool! I got about as many as Scott did on his first try!" I'll talk about the survey size more in the Conclusion section. For now, on with the numbers.

Previous LessWrong Surveys:
No: 131, 72.0%
Prefer not to answer: 6, 3.3%
Yes: 45, 24.7%

Did you also take the ACX 2022 Survey?
No: 93, 50.5%
Prefer not to answer: 3, 1.6%
Yes: 88, 47.8%

I. Demographics (Not counting sex and gender)

There's a format I'll use in a few places throughout this summation. If you see a sequence of numbers like "30.1 + 8.9 (24, 28, 34) [n=186]" those numbers are "Mean + standard deviation (1st quartile, 2nd quartile, 3rd quartile) [n= number responding]."

Age:
30.1 + 8.9 (24, 28, 34) [n=186]

I broke this out into buckets by decade so I could get a better sense of what was going on.

Under 20: 13, 7.0%
20 to 29: 91, 49.2%
30 to 39: 57, 30.8%
40 to 49: 14, 7.6%
50 to 59: 8, 4.3%
60 to 69: 2, 1.1%

The 2009 to 2014 surveys consistently found the mean age was 27, and in 2016 the mean age had only gone up to 28. This is an interesting property for a community to have! It's far from unusual: any survey of a college campus will find the mean age stays about the same, and for different reasons so can the mean age of countries. A survey of Britney Spears fans probably finds that the mean age goes up over time, and many churches observe the same pattern. It's one of several high level ways to look at a community: are you replacing yourself about as fast as you lose people?

LessWrong is probably a community people find and age out of. We're a bit older. In 2011, that 20 to 29 bucket was about 60%. Still, teenagers and college students are still finding us.

Race:
White (non-Hispanic): 148, 82.7%
Asian (East Asian): 6. 3.4%
Asian (Indian subcontinent): 6, 3.4%
Hispanic: 5, 2.8%
Middle Eastern: 3, 1.7%
Other: 14, 7.8%

Country:
United States: 95, 51.1%
United Kingdom: 20, 10.8%
Australia: 12, 6.5%
Germany: 11, 5.9%
New Zealand: 6, 6.6%
France: 6, 6.6%
Canada: 5, 5.5...
Use the Nato Alphabet
by Cedar — 2023-03-08T19:14:48.008Z

I plan to extend this into a longer post, but for now I want to tell a story.

I have a name that is pretty difficult for English speakers to just "know" the spelling of from the pronounciation.

Recently I started using the Nato alphabet. It really greatly reduces the friction of e.g. communicating to a telecom provider over the phone.

If you, like me, struggle to find words for spelling words, try:

http://natoalpha.bet/

Disclaimer: I made it after struggling way too much with remembering / coming up with the words.
LessWrong needs a sage mechanic
by lc — 2023-03-08T18:57:34.080Z

Most forums work in a similar way: if you reply to a post, it gets bumped to the top of the feed. This is usually good, from the perspective of the commenter; people generally want others to see their comments.

There are obvious problems, though. On every forum with enough volume, there will be people who (deliberately or not) create reply-bait threads that drown out more interesting content. The best solution is not to reply - but people sometimes realllly want to do that, despite not wanting to boost the threads' half life or add discussion volume.

4chan has a hidden feature to help with this and make the boards a little less spammy: the sage tag. If you wish, you can type "sage" in the options field, and your reply will not bump the topic to the front page as usual. Every forum ought to have a "sage" feature, but LessWrong needs it in particular, to fight a natural trend away from its sometimes dry topics toward high context drama and political discussions.
Mathematical models of Ethics
by Victors — 2023-03-08T17:40:18.106Z

Hello, I am currently trying to find all the attempts at mathematical modeling of ethics (utilitarianism, axiologies, etc.) I have found several, do you have any ideas of keywords to type or even articles to suggest here ?

An example of a (simple) model is the notion of utility in the Von Neumann-Morgenstern Theorem, but I am interested in any attempt to formalize ethical situations.

More fundamentally, formalizing rules allowing to order the possible worlds (resulting from actions that an agent does or does not do).

A kind of mathematical model of the Must Be by ordering desired consequences (state of the world).

For the moment I found these, for example:

https://www.lesswrong.com/posts/hbmsW2k9DxED5Z4eJ/impossibility-results-for-unbounded-utilities

https://www.lesswrong.com/posts/5iZTwGHv2tNfFmeDa/on-infinite-ethics

https://forum.effectivealtruism.org/posts/FEJLFr5ef82FSY8vr/minimalist-extended-very-repugnant-conclusions-are-the-least
Against LLM Reductionism
by Erich_Grunewald — 2023-03-08T15:52:38.741Z
Link: https://www.erichgrunewald.com/posts/against-llm-reductionism/

Summary

Large language models (henceforth, LLMs) are sometimes said to be "just" shallow pattern matchers, "just" massive look-up tables or "just" autocomplete engines. These comparisons amount to a form of (methodological) reductionism. While there's some truth to them, I think they smuggle in corollaries that are either false or at least not obviously true.

For example, they seem to imply that what LLMs are doing amounts merely to rote memorisation and/or clever parlour tricks, and that they cannot generalise to out-of-distribution data. In fact, there's empirical evidence that suggests that LLMs can learn general algorithms and can contain and use representations of the world similar to those we use.

They also seem to suggest that LLMs merely optimise for success on next-token prediction. It's true that LLMs are (mostly) trained on next-token prediction, and it's true that this profoundly shapes their output, but we don't know whether this is how they actually function. We also don't know what sorts of advanced capabilities can or cannot arise when you train on next-token prediction.

So there's reason to be cautious when thinking about LLMs. In particular, I think, caution should be exercised (1) when making predictions about what LLMs will or will not in future be capable of and (2) when assuming that such-and-such a thing must or cannot possibly happen inside an LLM.

Pattern Matchers, Look-up Tables, Stochastic Parrots

My understanding of what goes on inside machine learning (henceforth, ML) models, and LLMs in particular, is still in many ways rudimentary, but it seems clear enough that, however tempting that is to imagine, it's little like what goes on in the minds of humans; it's weirder than that, more alien, more eldritch. As LLMs have been scaled up, and more compute and data have been poured into models with more parameters, they have undergone qualitative shifts, and are now capable of a range of tasks their predecessors couldn't even grasp, let alone fail at, even as they have retained essentially the same architecture and training process.[1] How do you square their awesome, if erratic, brilliance with the awareness that their inner workings are so ordinary?

One route would be to directly deny the brilliance. Gary Marcus does this, pointing out, and relishing in, the myriad ways that LLMs misfire. Their main limits are, he says, that they are...
Agency, LLMs and AI Safety - A First Pass
by Giulio — 2023-03-08T15:42:59.123Z
Link: https://www.giuliostarace.com/posts/aisafety-llm-agency/

The following is an adaptation of part of my (unsuccessful) application to the
Winter 2022 cohort of SERI MATS, as prompted by
Victoria Krakovna's "What is your favorite
definition of agency, and how would you apply it to a language model?"

This is far from a perfect treatment of the topic, but I am publishing it to
document my early days in the field. Or should I say "Epistemic status: I am
learning" 😉.

I am interested in this question because it is one which came up over the summer
when I got more interested in AI Safety. In my readings of LessWrong and
technical AI Safety Research surveys, I started to question the need for agency
in AI models in the first place.
It seemed to me that most hypotheses about future AGI scenarios involved a notion of agency
in the model(s) considered, but it was unclear to me what advantages developing
agentic models would provide to humans. I thought that if agency was not
essential, then perhaps a “compromise” between safety and AGI capabilities could
be achieved by relying on oracle-like models, devoid of agency but useful for
guidance and whose answers and recommendations we could ultimately
reject[1]. I asked a
related question on the AI stackexchange forum
but unfortunately was met with confusion and ultimately my question was closed
as I had made the very obvious mistake of not defining my terms. In an attempt
(which never succeeded) to resuscitate the discussion, I proposed the following
definition of agency:

I define agency as the ability to autonomously perceive and interact
with a given environment. Anything capable of agency is then an agent.
Furthermore, I define

autonomous perception: the ability to perceive a given environment
without the need of an external agent

interaction: the ability to change the state of the environment

I liked (and still like) my definition, but admittedly I have not tried
evaluating it[2]. One issue is that I did not develop it in
isolation: my bias was that language models are tools, not agents and purposely
defined agency backwards from this claim. As such it is (perhaps artificially)
difficult to apply this definition of agency to language models. The choice of
environment for a language model is trivial: it can be anything whose state can
be represented (noisily) through language. The “interaction” requirement of the
definition can in a sense be m...
Why Uncontrollable AI Looks More Likely Than Ever
by otto.barten — 2023-03-08T15:41:33.122Z
Link: https://time.com/6258483/uncontrollable-ai-agi-risks/

This is a crosspost from Time Magazine, which also appeared in full at a number of other unpaid news websites.
BY OTTO BARTEN AND ROMAN YAMPOLSKIY

Barten is director of the Existential Risk Observatory, an Amsterdam-based nonprofit.

Yampolskiy is a computer scientist at the University of Louisville, known for his work on AI Safety.

“The first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control,” mathematician and science fiction writer I.J. Good wrote over 60 years ago. These prophetic words are now more relevant than ever, with artificial intelligence (AI) gaining capabilities at breakneck speed.

In the last weeks, many jaws dropped as they witnessed transformation of AI from a handy but decidedly unscary recommender algorithm, to something that at times seemed to act worryingly humanlike. Some reporters were so shocked that they reported their conversation histories with large language model Bing Chat verbatim. And with good reason: few expected that what we thought were glorified autocomplete programs would suddenly threaten their users, refuse to carry out orders they found insulting, break security in an attempt to save a child’s life, or declare their love to us. Yet this all happened.

It can already be overwhelming to think about the immediate consequences of these new models. How are we going to grade papers if any student can use AI? What are the effects of these models on our daily work? Any knowledge worker, who may have thought they would not be affected by automation in the foreseeable future, suddenly has cause for concern.

Beyond these direct consequences of currently existing models, however, awaits the more fundamental question of AI that has been on the table since the field’s inception: what if we succeed? That is, what if AI researchers manage to make Artificial General Intelligence (AGI), or an AI that can perform any cognitive task at human level?

Surprisingly few academics have seriously engaged with this question, despite working day and night to get to this point. It is obvious, though, that the consequences will be far-reaching, much beyond the consequences of even today’s best large language models. If remote work, for example, could be done just as well by an AGI, employers may be able to simply spin up a few new digital employees to per...
Universal Modelers
by George3d6 — 2023-03-08T15:39:38.754Z
Link: https://epistem.ink/570c388dff0e4eeb9de16c35129baac2

Alternative title: A simple framework for thinking about applying machine learning to science and engineering in a way that both produces reliable results and poses comparatively ~0 risk of destroying humanity

Mathematics was designed for the human brain.

In our times, this seems like a rather silly design decision. Applied mathematics, be it for physics, chemistry, or economics, run on computers.

To the extent applied mathematics has adopted computers, it was to speed up the solving of problems relevant principally to a pre-computer world. As if our ancestors discovered the combustion engine, then used it to boost the speed of a horse-drawn carriage. To a limited degree, this might work, but the horse becomes a pointless constraint in your design when the engine becomes thousands of times more powerful than the horse.

Motivated reasoning could keep justifying the horse’s existence. Oh, it helps in edge cases where you run out of fuel, it’s easier for humans to “understand” the carriage if it has a horse, and the constraints provided by the horse are ultimately useful in coming up with better designs.

But we don’t, we build cars.

The engine of mathematics now exists, and we ought to think really hard about how to develop mathematics around it, rather than retrofit computers onto mathematics designed for the CNS of apes.

One amendment I should make is that I am not claiming our imagination or thinking is so trivial as to be easily replaced by computers as they are now, even the most powerful and adeptly programmed. I argue that* the** constraint of human minds that have to add, subtract and memorize numbers is limiting our mathematics.*

1. Leaky Abstractions

Take something like sigma summation (∑), at first glance it looks like an abstraction, but it’s too leaky to be a good one.

Leaky is a term from software engineering; it refers to a fault of an abstraction called ‘interface’ which helps us interact with a system. An interface is ‘leaky’ if it does not properly cover the system, thus sometimes compelling us to use direct parts of the system.

Why is it leaky? Because all the concepts it “abstracts” over the need to be understood in order to use the sigma summation. You need to "understand" operations, operators, numbers, ranges, limits, and series.

Or take integrals (∫); To master the abstraction one needs to understand all of calculus and all that is foundational to c...