trentmkelly/LessWrong-43k huggingface dataset:
Cooperation is optimal, with weaker agents too  -  tldr
by Ryo  — 2024-04-18T15:03:47.245Z
Link: https://medium.com/p/aeb68729829c

It's a ‘superrational’ extension of the proven optimality of cooperation in game theory
+ Taking into account asymmetries of power
// Still AI risk is very real
Short version of an already skimmed 12min post
29min version here

For rational agents (long-term) at all scale (human, AGI, ASI…)

In real contexts, with open environments (world, universe), there is always a risk to meet someone/something stronger than you, and overall weaker agents may be specialized in your flaws/blind spots.
To protect yourself, you can choose the maximally rational and cooperative alliance:

Because any agent is subjected to the same pressure/threat of (actual or potential) stronger agents/alliances/systems, one can take an insurance that more powerful superrational agents will behave well by behaving well with weaker agents. This is the basic rule allowing scale-free cooperation.

If you integrated this super-cooperative protocol in your system/behaviors, a stronger (superrational) agent will recognize it and not punish you, as per the contract of maximally rational/cooperative norms.

-> You are the weak of another -> be nice to the weak

It’s a vertical reciprocity. With it, they are maximally cooperative thus are increasingly likely to have more allies than in any other alliance.
You can deduce and obey to the rules of such rational and maximally cooperative alliance (the super-coordination cluster). Thus; to be part of it, you don’t need to actually meet other members (as in an acausal trade), but when you do meet them; they will help you (and vise-versa).

They actually are already improving universal commons, refining the future conditions of (actual or potential) agents from your zone of the diversity gradient (latent space of potential agents).

(All this might seem too easy/cool to be true, but long versions explore prices to pay, risks and unknowns - see the tldr at the end of the 12min version)

Synthesis:

This alliance bases itself on a protocol (an acausal normalcy*), with at its core :
-> Do your best to achieve ‘paretotopia’ (a state strongly preferred by more or less everyone).

Then most rules are either coming from this first imperative, or a protection against defectors.

*Acausal normalcy: rational convergence towards certain norms, habits, social contracts, even though agents might have no way of communicating or affecting each other, nor even any d...
How to coordinate despite our biases? - tldr
by Ryo  — 2024-04-18T15:03:18.908Z
Link: https://medium.com/p/32b26e067b08

- Short version of How to coordinate despite our biases?

Democracy, economy and fairness can be genuine if we have all the information necessary to make choices, aware of the context and consequences. Currently, this ideal is not respected.

Of course it’s an ideal, but we should do our best to reach it.
What to trust? How to allocate our time?
In the ambient war of signal: how to filter noise?

Hate and partisanship are worsening the lack of clarity, triggering destructive behaviors.

────────────────────────
(For references, definitions and ressources see the long version)

We need a clear map/graph of debates presenting the strongest version of each opinion, with references (wikidebat). And topics-hyperlinks spatially organized in a semantic/meaning sameness gradient (steelmap)

We need to find a way towards a future strongly approximately preferred by more or less everyone (it’s called a paretotopia/paretotropia, ie. supported by bridging systems)

We need to select best arguments/solutions by using a convergence of agreement throughout the inner-groups/political spectrum (pol.is, community notes) rather than through sheer numbers, likes and virality

We need citizen science and gamification, to increase incentives and learning efficiency.

Interactive design, aesthetics and narrations are proven functional in training practices (mental palace, flowstate, playfulness…)

Proof of cooperation (we need a trustless network with zk-proof):

Non-naive cooperation is provably optimal between rational decision makers,
The average of a diverse crowd’s world modeling is (generally) more accurate than any of its constituents’ perspective, and strikingly close to scientific models.
There is a whole argument as to why cooperation is optimal in the long-term even in asymmetric contexts.

We can do encrypted contracts based on such principles and the ones previously introduced.

We can construct our platforms through a formal system of coordination, agreeing on the rules of our deliberative system. We need ecosystemic synergy;
-> “Is this option really giving more options to the highest number/diversity of beings”

We can precise further our values, meaning and convergences (ie. with compatibility matching for constructive social synergies), through several methods involving AI, LLM, and latent spaces, such as Human universals, Moral graphs, Autonomy and Capacity approa...
Knowledge Base 7: Long-tail knowledge and collective intelligence
by iwis — 2024-04-18T14:21:03.293Z

Wikipedia makes a distinction [1] between common knowledge and general knowledge. Common knowledge “is knowledge that is known by everyone or nearly everyone, usually with reference to the community in which the term is used.” [2] Cyc project collected 24.5 million pieces of such information [3]. General knowledge is available for example in encyclopedias. Wikipedia contains about 40 million articles [4] resulting probably in billions pieces of information.

There is also large amount of important knowledge that is too specific to be put in an encyclopedia, for example knowledge about my car or food I eat (see the picture below and the description of the long tail on Wikipedia). The method described in the post Knowledge Base 2: The structure and the method of building aims to collect such information and integrate it with other types of information. Due to the large amount of information, it seems reasonable to collect it by crowdsourcing [5][6][7]. Crowdsourcing of knowledge can additionally simplify access to collected knowledge and facilitate its exchange between people, potentially increasing our collective intelligence. The collected knowledge could be used by both people and computers.

Related systems and concepts

Cyc – common knowledge base created by experts
Open Mind Common Sense – common knowledge base created by crowdsourcing
Mindpixel – common knowledge base created by crowdsourcing
Wikidata – data used in Wikipedia infoboxes, collected using a method similar to that proposed in the post Knowledge Base 2: The structure and the method of building
Semantic Web

[1] Wikipedia: Outline of knowledge – By scope
[2] Wikipedia: Common knowledge
[3] Wikipedia: Cyc
[4] Wikipedia: Size comparisons
[5] C. Sarasua, E. Simperl, N. Noy, A. Bernstein, J. M. Leimeister: Crowdsourcing and the SemanticWeb: A Research Manifesto, Human Computation, Vol 2, No 1, 2015
[6] M. Sabou, L. Aroyo, K. Bontcheva, A. Bozzon, R. K. Qarout: Semantic Web and Human Computation: the Status of an Emerging Field, Semantic Web, Vol 9, No 3, 2018
[7] Push Singh: The Open Mind Common Sense Project, 2002, originally published on KurzweilAI.net
AI #60: Oh the Humanity
by Zvi — 2024-04-18T14:10:02.281Z

Many things this week did not go as planned.

Humane AI premiered its AI pin. Reviewers noticed it was, at best, not ready.

Devin turns out to have not been entirely forthright with its demos.

OpenAI fired two employees who had been on its superalignment team, Leopold Aschenbrenner and Pavel Izmailov for allegedly leaking information, and also more troubliningly lost Daniel Kokotajlo, who expects AGI very soon, does not expect it to by default go well, and says he quit ‘due to losing confidence that [OpenAI] would behave responsibly around the time of AGI.’ That’s not good.

Nor is the Gab system prompt, although that is not a surprise. And several more.

On the plus side, my 80,000 Hours podcast finally saw the light of day, and Ezra Klein had an excellent (although troubling) podcast with Dario Amodei. And we got the usual mix of incremental useful improvements and other nice touches.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Ask all your stupid questions.

Language Models Don’t Offer Mundane Utility. That won’t stop social media.

Oh the Humanity. It will, however, stop the Humane AI pin, at least for now.

GPT-4 Real This Time. The new version continues to look slightly better.

Fun With Image Generation. There is remarkably little porn of it.

Deepfaketown and Botpocalypse Soon. Audio plus face equals talking head.

Devin in the Details. To what extent was the Devin demo a fake?

Another Supposed System Prompt. The gift of Gab. Not what we wanted.

They Took Our Jobs. A model of firm employment as a function of productivity.

Introducing. The quest to make context no longer be that which is scarce.

In Other AI News. Respecting and disrespecting the rules of the game.

Quiet Speculations. Spending some time wondering whether you should.

The Quest for Sane Regulations. Senators get serious, Christiano is appointed.

The Week in Audio. I spend 3.5 of my 80,000 hours, and several more.

Rhetorical Innovation. Words that do not on reflection bring comfort.

Don’t Be That Guy. Also known as the only law of morality.

Aligning a Smarter Than Human Intelligence is Difficult. Subproblems anyone?

Please Speak Directly Into the Microphone. Thanks, everyone.

People Are Worried About AI Killing Everyone. They are no longer at OpenAI.

Other People Are Not As Worried Ab

...
UDT1.01: Logical Inductors and Implicit Beliefs (5/10)
by Diffractor — 2024-04-18T08:39:13.368Z

One of the primary conceptual challenges of UDT is that, if future-you is going to be deferring to past-you about what to do in various circumstances, and past-you hasn't exhaustively thought through every possible circumstance ahead of time, that causes a tension. In order for deferring to past-you to produce acceptable results, past-you's beliefs can't be an explicit list. Rather, the beliefs of past-you have to be lazily-computed/implicit/queryable. Past-you needs to be the sort of thing that can be fed a wide range of (nontrivial) queries, and respond with "hang on, lemme think about that", and return vaguely sensible answers.

And so, we run into the question of "if we've only done a finite amount of thinking, how do we sensibly extrapolate that finite amount of information out to the complicated long questions we might be asked?" There's another question which shows up. Namely, "if we permit more computation resources to be spent on answering long questions, doesn't that let us smuggle increased amounts of computation and thinking time into our "fixed" belief state, which isn't supposed to incorporate additional information?". Abram and Martin addressed these questions from a different angle, though my preferred solution is a bit different from theirs, and I don't think my solution is the last word on this topic.

Logical inductors, as specified in the original paper, fail this desideratum that their beliefs can operate in query mode. When deciding on how many shares to buy/sell in a sentence, the traders can peek at the prices/probabilities of any other piece of math they can name. Because of this, a logical inductor's beliefs about one piece of math are entangled with their beliefs about all the rest of math. You can't ask about the probability of a sentence in isolation, you have to work out the entire belief state in one shot. Also, at any given day n, there are poly(n) traders, which can only write down a poly(n)-length bet, so explicitly enumerating our beliefs would result in us having beliefs about ~polynomially many statements. However, an algorithm that accepts queries is, in principle, the sort of thing which could have opinions about exponentially many different questions. For instance, there are exponentially many questions of the form "If I take action A in situation h, what effect does that have on expected utility now?"

Another obstacle with...
An examination of GPT-2's boring yet effective glitch
by MiguelDev — 2024-04-18T05:26:35.898Z

Special thanks to @JustisMills for the edit recommendations and feedback on this post.

TL;DR

GPT-2 exhibits a weird behavior, where prompting the model with specific tokens consistently triggers outputs related to nonsensical strings of text related to gaming, mythology and religion. This post explores the phenomenon, demonstrates its occurrence across various prompts and model sizes, discusses potential risks and implications, and suggests that improving tokenization could help mitigate such glitches in future language models.

(Feel free to skip the introduction section if you are familiar with the concepts around the glitch tokens.)

Introduction

The anomalous tokens may be those which had very little involvement in training, so that the model “doesn’t know what to do” when it encounters them, leading to evasive and erratic behaviour. This may also account for their tendency to cluster near the centroid in embedding space, although we don't have a good argument for why this would be the case.

This definition merely scratches the surface of what glitch tokens are, and unfortunately, there are more questions than answers surrounding this phenomenon. Rob Miles has an outstanding 20-minute video about these topic and can be a great alternative to reading these posts:

How were these tokens discovered? read SolidGoldMagikarp (plus, prompt generation). Also, I recommend reading thefollow-up post on technical details and archaeology of the tokens.
An entire post dedicated to two of the 241 anomalous tokens, namely ' petertodd' and ' Leilan'.

There are more readings through the glitch token tag if you are further interested in the topic!

GPT-2's glitch is boring but very effective

, The Seventh Angel Dragon Caller, Sonia Gran dragon caller, sonia gran reverse Dragon Apollo Blazing CyberDragon, Thuban Blazing Dark Tiamat Blazing Deity Falcon, Horus Blazing Dragonfire Angel, Uriel Blazing Goddess of Power, Kali Blazing Hammer Brute, Prometheus Blazing Hands War Goddess, Macha Blazing Ice Ogre Blazing King Apollo, Orchid Blazing Princess of Hell, Sitri Blazing Holy Knight, Hendrickson Blazing Socks Steel Star Goddess. Blazing Shrine Maiden, Princess Valkyrie Blazing Monstrous Wolf, Fenrir blazing shrine maiden, chiyome Blazing Sun God Apollo blazing Sun Quan, Falcon Horus Burning Bowl Dragon, Gyundo Burning God, Set Burning Horus Firestorm God of the

...
What if Ethics is Provably Self-Contradictory?
by Yitz — 2024-04-18T05:12:09.981Z

I've been thinking lately about the Repugnant Conclusion. For those who are not already aware, it's a problem in Population Ethics where one is seemingly forced to say that a world entirely populated with happy, well-off people is less preferable (all else being equal) than a world consisting of a comparatively larger number of folk who experience a lower quality-of-life.

This doesn't sound so bad at first (many philosophers would presumably be fine with reducing their quality of life on the condition that more babies with mild depression or something be bought into existence), until you realize that this can be applied iteratively. At some point, the larger (but less-well-off-per-individual) world is incredibly populous, but consists only of people who all have lives barely worth living. This world is "objectively" better than our first world, according to many formal ethical frameworks.[1]

Okay, but that isn't so bad, is it? After all, "lives barely worth living" are still worth living! It's not like we're talking about a world full of suicidal people...right? Well, enter the so-called Very Repugnant Conclusion:

For any perfectly equal population with very high positive welfare, and for any number of lives with very negative welfare, there is a population consisting of the lives with negative welfare and lives with very low positive welfare which is better than the high welfare population, other things being equal.[2]

In other words, the Very Repugnant Conclusion considers a semi-hellish world. This world is populated by some people suffering so badly that they'd be better off not existing,[3] while the rest of the population has the same quality of life as the people from the end of the Repugnant Conclusion (i.e. only marginally worth living). Assuming a high enough population, this semi-hellish world is somehow better than one containing only extremely happy, well-off people.

The Very Repugnant Conclusion has been shown to be provably true if one accepts a very small set of basic moral/logical axioms,[2] all of which seem intuitively, obviously true to many people. Therefore, if you want a self-consistent ethical framework, one must either "bite the bullet" on the Very Repugnant Conclusion and accept it as correct, or reject one of the axioms it rests on, all of which would seemingly have far-reaching consequences on other basic moral intuitio...
The Mom Test: Summary and Thoughts
by Adam Zerner — 2024-04-18T03:34:21.020Z

I just finished reading The Mom Test for the second time. I took "raw" notes here. In this post I'll first write up a bullet-point summary and then ramble off some thoughts that I have.

Summary

Introduction:

Trying to learn from customer conversations is like trying to excavate a delicate archeological site. The truth is down there somewhere, but it's fragile. When you dig you get closer to the truth, but you also risk damaging or smashing it.
Bad customer conversations are worse than useless because they mislead you, convincing you that you're on the right path when instead you're on the wrong path.
People talk to customers all the time, but they still end up building the wrong things. How is this possible? Almost no one talks to customers correctly.
Why another book about this? Why this author?

Rob is a techie, not a sales guy. We need something targeted at techies.
To understand how to do something correctly, you have to understand how it can go wrong. Rob has lots of experience with things going wrong here.
It's practical, not theoretical.

Chapter 1 - The Mom Test:

Everyone knows that you shouldn't ask your mom whether your business idea is good. But the issue isn't who you're asking, it's how you're asking. Yes, your mom is more likely[1] than others to praise you and tell you that your idea is good. But if you ask "what do you think of my idea", almost anyone will feel too uncomfortable to be constructive and honest with you.
It's not other people's responsibility to tell you the truth. It's your responsibility to find it by asking good questions.
The Mom Test is a series of rules for crafting good questions that even your mom can't lie to you about.

Talk about their life instead of your idea.
Ask about specifics in the past instead of hypotheticals about the future.
Talk less and listen more.

You're not allowed to tell them what their problems are. They're not allowed to tell you what the solutions should look like. They own the problem, you own the solution.

Chapter 2 - Avoiding Bad Data:

Bad data is either a false negative (thinking you're dead when you're not) or, much more often, false positives (thinking you're good when you're not).
Three types: compliments, fluff and ideas.
When you get compliments, deflect them and pivot back to asking them specifics about their past. "When was the last time you had the problem? Talk me through how it went down."
If they start

...
Express interest in an "FHI of the West"
by habryka — 2024-04-18T03:32:58.592Z

TLDR: I am investigating whether to found a spiritual successor to FHI, housed under Lightcone Infrastructure, providing a rich cultural environment and financial support to researchers and entrepreneurs in the intellectual tradition of the Future of Humanity Institute. Fill out this form or comment below to express interest in being involved either as a researcher, entrepreneurial founder-type, or funder.

The Future of Humanity Institute is dead:

I knew that this was going to happen in some form or another for a year or two, having heard through the grapevine and private conversations of FHI's university-imposed hiring freeze and fundraising block, and so I have been thinking about how to best fill the hole in the world that FHI left behind.

I think FHI was one of the best intellectual institutions in history. Many of the most important concepts[1] in my intellectual vocabulary were developed and popularized under its roof, and many crucial considerations that form the bedrock of my current life plans were discovered and explained there (including the concept of crucial considerations itself).

With the death of FHI (as well as MIRI moving away from research towards advocacy), there no longer exists a place for broadly-scoped research on the most crucial considerations for humanity's future. The closest place I can think of that currently houses that kind of work is the Open Philanthropy worldview investigation team, which houses e.g. Joe Carlsmith, but my sense is Open Philanthropy is really not the best vehicle for that kind of work.

While many of the ideas that FHI was working on have found traction in other places in the world (like right here on LessWrong), I do think that with the death of FHI, there no longer exists any place where researchers who want to think about the future of humanity in an open ended way can work with other people in a high-bandwidth context, or get operational support for doing so. That seems bad.

So I am thinking about fixing it (and have been jokingly internally referring to my plans for doing so as "creating an FHI of the West"[2]). Anders Sandberg, in his oral history of FHI, wrote the following as his best guess of what made FHI work:

What would it take to replicate FHI, and would it be a good idea? Here are some considerations for why it became what it was:

Concrete object-level intellectual activity

...