trentmkelly/LessWrong-43k huggingface dataset:
The subset parity learning problem: much more than you wanted to know
by Dmitry Vaintrob — 2025-01-03T09:13:59.245Z

Imagine that you’re looking for buried treasure on a large desert island, worth a billion dollars. You don’t have a map, but a mysterious hermit offers you a box with a button to help find the treasure. Each time you press the button, it will tell you either “warmer” or “colder”. But there’s a catch. With probability 2−100, the box will tell you the truth about whether you’re closer than you were last time you pressed. But with the remaining probability of .9999999999999999999999999999992, the box will make a random guess between “warmer” and “colder”. Should you pay $1 for this box?

Keep this in mind as we discuss the closely related problem of parity learning.

In my experience of interacting with the ML and interpretability communities, the majority of people don’t know about the impossibility result of the parity learning problem. The ones who do will often assume that this is a baroque, complicated result that surely doesn’t have a simple proof (another surprising opinion I’ve heard is of people knowing about the result, but saying that “there’s some new architecture that seems to solve it, actually”, which is somewhat indicative of people’s trust in the concept of “proof” in the ML community).

Recently I was pleasantly surprised to realize that the impossibility of solving this problem (via a gradient-based learning algorithm in polynomial time) actually admits a pretty nice and understandable proof. The gist of it boils down to the silly “pirate treasure” story above: the answer, of course, is that you shouldn’t buy the box (at least if you’re trying to maximize your expected income), and for the same reason you can’t build a cool new architecture that solves the parity learning problem.

In this post I’ll briefly explain the problem, how it’s different from some other impossibility results, and why I think it’s important. This will later tie in to a series of posts about the insufficiency of Bayesian methods in (realistic) ML contexts.

What is the parity learning problem?

This is a post where a bit of math can go a long way, but I’ll try to make it approachable to anyone with either a bit of math or a bit of ML background. First, the XOR parity function is a function πS:{0,1}n→{0,1} from a length-n boolean input to a single boolean output. This function depends on a “secret” variable S, which is a subset: S⊂[n] (here [n]={0,…,n−1} ...
Building AI safety benchmark environments on themes of universal human values
by Roland Pihlakas — 2025-01-03T04:24:36.186Z
Link: https://docs.google.com/document/d/1lg9C7FznXR908U30hZ_KkSh6na8U515z_jgjeVwZsFY/edit?pli=1&tab=t.0

This is an AI Safety Camp 10 project that I will be leading. With this post, I am looking for external collaborators, ideas, questions, resource suggestions, feedback, and other thoughts.

Summary

Based on various sources of anthropological research, I have compiled a preliminary list of universal (cross-cultural) human values. It seems to me that various of these universal values resonate with concepts from AI safety, but use different keywords. It might be useful to map these universal values to more concrete definitions using concepts from AI safety.

One notable detail in this research is that in case of AI and human cooperation, the values are not symmetric as they would be in case of human-human cooperation. This arises because we can change the goal composition of agents, but not of humans. Additionally there is the crucial difference that agents can be relatively easily cloned, while humans cannot. Therefore, for example, a human may have a universal need for autonomy, while an AI agent might imaginably not have that need built-in. If that works out, then the agent would instead have a need to support human autonomy.

The objective of this project would be to implement these mappings of concepts into tangible AI safety benchmark environments.

The non-summary

A related subject is balancing multiple human values (as the title says, it is in plural!). The human values and needs have to be met to a reasonable degree, that is, considering balancing all other human values as well. In this context, balancing is not the same as “tradeoff”. In some interpretations and use cases, tradeoff means linear rate of substitution between objectives, but as economists know well - generally humans prefer averages in all objectives to extremes in a few objectives. This means a naive approach of summing up the rewards of an AI agent would not yield aligned results. It is essential to use nonlinear utility functions for transforming the rewards before summing them up in the RL algorithm.

The current compiled list of universal human values is available in this document: "Universal ethical values - Survey of values"

https://docs.google.com/document/d/1ZZiToC149g9vKJGZRhktFmLYdB5J63nbClvCN_CxqAM/edit?usp=sharing

(We may publish it as a separate LW post in the future).

It might be also interesting to consider how agents could internally represent the diversity of hum...
Emotional Superrationality
by nullproxy — 2025-01-02T22:54:53.303Z

I'm going to start with a big claim here; your emotions are not irrational, they're superrational. You're using them wrong.

Superrationality is the term for perfect rationality, the state which maximizes utility because it assumes that all other players are also superrational, and thus all superrational players will come up with the same strategy if they assume that all other players are superrational too. Different choices are impossible, because each player is choosing for all players utilizing perfect rationality. The prisoner's dilemma effectively disappears under this assumption; a superrational player only has to think "If I defect, they defect. But if I cooperate, then they cooperate. Therefore I'm really choosing between (defect, defect) and (cooperate, cooperate)." Since (cooperate, cooperate) has a better payoff than (defect, defect), a superrational player chooses to cooperate.

In this case, you'll notice that different choices are impossible because the choice that emerges is most beneficial personally. It's also most beneficial societally, but that's somewhat of a secondary benefit. It's most beneficial personally because you are inherently drawn to staying in a state of superrationality, if you can get there. If you're in a state of superrationality, you have to exert less effort to get an outcome that benefits you personally, you experience less fear of other agents, you don't have to work as hard at your internal calculations, and you can navigate to a pleasant reality where you have more time outside of prison to do whatever you want to. These are all good things. These are not only personally beneficial, but also have a cascading effect on other agents. When other agents see your strategy, they are drawn to cooperate with you and attempt to align with your superrationality, because you have found a truthful and sustainable method of self-stabilizing coordination that feels good to be in. A state that feels better than defecting does.

But this is assuming that the opposing player is superrational. And this isn't true in the real world, not by a long shot.

Or is it?

Empathy is the ability to share another person's feelings, experiences, and emotions. It's the ability to "put yourself in someone else's shoes" so to speak. Often, this is a process that is semi-automatic, inherent, can be developed, but requires an increase of emotional intelligence....
Playing with Otamatones
by jefftk — 2025-01-02T19:50:01.781Z

A couple months ago Nora (3y) got very into

Otamatones. She
wanted to watch lots of videos, primarily

TheRealSullyG.
She asked for one for Christmas, and so did I:

They're a lot of fun, but I haven't yet figured out if it's an
instrument I'll ever be able to reliably play in tune.

The basic idea is you have a touch sensitive ribbon which triggers a
pitch dependent on the location of your finger, and then you squeeze
the cheeks to control the volume. Nora's small pink one is the
standard size, and I find it quite hard to use with grownup
fingers. My big red one is the "deluxe" model, and at the larger size
the ribbon is quite playable.

Lily made a video of the Mary Ellen
Spider song with an Otamatone actor, though (perhaps for the best)
the instrument is not featured musically:

I measure both models as having 29-31 half steps (2.5 octaves)
depending on which setting you're using:

Standard, low: A#1 (59Hz) to D#4 (312Hz)

Standard, medium: G#3 (213Hz) to D6 (1,184 Hz)

Standard, high: A#5 (920Hz) to D#8 (4,980Hz)

Deluxe, low: F1 (44Hz) to C4 (265Hz)

Deluxe, medium: D#3 (158Hz) to B5 (980Hz)

Deluxe, high: E5 (671Hz) to A#7 (3,676Hz)

Overall it's a fun and silly instrument, though there are a few
smallish places it could be better:

It's a little quiet for group playing, much quieter than a
fiddle or mandolin. If I'm playing with others I'm generally on the loudest
setting and wishing it were a little louder. This is less of a
problem now when I'm kind of terrible at it, but if I do get good it
will be limiting.

You can hook the "deluxe" model up to an external speaker, at
the cost of losing control over cheek articulation. And I was hoping
the bass would sound better with a speaker intended for that range,
but it turns out that's not a voice they've gone for.

Instead of an octave selector I want either a pitch range knob
or a five-setting switch: three settings is often not enough to place
the 2.5 octave range where I want it.

Still, I'm happy! And now Anna is asking for one for her birthday,
which would let us play as a trio.

Comment via: facebook, mastodon, bluesky
7. Iterate the Game: Racing Where? 
by Allison Duettmann — 2025-01-02T19:06:22.165Z

We started this book with the observation that different players of Civilization value things differently. The game of Civilization emerges from the composition of these valuations. The strategy of voluntary cooperation serves each player a great deal. We should uphold it as new AI players enter the game. If we succeed and get better at the game, what awaits in future rounds of play?

Robin Hanson suggests that:

“... in the distant future, our descendants will probably have spread out across space, and redesigned their minds and bodies to explode Cambrian-style into a vast space of possible creatures. If they are free enough to choose where to go and what to become, our distant descendants will fragment into diverse local economies and cultures. Given a similar freedom of fertility, most of our distant descendants will also live near a subsistence level. Per-capita wealth has only been rising lately because income has grown faster than population. But if income only doubled every century, in a million years that would be a factor of 103000, which seems impossible to achieve with only the 1070 atoms of our galaxy available by then. Yes, we have seen a remarkable demographic transition, wherein richer nations have fewer kids, but we already see contrarian subgroups like Hutterites, Hmongs, or Mormons that grow much faster. So unless strong central controls prevent it, over the long run such groups will easily grow faster than the economy, making per person income drop to near subsistence levels.”

In this Malthusian future, civilization, left to emergent phenomena, leads to a race to subsistence. Much of our planet’s history from bacteria to civilization occurred at subsistence levels. If being far from subsistence is exceptional, we shouldn't expect an imaginary future where most activity is far from it. More efficient activity outcompetes less efficient activity. In other words, we are racing toward competitive equilibria, which is at subsistence levels.

Competitive Equilibria: Subsistence without Suffering

Should we be worried? Only if subsistence means suffering. We find subsistence intuitively repugnant because our past history causes us to equate subsistence with suffering. Given the increasing automatability of manual tasks, future activity will not involve back-breaking physical activity but rather knowledge-based work. Does know...
6. Increase Intelligence: Welcome AI Players
by Allison Duettmann — 2025-01-02T19:06:01.521Z

Voluntary cooperation is a main feature of the civilizational game. It got us to where we are today. We explored how to improve and defend this dynamic from within the game. Gradually, non-human intelligent players are entering the playing field. In a few more iterations, they will be major players. If we want to continue our Paretotropian ascent, we better make sure our cooperative framework is set up to handle a diversity of intelligences pursuing a diversity of goals.

AI Threats

Let’s pretend we have achieved computer security. Are we set for a cooperative long-term future? It is worth revisiting Toby Ord’s AI takeoff scenario from the last chapter. According to Ord, once an AI breaches computer security vulnerabilities, it could re-escalate its power.

“This is more speculative, but there are many plausible pathways: by taking over most of the world’s computers, allowing it to have millions or billions of cooperating copies; by using its stolen computation to improve its own intelligence far beyond the human level; by using its intelligence to develop new weapons technologies or economic technologies; by manipulating the leaders of major world powers (blackmail, or the promise of future power); or by having the humans under its control use weapons of mass destruction to cripple the rest of humanity. Of course, no current AI systems can do any of these things. But the question we’re exploring is whether there are plausible pathways by which a highly intelligent AGI system might seize control. And the answer appears to be “‘yes.’” ”

While “AGI” sometimes describes an Artificial General Intelligence of mere human-level intelligence, many assume an AGI reaching that level will eventually exceed it in most relevant intellectual tasks. An AGI that displaces human civilization as the overall framework of relevance for intelligence and dominates the world can be described as an AGI singleton. This scenario carries two threats worth unpacking; the first-strike instabilities generated by a mere possibility of such a takeover, and the value alignment problems resulting from a successful takeover.

AI First Strike Instabilities

An AGI singleton potentially conquering the world is also a threat of a unitary permanent military takeover, discussed in an earlier chapter. We live in a world where multiple militaries have nuclear weapon delivery capabilitie...
5. Uphold Voluntarism: Digital Defense
by Allison Duettmann — 2025-01-02T19:05:33.963Z

Security in the physical realm is an extraordinarily hard problem, but an inescapable one. Our civilization now also rests on software infrastructure. Security in the digital realm is now also an inescapable problem. The current software infrastructure is not just insecure, it is insecurable. Digital security is extraordinarily hard, but it is differently hard than physical security. Reasoning by analogy is fraught with peril. Let’s begin by explaining how these realms differ, and how to approach computer security.

Security in Digital vs. Physical Realms

Security in the digital realm differs in fundamental ways from security in the physical realm. But all security starts with defensible boundaries.

Physics does not let us build impenetrable walls. We can build stronger and stronger walls, but no matter how strong the wall, there is always an ever stronger force that can break through it. In the digital realm, perfect boundaries are cheap and plentiful. All modern CPUs support address space separation and containment of user-mode processes. Many programming languages support memory safety and object encapsulation. Modern cryptographic algorithms give us separation that seems close enough to perfect. The software built on these boundaries—operating systems, application software, cryptographic protocols—thus could be built securely.

In the physical realm, an attack is costly for the attacker. There is a marginal cost per victim, if nothing else, of the attacker’s attention. A good defense raises the marginal cost of attack. By contrast, software attacks typically have zero marginal cost per victim. Once malware works, its damage can be multiplied by billions using only the victim’s resources. Any vulnerable software system exposed to the outside world will eventually be attacked. We must build invulnerable systems when we can, and otherwise minimize the damage of a successful attack.

With perfect boundaries to build on, why is so much software so hopelessly insecure? The richness of modern software comes from composing specialized software building blocks written by others. These building blocks embody specialized domain knowledge. We compose them so they cooperate and bring more knowledge to bear on the task at hand. Boundaries alone only prevent involuntary interference. To enable cooperation, we poke holes in the boundaries. Without the right architect...
4. Uphold Voluntarism: Physical Defense
by Allison Duettmann — 2025-01-02T19:04:53.675Z

We started this book with the observation that, historically, violence has been declining, allowing us to make progress on a variety of factors that humans value. In the last chapter, we explored how to cooperate better to further advance such progress. Most progress will come from technological improvements. Biotechnology will unlock healthier lives for more people, while remote sensing and robotics will enable production at lower cost, using less energy, and creating less waste. Soon, we may be able to produce entirely new materials with molecular precision. We need technological progress to enable more valuing entities achieve what they value.

So far, so good. Now let’s look at the dark side. Technology is raising the stakes of our civilizational game, and could threaten its very existence. There are at least two traps that are important to steer away from:

Avoid Trap 1: ‘Small Kills All’ Via Technological Proliferation

Technology is becoming cheaper and more widely available. CRISPR developed from the biotechnology achievement of the decade to being used routinely by lab students. Democratization of powerful technologies also means a proliferation of dangers. Nuclear weapon development is a large undertaking employing special materials to make distinctive products. In contrast, the cost to chemically manufacture strands of DNA could soon be affordable to the point of printing targeted deadly proteins or cells at home. Further out, atomically precise manufacturing could allow ordinary humans employ ordinary materials to create entirely unexplored military applications.

The risks from increasingly widespread weaponizable technology are sometimes summarized under the dynamic of “small kills all”, i.e. fewer people can cause ever greater destruction. It may only be a question of time until one of the 7 billion players of Civilization seeks large-scale civilizational destruction and can acquire the technology to cause it. To understand the danger, let’s zoom in on a few risky technological dynamics from nukes, to robotic violence, to biotechnology and nanotechnology.

Dangerous Dynamics

Nuclear Weapons

Nukes are a great example of how we historically fared with massive risks for large-scale violence. The protections against an accidental nuclear launch were terrifyingly weak, while the degree of lying to the publ...
3. Improve Cooperation: Better Technologies
by Allison Duettmann — 2025-01-02T19:03:16.588Z

Having read the manual for civilization and analyzed previous rounds of the game, let's turn to how to make the most of future rounds in this chapter. Over the course of civilization, we developed rights, contracts and various other mechanisms and institutions to help us cooperate. But challenges remain: it is costly to search for other players to cooperate with, negotiate an agreement that is attractive to all sides, credibly commit to it, and enforce the terms once they are set. Fortunately, we can look to technologies to lower these costs and open up new levels of cooperation.
Note: This chapter deals with standard technologies for improving cooperation, while the later chapter Increase Intelligence focuses on how AI might impact cooperation.

Free Information

Legal systems emerged to establish neutral rule systems that support cooperation without violence. Governments and their rules are based on jurisdictions, yet many of us no longer know or care where the people we communicate with are. The internet not only makes it dramatically easier to find others to collaborate with but it also produces a permanent system of voluntary rules that is very costly to suppress.

Free speech is a great example. Governments imprison people for violating various national speech codes. Edward Snowden is in political asylum because he is rightly worried the US won’t grant him a fair trial for releasing information to the public. Despite this, all the servers storing the released information, and all the routers transmitting that information create an emergent phenomenon by which the information stays public.

It would require global totalitarianism to reliably and permanently remove information from public availability. In the absence of that, any single entity that tries to erase widely dispersed information renders itself irrelevant, or worse, achieves the opposite of their desire; the Barbara Streisand Effect.[1]

Prediction Markets

Free information is a foundational pillar of cooperation but today's information is often rather disorganized. Can we do better by supercharging how we make sense of it?Philosopher Karl Popper observed that knowledge, much as biology, evolves by a process of variation, replication, and selection. Variation of knowledge as in tossing new ideas out there, replication of knowledge as in spreading ideas, and selection...