trentmkelly/LessWrong-43k huggingface dataset:
Sanity-checking in an age of hyperbole
by Ciprian Elliu Ivanof — 2022-10-31T20:04:44.911Z

I grew up in a range of environments including a high-crime and poor neighborhood. It was the sort of place with a lots of impulsive crime such as fights and a stabbing (and arson incident) driven by personal animosity. Suffice to say, I consider the reality of crime to be obvious and not inherently tied to wealth (for I have seen much poorer people behave better) and tragic in its consequences.

Rather than describe what I witnessed and my factual assessments, it is clear I tend to focus on the wording of a directive, not some unknowable imputation. When trying to find out what the lethally enforced social norms are, I don't just look at the legal system, I look to human behavior such as seeming tolerance for physical attacks before anybody might bother reporting it to the police.

I know I struggle in figuring out directives from peers since I may assume a wider range of expected behavior (talking about a stick-up seemed to shock some classmates). When broadly worded directives get issued in the imperative such as "always be tolerant" or "never call the cops on a Black person", I assume the scope of the wording applies, not some unspoken restrictive clause or that "always" means "generally" and "never" means "generally not".

The sense of being in a double-bind has not been improved since the broad wording issued in the wake of the murder of George Floyd.

I may or may not be rational in how I process the wording of the directives but what can be done? Importing alleged goals seems like a recipe for wishful thinking or utterly disregarding the expressed wishes of people (indeed, "imposing your values on other people is specifically contraindicated). Being cynical and saying "people don't understand what they are demanding" seems to dehumanize people clearly capable of reason. When facts about specific incidents, it seems like dangerous mind-reading to assume that they are unfamiliar with specific incidents they are complaining about (like complaining the police shot someone in the middle of a stabbing but took the surrender of someone with his hands up).

The path in between seems dangerously like laughing manically and screaming "I know better than everybody else". While in the past, one could perhaps assert a degree of selectivity in criticism and some attention to the words used, that may not be possible here of the norms of communication have ceased to view the ...
Why Aren't There More Schelling Holidays?
by johnswentworth — 2022-10-31T19:31:55.964Z

A common pattern when working in teams: when one person is out on vacation, there’s a disproportionate drop in productivity for the team as a whole. Lots of things end up blocked on the person who’s out. For instance, in a small software team, maybe the developer who owns a particular API is out, and nobody else knows that API well enough to confidently make changes, so anything involving changes in that API ends up blocked until the owner is back. Or, even if someone else steps in to handle changes to the API, they’re much more likely to introduce bugs.

To some extent, we can structure teams to mitigate that kind of problem. “Everyone does everything” is very costly, but underrated. Weaker versions like “At least three people can cover any given thing” are also costly, but underutilized.

But there’s a much less costly strategy which gains a decent chunk of the same benefits: coordinate vacation. If everyone goes on vacation 3 weeks per year on the same 3 weeks, then that’s only 3 weeks of nonproductivity for the team; the other 49 weeks are full steam. If everyone on a 10-person team goes on vacation 3 weeks per year at different times, then that’s 30 weeks per year of disproportionate productivity-loss; the team is at full steam less than half the time.

Thus the case for national holidays: it’s not like everyone is required to get the day off (at least in the US), but it’s a Schelling point for lots of people to take a vacation at the same time. People get their much-needed recovery time simultaneously, so teams can spend more time working at full capacity.

But in practice, most people take a lot more vacation days than there are national holidays - which suggests that the current number of holidays is too small.

It seems like companies (or nonprofit orgs) could profit by filling that gap at the company level: declare official “company holiday” weeks, and offer incentives for employees to take their vacations during those weeks. Unlike national holidays, the company holidays won’t necessarily line up neatly with good vacation times for spouses/children/parents/friends. But on the upside, since a company holiday need not be at the same time as the rest of the country, traveling should be easier.

Expanding from there, one could imagine a few companies/orgs which need to interface with each other a lot coordinating their Schelling holidays. For instance, when ...
The circular problem of epistemic irresponsibility
by Roman Leventov — 2022-10-31T17:23:50.719Z

Problem statement

The problem of epistemic irresponsibility, as I will call it in this post, is that people sometimes agree that their reasoning (and hence models that they hold and use as the basis of their action) is irrational or illogical. For example, people knowingly use theories that have already been refuted, such as astrology. Yet, they fail to act on this knowledge and don’t update their models and their action plan accordingly.

A “softer” version of this problem arises when people agree that they don’t know if their reasoning is rational, either because they didn’t seek any evidence or didn’t try to come up with any arguments refuting their reasoning or models, or because they don’t know how to know if their theory is true or false, i. e., don’t have the necessary epistemological apparatus (though, these people would seldom think in these terms). Then, people fail to act on this knowledge. The examples are people’s uncritical perception of news in the media or claims made by politicians, and, to some degree, the unsound scientific practices, such as those described in Richard Feynman’s “Cargo Cult Science”.

The problem of epistemic irresponsibility, or, rather, its inverse, epistemic responsibility, belongs to the discipline of normative logic, i. e. the “pre-Frege” logic, as John Sowa highlighted here:

The three kinds of value judgments are Beauty, Goodness, and Truth. They determine the three kinds of normative science: Aesthetics, Ethics, and Normative logic. Peirce equated normative logic with logic as semiotic. But all sciences, including the normative sciences, depend on mathematics and mathematical logic (AKA formal logic).

All empirical sciences, including the normative sciences, depend on phenomenology for the analysis and interpretation of perception. The three parts of normative logic (AKA logic as semiotic) are (1) Critic, which is formal logic; (2) Grammar; and (3) Methodeutic, which is Peirce's name for the methodology of science.

All these issues were discussed and analyzed in detail by Aristotle, debated for centuries by the Greeks, Romans, and Arabs, and developed to a high level of sophistication by the medieval Scholastics. The books called "logic'' from the 13th to the 19th centuries discussed all these issues. But the 20th c. logicians ignored all but the formal logic. They did a lot of good work on logic, but they also lost a great

...
AI as a Civilizational Risk Part 3/6: Anti-economy and Signal Pollution
by PashaKamyshev — 2022-10-31T17:03:00.179Z

Any social problem in your society can and will be made worse by adding AI. Just because we tolerate a problem without society collapsing does not mean we can tolerate an AI-enhanced version of it.

Spam as an example of Anti-Economy

We can talk about the problems of behavioral modification in economic terms. Propaganda and engagement AIs are centralizing benefits and distributing costs, a well-known economic issue of externalities. The tools we have created do not price in those externalities. The externalities may be so significant that large companies are potentially net negative for humanity if they use too many behavior modification AIs. I like to call this the anti-economy. A new term highlights that the magnitude of distributed costs has become so high as to create its own "externality economy."

The most obvious example of the anti-economy is spam. Spam and its related issue of scams use a lot of automated tools. Many of them are not AIs yet, but they will become AI in a reasonably short time frame. They centralize a tiny benefit to their creators, usually through illegal or borderline illegal means, and they destroy a lot of people's valuable attention and time, thus distributing massive amounts of cost. The entire mediums of phone calls and paper mail have significantly underperformed compared to their potential utility due to a large portion of unrecognized phone calls being spam.

Spam phone calls can have deadly consequences

.

Anti-economy of online bots, scams, and spam is a clear net negative for humanity. In addition, many legitimate or quasi-legitimate businesses resort to spam or scam-like tactics.

Hidden service costs

and bot-driven fake reviews create a wrong perception of product quality. The economy intersects with the anti-economy in that something might not be a total scam and be a product. However, because this product's creator has solicited some narrow AI's help to write fake reviews, it is crossing over into borderline scam territory. The fake reviews, articles, and news are drowning signals of actual knowledge with noise.

They are parodied here

. Signal pollution is reducing real economic output, and it is potentially destroying actual physical and mental health.

These spam algorithms will increasingly utilize GPT-like AIs, thus imposing further costs ...
Average utilitarianism is non-local
by Yair Halberstadt — 2022-10-31T16:36:09.406Z

This is a pretty obvious point, which is sort of made here but I haven't seen anyone lay out explicitly.

Under average utilitarianism the morality of having a child depends on whether a billion light years away there's any sentient aliens, how many there are, and whether their average happiness is greater or less than your child's would be. This despite your actions having zero impact on them, and vice versa.

This, far more than the sadistic conclusion, seems to me to sound the death knell of average utilitarianism. If your formalisation of utilitarianism both completely diverges from intuition in a straightforward situation (should I have kids), and is incalculable not just practically, but even in theory (it requires knowing the contents of the whole universe), what's the point?
Marvel Snap: Phase 1
by Zvi — 2022-10-31T15:20:00.442Z

Usual game approach. I’ll start off spoiler-free, then proceed to the semi-spoiled section, then to the Here Be Spoilers section.

Maximally Spoiler-Free (1 Bit of Info): Should You Play?

No.

Spoiler-Free (3 Bits of Info): What Tier Is It?

Tier 3. It is a good game.

I want to be able to put the game into Tier 2 despite its severe problems. I can’t.

I can see this becoming a Tier 2+ game over time if the outer loops are improved.

Spoiler-Minimized Overview (Significant Bits)

Marvel Snap is a beautiful mess.

There are things I and everyone else I talk about it with truly love about it.

There are other things about it I, and it seems pretty much everyone else as well, wish would die in a fire.

The thing to love about the game is its core gameplay. The inner loop.

Marvel Snap games are elegant, fun, engaging and involve interesting decisions, while also being super duper fast.

Decks are twelve unique cards. Games last at most six turns. You start with three cards and draw one each turn. Each game there are three locations, each with an ability, which get revealed one per turn for the first three turns. On turn X, you get X energy to spend on cards, which you can deploy at one of the three locations, maximum four cards per location per side. Cards do things, and provide power. Whoever has more power in two out of three sites wins.

The central twist, the ‘snap,’ is the doubling cube. You play games for cubes. At first the stakes are one cube. Each player can snap once per game, doubling that to two once the current turn ends. Stakes automatically double after turn six ends, as well, max stakes is 8 cubes. At any time, you can retreat and only lose current stakes. The decisions on when to double, including when your situation is ‘too good’ to double, and when to decline a double, get fascinating once your opponents are not rocks.

That’s the whole game. All of that is great, at the early card mix is chosen very well to ensure interesting game play. Giving everyone the same starting cards keeps it fair, at least initially.

Also, the Marvel flavor is light, but parts of it are pretty damn cool.

The thing to hate about the game is its monetization models and incentive structures designed to keep you playing. The outer loop.

This is a Gotcha Gacha Gotcha game (can we make this name change happen?). Your core outer loop experience, to the extent you pay attentio...
Boundaries vs Frames
by Scott Garrabrant — 2022-10-31T15:14:37.312Z

This post is partially in response to Critch's boundaries sequence. My best guess is that he would agree with most of it in theory, but disagree with some of it in practice due to tradeoffs with other considerations in defining the concepts.

Boundaries and Frames

Imagine a world consisting of some atomic objects R={r0,r1,…}. For example, you can think of the ri as physical atoms or cells in the game of life. Each object comes along with a collection of states it can be in ri={s0i,s1i,…}.

(I am not committing now on whether or not we are thinking about our model as timeless. Maybe ri should be thought about as a cell in the game of life that passes through time, or maybe it should be thought about as a (cell, time) pair.)

There are two sets we naturally want to associate with our world. First, we have R={r0,r1,…}, which I will call the object space. Second, we have S=Πr∈R r, which is the set of all ways to assign state to each object in R. I will call this the state space. Note that partitions of R correspond to factorizations of S.

If I want to point at (for example) an agent in R, I might tell you what atoms are inside that agent, and thus express R in the form R=a⊔e, where a is the set of all atoms that are in the agent and e is the set of all atoms that are outside of the agent (and thus in the environment).

The agent then has its own state space A=Πr∈a r, while the environment has its own state space, E=∏r∈e r. Now, to point at this agent in S, I can express S in the form S=A×E.

Instead of specifying a list of objects in the agent, I specify the state space of the agent. Instead of thinking of the environment as the result of subtracting the agent object out of the world, I think of the environment as the result of quotienting out the agent's state space from the world's state space.

Instead of defining a Cartesian Boundary, I am defining a Cartesian Frame.

Playing on this naming scheme, in general (when not necessarily talking about agents), I will use the word boundary when talking about partitioning the object space into a disjoint inside and outside, and I will use the word frame when talking about factoring the state space into an independent inside and outside. (This naming might end up only being...
Embedding safety in ML development
by zeshen — 2022-10-31T12:27:13.055Z

This post was written as part of Refine. Thanks to Adam Shimi, Alexander Gietelink Oldenziel, and Vanessa Kosoy for helpful discussion and feedback.

Summary

This post aims to:

Advocate for embedding safety into development of machine learning models
Propose a framing on how to think about safety, where three factors contribute to an AI being dangerous, namely misalignment, optimization, and influence
Discuss the pros and cons of this framing

None of the ideas in this post are novel, and some of them may not even be practical, but they may be a useful framing to the problem of AI safety.

Introduction

Motivations

There are many ways to get to AGI. It could be

recursively self improving

AI, or something more

brain-like

, or something that comes from improvements in machine learning systems, or something else much more alien to us. I don’t know which pathways are the ones our world will eventually end up in, but given recent rapid advances in ML recently, in this post I will address the scenario where the first dangerous AGI is achieved by mostly scaling up and improving upon present-day ML systems.

In the absence of a robust solution to the

hard problem

of alignment, I think we should at least try to make the development of machine learning models go safely for as long as possible. This may be especially valuable if the first AGI we get is

not yet an all-powerful superintelligent

. It might be a fairly weak AGI capable of inflicting a lot of damage, but the damage could have been prevented. Not having a solution to the hard problem should not mean getting ourselves into preventable disasters.

What safety means in this context

Most safety fields in the real world don’t solve safety with guarantees. Whether it is flying an aircraft, operating a nuclear plant, or constructing a building, safety is all about

effective risk management

. This framing of safety has not been commonly used in the field of AI safety, for very good reasons, as the problem of a misaligned AGI is an

unbounded

one, where a ‘safety failure’ with a strong form of AGI could mean human extinction even with just a tiny degree of misalignment.

For the purpose of this post, I am using the word “safety” in a more narrow sense than most people would. In terms of time-scale, I refer to “we are safe for now” rather than

“things go great in the long run”

; while...
[Book] Interpretable Machine Learning: A Guide for Making Black Box Models Explainable
by Esben Kran — 2022-10-31T11:38:45.404Z
Link: https://christophm.github.io/interpretable-ml-book/

This is a book by Christoph Molnar about interpretability. It covers a lot of traditional, non AI safety interpretability. Some recommendations for chapters to read are in an excerpt from our summary of it in the interpretability starter resources:

An amazing and up-to-date introduction to traditional interpretability research. We recommend reading the taxonomy of interpretability and about the specific methods of PDP, ALE, ICE, LIME, Shapley values, and SHAP . Also read the chapter on neural network interpretation such as saliency maps and adversarial examples. He has also published a "Common pitfalls of interpretability" post (and its paper) that is recommended reading.

Why share it here? It is a continuously-updated, high quality book on interpretability as seen from outside AI safety which seems very relevant to understanding what the field in general looks like. It has 78 authors and seems very canon. The first edition is from Apr 11, 2019 while the second edition (latest version) is from Mar 04, 2022. It was last updated on the Oct 13, 2022.

From the introduction of the book:

Machine learning has great potential for improving products, processes and research. But computers usually do not explain their predictions which is a barrier to the adoption of machine learning. This book is about making machine learning models and their decisions interpretable.

After exploring the concepts of interpretability, you will learn about simple, interpretable models such as decision trees, decision rules and linear regression. The focus of the book is on model-agnostic methods for interpreting black box models such as feature importance and accumulated local effects, and explaining individual predictions with Shapley values and LIME. In addition, the book presents methods specific to deep neural networks.

All interpretation methods are explained in depth and discussed critically. How do they work under the hood? What are their strengths and weaknesses? How can their outputs be interpreted? This book will enable you to select and correctly apply the interpretation method that is most suitable for your machine learning project. Reading the book is recommended for machine learning practitioners, data scientists, statisticians, and anyone else interested in making machine learning models interpretable.
My (naive) take on Risks from Learned Optimization
by Artyom Karpov — 2022-10-31T10:59:39.727Z

This is my current understanding of Risks from Learned Optimization. I'm pretty sure there are mistakes here and there. I also list things I agree or not with in the paper as I think it might be too large for one comment. So feel free to point out where I'm wrong (and why)!

Parent optimization process might intentionally or unintentionally create another child optimization process. It's more likely if a problem couldn't be fully solved by a parent optimizer, there are too many policies to create; the smaller parent the more likelihood of child optimizer appearance and vice versa; the parent is biased towards simplicity. Less likely if the parent has less state or not enough power to find a child. Examples of child optimization are evolution and human brain, teacher and student, society and corporation, programmer and a program that searches solutions to NP problem, and so on.

A child is likely to have goals different to parent's goals when it is free from parent's control or deployed in another environment. In that case a child's goals might be totally different from parent's ones, or be near them, or be not optimal. If a child's objective is totally different from the parent's then it might nevertheless increase the parent's objective if it doesn't interfere with its own goals or if it can pursue its own goals afterwards. It is unclear how to match a child's objective, which is indirectly learned, to the parent's one (inner alignment). Especially risky is a child's deception that happens if the child can project or plan many steps ahead, knows the parent’s objective, and there is a threat of shutting down or modification. Deception is more likely as we create objective proxies, or not train well before deployment.

It is likely that without child optimization systems will be much less useful but with this optimization in place the existential risk from AI increases.

1.

In the glossary:

An optimizer is a system that internally searches through some space...

But shouldn't it be a process and not a system? Shouldn't it be only some computational process? To my mind the optimizer in this work is a computational process because it does a calculation to find an optimal solution. Still it is arguable if evolution is a computational process but I think it is in accordance with the Church-Turing thesis as optimization is a close ended and physical process. In that sen...