trentmkelly/LessWrong-43k huggingface dataset:
Fractional Resignation
by jefftk — 2022-12-27T02:30:01.240Z

When you're playing games you generally want to spend your time
exploring interesting situations, which are typically ones where there
are a range of outcomes. If someone is definitely going to win it's
not usually much fun to keep going, and in many gaming cultures when
you get to this point it's polite to resign. The problem is, what do
you do when it's only very likely that someone will win?

In some games, this is still a very interesting area to explore, with
lots of interesting choices even for the losing player, but in others
(Risk, Go, etc) it would usually be better to stop and play something
else. In these cases, bargaining over fractional resignations can
work well.

For example, I might think I have an 80%
chance of losing, and if further play doesn't seem fun I could
offer my opponent 80% of a resignation. This is something that we
would think of as essentially equivalent to them winning four games
and me winning one game. If they think they actually have a 90%
chance of winning we could bargain and maybe end up at 85%, but
usually that isn't needed.

This does mean that more games end in resignations, but only in cases
where that's what everyone wants. It works especially well if you're
gaming with someone who doesn't like to resign if there's a chance,
but you're playing games where finishing them off can be just really
tedious.

In games with more than two players, negotiating fractional
resignation should still be possible, but I don't think I've ever done
it. It's more common for these games to have dynamics that keep it
interesting for everyone, however, so this is often less needed.

(Avoiding the need for this usually looks like keeping the possibility
of anyone winning relatively live until the very end, through some
combination of randomness, secrecy, and last-shall-be-first dynamics.
For example, when I play Modern Art we usually don't know who will win
until we count up the score because people's amounts of money are too
hard to keep track of, in Ticket to Ride you don't know what sort of
progress opponents are making towards their tickets, and in Power Grid
the "penalize the apparently leading player" dynamic is a strong
equalizer.)
What policies have most thoroughly crippled (otherwise-promising) industries or technologies?
by benwr — 2022-12-27T02:25:44.376Z

(cropped from an image by DALL-E 2)

In order to seriously consider promoting policies aimed at slowing down progress toward transformative AI, I want a better sense of the reference class of such policies.

What policies do you know of that have "done the most damage" to industry or progress in some restricted domain?
(optional) Exactly what did those policies "accomplish" and how? How would you measure their impact?
(optional) Was the crippling effect intentional on the part of the policymakers?
Recent advances in Natural Language Processing—Some Woolly speculations (2019 essay on semantics and language models)
by philosophybear — 2022-12-27T02:11:36.960Z

This essay was written back in 2019, not too long after GPT-2 came out. Although parts of it are dated- mostly the list of new achievements in Natural Language Processing- on the whole, it’s held up really well and outlines the core of my view on questions regarding semantics, philosophy of language, and natural language processing. I thought it was quite forgotten, but I saw recently that an essay by

Dragon God that mentioned the core of the idea in passing:

“Premise 1: Modelling is transitive. If X models Y and Y models Z, then X models Z.

Premise 2: Language models reality. "Dogs are mammals" occurs more frequently in text than "dogs are reptiles" because dogs are in actuality mammals and not reptiles. This statistical regularity in text corresponds to a feature of the real world. Language is thus a map (albeit flawed) of the external world.

Premise 3: GPT-3 models language. This is how it works to predict text.”

Which is the argument of the text- indeed almost the same words in parts. Perhaps the text is outdated now, but if so, I’d like ot think it’s because the ideas within it have entered the water.

I’m putting my existing work on AI on Less Wrong, and editing as I go, in preparation to publishing a collection of my works on AI in a free online volume. If this content interests you, you could always follow my Substack, it's free and also under the name Philosophy Bear.

1. Recent achievements in Natural Language Processing

[2022 Edit: Contemporary readers could skip this section, although it may be a useful reminder of how quickly things have moved since then.]

Natural Language Processing (NLP) per Wikipedia:

“Is a sub-field of linguistics, computer science, information engineering, and artificial intelligence concerned with the interactions between computers and human (natural) languages, in particular how to program computers to process and analyze large amounts of natural language data.”

The field has seen tremendous advances during the recent explosion of progress in machine learning techniques.

Here are some of its more impressive recent achievements:

A) The Winograd Schema is a test of common sense reasoning—easy for humans, but historically almost impossible for computers—which requires the test taker to indicate which noun an ambiguous pronoun stands for. The correct answer hinges on a single word, which is different between two separate versions of t...
Against Agents as an Approach to Aligned Transformative AI
by DragonGod — 2022-12-27T00:47:03.706Z

Epistemic Status

Written (very) quickly[1].

Thesis

The creation of superintelligent agents — even aligned ones — would with high probability entail a kind of existential catastrophe[2].

Introduction

Agents seem kind of bad. Even assuming we solve intent alignment[3] for agents, I don't know that I would be happy with the outcome.

I'm not convinced that my vision of a flourishing future is at all compatible with the creation of agentic superintelligences.

I've had many discussions about what the post singularity landscape would look like in a world where alignment succeeded, and I've not really been satisfied with the visions painted. It just doesn't seem to admit my values.

Why Do I Oppose Agents?

I strongly value autonomy and enhancing individual and group autonomy is essential to what I think a brighter future looks like. Superhuman (aligned) agents seems liable to hard break autonomy (as I conceive of it).

I think the crux of my disagreement with aligned superhuman agents is human obsoletion. In a world with strongly superhuman agents, humanity would quickly be rendered obsolete.

The joys of invention, the wonders of scientific discovery, the triumph of creation, many forms of self actualisation would be placed forever beyond our grasp. We'd be rendered irrelevant.

The idea of other minds accomplishing all of those on our behalf? It just isn't the same. It doesn't just matter that some accomplishments were made, but that I (or you, or others) made those accomplishments.

I do not want a corrigible intent aligned godlike nanny serving my every whim; I want to be a god myself goddammit.

When I try visualising the future in a world of aligned superhuman agents, I find the prospect so bleak. So hopeless, as if life has been robbed of all meaning.

"Our final invention" is not something I rejoice at, but something that grips me with despair. It would be an existential catastrophe in its own right.

What Do I Hope For?

My desired outcome for the AI project is to develop superhuman general intelligences that are not agents[4], that have no independent volition of their own. I want AI systems to amplify human cognition, not to replace it or supersede it.

I want humans to ultimately supply the volition. To retain autonomy and control over our future.

Agents Are Not Necessary?

This is not an impossible ask. Agents do not seem strictly necessary to attain AI systems...
Can we efficiently distinguish different mechanisms?
by paulfchristiano — 2022-12-27T00:20:01.728Z

(This post is an elaboration on “tractability of discrimination” as introduced in section III of Can we efficiently explain model behaviors? For an overview of the general plan this fits into, see Mechanistic anomaly detection and Finding gliders in the game of life.)

Background

We’d like to build AI systems that take complex actions to protect humans and maximize option value. Powerful predictive models may play an important role in such AI, either as part of a model-based planning algorithm or as part of a learned policy. Either way, this introduces the risk that we will select actions predicted to lead to good-looking outcomes without being able to distinguish predicted outcomes that are actually good from predicted outcomes where our measurements were corrupted. If tampering with sensors is easier than actually achieving our goals then we may inadvertently build very powerful systems taking creative actions to corrupt our measurements. If we iteratively improve and harden our measurements, this can lead to AI systems that work well for a long time before abruptly and catastrophically disempowering humanity.

I consider this one of the conceptually cleanest alignment problems, and I expect similar dynamics to play a role in realistic alignment failures even if those failures aren’t this simple. ARC’s current work is focused on decisive solutions to this problem, though it looks like the same approach may also apply directly to identifying treacherous turns more generally.

Are distinct mechanisms enough?

ARC has been looking for training strategies that avoid this problem by leveraging only the fact that sensor tampering is “weird,” i.e. conceptually distinct from the normal mechanism giving rise to predictions of good-looking outcomes on the training distribution.

More specifically, at training time our model predicts coherent sensor readings because it predicts that sensors reflect coherent structure in the world. But if someone tampers with sensors to show a convincing fiction, then the predicted observations are coherent because the fiction was designed to look coherent. This suggests that different mechanisms are responsible for (actions that lead to good-looking outcomes for the normal reasons) and (actions that lead to good-looking outcomes via sensor tampering). If we are able to detect that difference by looking at the internal behavior of a predictor, t...
Air-gapping evaluation and support
by Ryan Kidd — 2022-12-26T22:52:29.881Z

This blog post was written fast to communicate a concept I think is important. I may edit this post for legibility later.

I think evaluation and support mechanisms should be somewhat “

air-gapped

,” or isolated, in their information-gathering and decision-making processes. The incentives of optimal evaluators (to critique flaws) seem to run counter to the incentives of optimal supporters (to improve flaws). Individuals who might benefit from support may be discouraged from seeking it by fear of harsher evaluation if their private struggles are shared with evaluators. Evaluators who want to provide support may worry about compromising their evaluation ability if they make inconsistent exceptions. To optimally evaluate and support individuals, I believe that it is necessary to establish and declare appropriate information air gaps between different ecosystem roles.

Evaluation mechanisms, such as academic exams, job interviews, grant applications, and the peer review process, aim to critique an individual or their output. To be maximally effective, evaluation mechanisms should be somewhat adversarial to identify flaws and provide useful criticism. It is in the interests of evaluators to have access to all information about a candidate; however, it is not always in the candidate’s best interests to share all information that might affect the evaluation. It is also in the interests of evaluators for candidates to get access to all the support they need to improve.

If an attribute that disadvantages a job candidate (e.g., a disability) is protected by antidiscrimination law, an evaluator may be biased against the attribute either unconsciously or on the basis that it might genuinely reduce performance. Of course, evaluators should be required to ignore or overcome biases against protected attributes, but this “

patch

” may break or fail to convince candidates to divulge all evaluation-relevant information. Additionally, in the case that a candidate shares sensitive information with an evaluator, they might not have the appropriate resources or experience to provide beneficial support. Thus, an independent support role might benefit the interests of evaluators.

Support mechanisms, such as psychological counseling, legal support, and drug rehabilitation programs, aim to help individuals overcome their personal challenges, often to improve their chances a...
Slightly against aligning with neo-luddites
by Matthew Barnett — 2022-12-26T22:46:42.693Z

To summarize,

When considering whether to delay AI, the choice before us is not merely whether to accelerate or decelerate the technology. We can choose what type of regulations are adopted, and some options are much better than others.
Neo-luddites do not fundamentally share our concern about AI x-risk. Thus, their regulations will probably not, except by coincidence, be the type of regulations we should try to install.
Adopting the wrong AI regulations could lock us into a suboptimal regime that may be difficult or impossible to leave. So we should likely be careful not endorse a proposal because it's "better than nothing" unless it's also literally the only chance we get to regulate AI.
In particular, arbitrary data restrictions risk preventing researchers from having access to good data that might help with alignment, potentially outweighing the (arguably) positive effect of slowing down AI progress in general.

It appears we are in the midst of a new wave of neo-luddite sentiment.

Earlier this month, digital artists staged a mass protest against AI art on ArtStation. A few people are reportedly already getting together to hire a lobbyist to advocate more restrictive IP laws around AI generated content. And anecdotally, I've seen numerous large threads on Twitter in which people criticize the users and creators of AI art.

Personally, this sentiment disappoints me. While I sympathize with the artists who will lose their income, I'm not persuaded by the general argument. The value we could get from nearly free, personalized entertainment would be truly massive. In my opinion, it would be a shame if humanity never allowed that value to be unlocked, or restricted its proliferation severely.

I expect most LessWrong readers to agree with me on this point — that it is not worth sacrificing a technologically richer world just to protect workers from losing their income. Yet there is a related view that I have recently heard some of my friends endorse: that nonetheless, it is worth aligning with neo-luddites, incidentally, in order to slow down AI capabilities.

On the most basic level, I think this argument makes some sense. If aligning with neo-luddites simply means saying "I agree with delaying AI, but not for that reason" then I would not be very concerned. As it happens, I am sympathetic to many of the arguments in Katja Grace's recent post about delaying AI in order t...
Avoiding perpetual risk from TAI
by scasper — 2022-12-26T22:34:48.565Z

Stephen Casper,

scasper@mit.edu

. Thanks to Rose Hadshar and Daniel Dewey for some discussions and feedback.

The goal of this post is to sort through some questions involving the difficulty of avoiding perpetual risk from transformative AI. Feedback in the comments is welcome!

TL;DR

Getting AI to go well means that at some point in time, the acute period of risk posed by the onset of transformative AI must end. Ending that period will require establishing a regime for transformative AI that is exclusive, benevolent, stable, and successful at alignment. This post argues that this type of regime may be very difficult to establish and that this will largely be an AI governance problem. If so, this gives an argument for emphasizing work on AI safety challenges other than just alignment.

Intro

Often–at least in conversation–we talk about how we need “a solution” to AI safety. And maybe AI safety is a problem that has a once-and-for-all solution. If we can avoid doom, maybe we can use aligned AI to solve our key challenges, avoid all other X-risks, and set ourselves on a sustainable course for the future. This seems possible, and it would be great. Thinking in these terms stems back at least to Nick Bostrom’s Superintelligence. Bostrom discusses the possibility that in the same way that extinction is an attractor state, meeting our cosmic endowment may be as well. If we can get AI right just once, maybe that’s the last key challenge we need to solve.

From Superintelligence:

However, achieving immortality would mean a lot of things would have to go right. Establishing a regime that avoids perpetual risk will be complex, and it will definitely involve more than just figuring out how to align AI systems with our goals.

Five requirements for avoiding perpetual risk

Suppose that highly transformative AI (TAI) technologies are someday developed and that they are powerful enough to do catastrophically dangerous things. By definition I will refer to whatever set of institutions that control the TAI and could cause catastrophic risks with it as the TAI regime. The regime could consist of human institutions, AI institutions, or both:

TAI regime = whatever set of actors controls TAI

To avoid perpetual AI risk, five things need to be true about the TAI regime. And to the extent that any are false, the probability of perpetual risk will substantially incre...
Announcing: The Independent AI Safety Registry
by Shoshannah Tekofsky — 2022-12-26T21:22:18.381Z

Are you an independent AI Safety Researcher who would like to get in touch with other people in the same position? Fill out your location, work details, and collaboration interests, and you'll be added to a public google sheets file where Independent AI Safety Researchers can more easily get in touch with each other.

Here you can browse the entries so far. In a couple of months I'll lock the file and the form, cause there is no updating mechanism for old entries. If the concept of this registry is wildly successful, I'll look into a more persistent format at that point.

Enjoy!

Form - So You Can Join
File - So You Can Browse
Are men harder to help?
by braces — 2022-12-26T21:11:08.036Z

Of Boys and Men is a new book about how the outcomes of American men and boys are lagging in several ways. Most of it concerns aggregate trends that are irrefutable, such as the gender gap in college graduation. But books like these are often useful for testable hypotheses rather than reliable facts. Case in point: Reeves makes the odd claim that "men are hard to help":

[A] startling number of social programs seem to work well for girls and women, but not for boys and men — among which are a student-mentoring scheme in Fort Worth, Texas; a school-choice program in Charlotte, North Carolina; an income boost to low-wage earners in New York City; and many more.

The failure of these programs to help boys and men is a big problem, given that in many cases they are the ones who need the most help. But the problem rarely receives any attention, not least because almost nobody knows about it. (source)

I was startled when I read this because it seemed like such a textbook example of cherry-picking. Of the many studies out there we should probably have priors that the effects are about even on average. Reeves cites around eight individual evaluations that found larger benefits for females compared to males, but he never points to any attempt at aggregation.

The cited studies cover four broad areas:

Free college tuition with other supports (Kalamazoo Promise and Stay the Course at Tarrant County College in Forth Worth, Texas)
Pre-school programs (a study of the Abecedarian, Perry, and the early Training Project and one of Project READS in North Carolina)
Mentoring programs and boarding schools (just mentions that they are in New Hampshire, Baltimore, and Washington, D.C.)
Wage subsidies (the Paycheck Plus pilot)

These all find the effect difference that Reeves claims to have discovered. The problem is that “men are harder to help” doesn’t seem to be true in general. For each of the four areas, I looked for meta-analyses or similar studies that estimated gender differences in the treatment effects:

Free college tuition: a rigorous trial of the Susan Thompson Buffett Foundation, which gives large grants to Nebraska high schoolers planning to attend public colleges in the state, finds if anything larger impacts on men compared to women. A meta-analysis discussing this question finds a mix favoring men, women, or neither and concludes that there's not enough evidence to say.
Pre-scho

...
How much should I update on the fact that my dentist is named Dennis?
by MichaelDickens — 2022-12-26T19:11:07.918Z

According to some psychology research (eg Pelham, Mirenberg, and Jones, people are slightly more likely to choose occupations that sound like their names: "people named Dennis or Denise are overrepresented among
dentists". Other research (eg Simonsohn (2011)) claims this result is spurious. I haven't read these papers and I probably don't have enough expertise to judge them, so my prior on this hypothesis is somewhere around 50%.

But my own dentist is named Dennis.

Dennis isn't that common of a name. Should this be a strong update in favor of the hypothesis? Or does it not matter that much?

The Pelham paper found that Dennises were only about 1% more likely than the base rate to go into dentistry, so even if the hypothesis is true, it's improbable that my dentist would be named Dennis. So perhaps I should believe that the true effect size is even larger than what Pelham found?