trentmkelly/LessWrong-43k huggingface dataset:
On "aiming for convergence on truth"
by gjm — 2023-04-11T18:19:18.086Z

Background

Duncan Sabien wrote a list of proposed "basics of rational discourse" guidelines. Zack M Davis disagrees with (his interpretation of) one of the guidelines. I think the question is interesting and don't feel that those two posts and their comments resolve it. (Spoiler: I largely agree with Duncan on this.)

So, Duncan says that we should

aim for convergence on truth, and behave as if your interlocutors are also aiming for convergence on truth

and that we should care whether an interlocutor is

present in good faith and genuinely trying to cooperate.

Zack, on the other hand, holds (as I understand him) that

we should aim for truth and not particularly care about convergence
in cases where we have something valuable to contribute, it should be precisely because there is divergence between what others think and our view of the truth
a good frame for thinking about this is that debates are like prediction markets
it shouldn't matter much whether someone else is "in good faith" or "trying to cooperate"; the merits and defects of their arguments are whatever they are, regardless of intentions, and we should be responding to those
talk of "cooperation" and "defection" implies a Prisoner's Dilemma situation, and attempted rational discourse doesn't generally have the properties of a Prisoner's Dilemma.

I will argue

that we should aim for truth rather than for convergence as such
that I think Zack has misunderstood what Duncan means by aiming for convergence
that what Duncan means by "aiming for convergence" is a reasonable goal
that debates are importantly unlike prediction markets (and even prediction markets don't necessarily have quite the incentives Zack assumes)
that there are specific forms of not-aiming-for-convergence that are more common than Zack's "aiming for divergence"

and they are usually harmful when they occur
and when they do, it often becomes necessary to adopt modes of discourse that attend explicitly to status, politeness, intentions, etc., which makes everything worse
and this failure mode is made less likely if (as Duncan proposes) we avoid those forms of not-aiming-for-convergence and assume until we get strong contrary evidence that others are also avoiding them

that while in principle we can just respond to others' arguments and ignore their (actual or perceived) intentions, in practice we can't and shouldn't
that informal talk of cooperation and d

...
In favor of accelerating problems you're trying to solve
by Christopher King — 2023-04-11T18:15:07.061Z

John von Neumann, a renowned Hungarian-American mathematician and physicist, played a critical role in the Manhattan Project, the top-secret research effort during World War II that led to the development of the first atomic bombs. As a key contributor, he provided important insights into the mathematical modeling of nuclear chain reactions, which were instrumental in the design and construction of the weapons. After the war, von Neumann continued to shape nuclear deterrence policy, advocating for a strategy of mutually assured destruction (MAD) to prevent large-scale conflict. By emphasizing the catastrophic consequences of a full-scale nuclear exchange, MAD established a balance of power that, in turn, helped avert the existential risk of nuclear war. Von Neumann's early research and development of primitive nuclear weapons thus contributed significantly to global stability in the face of an unprecedented threat.

Don't accelerate problems you're trying to solve argues for the relatively intuitive notion that we shouldn't accelerate problems we are trying to solve. In this post, I will argue in favor of acceleration, and explain how to do it properly.

"Don't accelerate" seems like a default and conservative option. I think this causes people to fail to use a security mindset when thinking about it, even when they normally pretty good at it.

However, it doesn't take much creativity to see potential catastophic risks from this strategy. We can just take examples from history:

Alcohol prohibition in the US resulting in a large increase in organized crime.
"The Smokey Bear Effect": fighting fires too aggressively leads to an increase in risk of extremely large fires.
Eliezer Yudkowsky causing various AGI firms to get founded and funded.

So we can't automatically treat "Don't accelerate" as the safe option.

The key, I think, is differential acceleration. Society is a complex adaptive system. The key metric is how much your accelerating causes society to adapt in favor of AI safety.

First I will give a minimal toy example, and then I will give a recent real life example that is very strong: stable diffusion. I will also point out OpenAI's short comings.

I will also gives examples of respected roles in society that involve making society adapt, and lessons we can draw from them.

Toy Example: A pendulum, two buttons, and a 10 minute timer

There is a large and heavy pendu...
[Interview w/ Jeffrey Ladish] Applying the 'security mindset' to AI and x-risk 
by fowlertm — 2023-04-11T18:14:34.059Z

Though I've been following the AI safety debate for a decade or so, I've had relatively few conversations with the relevant experts on the Futurati Podcast.

Having updated with the release of GPT-4, however, I'm working to change that.

I recently had a chance to sit down with Jeffrey Ladish to talk about global catastrophic risk, the economic incentives around building goal-directed systems, fragile values, the prospects of being able to predict discontinuities in ability, how far scaling can take us, and more.

Though I imagine most of this will be review for the LW crowd, if you think there's anyone else who would enjoy the conversation, consider sharing it. I'd like to devote more time to AI Safety and x-risk, but I won't do that unless I can see that people are getting value out of it (and I operationalize 'people getting value out of it' with view counts and tweets.)
Request to AGI organizations: Share your views on pausing AI progress
by Orpheus16 — 2023-04-11T17:30:46.707Z

A few observations from the last few weeks:

On March 22, FLI published an open letter calling for a six-month moratorium on frontier AI progress.
On March 29, Eliezer Yudkowsky published a piece in TIME calling for an indefinite moratorium.
To our knowledge, none of the top AI organizations (OpenAI, DeepMind, Anthropic) have released a statement responding to these pieces.

We offer a request to AGI organizations: Determine what you think about these requests for an AI pause (possibly with uncertainties acknowledged), write up your beliefs in some form, and publicly announce your position.

We believe statements from labs could improve discourse, coordination, and transparency on this important and timely topic.

Discourse: We believe labs are well-positioned to contribute to dialogue around whether (or how) to slow AI progress, making it more likely for society to reach true and useful positions.

Coordination: Statements from labs could make coordination more likely. For example, lab A could say “we would support a pause under X conditions with Y implementation details”. Alternatively, lab B could say “we would be willing to pause if lab C agreed to Z conditions.”

Transparency: Transparency helps others build accurate models of labs, their trustworthiness, and their future actions. This is especially important for labs that seek to receive support from specific communities, policymakers, or the general public. You have an opportunity to show the world how you reason about one of the most important safety-relevant topics.

We would be especially excited about statements that are written or endorsed by lab leadership. We would also be excited to see labs encourage employees to share their (personal) views on the requests for moratoriums.

Sometimes, silence is the best strategy. There may be attempts at coordination that are less likely to succeed if people transparently share their worldviews. If this is the case, we request that AI organizations make this clear (example: "We have decided to avoid issuing public statements about X for now, as we work on Y. We hope to provide an update within Z weeks.")

At the time of this post, the FLI letter has been signed by 7 DeepMind research scientists/engineers, probably 0 OpenAI research scientists and 0 Anthropic employees.

See also:

Let's think about slowing down AI
A challenge for AGI organizations

...
Core of AI projections from first principles: Attempt 1
by tailcalled — 2023-04-11T17:24:27.686Z

I have at various times commented on AI projections and AI risk, using my own models of the key issues. However I have not really crystallized any overview of what my models imply, partly because I am uncertain about some key issues, but also partly because I just haven't sat down to work through the full implications of my views.

For the next while, I plan to make a post like this every other week, where I try to describe the core of the issues, and open up for debate, so I can hopefully crystallize something more solid.

Currently, in order to get pretty much anything[1] done, the most effective method is to get a human to handle it, either by having them do it directly, or having them maintain/organize human-made systems[2] which do it. This is because humans contain reasonably high-fidelity sensors for the most commonly useful modalities[3], flexible and powerful information-processing and decision-making capabilities, flexible actuators[4], and comprehensive self-repair/self-maintenance mechanisms.

These features together make humans capable of contributing to pretty much any task we can solve, and there's nothing else with these features that can compete with humans. I believe humans are on the (quality, flexibility) Pareto frontier in terms of actuators and information-processing/decision-making[5], but that human-made technology exceeds humans for a lot of different sensor types.

Most of the marginal value seems to come from the information-processing and decision-making, aka intelligence[6]. This is partly because coordination is very scarce. But it is also because interfaces are very scarce (in particular while sensors and actuators might seem to have a lot of independent value, non-human sensors and actuators have much room for improvement, which more intelligence applied to technical advancements could achieve, so intelligence "picks up" value from the value of actuators and sensors).

Eventually we will get superhuman artificial intelligence.[7] Some people say that will go badly for humanity, but let's start with the main (and only?) analysis for why it will go well for humanity. The econ 101 analysis is that the price of labor will massively decrease compared to the price of capital, and that humanity will therefore switch from living off our labor to living off rents from capital.[8] The core of the AI debate is then whether this...
What Jason has been reading, April 2023
by jasoncrawford — 2023-04-11T16:29:28.796Z
Link: https://rootsofprogress.org/reading-2023-04

A monthly feature. Note that I generally don’t include very recent writing here, such as the latest blog posts (for those, see my Twitter digests); this is for my deeper research.

AI

First, various historical perspectives on AI, many of which were quite prescient:

Alan Turing, “Intelligent Machinery, A Heretical Theory” (1951). A short, informal paper, published posthumously. Turing anticipates the field of machine learning, speculating on computers that “learn by experience”, through a process of “education” (which we now call “training”). This line could describe current LLMs:

They will make mistakes at times, and at times they may make new and very interesting statements, and on the whole the output of them will be worth attention to the same sort of extent as the output of a human mind.

Like many authors who came before and after him, Turing speculates on the machines eventually replacing us:

… it seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers. There would be no question of the machines dying, and they would be able to converse with each other to sharpen their wits. At some stage therefore we should have to expect the machines to take control, in the way that is mentioned in Samuel Butler’s Erewhon.

(I excerpted Butler’s “Darwin Among the Machines” in last month’s reading update.)

Irving John Good, “Speculations Concerning the First Ultraintelligent Machine” (1965). Good defines an “ultraintelligent machine” as “a machine that can far surpass all the intellectual activities of any man however clever,” roughly our current definition of “superintelligence.” He anticipated that machine intelligence could be achieved through artificial neural networks. He foresaw that such machines would need language ability, and that they could generate prose and even poetry.

Like Turing and others, Good thinks that such machines would replace us, especially since he foresees the possibility of recursive self-improvement:

… an ultra-intelligent machine could design even better machines; there would then unquestionably be an “intelligence explosion,” and the intelligence of man would be left far behind…. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.

(See also Verner Vinge on the Singularity,...
Contra-Berkeley
by False Name — 2023-04-11T16:06:36.286Z

Abstract: This is mostly trivia; but, reference to it is made elsewhere, so may as good put it up. Besides, it's oblique support of "rationality" against the converse.

Bishop George Berkeley's supposedly incontrovertible idealism, can be undone as follows:

Berkeley maintains, in essence, that all human perceptions are subsets of the perceptions of the deity, by whose will all perceptions are effectuated, and comport with one another to create the phenomenal world. And, the benevolence and omnipotence of the deity taken as given, perceptions can be relied upon as reality of that phenomenal world, forever after.

This scheme is objected to as, per the admission of the unrestricted Law of the Excluded Middle, either the deity perceives itself, or it does not.

But, if the deity perceives itself, in sensory fashion, yet humans do not. That being so, the lack, is by assumption from the will of the divine, that it be not perceived. However, then, by will also, the deity could at-once deprive of human perception any-thing; that is, it could exclude from perception anything it wishes, at its will.

This being so, however, the deity surely cannot be relied upon to maintain the phenomenal world inviolate, since, rather, it might be altered in any way whatever for reasons humans cannot ken, and, if thoughts make up our minds, and thoughts derived from imitation or impression of sense-objects, our minds cannot be relied upon to subsist for any term of time. And, that being so, it cannot be guaranteed that Berkeley's mind, nor any other, can be relied upon to reason, or to have the reason sustain. So that, as guarantor of the phenomenal world, the deity cannot be relied upon so to sustain (nor the arguments of Berkeley's thoughts subsist; that they do and will ever-after implies, the deity cannot alter them, contrary to the assumption of its omnipotence).

Converse proposition, that the deity cannot perceive itself. However, in this case, there are limits to the deity’s perceptions, and so, whatever may subsist in even the phenomenal world, deity cannot be relied upon to perceive, nor to transmit such perceptions to humankind - so that, again, the deity cannot be relied upon to furnish for humans the unerring perceptual data of the phenomenal world.

Amusingly, David Hume's skepticism as to sensory data - essentially, agreeing with Berkeley that our sense data is furnished for...
[MLSN #9] Verifying large training runs, security risks from LLM access to APIs, why natural selection may favor AIs over humans
by Dan H — 2023-04-11T16:03:31.061Z
Link: https://newsletter.mlsafety.org/p/ml-safety-newsletter-9

As part of a larger community building effort, CAIS is writing a safety newsletter that is designed to cover empirical safety research and be palatable to the broader machine learning research community. You can subscribe here or follow the newsletter on twitter here. We also have a new non-technical newsletter here.

Welcome to the 9th issue of the ML Safety Newsletter by the Center for AI Safety. In this edition, we cover:

Inspecting how language model predictions change across layers

A new benchmark for assessing tradeoffs between reward and morality

Improving adversarial robustness in NLP through prompting

A proposal for a mechanism to monitor and verify large training runs

Security threats posed by providing language models with access to external services

Why natural selection may favor AIs over humans

And much more...

We have a new safety newsletter. It’s more frequent, covers developments beyond technical papers, and is written for a broader audience.

Check it out here:

AI Safety Newsletter

.

Monitoring

Eliciting Latent Predictions from Transformers with the Tuned Lens

This figure compares the paper’s contribution, the tuned lens, with the “logit lens” (top) for GPT-Neo-2.7B. Each cell shows the top-1 token predicted by the model at the given layer and token index.

Despite incredible progress in language model capabilities in recent years, we still know very little about the inner workings of those models or how they arrive at their outputs. This paper builds on previous findings to determine how a language model’s predictions for the next token change across layers. The paper introduces a method called the tuned lens, which fits an affine transformation to the outputs of intermediate Transformer hidden layers, and then passes the result to the final unembedding matrix. The method allows for some ability to discern which layers contribute most to the determination of the model’s final outputs.

[

Link

]

Other Monitoring News

[

Link

] OOD detection can be improved by projecting features into two subspaces - one where in-distribution classes are maximally separated, and another where they are clustered.

[

Link

] This paper finds that there are relatively low-cost ways of poisoning large-scale datasets, potentially compromising the security of models trained with them.

Alignment

The Machiavelli Benchmark: Trade Offs Between Rewards and Ethical Behavior

General-purpose models l...
Where's the foom?
by Fergus Fettes — 2023-04-11T15:50:43.461Z

"The first catastrophe mechanism seriously considered seems to have been the possibility, raised in the 1940s at Los Alamos before the first atomic bomb tests, that fission or fusion bombs might ignite the atmosphere or oceans in an unstoppable chain reaction."[1]

This is not our first rodeo. We have done risk assessments before. The best reference-class examples I could find were the bomb, vacuum decay, killer strangelets, and LHC black holes (all covered in [1]).

I was looking for a few days, but didn't complete my search, but I decided to publish this note as now Tyler Cowen is asking too: "Which is the leading attempt to publish a canonical paper on AGI risk, in a leading science journal, refereed of course. The paper should have a formal model or calibration of some sort, working toward the conclusion of showing that the relevant risk is actually fairly high. Is there any such thing?"

The three papers people replied with were:
- Is Power-Seeking AI an Existential Risk?
- The Alignment Problem from a Deep Learning Perspective
- Unsolved Problems in ML Safety

Places I was looking so far:
- The list of references for that paper[2]
- The references for the Muehlhauser and Salamon intelligence explosion paper[3]
- The Sandburg review of singularities[4] and related papers (these are quite close to passing muster I think)

Places I wanted to look further:

- Papers by Yampolsky, aka[5]
- Papers mentioned in there by Schmidhuber (haven't gotten around to this)
- I haven't thoroughly reviewed Intelligence Explosion Microeconomics, maybe this is the closest thing to fulfilling the criteria?

But if there is something concrete in eg. some papers by Yampolsky and Schmidhuber, why hasn't anyone fleshed it out in more detail?

For all the time people spend working on 'solutions' to the alignment problem, there still seems to be a serious lack of 'descriptions' of the alignment problem. Maybe the idea is, if you found the latter you would automatically have the former?

I feel like something built on top of Intelligence Explosion Microeconomics and the Orthogonality Thesis could be super useful and convincing to a lot of people. And I think people like TC are perfectly justified in questioning why it doesn't exist, for all the millions of words collectively written on this topic on LW etc.

I feel like a good simple model of this would be much more useful than another ...
"The Need for Long-term Research" - Seeds of Science call for reviewers
by rogersbacon — 2023-04-11T15:37:58.108Z

Abstract

This article proposes the idea of creating specialized institutions for long-range research with a horizon of 25 years or more. While universities are traditionally seen as the primary institutions for research, their focus tends to be on shorter-term projects due to factors such as PhD and post-doctoral researcher timelines. Long-term research, which may involve ambitious multi-generation projects or longitudinal studies, is essential for making discoveries that require extended durations and resources beyond the scope of individual investigators.

---

Seeds of Science is a journal (funded through Scott Alexander's ACX grants program) that publishes speculative or non-traditional articles on scientific topics. Peer review is conducted through community-based voting and commenting by a diverse network of reviewers (or "gardeners" as we call them). Comments that critique or extend the "seed of science" in a useful manner are published in the final document, right after the main text.

We have just sent out a short article for review ("The Need for Long-term Research") that may be of interest to some in the LessWrong community, so I wanted to see if anyone - particularly those with interest in futurism/long-termism - would be interested in joining us as a gardener and providing feedback on the article. As mentioned above, this is an opportunity to have your comment recorded in the published scientific literature (comments can be made with real name or pseudonym).

It is free to join as a gardener and anyone is welcome (we currently have gardeners from all levels of academia and outside of it). Participation is entirely voluntary - we send you submitted articles and you can choose to vote/comment or abstain without notification (so it's no worries if you don't plan on reviewing very often but just want to take a look here and there at the articles people are submitting).

To register, you can fill out this google form. From there, it's pretty self-explanatory - I will add you to the mailing list and send you an email that includes the manuscript, our publication criteria, and a simple review form for recording votes/comments. If you would like to just take a look at this article without being added to the mailing list, then just reach out (info@theseedsofscience.org) and say so.

Happy to answer any questions about the journal through email or in...
NTIA - AI Accountability Announcement
by samshap — 2023-04-11T15:03:23.966Z
Link: https://www.ntia.doc.gov/issues/artificial-intelligence/announcement-event

The Department of Commerce is announcing a new policy on AI Accountability today, with an event at 12pm EST, April 11th.

The linked page also includes a link to the new policy.

There is an opportunity to leave comments on the policy until June 10th.