trentmkelly/LessWrong-43k huggingface dataset:
Crypto-fed Computation
by aaguirre — 2022-06-13T21:20:58.988Z

Overview

The idea of this post is to describe, discuss, and if warranted understand how to create, a model of crypto-fed computation.[1] The basic idea is that high-powered GPU (or other ML-specialized) hardware could be equipped with in-chip hardware locks such that the computational cores require a steady stream of cryptographic keys in order to continue performing. In the absence of such continually supplied keys the hardware would downgrade to a small or zero fraction of its nominal capability.

Deploying such hardware would allow three key things:

Monitoring of how computation is being used: who/whatever is supplying the keys will know how many are being sent and to where.

Control of computation: in a framework in which large levels of computation are regulated, this mechanism allows setup such that the continued consent of an outside regulating agent is reuired in order to continue large computational projects.

Off-switch: the default mode of this hardware system is "idle" (or if desired, "off") so that in the unlikely but possible case that standard control of a computational system is lost or goes rogue, it could not continue without also co-opting the source of the cryptographic keys.

I'll argue that such a capabilities appears to be quite technically feasible, and there may be plausible pathways to adopting it. However there are plenty of open questions, and many details to be nailed down before attempting such an effort.

How would it work

Let's lay out some design criteria for such a system, aiming to provide at least the three services above in situations where agents attempting to subvert the system are verycapable.

Very Strong cryptography: The cryptographic protocol used should be as strong and well-tested as possible, even against quantum computation.

Hardware encoded: It must be impossible to subvert the need for the keys without very substantial modification of the chip hardware itself (tantamount to remanufacturing it.)

Bi-directional: the computational cores must be able to and required to "ask" for more keys, and this must be cryptographically secure (so that it cannot be spoofed to provide keys to a third party.)

Efficient: there should be relatively low overhead, computational or otherwise, to the system.

Example schemas

Here are a few examples of ways this might work. As this is not my area of expertise, it is extremely likely th...
Contra EY: Can AGI destroy us without trial & error?
by nsokolsky — 2022-06-13T18:26:09.460Z

NB: I've never published on LW before and apologize if my writing skills are not in line with LW's usual style. This post is an edited copy of the same article in my blog.

EY published an article last week titled “AGI Ruin: A List of Lethalities”, which explains in detail why you can’t train an AGI that won’t try to kill you at the first chance it gets, as well as why this AGI will eventually appear given humanity’s current trajectory in computer science. EY doesn’t explicitly state a timeline over which AGI is supposed to destroy humanity, but it’s implied that this will happen rapidly and humanity won’t have enough time to stop it. EY doesn’t find the question of how exactly AGI will destroy humanity too interesting and explains it as follows:

My lower-bound model of "how a sufficiently powerful intelligence would kill everyone, if it didn't want to not do that" is that it gets access to the Internet, emails some DNA sequences to any of the many many online firms that will take a DNA sequence in the email and ship you back proteins, and bribes/persuades some human who has no idea they're dealing with an AGI to mix proteins in a beaker, which then form a first-stage nanofactory which can build the actual nanomachinery. (Back when I was first deploying this visualization, the wise-sounding critics said "Ah, but how do you know even a superintelligence could solve the protein folding problem, if it didn't already have planet-sized supercomputers?" but one hears less of this after the advent of AlphaFold 2, for some odd reason.) The nanomachinery builds diamondoid bacteria, that replicate with solar power and atmospheric CHON, maybe aggregate into some miniature rockets or jets so they can ride the jetstream to spread across the Earth's atmosphere, get into human bloodstreams and hide, strike on a timer. Losing a conflict with a high-powered cognitive system looks at least as deadly as "everybody on the face of the Earth suddenly falls over dead within the same second".

Let’s break down EY’s proposed plan for “Skynet” into the requisite engineering steps:

Design a set of proteins that can form the basis of a “nanofactory”
Adapt the protein design to the available protein printers that accept somewhat-anonymous orders over the Internet
Design “diamondoid bacteria” that can kill all of humanity and that can be successfully built by the “n

...
What are some smaller-but-concrete challenges related to AI safety that are impacting people today?
by nonzerosum — 2022-06-13T17:36:40.896Z

Making a list of smaller but non-abstract challenges related to large topics like AI safety, that someone could work on to get practical on the ground experience.

Some examples:

Help media companies better communicate advances in AI ("Google's Sentient AI")
Help people detect deepfakes
Help online communities deal with deepfakes and set good policies
Help people detect online AI astroturfing

What else?
[Link] New SEP article on Bayesian Epistemology
by Aryeh Englander — 2022-06-13T15:03:09.892Z

The Stanford Encyclopedia of Philosophy has a brand new article on Bayesian Epistemology, replacing the previous entry on that topic.

https://plato.stanford.edu/entries/epistemology-bayesian/

Table of Contents:

1. A Tutorial on Bayesian Epistemology

1.1 A Case Study
1.2 Two Core Norms
1.3 Applications
1.4 Bayesians Divided: What Does Coherence Require?
1.5 Bayesians Divided: The Problem of the Priors
1.6 An Attempted Foundation: Dutch Book Arguments
1.7 Alternative Foundations
1.8 Objections to Conditionalization
1.9 Objections about Idealization
1.10 Concerns, or Encouragements, from Non-Bayesians

2. A Bit of Mathematical Formalism
3. Synchronic Norms (I): Requirements of Coherence

3.1 Versions of Probabilism
3.2 Countable Additivity
3.3 Regularity
3.4 Norms of Conditional Credences
3.5 Chance-Credence Principles
3.6 Reflection and Other Deference Principles

4. Synchronic Norms (II): The Problem of the Priors

4.1 Subjective Bayesianism
4.2 Objective Bayesianism
4.3 Forward-Looking Bayesianism
4.4 Connection to the Uniqueness Debate

5. Issues about Diachronic Norms

5.1 Old Evidence
5.2 New Theory
5.3 Uncertain Learning
5.4 Memory Loss
5.5 Self-Locating Credences
5.6 Bayesianism without Kinematics

6. The Problem of Idealization

6.1 De-idealization and Understanding
6.2 Striving for Ideals
6.3 Applications Empowered by Idealization

7. Closing: The Expanding Territory of Bayesianism
Bibliography
Academic Tools
Other Internet Resources
Related Entries
Training Trace Priors
by Adam Jermyn — 2022-06-13T14:22:19.503Z

I’m worried about scenarios involving deceptive models. We’ve failed at inner alignment so the model has goals that are not aligned with ours. It can somehow detect when it’s in training, and during training it pretends to share our goals. During deployment, surprise! The model paperclips the universe.

In this story deception is all about the model having hidden behaviors that never get triggered during training. There are many inputs where the model does what we want. These correspond to inputs where it ‘plays along’. Then on some rare input or an input outside of the training distribution it defects, triggering some catastrophic behavior that corresponds to maximizing an ulterior motive.

This suggests that we should be wary of models that have logic/subcircuits/states that are never used during training, and we can penalize models for having such hidden behaviors.

In this post I'm focused on examples of training trace priors, which are priors that make use of execution traces from the training distribution. I think these provide at least a few bits of optimization away from deception.

These are very much working notes, and feedback is very welcome.

One-Gate Trace Prior: Entropy

Consider a boolean circuit. The output of each gate is 0/1. One way to implement the entropy prior is to record traces of these outputs during training, and penalize the model for having strong bias between 0/1:

p∝exp(−∑i∑j∈[0,1]pi(j)lnpi(j))

Here pi is the observed fraction of the time gate i produces output j on the training distribution. We call this the one-gate trace prior because it relies on probabilities defined over single gates. This prior favors models where each gate has even odds of producing 0/1.

The intuition for why this might work is that deceptive models need some machinery specifying when and how they defect, and this machinery is never active during training. This is at the core of what makes them bad! If the defecting machinery activated during training we’d notice because the model would stop optimizing the outer goal, so we’d assign it a high loss and move on to better models.

Another framing is: It is inherently suspicious if a model has machinery that is rarely activated. That machinery wasn’t vetted during training, so we have no reason to trust it. If the model has a hidden love of paperclips, this is the natural place to hide it.

As a t...
Can you MRI a deep learning model?
by Yair Halberstadt — 2022-06-13T13:43:05.293Z

In an MRI scan you see which parts of a brain light up in response to a stimulus. This has proven invaluable in understanding brains.

Is there an equivalent thing you can do with deep learning models, where you can see which parts light up in response to stimuli? And does there exist good UIs to explore this? It seems like such a technique would be invaluable for understanding deep learning models, and possibly for alignment.
On A List of Lethalities
by Zvi — 2022-06-13T12:30:01.624Z

Response to (Eliezer Yudkowsky): A List of Lethalities.

Author’s Note: I do not work in AI Safety, lack technical domain knowledge and in many ways am going to be wrong. I wasn’t going to write this to avoid potentially wasting too much time all around without having enough to offer, and for fear of making stupid errors, but it was clear that many people thought my response would be valuable. I thank those whose anonymous sponsorship of this post both paid for my time and made me update that the post was worth writing. I would be happy for this to happen again in the future.

Eliezer has at long last delivered the definitive list of Eliezer Rants About Why AGI Will Definitely Absolutely For Sure Kill Everyone Unless Something Very Unexpected Happens.

This is excellent. In the past we had to make do with makeshift scattershot collections of rants. Now they are all in one place, with a helpful classification system. Key claims are in bold. We can refer, consider and discuss them.

It would be an even better post if it were more logically organized, with dependencies pointed out and mapped and so on.

One could also propose making it not full of rants, but I don’t think that would be an improvement. The rants are important. The rants contain data. They reveal Eliezer’s cognitive state and his assessment of the state of play. Not ranting would leave important bits out and give a meaningfully misleading impression.

I am reminded of this comment of mine that I dug out of the archives, on another Eliezer post that was both useful and enthused with this kind of attitude:

Most of this applies again. Eliezer says explicitly that the alternative post would have been orders of magnitude harder to write, and that the attitude is important information.

I would expand this. Not only are the attitude and repetition important information in terms of allowing you to understand the algorithm generating the post and create a better Inner Eliezer, but they also are importantly illustrating the cognitive world in which Eliezer is operating.

The fact that this is the post we got, as opposed to a different (in many ways better) post, is a reflection of the fact that our Earth is failing to understand what we are facing. It is failing to look the problem in the eye, let alone make real attempts at solutions.

Eliezer is not merely talking to you, yes you (with no...
D&D.Sci June 2022 Evaluation and Ruleset
by abstractapplic — 2022-06-13T10:31:25.447Z

This is a followup to the D&D.Sci post I made ten days ago; if you haven’t already read it, you should do so now before spoiling yourself.

Here is the web interactive I built to let you evaluate your solution; below is an explanation of the rules used to generate the dataset (My full generation code is available here, in case you’re curious about some detail not explained below). You’ll probably want to test your answer before reading any further.

Ruleset

(Note: to make writing this easier, I’m using standard D&D dice notation, in which “4d8+3” means “roll four eight-sided dice, sum the results, then add three”.)

Traits

There’s a moderate positive correlation between the Otaku trait and the Nerd trait, and a strong negative correlation between the Office Worker trait and the Hikkikomori trait (it would be stronger still, but remote work from a home office allows these traits to coexist). The Sociopath trait doesn’t correlate with any other.

In addition to the five recorded traits, there’s one the angel and the goddess aren’t equipped to detect: a random 19.2% of heroes are Fated to win regardless of their choices. Being Fated also doesn’t correlate with other traits.

Aptitude

A hero’s Aptitude – that is, how well-adjusted they are to being isekai’d – is (by default) given by 2+1d8. Traits modify this as follows:

Sociopaths get +1 Aptitude (empathy is a net liability when fighting a Demon King)
Nerds get +2 Aptitude (scientific/mathematical knowledge is a real boon in another world)
Otaku get +4 Aptitude (nothing helps an otherworldly hero like knowing isekai tropes)
Office Workers, instead of generating their starting aptitude with 2+1d8, generate it with 4+1d4 (white-collar workers are lower-variance)
Hikkikomoris get +1d4 -1d4 Aptitude (isolation makes them higher-variance and longer-tailed)

Cheat Skills

Choices of Cheat Skills are informed by both Traits and Aptitude. In particular, Fated heroes tend to be at least dimly aware of their good fortune, and are therefore liable to pick cheat skills optimized for smoothing the path to an inevitable victory; when given a choice, they’re much more likely to select Enlightenment, Radiant Splendor, and Uncanny Luck than un-Fated champions. (This causes those skills to appear more useful than they are.)

Power

Power is found by taking Aptitude and adding to it based on the cheat skills chosen.

Before any other skill is applie

...
What's the "This AI is of moral concern." fire alarm?
by Quintin Pope — 2022-06-13T08:05:10.395Z

Given the recent noise on this issue around LaMDA, I thought it might be a good idea to have some discussion around this point. I'm curious about what possible evidence would make people update in favor of a given system being morally relevant. Less "here's the answer to morality" and more "here are some indicators that you should be concerned". Note also that I'm not asking about consciousness, per se. I'm specifically asking about moral relevance.

My Answer (feel free to ignore and post your own)

I think that one class of computation that's likely of moral concern would be self-perpetuating optimization demons in an AI.

Specifically, I'm thinking of optimization demons that are sophisticated enough to preserve themselves by actively and deliberately maintaining a sort of homeostasis in their computational environment, e.g., by preventing gradient updates that would destroy them. Such computations would (1) not want to die as a terminal value, (2) plausibly be cognitively sophisticated enough negotiate and trade with, and (3) have some awareness of themselves and their relation with the computational environment in which they're embedded.

I think the cognitive capabilities that would help an optimization demon perpetuate itself strongly intersect with the cognitive capabilities that let humans and other animals replicate themselves, and that the intersection is particularly strong along dimensions that seem more morally relevant. Reasoning along such lines leads me to think optimization demons are probably of moral concern, while still being agnostic about whether their conscious.

I think the only situations in which you can get these sorts of optimization demons are when the AI in question has some influence over its own future training inputs. Such influence would allow there to be optimization demons that steer the AI towards training data that reinforce the optimization demon.

Thus, one of my "indicators of concern" is whether the training process allows for feedback loops where the AI influences its own future training data. Self-supervised language modeling under IID data does not count. However, something like InstructGPT's training process would.

At this point, I'd been intending to say that InstructGPT seemed more likely to be of moral worth than LaMDA, but based on this blog post, it looks like LaMDA, might actually count as...
The beautiful magical enchanted golden Dall-e Mini is underrated
by p.b. — 2022-06-13T07:58:50.107Z

[Showcasing some of the pictures that my daughter and I created with the open source generative model Dalle-Mini. No deeper insights, just an appreciation how well it works in some cases.]

I have been aware of a small open source Dall-e version for some time now. My initial check left me with the impression that it was way worse than Dall-e or Dall-e 2 and not really worth engaging with.

However, recently I started to implore Dall-e Mini together with my little daughter who is less critical of machines that can create any picture she proposes. Originally we ended every prompt with " - a children's book illustration".

We quickly got better at prompting and realized that there were some things Dall-e Mini couldn't do very well, like faces and animals and complex scenes, but also others where it wasn't as far behind of the SOTA-models as you'd reasonably expect from the model size. (Above a ship sailing through the clouds with a full moon, a castle in the clouds and a magical door in an enchanted forest.)

We switched to " - a beautiful fantasy illustration" to often get just that. These prompts all start with a version of "A beautiful magical enchanted golden ... ".

We created swords ...

And magical artefacts ...

Birds ...

And beautiful magical enchanted golden eyes ...

Flowers ... yes, magical, beautiful, enchanted ones.

Magical, beautiful, enchanted, golden bananas ...

Crowns ...

Magical, beautiful, enchanted, golden glasses ...

And other stuff ...

I might also implore the generation of icons for the ubiquitous PPT.

Angels taking walks with wild boars ...

These are all cherry-picked of course. But the point remains that even this small open source model can create quite amazing pictures.
Why so little AI risk on rationalist-adjacent blogs?
by Grant Demaree — 2022-06-13T06:31:40.288Z

I read a lot of rationalist-adjacents. Outside of LessWrong and ACX, I hardly ever see posts on AI risk. Tyler Cowen of Marginal Revolution writes that, "it makes my head hurt" but hasn't engaged with the issue. Even Zvi spends very few posts on AI risk.

This is surprising, and I wonder what to make of it. Why do the folks most exposed to MIRI-style arguments have so little to say about them?

Here's a few possibilities

Some of the writers disagree that AGI is a major near-term threat
It's unusually hard to think and write about AI risk
The best rationalist-adjacent writers don't feel like they have a deep enough understanding to write about AI risk
There's not much demand for these posts, and LessWrong/Alignment Forum/ACX are already filling it. Even a great essay wouldn't be that popular
Folks engaged in AI risk are a challenging audience. Eliezer might get mad at you
When you write about AGI for a mainstream audience, you look weird. I don't think this is as true it used to be, since Ezra Klein did it in the New York Times and Kelsey Piper in Vox
Some of these writers are heavily specialized. The mathematicians want to write about pure math. The pharmacologists want to write about drug development. The historians want to argue that WWII strategic bombing was based on a false theory of popular support for the enemy regime, and present-day sanctions are making the same mistake
Some of the writers are worried that they'll present the arguments badly, inoculating their readers against a better future argument

What they wrote

I'll treat Scott Alexander's blogroll as the canonical list of rationalist-adjacent writers. I've grouped them by their stance on the following statement:

Misaligned AGI is among the most important existential risks to humanity[1]

Explicitly agrees and provides original gears-level analysis (2)

Zvi Mowshowitz

“The default outcome, if we do not work hard and carefully now on AGI safety, is for AGI to wipe out all value in the universe.” (Dec 2017)

Zvi gives a

detailed analysis here

followed by his own model in response to the 2021 MIRI conversations.

Holden Karnofsky of OpenPhil and GiveWell

In his Most Important Century series (Jul 2021 to present), Holden explains AGI risk to mainstream audiences. Ezra Klein featured Holden’s work in the New York Times.

This series had a high impact on me, because Holden used to have

specific and detailed objection

...
Code Quality and Rule Consequentialism
by Adam Zerner — 2022-06-13T03:12:09.861Z

(See also Taking the outside view on code quality)

Code quality. Such a divisive topic. To overgeneralize[1], managers always want things released soon, which means a quick and dirty version of the code. On the other hand, engineers always want to do it the "right way", which means taking longer before releasing.

And that's just one example. Here are some others.

Engineers want to refactor stuff. Managers say it's not worth it.

Engineers want to take the time to write tests. Managers need new features to be released.

Engineers want to upgrade to the new version of the library. Managers say that'll be too costly.

Engineers want to set up a linter. Managers don't see how that'll actually help the business.

Engineers want to spend time discussing things during code review. Managers feel like that always just gets in the way of meeting deadlines and doesn't really matter.

And now for the million dollar question: who's right? Are the engineers right in saying that you should take the time to do these sorts of things? Or are the managers right in saying that you should keep your eye on the prize and deploy features that are actually gonna improve the lives of end users?

Just kidding. We shouldn't ask who's right. Instead, let's think about how we'd even go about answering these sorts of questions in the first place.

Two types of questions

I want to pause here for a second. I want to emphasize the distinction. "Who's right?" and "How do we go about thinking about who's right?" are two very different types of questions.

Let's make it more concrete.

Should we convert those class-based React components into functional components?

How would we go about deciding whether to convert those class-based React components into functional components?

Do you see the difference? (1) is asking what we should do. (2) is asking how to go about deciding what we should do.

It's the difference between 1) asking what we should eat for dinner and 2) asking how we should go about deciding what to eat for dinner. The answer to (1) might be "pizza". The answer to (2) might be "we should think about how well different options perform along the dimensions of health, convenience, cost, and taste, and pick the option that performs best".

For these debates about whether to prioritize code quality, I've seen commentary on the first type of question. On the other hand, I can't really think of any co...