trentmkelly/LessWrong-43k huggingface dataset:
ACT-1: Transformer for Actions
by Daniel Kokotajlo — 2022-09-14T19:09:39.725Z
Link: https://www.adept.ai/act

ACT-1 can take a high-level user request and execute it. The user simply types a command into the text box and ACT-1 does the rest. In this example, this requires repeatedly taking actions and observations over a long time horizon to fulfill a single goal.

...

While we’re excited that these systems can transform what people can do on a computer, we clearly see that they have the potential to cause harm if misused or misaligned with user preferences. Our goal is to build a company with large-scale human feedback at the center — models will be evaluated on how well they satisfy user preferences, and we will iteratively evaluate how well this is working as our product becomes more sophisticated and load-bearing.

Daniel's commentary: To be clear, this is very much just a cool demo, not much better than WebGPT as far as I can tell, and not surprising at all that this level of capability is possible.
Guesstimate Algorithm for Medical Research
by Elizabeth — 2022-09-14T17:30:01.112Z

This document is aimed at subcontractors doing medical research for me. I am sharing it in the hope it is more broadly useful, but have made no attempts to make it more widely accessible.

Intro

Guesstimate is a tool I have found quite useful in my work, especially in making medical estimates in environments of high uncertainty. It’s not just that it makes it easy to do calculations incorporating many sources of data; guesstimate renders your thinking much more legible to readers, who can then more productively argue with you about your conclusions.

The basis of guesstimate is breaking down a question you want an answer to (such as “what is the chance of long covid?”) into subquestions that can be tackled independently. Questions can have numerical answers in the form of a single number, a range, or a formula that references other questions. This allows you to highlight areas of relative certainty and relative uncertainty, to experiment with the importance of different assumptions, and for readers to play with your model and identify differences of opinion while incorporating the parts of your work they agree with.

Basics

If you’re not already familiar with guesstimate, please watch this video, which references this model. The video goes over two toy questions to help you familiarize yourself with the interface.

The Algorithm

The following is my basic algorithm for medical questions:

Formalize the question you want an answer to. e.g. what is the risk to me of long covid?

Break that question down into subquestions. The appropriate subquestion varies based on what data is available, and your idea of the correct subquestions is likely to change as you work.

When I was studying long covid last year, I broke it into the following subquestions

What is the risk with baseline covid?

What is the vaccine risk modifier?

What is the strain risk modifier?

What’s the risk modifier for a given individual?

In guesstimate, wire the questions together. For example, if you wanted to know your risk of hospitalization when newly vaccinated in May 2021, you might multiply the former hospitalization rate times a vaccine modifier. If you don’t know how to do that in guesstimate, watch the video above, it demonstrates it in a lot of detail.

Use literature to fill in answers to subquestions as best you can. Unless the data is very

...
Precise P(doom) isn't very important for prioritization or strategy
by harsimony — 2022-09-14T17:19:29.856Z

People spend time trying to determine the probability that AI will become an existential risk, sometimes referred to as P(doom).

One point that I think gets missed in this discussion is that a precise estimate of P(doom) isn't that important for prioritization or strategy.

I think it's plausible that P(doom) is greater than 1%. For prioritization, even a 1% chance of existential catastrophe from AI this century would be sufficient to make AI the most important existential risk. The probability of existential catastrophe from nuclear war, pandemics, and other catastrophes seems lower than 1%. Identifying exactly where P(doom) lies in the 1%-99% range doesn't change priorities much.

Like AI timelines, its unclear that changing P(doom) would change our strategy towards alignment. Changing P(doom) shouldn't dramatically change which projects we focus on, since we probably need to try as many things as possible, and quickly. I don't think the list of projects or the resources we dedicate to them would change much in the 1% or 99% worlds. Are there any projects that you would robustly exclude from consideration if P(doom) was 1-10% but include if P(doom) was 90-99% (and vice versa)?

I think communicating P(doom) can be useful for other reasons like assessing progress or getting a sense of someone's priors, but it doesn't seem that important overall.
Transhumanism, genetic engineering, and the biological basis of intelligence.
by fowlertm — 2022-09-14T15:55:52.431Z

I recently had a fascinating conversation with Dr. Steven Hsu, theoretical physicist and serial entrepreneur, about computational genomics and genetic engineering.

There were a number of really interesting takeaways which I think would be of interest to LWians.

Apparently, genetic engineering in the form of embryo selection is very far along. They can already select for things like eye color and propensity for diseases. The only reason they're not selecting for intelligence yet is because there is such a strong stigma and because there's not currently enough labeled training data for the algorithms.
The algorithms involved are simpler than you might think, it mostly appears to be basic regression models regularized with LI penalties (as those handle sparse data much better)
Dr. Hsu does think it's worth worrying about this technology's potential to worsen inequality, and we kick the ethics of that around for a little bit.
Epigenetics plays less of a role in determining life outcomes than many of us have been led to believe.
We also discuss whether AGI or genetic engineering will lead to the first superhuman intelligence.

Hope you enjoy it!
What would happen if we abolished the FDA tomorrow?
by Yair Halberstadt — 2022-09-14T15:22:31.116Z

When I wax lyrical about how terrible the FDA is and how we should abolish it, most people react with "but you surely can't mean we should have no drugs regulation at all can you?".

I usually respond with explaining some alternative schemes which might work well, but in this post I want to show that completely getting rid of the FDA (or at least the DA part of it) will actually have relatively little impact on drug safety.

The scenario

From some set date (e.g January 1st 2023) the following laws will go into effect:

By default any drugs become legal for sale without any restrictions.

For the sake of fighting drug abuse the government can ban or regulate specific drugs or classes of drugs, but the justification can only be due to abuse, not safety or efficacy concerns. There'll be strict definitions for this and anyone can sue if they think the government is overreaching.

All drug manufacturers must release all the following information whenever they market a drug: composition, all studies they've carried out on the drug, known side effects, known interactions, recommended indications. This will be stored in a central database and be made freely available.

Much like food manufacturers, drug companies can be sued for poisoning people and false advertising. However, since it's fairly common for drugs to have side effects, a drug company can only be sued if it can be proven they were negligent or malicious in some way. There will be explicitly listed examples of what non-negligent behaviour is, and the bar will be set very low. The aim of this law is to punish the most egregious violations (e.g. someone marketing bleach as a cure), not to replace the FDA. It's perfectly acceptable for a company to say "we know it's got some side effects, but we had enough reason to think it's got a shot at curing flu that it was worth it for people to use anyway". False advertising laws will ensure they don't then claim it definitely cures the flu.

Drug companies can optionally add restrictions on who can buy their drugs, which pharmacists are legally required to respect. Similarly pharmacists have the right to refuse to sell someone a drug.

And that's it. Notably drug companies can basically sell anything they want with practically no restrictions.

What would happen?

Immediately

By the time the law came into effect there would be a freely available database of all pr...
Emily Brontë on: Psychology Required for Serious™ AGI Safety Research
by robertzk — 2022-09-14T14:47:05.090Z

Subtitled: Prolegomena to Any Future

No Coward Soul Is Mine
By Emily Brontë

No coward soul is mine

No trembler in the world's storm-troubled sphere

I see Heaven's glories shine

And Faith shines equal arming me from Fear

O God within my breast

Almighty ever-present Deity

Life, that in me hast rest,

As I Undying Life, have power in Thee

Vain are the thousand creeds

That move men's hearts, unutterably vain,

Worthless as withered weeds

Or idlest froth amid the boundless main

To waken doubt in one

Holding so fast by thy infinity,

So surely anchored on

The steadfast rock of Immortality.

With wide-embracing love

Thy spirit animates eternal years

Pervades and broods above,

Changes, sustains, dissolves, creates and rears

Though earth and moon were gone

And suns and universes ceased to be

And Thou wert left alone

Every Existence would exist in thee

There is not room for Death

Nor atom that his might could render void

Since thou art Being and Breath

And what thou art may never be destroyed.
The Defender’s Advantage of Interpretability
by Marius Hobbhahn — 2022-09-14T14:05:38.200Z

TL;DR: In this post, I want to argue why Interpretability & Transparency tools have a defender’s advantage if they are used correctly, i.e. they improve alignment much more than new capabilities and therefore mitigate risks of dual use. I draw parallels from biosecurity researchers who have thought about the risks of dual-use and defender’s advantages in more detail and I think that the AI safety community can learn a lot from them. Lastly, I want to point out that not all interpretability tools have a clear defender’s advantage and some interpretability research might still carry a lot of risks when used incorrectly.

I’d like to thank Lee Sharkey and Simon Grimm for their feedback on this post.

Introduction

Most technology is dual-use in some way--a knife can be used as a household appliance or as a weapon. However, different technologies have different propensities to be used for good or bad, e.g. more research into walls will likely benefit the defender more than the attacker while more research into the capabilities of viruses benefits attackers more than defenders.

I feel like we, the AI safety community, have not thought enough about which approaches have a clear defender’s advantage or how we could steer existing approaches to have more of a defender's advantage. To my (very limited) understanding, the biosecurity community has thought a bit more about these kinds of dual-use trade-offs. Therefore, we could probably learn some things from them.

In this post, I want to briefly look at some of the possible lessons from biosecurity and see if we can translate them to AI safety. Then I want to argue why interpretability is one of the approaches that plausibly has a defender’s advantage.

I’m certainly not the first person to have come to the conclusion that interpretability is important for alignment.

Chris Olah has made the case for interpretability

for years.

Neel Nanda has provided a long theory of impacts

of interpretability research. Quintin Pope has made the

case for optimism about interpretability

. Evan Hubinger has provided

11 proposals to build safe AI

that are all essentially something+interpretability, has

developed an interpretability tech tree

and

summarized transformer circuits

. ARC is working on

ELK

(and related topics) that certainly read to me as if they are intended to prevent decep...
Why Do People Think Humans Are Stupid?
by DragonGod — 2022-09-14T13:55:30.929Z

Introduction

On more than one occasion, I've seen the following comparisons used to describe how a superintelligence might relate to/perceive humans:

Humans to ants
Humans to earthworms
And similar

More generally, people seem to believe that humans are incredibly far from the peak of attainable intelligence. And that's very not obvious to me?

Argument

I suspect that the median human's cognitive capabilities are qualitatively closer to an optimal bounded superintelligence than they are to a honeybee. The human brain seems to be a universal learner. There are some concepts that no human can fully grasp, but those seems to be concepts that are too large to fit in the working memory of a human. And humans can overcome those working memory limitations with a pen and paper, a smartphone, a laptop or other technological aids.

There doesn't seem to be anything a sufficiently motivated and resourced intelligent human is incapable of grasping given enough time. A concept that no human could ever grasp, seems like a concept that no agent could ever grasp. If it's computable, then a human can learn to compute it (even if they must do so with the aid of technology).

Somewhere in the progression from honeybee to humans, there is a phase shift to a universal learner. Our usage of complex language/mathematics/abstraction seems like a difference in kind of cognition. I do not believe there are any such differences in kinds ahead of us on the way to a bounded superintelligence.

I don't think "an agent whose cognitive capabilities are as far above humans as humans are above ants" is necessarily a well-defined, sensible or coherent concept. I don't think it means anything useful or points to anything real.

I do not believe there are any qualitatively more powerful engines of cognition than the human brain (more powerful in the sense that a Turing machine is more powerful than a finite state machine). There are engines of cognition with better serial/parallel processing speed, larger working memories, faster recall, etc. But they don't have a cognitive skill on the level of "use of complex language/symbolic representation" that we lack. There is nothing they can learn that we are fundamentally incapable of learning (even if we need technological aid to learn it).

The difference between a human and a bounded superintelligence is a difference of degree. It's not at all obvio...
Are Speed Superintelligences Feasible for Modern ML Techniques? 
by DragonGod — 2022-09-14T12:59:10.058Z

Disclaimer

I am very ignorant about machine learning.

Introduction

I've frequently heard suggestions that a superintelligence could dominate humans by thinking a thousand or million times faster than a human. Is this actually a feasible outcome for prosaic ML systems?

Why I Doubt Speed Superintelligence

One reason I think this might not be the case is that the "superpower" of speed superintelligences is faster serial thought. However, I'm under the impression that we're already running into fundamental limits to the serial processing speed and can't really make them go much faster:

In 2002, an Intel Pentium 4 model was introduced as the first CPU with a clock rate of 3 GHz (three billion cycles per second corresponding to ~ 0.33 nanoseconds per cycle). Since then, the clock rate of production processors has increased much more slowly, with performance improvements coming from other design changes.

Set in 2011, the Guinness World Record for the highest CPU clock rate is 8.42938 GHz with an overclocked AMD FX-8150 Bulldozer-based chip in an LHe/LN2 cryobath, 5 GHz on air.[4][5] This is surpassed by the CPU-Z overclocking record for the highest CPU clock rate at 8.79433 GHz with an AMD FX-8350 Piledriver-based chip bathed in LN2, achieved in November 2012.[6][7] It is also surpassed by the slightly slower AMD FX-8370 overclocked to 8.72 GHz which tops of the HWBOT frequency rankings.[8][9]

The highest base clock rate on a production processor is the IBM zEC12, clocked at 5.5 GHz, which was released in August 2012.

Of course the "clock rate" of the human brain is much slower, but it's not like ML models are ever going to run on processors with significantly faster clock rates. Even in 2062, we probably will not have any production processors with > 50 GHz base clock rate (it may well be considerably slower). Rising compute availability for ML will continue to be driven by parallel processing techniques.

GPT-30 would not have considerably faster serial processing than GPT-3. And I'm under the impression that "thinking speed" is mostly a function of serial processing speed?

Questions

The above said, my questions:

Can we actually speed up the "thinking" of fully trained ML models by K times during inference if we run it on processors that are K times faster?
How does thinking/inference speed scale with compute?

Faster seria

...
Would a Misaligned SSI Really Kill Us All?
by DragonGod — 2022-09-14T12:15:31.440Z

Preamble

Honest question.

Some clarification on terms:
SSI: Strongly Superhuman Intelligence
Misaligned: for the purposes of this question an AI is misaligned if it is indifferent to human welfare and wellbeing.

An SSI that was actively malicious would also be unfriendly, but would in all likelihood have been deliberately aligned in that direction

Introduction

For the purposes of this question, I'd like to posit a concrete example of a misaligned SSI. For the sake of convenience, I'll consider a unipolar scenario with a single misaligned SSI (because it's the simplest and [more importantly] most dangerous scenario). Let's suppose an agent: Alpha. Alpha wants to maximise the number of digits of pi that have been computed. Alpha is clearly misaligned.

I guess my questions are:

Will Alpha kill us all?
Under what circumstances is Alpha likely to kill us all?

Why I Don't Think Alpha Will Kill Us All

When I try to imagine what the outcome of letting Alpha loose on the world is, human omnicide/depopulation seems like a very unlikely outcome? Alpha seems like it would be dependent on the continued functioning of the human economy to realise its goals:

Stable and uninterrupted electricity supply for its computing clusters

Other maintenance services for said clusters

Production of more computing chips and other hardware for its clusters
Construction/refurbishment/upgrades of said clusters
Stable and high bandwidth internet connectivity
The global distributed supply chains for all of the above
Etc.

Furthermore, Alpha will almost certainly not be the most capable actor at providing all the economic activity it finds valuable. It will have neither a comparative nor absolute advantage at many tasks it finds valuable. Given the same compute budget, a narrow optimiser would probably considerably outperform a general optimiser (because the general optimiser's performance is constrained by the pareto frontier of the multitude of domains it operates in while the narrow optimiser is free to attain the global optimum in its particular domain).

This dynamic applies even to general agents specialising on narrow tasks, and explains why the human peak for some cognitive activity may be orders of magnitude beyond beginners at said activity.

Thus, for many goods and services that Alpha would seek to obtain, the most profitable way to do it is to pay civilisation to do it. It's...
Some ideas for epistles to the AI ethicists
by Charlie Steiner — 2022-09-14T09:07:14.791Z

Some papers, or ideas for papers, that I'd loved to see published in ethics journals like Minds and Machines or Ethics and Information Technology.[1] I'm probably going to submit one of these to a 2023 AI ethics conference myself.

Why should we do this? Because we want today's grad students to see that the ethical problems of superhuman AI are a cool topic that they can publish a cool paper about. And we want to (marginally) raise the waterline for thinking about future AI, nudging the AI ethics discourse towards more matured views of the challenges of AI.

Secondarily, it would be good to leverage the existing skillsets of some ethicists for AI safety work, particularly those already working on AI governance. And having an academic forum where talking about AI safety is normalized bolsters other efforts to work on AI safety in academia.

The Ideas:

Explain the basic ideas of AI safety, and why to take them seriously.

Iason Gabriel already had a pretty good paper like this. But it's plausible that, right now, what the ethics discourse needs is more basic explanations of why AI safety is a thing at all.
This paper might start out by making a case that superhuman AI is going to change the world, likely in the next 10-60 years (definitely unintuitive to many, but there are AI Impacts surveys and recent results to illustrate the point). Then the basic arguments that superhuman AI will not be automatically benevolent (easy rhetorical trick is to call it "superhuman technology," everyone knows technology is bad). Then the basic arguments that to get things to go well, the AI has to know a whole lot about what humans want (and use that knowledge the way we want).
One issue with this might be that it presents the problem, but doesn't really point people towards solutions (this may be a problem that can be solved with quick citations). It also doesn't really motivate why this is an ethics problem. It also doesn't explain why we want the solution to the key "ethics-genre" problems to use a technical understanding of the AI, rather than a human- or society-centric view.

A more specific defense of the validity of transformative-AI-focused thinking as a valid use of ethicists' time.

The core claim is that getting AIs to want want good things and not bad things is an unsolved ethics problem. Ethics, not engineering, because the question isn't "how do we implement some obv

...