trentmkelly/LessWrong-43k huggingface dataset:
Cryptographic Life: How to transcend in a sub-lightspeed world via Homomorphic encryption
by Golol — 2022-06-14T19:22:47.957Z

Hello. This is a concept that I have not seen formulated before and I think the people here might find interesting. I am a new LW user and so I do not know if this has been formulated before. In this case I suppose you can refer me to it and close this post or something like that.

This "cryptographic life" I will describe now should be seen as a plausible example of how a lifeform/consciousness could span a volume measured in hundreds of lightyears without being fixed to any particular physical objects.

Here are my premises:

P1: We assume that AGI is of course possible and can be implemented on classical computers. It is also possible to implement these AGIs as distributed/decentralized computations. You could, for example, have an AI which runs different aspects of its computation on hundreds of different physical computers, constantly sending data between them to keep its overall state of mind coherent. It can dynamically move its computation onto new hardware and integrate new sensory data sources.

P2: We assuming as given a galaxy filled with advanced civilizations that communicate at lightspeed and have access to large amounts of classical, digital compute power. THeir communications technology is sufficiently advanced to send and receive high bandwidths over hundreds of lightyears.

P3: Full Homomorphic encryption is possible. According to Wikipedia: "Homomorphic encryption is a form of encryption that permits users to perform computations on its encrypted data without first decrypting it." As far as I know we only know of such methods that allow for limited arithmetic operations to be performed "under the hood".

We assume that "full homomorphic encryption" is possible in a "public-private" key manner. This means that a customer can give you a description of any program (any turing machine) which has been homomorphically encrypted with a private key, so that you can take an input, encrypt it with the public key, and the let the program run on this input by performing certain fixed computation steps. Then you can return the output to the customer and the customer can decrypt it, getting his answer. Homomorphic encryption of this kind allows cloud computation to become secure: You can give your program to other people and you know that they will have to correctly execute it. If someone tries to modify the program during its execution then the result will not be...
Blake Richards on Why he is Skeptical of Existential Risk from AI
by Michaël Trazzi — 2022-06-14T19:09:26.783Z
Link: https://theinsideview.ai/blake

I have recently interviewed Blake Richards, an Assistant Professor in the Montreal Neurological Institute and the School of Computer Science at McGill University and a Core Faculty Member at MiLA. Below you will find some quotes summarizing his takes on AGI.

Blake is not really concerned about existential risk from AI. Like Yann LeCun, he finds that AGI is not a coherent concept, and that it would be impossible for an AI to be truly general (even if we restrict the no free lunch theorem to economically valuable tasks).

Why I Interviewed Blake

Although I do not agree with everything he says, I think there is value in trying to interact with AI researchers outside of the AI Alignment bubble, understanding exactly what arguments they buy and do not buy, eventually nailing down some cruxes that would convince them that AI existential risk is worth thinking about.

Better understanding LeCun's position has been valuable for many on LessWrong (see for instance the 2019 debate with Bengio and Russell), and Blake thinking is close to Yann's, given they are part of a similar philosophical bent.

Why you Might Want to Talk to Skeptics

Another exercise I found insightful was (mostly incorrectly) assessing people's views on AI Alignment and AI timelines, which made me understand better (thanks Cunningham's law!) the views of optimists (they turned out to be pretty close to Richard Ngo's reasons for optimism at 11:36 here).

In any case, I recommend to people who are in touch with ML researchers or practitioners to 1) get to a level where they feel comfortable steelmanning them 2) do a write-up of their positions on LW/EAF. That would help nail down the community's understanding of what arguments are convincing or not, and what would make them change their mind.

To that end, here are what Blake has to say about his position on AGI and what could make his change his mind about existential risk.

Generalizing to "All Sort of Tasks We Might Want It To do"

"We know from the no free lunch theorem that you cannot have a learning algorithm that outperforms all other learning algorithms across all tasks. [...] Because the set of all possible tasks will include some really bizarre stuff that we certainly don’t need our AI systems to do. And in that case, we can ask, “Well, might there be a system that is good at all the sorts of tasks that we might want it to do?” Here, we don’t have a m

...
How Do You Quantify [Physics Interfacing] Real World Capabilities?
by DragonGod — 2022-06-14T14:49:47.264Z

Disclaimer

Sorry for the length of this "question". The post in which I cover the preliminary context hasn't been published yet (4,000+ words and I'm < 40% through my outline).

Introduction

I have done some thinking of how one might quantify the capabilities of an AI system to influence the "real world" environment.

I have identified three broad interfaces for executing capabilities:

Humans (and groups thereof)
Human infrastructure
Physics/Bare reality

Humans

The AI could influence the world by convincing humans to do things that it wants.

Some relevant skills:

Communication
Negotiation/Bargaining (broadly construed)
Trade (broadly construed)
Persuasion
Flirtation
Deception
Manipulation
Etc.

The idea here is influence the real world via influencing individual humans or groups of humans

Quantifying Human Interfacing Capabilities

To a first approximation, a function to quantify human capabilities might be a positive (aggregate) function of something like: “the likelihood that the AI could convince a(n arbitrary) human to perform a(n arbitrary) act” (let's call this the likelihood of successful persuasion [LSP]). The function may apply suitable modifications to LSP such as:

Positive weighted by the power/influence of the human(s)
Inversely weighted by the time taken to convince said human(s)
Positive weighted by the influence of the act(s)
Inversely weighted by the human('s/s') disposition towards the act(s)

Defence of the Method

The human interface is just the AI influencing the world via influencing humans. A naive way to quantify the AI's ability to influence humans is something like an aggregate of LSP. Upon thinking on it for a few seconds, you'd like to modify the LSP in some ways.

This is vague, because I am not trying to design a concrete measure of "ability to influence humans", at this stage of my thinking, I'm fine with a high level abstraction of what such a measure might look like.

Human Infrastructure

The AI could influence the world via the levers of human civilisation. A non-exhaustive list of relevant human infrastructure follows:

Social
Political
Cultural
Economic and financial
Security
Information Technology
Legal
Etc.
Subclasses of the above
Superclasses of the above
Intersections of the above
Unions of the above

An example of influencing the world via human infrastructure is hacking into a web server. I'm not going to attempt to list the releva...
Was the Industrial Revolution The Industrial Revolution?
by Davis Kedrosky — 2022-06-14T14:48:44.700Z
Link: https://daviskedrosky.substack.com/p/was-the-industrial-revolution-the?s=w

There was an Industrial Revolution, and it was slow.

The debate’s been over for two decades. The gradualists, armed with new growth accounting techniques, have won, exorcising forever the Ashtonian vision of an eighteenth-century leap into exponential growth. Where historians once thought the spinning jenny, steam engine, and factory system the immediate preconditions of England’s dramatic transformation, they now accept that these advances emerged in a trickle in a few tiny sectors. Nicholas Crafts, a primary exponent of this “limited-gradualist” view, opens a recent survey on productivity growth during the Industrial Revolution on a triumphalist note:

The British industrial revolution was a landmark event in world economic history. While it was once seen as a period of dramatic ‘take-off’, it is now generally accepted that it was an episode of gradual acceleration in labour productivity growth which eventually led to sustained increases in living standards (Crafts 2021, p. 1).

Crafts suggests that the research frontier on the 1750-1850 expansion has shifted. Economic historians should now be asking the following questions: “why did productivity growth accelerate?”, “how soon did workers benefit from faster productivity growth?”, and “why did productivity growth experience a major slowdown as soon as the 1870s?” Having exorcised the ghost of Rostow and established the empirical facts of Britain’s output trajectory, causality and distribution necessarily become the chief problems for the field. We can move on.

The macroeconomic figures are largely settled, for the moment; a substantial revision by Broadberry et al (2013) points to faster growth, but not much. That said, Crafts still tries to justify the low numbers—re-revised substantially since his 1985 “revision”—in light of qualitatively obvious increase in productivity-enhancing inventive activity. But what are the consequences for the history of English economic development in the eighteenth century if the decisive changes came after the middle of the nineteenth? Indeed, given that the latest statistics show sluggish growth until 1850 and stagnation after 1870, we appear to have shortened the Industrial Revolution out of existence.

Yet Britain did eventually start climbing onto the “inverted hockey curve” of rising living standards. Today, Manchester and Birmingham are modern cities with cars, helicopters, a...
Investigating causal understanding in LLMs
by Marius Hobbhahn — 2022-06-14T13:57:59.430Z

This is a project by Marius Hobbhahn and Tom Lieberum. David Seiler contributed ideas and guidance.

Executive summary

We want to investigate the quality of LLMs causal world models in very simple settings. To this end, we test whether they can identify cause and effect in natural language settings (taken from

BigBench

) such as “My car got dirty. I washed the car. Question: Which sentence is the cause of the other?” and in toy settings such as the one detailed below. We probe this world model by changing the presentation of the prompt while keeping the meaning constant. Additionally, we test if the model can be “tricked” into giving wrong answers when we present the shot in a different pattern than the prompt. For example, you can see the 3 colored balls toy setup below.

We draw three main conclusions from the report:

Larger models yield better results. This has been consistent throughout all experiments and other work and is not surprising.
K-shot outperforms one-shot and one-shot outperforms zero-shot in standard conditions, i.e. where shot and prompt have a similar pattern. If the shot and prompt have a different pattern but similar content, this decreases the performance of the model (see figure below). Furthermore, we find that switching the order of presentation in the prompt decreases the zero-shot performance.

For an example of the scenarios see the table above. The -1 is because you can’t switch shot and prompt in zero-shot settings. We find that crossed settings (i.e. columns 2&3) perform worse than ones where shot and prompt follow the same pattern. Furthermore, zero-shot performance is worse in a switched setting.

We interpret these findings as preliminary evidence that the LLM does not fully answer based on content but also on the pattern suggested by the prompt. The decrease in zero-shot performance can be explained by the fact that the model often just responds with the first answer that matches the pattern, e.g. the first color it identifies, or that the pattern of the prompt primes it. Possibly this is due to a failure mode of the model’s induction heads which, given a sequence of tokens `A B … A`, increase the logit of token `B` (see

Olsson et al, 2022

for details).
When we increase the complexity of the task, e.g. use five instead of three colors for the toy experiments, the model performance gets muc

...
Why multi-agent safety is important 
by Akbir Khan — 2022-06-14T09:23:11.951Z

Alternative Title: Mo’ Agents, Mo’ Problems

This was a semi-

adversarial collaboration

with Robert Kirk; views are mostly Akbir’s, clarified with Rob’s critiques.

Target Audience: Machine Learning researchers who are interested in Safety and researchers who are focusing on Single Agent Safety problems.

Context: Recently I’ve been discussing how to meaningfully de-risk AGI technologies. My argument is that even in the situation where single agent alignment is solved, there is a complementary set of problems which must be addressed. Below I motivate what to expect in a multipolar scenario, how existing safety methods are likely unsatisfactory and describe new failure modes that will need to be addressed.

My Multipolar world

Why should we expect a multipolar scenario?

Much like today, the future will likely consist of many organisations (states and companies) enthusiastically building AI systems. Along this path, many notice they can improve their systems by scaling data/compute and as such will deploy these systems into the real-world to scale data collection.

When deployed in the real world, systems will become smarter much quicker - there will be a “takeoff” in capabilities. I claim the process will take months to years, with bottlenecks around collecting data (people are slow to adopt new products). As many see promise in this approach, many takeoffs are likely to happen and be competing with each other, as many consumers find themselves choosing between competing products - “shall I use Dall-e 2 or Imagen for my illustration?”. Because of the existence of multiple competing companies all building more and more general systems, and the (relatively low) speed[1] of these takeoffs, I believe there will be many competing AI systems of similar competencies all acting and learning in the real world. That is, even if one company is in the lead, this doesn’t mean that they will immediately achieve a decisive strategic advantage past a certain capabilities threshold; or equivalently that the threshold is at a later point on the capabilities axis than the capability required for these systems to have transformative impact on the world, and influence each other while learning. In VC speak: we’d say that data is the moat that keeps companies defensible against more capable systems.

My belief in the above scenario is rooted in pragmat...
Was Eliezer Yudkowsky right to give himself 10% to succeed with HPMoR in 2010?
by momom2 — 2022-06-14T07:00:29.955Z

In Hero Licensing, Eliezer Yudkowsky states that in 2010, he would have given himself 10% chance of HPMoR being very successful.

He then goes on to explain why he thought that instead of something lower; but I don't understand why he thought that instead of something higher: given that HPMoR did end up successful, it looks like it actually had higher than 10% chance of happening. Or maybe I'm by coincidence in the 1 in 10 worlds where it ended up successful? How can I tell?

If I try to use Bayes's law: let's call A "HPMoR is successful in 2022" and B "In 2010, there's at most 10% chance that HPMoR will be successful".
I want to update B based on A: P(B|A) = P(B)P(A|B)/P(A)
P(A|B) = 10%, P(A) = ~1
So it appears 1/10 as likely that EY was correct in 2010 about his prediction now than it appeared then.

Is this calculation correct?

Obviously, I'm not that interested about this particular result. In general, how do I improve my own prediction-making based on evidence?
Vael Gates: Risks from Advanced AI (June 2022)
by Vael Gates — 2022-06-14T00:54:25.448Z

I'm a postdoctoral researcher at Stanford HAI and CISAC. I recently gave a HAI Seminar Zoom talk, in which I lay out some of the basic arguments for existential risk from AI during the first 23m of the talk, after which I describe my research interviewing AI researchers and answer Q&A.
I recommend the first 23m as a resource to send to people who are new to these arguments (the talk was aimed at computer science researchers, but is also accessible to the public). This is a pretty detailed, current as of June 2022, public-facing overview that's updated with April-May 2022 papers, and includes readings, funding, additional resources at the bottom of the page. Also, if you want to give your own version of the talk, please be my guest! No attribution necessary for the first 23m (I would like attribution for the "interviews" part of the talk).

The slides are here.
An optional transcript of the entire talk is below (marked at the 23m mark); thanks to Jonathan Low for drafting it and Vaidehi Agarwalla for suggesting it.
[Link to post on the EA Forum]

Zoom Talk

Transcript

Dr. Vael Gates, a HAI-CISAC postdoc at Stanford University, describes their work interviewing researchers about their perceptions of risks from current and future AI. The transcript below runs over the first 23 minutes of the talk, in which they introduce some recent AI developments, researcher timelines for AGI, and the case for existential risk from non-aligned AGI. The latter part of the talk focuses on Gates’s preliminary research results, and audience Q&A.

The transcript, which has been edited for clarity, is below. Dr. Gates’s talk is available to watch in full on the Stanford HAI website, and on YouTube.

My talk today is called “Researcher Perceptions of Current and Future AI”, though it could also be called “Researcher Perceptions of Risks from Advanced AI”, as my talk is actually focused on risk from advanced AI.

The structure of this talk is as follows: I'm going to give some context for the study I did, I'll talk about the development of AI, the concept of AGI, and the alignment problem and existential risk. [Then I'll go on to the research methods I used in this study, some of the research questions I asked researchers, and the interim results, finishing with some concluding thoughts. We should have about 10-15 minutes o...
OpenAI: GPT-based LLMs show ability to discriminate between its own wrong answers, but inability to explain how/why it makes that discrimination, even as model scales
by Aditya Jain — 2022-06-13T23:33:13.049Z
Link: https://openai.com/blog/critiques/

This seems concerning. Not an expert so unable to tell how concerning it is. Wanted to start a discussion! Full text: https://openai.com/blog/critiques/

Edit: the full publication linked in the blog provides additional details on how they found this in testing. See Appendix C. I'm glad OpenAI is at least aware of this alignment issue and plans to address it with future language models, postulating how changes in training and/or testing could ensure there is greater/more accurate/more honest model outputs.

Key text:

Do models tell us everything they know? To provide the best evaluation assistance on difficult tasks, we would like models to communicate all problems that they “know about.” Whenever a model correctly predicts that an answer is flawed, can the model also produce a concrete critique that humans understand?

This is particularly important for supervising models that could attempt to mislead human supervisors or hide information. We would like to train equally smart assistance models to point out what humans don’t notice.

Unfortunately, we found that models are better at discriminating than at critiquing their own answers, indicating they know about some problems that they can’t or don’t articulate. Furthermore, the gap between discrimination and critique ability did not appear to decrease for larger models. Reducing this gap is an important priority for our alignment research.
Continuity Assumptions
by Jan_Kulveit — 2022-06-13T21:31:29.620Z

This post will try to explain what I mean by continuity assumptions (and discontinuity assumptions), and why differences in these are upstream of many disagreements about AI safety. None of this is particularly new, but seemed worth reiterating in the form of a short post.

What I mean by continuity

Start with a cliff:
This is sometimes called the Heaviside step function. This function is actually discontinuous, and so it can represent categorical change.
In contrast, this cliff is actually continuous:

From a distance or in low resolution, it looks like a step function; it gets really steep. Yet it is fundamentally different.

How is this relevant

Over the years, I've come to the view that intuitions about whether we live in a "continuous” or "discontinuous” world are one of the few top principal components underlying all disagreements about AI safety. This includes but goes beyond the classical

continuous

vs.

discrete

takeoff debates.
A lot of models of what can or can't work in AI alignment depends on intuitions about whether to expect "true discontinuities" or just "steep bits". This holds not just in one, but many relevant variables (e.g. the generality of the AI’s reasoning, the speed or differentiability of its takeoff).
The discrete intuition usually leads to sharp categories like:

Before the cliff	After the cliff
Non-general systems.
Lack the core of general reasoning, that which allows thought in domains far from training data	General systems.
Capabilities generalise far
Weak systems - that won't kill you, but also won't help you solve alignment	Strong systems - that would help solve alignment, but unfortunately will kill you by default, if unaligned
Systems which may be misaligned, but aren't competently deceptive about it	System which is actively modelling you at a level where the deception is beyond your ability to notice
Weak acts	Pivotal acts
…	…

In Discrete World, empirical trends, alignment techniques, etc usually don't generalise across the categorical boundary. The right is far from the training distribution on the left. Your solutions don't survive the generality cliff, there are no fire alarms - and so on.

Note that while assumptions about continuity in different dimensions are in principle not necessarily related, and you could e.g. assume continuity in takeoff and discontinuity in gener...