trentmkelly/LessWrong-43k huggingface dataset:
Updating Utility Functions
by JustinShovelain — 2022-05-09T09:44:46.548Z

This post will be about AIs that “refine” their utility function over time, and how it might be possible to construct such systems without giving them undesirable properties. The discussion relates to

corrigibility

,

value learning

, and (to a lesser extent)

wireheading

.

We (Joar Skalse and Justin Shovelain) have spent some time discussing this topic, and we have gained a few new insights we wish to share. The aim of this post is to be a brief but explanatory summary of those insights. We will provide some motivating intuitions, a problem statement, and a possible partial solution to the problem given in the problem statement. We do not have a complete technical solution to the problem, but one could perhaps be built on this partial solution.

Sections which can be skipped are marked with an asterisk (*).

Brief Background

*

This section says things that you probably already know. The main purpose of it is to prime you.

In the “

classical

”

picture

of AI systems, the AI contains a utility function that encodes a goal that it is trying to accomplish. The AI then selects actions whose outcome it expects will yield high utility (roughly). For example, the utility function might be equal to the number of paperclips in existence, in which case the AI would try to take actions that result in many paperclips.

In the “classical” picture, the utility function is fixed over time, and corresponds to an equation that at some point is typed into the AI’s source code. Unfortunately, we humans don’t really know what we want, so we cannot provide such an equation. If we try to propose a specific utility function directly, we

typically

get a function that would result in

catastrophic

consequences

if it were pursued with arbitrary competence. This is worrying.

This problem could perhaps be alleviated if we could construct AIs that can refine their utility function over time. For example, maybe we could create an AI that starts out with an imperfect understanding of human values, but then improves that understanding over time. Such an AI should ideally “want” to improve its understanding of human values (and actively come up with ways to do this), and it should at minimum not resist if humans attempt to update it. Unfortunately, it turns out to be difficult to design such systems. In this post we will talk more about this approach.

&nb...
Transcripts of interviews with AI researchers
by Vael Gates — 2022-05-09T05:57:15.872Z

tldr: I conducted a series of interviews with 11 AI reseachers to discuss AI safety, which are located here: TRANSCRIPTION LINK. If you are interested in doing outreach with AI researchers, I highly recommend taking a look!
[Cross-posted to the EA Forum.]

Overview

I recently conducted a series of interviews with 11 AI researchers, wherein I laid out some reasons to be concerned about long-term risks from AI.

These semi-structured interviews were 40-60 minutes long and conducted on Zoom. Interviewees were cold-emailed, were paid for their participation, and agreed that I may share their anonymized transcripts.

Six of the interviews were with researchers who had papers accepted at NeurIPS or ICML in 2021. Five of the interviews were with researchers who were informally categorized as “particularly useful to talk to about their opinions about safety” (generally more senior researchers at specific organizations).

I’m attaching the raw transcripts from these 11 interviews, at the following link. I’ve also included the approximate script I was following, post-interview resources I sent to interviews, and informal interview notes in the associated “README” doc. Ideally I’d have some analysis too, and hopefully will in the future. However, I think it’s useful— particularly for people who plan to start similar projects— to read through a couple of these interviews, to get an intuitive feel for what conversations with established AI researchers can feel like.

Note: I also interviewed 86 researchers for a more complete academic, under-IRB study (whose transcripts won’t be released publicly), whose results will be posted about separately on LessWrong once I finish analyzing the data. There will be substantially more analysis and details in that release; this is just to get some transcripts out quickly. As such, I won't be replying to a lot of requests for details here.

Thanks to Sam Huang, Angelica Belo, and Kitt Morjanova, who helped clean up the transcripts!

Personal notes

I worry that sometimes young EAs think of researchers like a crowd of young, influenceable proto-EAs. One of my models about community-building in general is that there’s many types of people, some who will be markedly more sympathetic to AI safety arguments than others, and saying the same things that would convince an EA to someone whose values don’t align will not be fruitful. A se

...
[Scribble] Bad Reasons Behind Different Systems and a Story with No Good Moral
by Rana Dexsin — 2022-05-09T05:21:53.221Z

I wrote this on impulse as a response to “A Reason Behind Bad Systems, and Moral Implications of Seeing This Reason”. It is essentially a first draft and is probably badly written. The plot also got away from me in the later sections; my mental processes surrounding a lot of these phenomena are too cross-linked. However, I sense it may be more useful for this to exist in the open than not, and I think if I decide to hide this away for rewriting, realistically I will not do so and it will languish forever… so here it is. Perhaps it can become something better later; perhaps not.

For copyright and moral rights purposes, as the sole author of everything in this post that is not a quotation, I place all such text in the public domain.

Epistemic status: fictional, quasi-allegorical, biased, exploratory.

I

The programmer with the messy program did what any sensible software engineer would have done in her shoes: Set aside the old software and built something useful, that did what was needed, using the full power of the company’s modern computers.

A month later, Evelyn's manager at Yottascape, Henrietta, stormed into Evelyn's office with a notepad about the hundred different telephone calls the department had gotten complaining that the results from the new system were subtly inconsistent with those from the old. Other people who had little to no understanding of the code behind the figures they were seeing had built processes and expectations based on patterns they observed, patterns which had stayed stable for years.

Some of the new outputs had caused cascading instabilities. One manufacturing location had run out of slack on a key inflow of materials, temporarily halting outflow of parts to a second location, which revealed that workers at a second location had become lax in ensuring that the machine they were using—which needed to be kept at a certain temperature or suffer expensive damage that would put it out of commission for months—was actually kept at that temperature. The intake at the second location had been so constant before that it created a self-regulating effect, and they had stepped down control checks to one third the original frequency, a cost-saving measure which had been widely praised by management. Now the damage would lead to broken promises which would have to be documented in their next SEC filing. Even before then, it would involve a flu...
What is the best day to celebrate Smallpox Eradication Day?
by Orborde — 2022-05-09T04:02:02.805Z

Several of my friends posted on Facebook/threw parties this weekend for a date of May 8. However, searching on Facebook, I find that I and several other friends have been posting celebratory messages on Facebook on December 9 in previous years.

This inconsistency is probably because there are two obvious candidate dates for eradication:

The global eradication of smallpox was certified, based on intense verification activities, by a commission of eminent scientists on 9 December 1979 and subsequently endorsed by the World Health Assembly on 8 May 1980.

"500 Million, But Not a Single One More", which is widely shared and has been adapted at least once for reading at a large Secular Solstice, uses the December 9 date.

I think creating more consensus on which date is the "official" one for celebration purposes would be a small improvement (at the very least: I would stop confusing my relatives by year-to-year inconsistency on when I message them a little commemoration). So, which is it?
A reason behind bad systems, and moral implications of seeing this reason
by Edward Pascal — 2022-05-09T03:16:50.434Z

(Epistemic status: Pretty sure the premises are all correct and the metaphor is accurate. Exploring the implications and conclusions.)

A programmer was given an aging piece of software to use to complete a project. To the company, using the old software seemed like the path of least resistance. But, opening the program and looking at the code, the programmer saw it had been hacked, edited, and added to by twenty or more programmers before her, each with their own style and vision of what a good program should be, each according to the conventions and concerns of the day.

"Ahhh, Legacy code," the programmer said. She frowned a little.

By now, the software was full of unused code and had a maze of lines of programming that referred to other lines that referred to yet other lines, and so forth. The programmer found it almost impossible to sort out exactly what many parts of the program were designed to do. Worse, though the computers used by the company were high-tech, parts of the software were designed to run on computers from decades before. The program was a nightmare to work with. Normally one had to doctor the finicky inputs, and often the software returned incorrect, puzzling, or even meaningless outputs.

When software like this does anything useful at all, it is a small miracle.

Most organizations, social hierarchies, and belief systems are heavy with garbage. There is no one to blame, because most of those systems are constructed from layers of intentions and interpretations spanning years, decades, or even centuries. Hence things like a legal system with thousands of pages of code and case law. Plenty of these systems evolved around social realities that have changed immensely since they were designed. Meanwhile, your own experience tells you that no matter how dysfunctional or downright it is, no organization, belief system, or social structure will ever agree with you that it is wrong. For the most part, none of these systems were designed to take the steps to eliminate themselves when they are no longer useful.

Once a system takes root in society and ages, it loses its agency, intelligence, and will. That system is usually very hard to remove even if it starts to work against its original intention. Most people either thoughtlessly adopt or begrudgingly embed themselves in whatever systems are presented by their culture and society. Doing so usually ...
An Alternative Interpretation of Physics
by dadadarren — 2022-05-09T00:52:14.352Z
Link: https://www.sleepingbeautyproblem.com/an-alternative-interpretaion-of-physics/

1. Physics With Minimal Assumptions

“I am this thing”. That is perhaps the most basic fact I know of. It is not a conclusion based on reasoning, but something much more instinctive: “I know how it feels.“. For me, the only subjective experience is from this thing. It’s at the very center of my perspective. Let’s call it “self”.

I learn about my surrounding by interacting with them. Via these interactions, I formed the conception of external objects and the world around me. If I study the interactions carefully, I would discover certain patterns: rules that can explain and predict them. These rules also help me to describe external objects as the interactions’ counterparties.

Then there’s a crucial realization: the first-person perspective I’m experiencing doesn’t have to be the only valid perspective. I realized external objects also interact with their environments. I can imagine thinking from a particular object’s viewpoint. So that thing would become the “self”, and I become an external object interacting with it.

Back to the rules that my interactions seem to follow. There are restrictive rules, that only work for a limited range of perspectives and interactions. And there are other, more general rules applicable to a wide range of perspectives. I should try to induce rules of this type as they give a deeper insight into how the world works. Let’s call the general rules “physical laws”, and the depictions of the world using these laws “physical descriptions”.

2. Classical Physics Is Weird

This parsimonious interpretation of physics has some implications. First of all, physical analysis has to be conducted from a prespecified perspective. It can be the natural first-person perspective of you or me or the perspective of any object. Which perspective is the reasoning to be conducted from i.e. what is the “self”, is an exogenous input, not something to be explained by physics.

Following the above, studying interactions received by the “self” (the thing at the perspective center) enables conceptualizing external objects as the actions’ counterparties. Physical descriptions of external objects are based on these interactions.

Finally, because objects are described based on their interactions with the self, the “self” itself is not describable. I.E. the “thing at the perspective center” is not within the domain of explanation. To physically analyze so...
Ion Implantation: Theory, Equipment, Process, Alternatives
by nomagicpill — 2022-05-08T22:30:51.878Z
Link: https://210ethan.github.io/research/implant.html

Ion implantation is a common process used in the semiconductor industry to change the properties of a material, namely silicon (the substrate). Physics, equipment used, process considerations, alternatives, and further resources are discussed.

Note: I copied this direct from my website and did not take the time to format the inline math. The symbols should be clear enough that subscripts/superscripts are not needed.

Contents

Physics

Dose vs. Concentration
Stopping
Activation Anneal

Equipment

Sources
Mass Analyzer
Acceleration
Endstation

Process Considerations

Implant Masks
Channeling
Uniformity

Doping Alternatives

Diffusion Doping
Spin-On Doping

Further Resources
See Also

Physics

First, why are ions shot into Si? What does it do from a physics perspective?

The reason has to do with the band gap, the energy range between the valence band and conduction band.

Within Si exist both electrons (negatively-charged) and electron hole (positively-charged). The hole is a bit of an oddity: it's an unoccupied space where an electron could exist, but doesn't at that moment. While not literally a particle, they can be treated and thought of as such. Current can be viewed as the flow of electrons or holes. When a bias is applied, electrons begin to move and occupy holes. When electron A moves to hole B, hole A is formed. This propagates and creates the flow of charge, or current.

What does this have to do with Si? Si has four valence electrons in its four states of 3s23p2. Those four valence electrons covalently bond with the surrounding Si crystal lattice structure, effectively filling that Si atom's valence shell. So, when an ion is introduced into the lattice that does not have four valence electrons, one of two things happen: there are free electrons (if the number of valence electrons is >4, such as in phosphorus's case) or free holes (valence electrons <4, such as boron). In the first case, four of P's electrons have bonded to surrounding Si atoms' valence electrons, leaving the fifth one free and able to contribute to current—a donor impurity, because it donates an electron. This creates an n(egative)-type semiconductor. In the second case, three of B's electrons have bonded to surrounding Si atoms, but one Si atom and the B atom both have incomplete valence shells (7/8, 6/8 filled, respectively). To fill the vacancy, the Si and B will pull a free electron from the lattice, leav...
Long COVID risk: How to maintain an up to date risk assessment so we can go back to normal life?
by Sameerishere — 2022-05-08T19:56:51.265Z

Despite Zvi's "Long Long Covid Post" concluding in February that Long COVID risk among healthy, vaccinated individuals is low enough that it's worth pretty much going back to normal life, I haven't felt comfortable doing so given the array of claims to the contrary.

Some of them have surfaced on LessWrong itself:

https://www.lesswrong.com/posts/emygKGXMNgnJxq3oM/your-risk-of-developing-long-covid-is-probably-high (March, by a poster who had not read Zvi's original post)
https://www.lesswrong.com/posts/vSjiTrHkckTPrirhS/hard-evidence-that-mild-covid-cases-frequently-reduce (May)

Others I have come across from friends or on Twitter.

My skills at carefully evaluating scientific research are fairly limited, and I'd also like to avoid spending all of my free time doing so, so I've been kind of stuck in this limbo for now.

Compounding the challenge of deciding what risks to take is that MicroCOVID doesn't seem to account for the increasing rate of underreporting or the much higher transmissibility of recent Omicron subvariants, making it really hard to decide what level of risk a given activity will pose. And given the transmissibility of those variants, and society's apparent decision to just ... ignore the risk of Long COVID and go back to normal, trying to avoid getting COVID going forward will be more and more socially costly.

I'm sure I'm not the only one in this situation.

So:

Is anyone confident going back to normal life despite claims to the contrary without feeling the need to read and evaluate each new study on Long COVID? Why? What logic / heuristics inform that assessment?

This seems to be Zvi's current stance, given he seems to be focused elsewhere with his recent posts, so Zvi, if you're reading this, I'd be curious to hear your thoughts!

Has anyone been tracking claims to the contrary and assessing their validity (e.g. based on the sorts of critiques Zvi covered in his post)?
Would anyone be interested in contributing to a systematic effort to do so?

Could we start some sort of centralized database of studies on Long COVID (a spreadsheet? a wiki?) and folks grab one or two here and there, evaluate them, and note their assessment / rationale?
Would folks be interested in contributing to a Kickstarter or something to pay a researcher (e.g. Elizabeth, Zvi, Scott - I don't know if any of them have bandwidth / a price at which they would be interested in d

...
Demonstrating MWI by interfering human simulations
by Yair Halberstadt — 2022-05-08T17:28:27.649Z

TLDR: A demonstration that, given some reasonable assumptions, a quantum superposition of a brain creates an exponential number of independent consciousnesses, and each independent consciousness experiences reality as classical.

The demonstration

I'm not going to explain the Many World's Interpretation of Quantum Physics, since much better expositions exist. Feel free to suggest your favourite in the comments.

Imagine we have the ability to simulate a conscious human in the future. This is almost definitely theoretically feasible, but it is unknown how difficult it is in practice.

I take it as obvious that a simulation of a conscious human will be conscious, given that it will answer exactly the same to any questions about consciousness as an actual human. So you should be as convinced it is conscious as you are about any other human.

Now we create a function that takes an input of length n bits and produces as an output whatever a given simulation responds to that input. For example the input could be an index into a gigantic list of philosophy questions, and the output would be whatever Eliezer Yudkowsky would have replied if you had texted him that question at exactly midnight on the 1st of January 2020.

We want to find the exact number of inputs that produce an output with a given property - e.g. the number of inputs that produce a yes answer, or the number where the answer starts with a letter from the first half of the alphabet, or whatever.

Classically it's obvious the only way to do this is to run the function 2^n times - once for every possible input, and then count how many have the desired property. Doing this will require creating 2^n separate consciousness, each of which live for the duration of the function call.

However, on a quantum computer we only need to run the function O(2^n/2) times using a quantum counting algorithm. So for example, if the inputs were 20 bits long, a classical computer would have to call the simulation function roughly a million times whilst the quantum computer would only use on the order of a thousand calls.

The way this works is by creating a superposition of all possible inputs, and then repeatedly applying the function in such a way as to create interference between the outputs which eventually leaves only the desired number.

So for any given property a simulation can exhibit, we can count how many of the 2^n possible sim...
Notes from a conversation with Ing. Agr. Adriana Balzarini
by Pablo Repetto — 2022-05-08T15:56:04.998Z
Link: https://pabloernesto.github.io/2022/05/08/interview-abalzarini.html

This post is part of my "research" series.

These are my notes from a 2022-04-22 conversation (in Spanish) with Adriana Balzarini Ingeniera Agrónoma. I met Adriana as part of the Board of Directors of the Partido de la Costa Science Club. The conversation wasn't recorded, so I've had to reconstruct some of the arguments from memory, and taken the opportunity to reformat and abridge the content. I thank her for the generous donation of so much of her time.

The Science Club ceased activities back in 2015 due to inhability to form a Board of Directors. We've managed to keep organizing voluntary participation in scientific work under the municipal government. The work we do is very fieldwork-oriented, so I don't know how much I can help you with bibliographic research.

The way we operate is as follows: our working group interfaces with research institutions such as CONICET (Consejo Nacional de Investigaciones Científicas y Técnicas, National Scientific and Technical Research Council) on behalf of the municipal government, and we offer our services for studies that need local data but would normally find it expensive -and early during the COVID-19 pandemic, outright impossible- to obtain. We then meet with the study authors to discuss goals and methodology, which then informs the fieldwork of our volunteers. Our work usually ends up credited as collaboration. The Club used to have some public presence, with school talks and a blog, but the restructuring depreoritized outreach activities.

As for the actual question of research, I rely heavily on expert opinion. I start by keyword-searching on Google/SciELO[1], which nets me relevant papers, organizations (specially research institutions), and -critically- researcher contact information. I then get in touch through whichever means they have made public and ask them for details of my topic of interest. Obviously I do some reading first; showing up for an interview knowing absolutely nothing is disrespectful and a waste of the expert's time.

It may be a different experience on mathematics or the humanities, but I doubt it. Knowledge is a collaborative enterprise. If you want to participate, you need to look for the communities where that knowledge lives. Even in dead disciplines you can find people studying them from a historical perspective. You'll notice you're there when you meet people who speak the jargon. Words that ...