trentmkelly/LessWrong-43k huggingface dataset:
Training My Friend to Cook
by lsusr — 2021-08-29T05:54:50.065Z

During pre-vaccination covid, my friend Brittany and I agreed to hang out once per week as quarantine buddies. Brittany has a laundry list of health problems, many of which are exacerbated by a poor diet. Brittany loves healthy food. She ate a diet based around TV dinners because she didn't know how to cook. My goal for covid lockdown was to train Brittany to cook.

It was summer. I started out by picnicing in the park together. I brought rice, beans, sauerkraut, sliced radishes, homegrown tomatoes, tortilla chips and homemade salsa. Though simple, the food I made was far tastier than anything Brittany was eating. It's not hard for fresh homemade food to beat TV dinners. After each picnic I'd send the leftovers home with her in lovely glass jars. This associated "homemade food" with sunshine, verdant trees, picnic tables and quality time with good friends.

I talked about how cheap it was to make food when your primary ingredients are rice, cabbage, onions and dried beans. This made an impact because Brittany was regularly paying $15 per meal for her TV dinners, a far inferior food. I never said "you should cook" because that would make Brittany feel bad for not cooking. I just talked about how great it was that I could cook. I showed her an ambition she could aspire to.

Brittany is a med student with very limited time. Going to a park takes time. Have already established the "homemade food" = "warm verdant picturesque parks" association, we moved our hangouts to her apartment. I brought the ingredients to her apartment and cooked them there. She insisted on splitting the grocery bill. A single night of my cooking would give her days of good food for the price of a single TV dinner.

Though I always provided Brittany with leftovers, I also made sure that they never lasted more than a few days. Brittany would eat delicious food for a few days and then she'd be back to her TV dinners. Brittany looked forward to our hangouts.

Brittany is Taiwanese and loves healthy food. I upgraded rice and beans to her favorite foods. I sautéed bok choy. I blanched Chinese broccoli. I figured out a mild mapo dofu. I learned how to prepare 絲瓜. After Brittany had watched me sauté vegetables a few times I encouraged her take over. I made a big show of how I got to relax while she did the cooking.

One day Brittany ordered a box of meal kits. I gently, but stubbornly, refused to help her wi...
Superintelligent Introspection: A Counter-argument to the Orthogonality Thesis
by DirectedEvolution — 2021-08-29T04:53:30.857Z

John Wentworth serendipitously posted How To Write Quickly While Maintaining Epistemic Rigor when I was consigning this post to gather dust in drafts. I decided to write it up and post it anyway. Nick Bostrom's orthogonality thesis has never sat right with me on an intuitive level, and I finally found an argument to explain why. I don't have a lot of experience with AI safety literature. This is just to explore the edges of an argument.

Here is Bostrom's formulation of the orthogonality thesis from The Superintelligent Will.

The Orthogonality Thesis

Intelligence and final goals are orthogonal axes along which possible agents can freely vary. In other words, more or less any level of intelligence could in principle be combined with more or less any final goal.

My Counter-Argument

Let's assume that an agent must be made of matter and energy. It is a system, and things exist outside the system of the agent. Its intelligence and goals are contained within the system of the agent. Since the agent is made of matter and energy, its intelligence and goals are made of matter and energy. We can say that it possesses an intelligence-device and a goal-device: the physical or energetic objects on which its intelligence and goals are instantiated.

In order for the orthogonality thesis to be true, it must be possible for the agent's goal to remain fixed while its intelligence varies, and vice versa. Hence, it must be possible to independently alter the physical devices on which these traits are instantiated. Note that I mean "physical devices" in the loosest sense: components of the intelligence-device and goal-device could share code and hardware.

Since the intelligence-device and the goal-device are instantiated on physical structures of some kind, they are in theory available for inspection. While a superintelligent agent might have the power and will to prevent any other agents from inspecting its internal structure, it may inspect its own internal structure. It can examine its own hardware and software, to see how its own intelligence and goals are physically instantiated. It can introspect.

A superintelligent agent might find introspection to be an important way to achieve its goal. After all, it will naturally recognize that it was created by humans, who are less intelligent and capable than the agent. They may well have used a suboptimal design for its intelligence, or oth...
Death by a Single Cut
by lsusr — 2021-08-29T03:58:27.332Z

There is a tactic in formal debate where you make several independent arguments to support a claim. Any one of the arguments is sufficient to prove your case. The arguments are redundant. Your wins even if all but one of your arguments for it is struck down.

Rationality is the opposite of debate. A rationalist should make only the strongest argument.

Suppose I believe "There is no monkey in my closet." There are two arguments I could put forth to support my claim.

I live in the Pacific Northwest. Monkeys don't live in the Pacific Northwest.

I looked in my closet and observed that there was no monkey in it.

If I was in a formal debate then I would put forth both claims. But I am a rationalist. My primary objective isn't to persuade other people. It is to identify my own reasons for believing things. I want to pinpoint the facts which, if inverted, would change my mind.

Which of my two contentions is stronger? Which bit of evidence, if inverted, would cause me to change my mind?

If I discovered that monkeys actually do live in the Pacific Northwest then I would continue to believe there is no monkey in my closet.

If I looked in my closet and saw a monkey then I would believe there is a monkey in my closet, ecology be damned.

A rationalist's arguments should be stripped down to the bare essentials. The whole lattice should collapse with the removal of a single argument. If you can't cut an argument down to its cruxes then you haven't identified your cruxes. If you haven't identified your cruxes then you don't know why you believe what you believe.

Statistical Evidence

What about something like "anthropogenic global warming is real"? Doesn't a lot of evidence go into a conclusion like that?

Since I'm not a climate scientist, I yield to the scientific consensus on climate science. My crux is: "The scientific consensus believes anthropogenic global warming is real." If I discovered the scientific consensus disbelieves "anthropogenic global warming is real" then I would change my mind.

I believe there is scientific consensus which believes in anthropogenic global warming because various trustworthy sources assure me there is one. No single trustworthy source is a crux. If <trustworthy news source x> reported there was no scientific consensus then my confidence that there is a scientific consensus would be weakened, but it would not instantly break.

Does this vi...
In search for plausible scenarios of AI takeover, or the Takeover Argument
by Ozyrus — 2021-08-28T22:30:34.827Z

I'm searching for writeups on scenarios of AI takeover. I'm interested in "?????" in
"1. AGI is created
2. ??????
3. AGI takes over the world"
Yeah, I played the paperclips game. But I'm searching for a serious writeup of possible steps rogue AI can take to take over the world and why(if) it inevitably will.
I've come to understand that it's kinda taken for granted that unaligned foom equals catastrophic event.
I want to understand if it's just me that's missing something, or if the community as a whole is way too focused on alignment "before", whereas it might also be productive to model the "aftermath". Make the world more robust to AI takeover, instead of taking for granted that we're done for if that happens.
And maybe the world is already robust to such takeovers. After all, there are already superintelligent systems out in the world, but none of them have taken it over just yet (I'm talking governments and companies).
To sum it up: I guess I'm searching for something akin to the Simulation Argument by Bostrom, let's call it "the Takeover Argument".
Brain-Computer Interfaces and AI Alignment
by niplav — 2021-08-28T19:48:52.614Z

cross-posted from niplav.github.io

I discuss arguments for and against the usefulness of brain-computer
interfaces in relation to AI alignment, and conclude that the path to
AI going well using brain-computer interfaces hasn't been explained in
sufficient detail.

Brain-Computer Interfaces and AI Alignment

As a response to Elon Musk declaring that NeuraLink's purpose is to
aid AI alignment,
Muehlhauser
2021
cites Bostrom 2014 ch.
2
for reasons why brain-computer interfaces seem unlikely to be helpful
with AI alignment. However, the chapter referenced concerns itself
with building superintelligent AI using brain-computer interfaces,
and not specifically about whether such systems would be aligned or
especially alignable.

Arguments against the usefulness for brain-computer interfaces
in AI alignment have been raised, but mostly in short form on
twitter (for example here).
This text attempts to collect arguments for and against brain-computer
interfaces from an AI alignment perspective.

Epistemic Status

I am neither a neuroscientist nor an AI alignment researcher (although I
have read some blogposts about the latter), and I know very little about
brain-computer interfaces (from now on abbreviated as “BCIs”). I have
done a cursory internet search for a resource laying out the case for the
utility of BCIs in AI alignment, but haven't been able to find anything
that satisfies my standards (I have also asked on the LessWrong open
thread
and on the AI alignment channel on the Eleuther AI discord server, and
not received any answers that provide such a resource (although I was
told some useful arguments about the topic)).

I have tried to make the best case for and against BCIs, stating some tree
of arguments that I think many AI alignment researchers tacitly believe,
mostly taking as a starting point the Bostrom/Yudkowsky story of AI risk
(although it might be generalizable to a
Christiano-like story; I don't know enough about
CAIS
or ARCHES to make a judgment about the applicability of the arguments).
This means that AI systems will be assumed to be maximizers, as
mathematical descriptions of other optimization idioms are currently
unsatisfactory.

Existing Texts

The most thorough argument for the usefulness of BCIs for AI alignment is
Urban
2017
(which I was pointed to by in this comment, thanks!).

The text mostly concerns itself with the current status of BCI techno...
How To Write Quickly While Maintaining Epistemic Rigor
by johnswentworth — 2021-08-28T17:52:21.692Z

There’s this trap people fall into when writing, especially for a place like LessWrong where the bar for epistemic rigor is pretty high. They have a good idea, or an interesting belief, or a cool model. They write it out, but they’re not really sure if it’s true. So they go looking for evidence (not necessarily confirmation bias, just checking the evidence in either direction) and soon end up down a research rabbit hole. Eventually, they give up and never actually publish the piece.

This post is about how to avoid that, without sacrificing good epistemics.

There’s one trick, and it’s simple: stop trying to justify your beliefs. Don’t go looking for citations to back your claim. Instead, think about why you currently believe this thing, and try to accurately describe what led you to believe it.

I claim that this promotes better epistemics overall than always researching everything in depth.

Why?

It’s About The Process, Not The Conclusion

Suppose I have a box, and I want to guess whether there’s a cat in it. I do some tests - maybe shake the box and see if it meows, or look for air holes. I write down my observations and models, record my thinking, and on the

bottom line

of the paper I write “there is a cat in this box”.

Now, it could be that my reasoning was completely flawed, but I happen to get lucky and there is in fact a cat in the box. That’s not really what I’m aiming for; luck isn’t reproducible. I want my process to robustly produce correct predictions. So when I write up a LessWrong post predicting that there is a cat in the box, I don’t just want to give my bottom-line conclusion with some strong-sounding argument. As much as possible, I want to show the actual process by which I reached that conclusion. If my process is good, this will better enable others to copy the best parts of it. If my process is bad, I can get feedback on it directly.

Correctly Conveying Uncertainty

Another angle: describing my own process is a particularly good way to accurately communicate my actual uncertainty.

An example: a few years back, I wondered if there were limiting factors on the expansion of premodern empires. I looked up the peak size of various empires, and found that the big ones mostly peaked at around the same size: ~60-80M people. Then, I wondered when the US had hit that size, and if anything remarkable had happened then which might suggest why earlier empires brok...
Moral Golems Assume Uncertainty
by Erich_Grunewald — 2021-08-28T14:05:38.909Z
Link: https://www.erichgrunewald.com/posts/moral-golems-assume-uncertainty/

[T]he Christian humanist and Hebraist Johannes Reuchlin (1492) report[ed] the creation of an artificial anthropoid whose forehead bore the Hebrew inscription "YHVH Elohim emeth" (The name of God is truth) and who fell dead to the ground when the aleph, the first letter forming the Hebrew word for truth, was removed.[1]

– Cathy Gelbin

I used to consider myself vaguely utilitarian, but after reading Fellow Creatures[2] and other texts by Christine M. Korsgaard I began to see myself more as a Kantian. Her ethics make a lot of sense to me, though I still find myself thinking at times in utilitarian or even Schopenhauerian terms. The weird thing is that I want to pick a favourite. Or maybe I should say "find myself wanting", because of course from some distance I can see that, when it comes to ethics, where evidence is scarce and p-values rarely cross the magical threshold, a stiff dose of uncertainty is a good thing. I suppose I find myself wanting to pick a favourite partly because I want to make it part of my identity (hmm, hmm) and partly because I feel that it is the necessary groundwork one needs to make before one can decide what to think about object-level moral issues.

That was the tension I was trying to resolve when I wrote the post about moral golems. I argued, or suggested perhaps, in it that we "should not search for the one true moral system, [but] seek many moral systems that each have a truth in them" and that "the project of ethics is not dissimilar from that of the social sciences, say, in that it tries to produce answers by modelling the essential characteristics of very complicated human structures". I likened such a moral system to a golem that we inscribe with truth, animate and instruct, and which, like a statistical model, does no more and no less than what we tell it to do. I was not aware at the time that a group of philosophers had been working on exactly this question under the heading moral uncertainty, but I am happy to report now that my discovering this caused me no annoyance, though it might have long ago.

How would the moral golem approach work in practice? Say Suleiman has decided that Utilitarianism, Kantianism and the ethics of Arthur Schopenhauer all have some truth in them. He is not sure about which one of them is true, if one is at all, but they seem like likely candidates. An Ottoman general knocks on his door and asks hi...
What are good alignment conference papers?
by adamShimi — 2021-08-28T13:35:37.824Z

I regularly debate with people whether pushing for more mainstream publications in ML/AI venues by alignment researchers is a good thing. So I want to find data: alignment papers published at NeurIPS and other top conferences (journals too, but there're less relevant in computer science) by researchers. I have already some ways of looking for papers like that (including the AI Safety Papers website), but I'm curious if people here have favorite that they think I should really know/really shouldn't miss.

(I volontarily didn't make the meaning of "alignment paper" more precise because I also want to use this opportunity to learn about what people consider "real alignment research")
The AI research process as incipient superintelligence
by Carlos Ramirez — 2021-08-27T20:16:20.513Z

EDIT: So this got upvotes while being a draft. Not sure what happened there, but it's now officially posted, even though it's incomplete. Won't be including it in the Guide for now, since the style clashes.

If I include it in the Guide I'll come back to this and give it a conclusion.

Summary

It is best for laypersons to understand the AI research process (henceforth the Process) as an AGI of unclear alignment that is working to produce a potentially superintelligent successor AGI.

The identity of AGI and AI research

Safety-pertinent traits of an AGI

1. Opacity

An AGI is necessarily a black box to any human interlocutors. Humans are relatively transparent to other humans solely because we have inbuilt unconscious heuristics that are adapted to making sense of other humans. These heuristics would not work at all when targeted at a non-human intellect, such as an AGI, creating an infinite chain of suspicion between the human and the non-human intellect. If the non-human intellect is also superintelligent though, the chain of suspicion is one-way: the human would be foolish to ever trust the non-human intellect (in any scenario other than one where there is somehow some guarantee of cooperation with the human), but the non-human intellect might be able to "trust" the human, to whatever extent the concept of "trust" can be said to apply to it.

One would think that, as a human construct, an AI would necessarily be completely transparent to at least some humans, but the current state-of-the-art techniques result in mathematical models that are thoroughly unintelligible when it comes to determining their decision making process, much like the human brain, which is the inspiration of those techniques. Though this does not mean an AI resembles a human brain much, the human brain does appear to place a lower bound on how complex and intelligible we should expect the decision making process of an AGI to be, that is, an AGI is unlikely to be simpler and more intelligible than a brain, purely because it would be capable of doing far, far more than a brain. At any rate, the state-of-the-art is not trending to simplicity and intelligibility, and more generally, and as a rule, human science and technology do not tend towards increasing simplicity and intelligibility, but to completely the opposite, any simplicity and intelligibility being purely an illusion for the benefit of ...
Can you control the past?
by Joe Carlsmith — 2021-08-27T19:39:29.993Z

(Cross-posted from Hands and Cities. Lots of stuff familiar to LessWrong folks interested in decision theory.)

I think that you can “control” events you have no causal interaction with, including events in the past, and that this is a wild and disorienting fact, with uncertain but possibly significant implications. This post attempts to impart such disorientation.

My main example is a prisoner’s dilemma between perfect deterministic software twins, exposed to the exact same inputs. This example that shows, I think, that you can write on whiteboards light-years away, with no delays; you can move the arm of another person, in another room, just by moving your own. This, I claim, is extremely weird.

My topic, more broadly, is the implications of this weirdness for the theory of instrumental rationality (“decision theory”). Many philosophers, and many parts of common sense, favor causal decision theory (CDT), on which, roughly, you should pick the action that causes the best outcomes in expectation. I think that deterministic twins, along with other examples, show that CDT is wrong. And I don’t think that uncertainty about “who are you,” or “where your algorithm is,” can save it.

Granted that CDT is wrong, though, I’m not sure what’s right. The most famous alternative is evidential decision theory (EDT), on which, roughly, you should choose the action you would be happiest to learn you had chosen. I think that EDT is more attractive (and more confusing) than many philosophers give it credit for, and that some putative counterexamples don’t withstand scrutiny. But EDT has problems, too.

In particular, I suspect that attractive versions of EDT (and perhaps, attractive attempts to recapture the spirit of CDT) require something in the vicinity of “following the policy that you would’ve wanted yourself to commit to, from some epistemic position that ‘forgets’ information you now know.” I don’t think that the most immediate objection to this – namely, that it implies choosing lower pay-offs even when you know them with certainty – is decisive (though some debates in this vicinity seem to me verbal). But it also seems extremely unclear what epistemic position you should evaluate policies from, and what policy such a position actually implies.

Overall, rejecting the common-sense comforts of CDT, and accepting the possibility of some kind of “acausal control,” leaves us in str...