trentmkelly/LessWrong-43k huggingface dataset:
Are there spaces for extremely short-form rationality content?
by Aleksi Liimatainen — 2022-06-22T10:39:30.259Z

I'm experimenting with very short-form, compressed evocations of various mental adjustments. (Yes, I spend a lot of time on Twitter. Why do you ask?) For example, there's an adjunct to the idea of carving reality at the joints that could go something like this: "Reality is interconnected. When you carve it at the joints, what are you severing and how will you stitch it back together?"
Do you think there is a purpose for extreme shortform like this? If yes, were should it go?
Solstice Movie Review: Summer Wars
by SebastianG  — 2022-06-22T01:09:26.749Z

If a classic is a movie everyone praises and no one watches, then Summer Wars is not a classic.

https://en.m.wikipedia.org/wiki/Summer_Wars

I have met only one person who knew the movie without me directing them to it - the one who directed me to it. Maybe I run in the wrong circles, though, and everyone else on LW knows the movie.

It should be a classic. A math-cryptography striver-class nerd is invited to his crush's aristocratic family home to pretend to be her boyfriend.

Meanwhile, the highly integrated web universe, OZ, is undergoing a minor crisis. A video game AI is going rogue and cannibalizing user accounts. It very quickly escapes to other parts of the metaverse, stealing accounts and creating havoc.

I won't say more about the plot. You should go watch the movie.

Instead I want to briefly discuss Summer Wars' vision of the internet. Firstly, this virtual world of OZ seems to have originally been a social network. The social network became a massive, multifaceted platform upon which people would manage many parts of their lives.

Secondly, this platform hosts more than individual personal information. Entire state and municipal governments are running their infrastructure on it as well, which means that platform is considered very secure.

One possibility is that although the VR platform appears like unto a FB/Meta, under the hood it is a decentralized platform with extreme interoperability and nonexistent transaction costs. This would explain some points of plot within the movie that I won't share.

The overall vision of this social media sphere is designed for cinematic action. More VR space, than text or code. It's a brilliant theatrical creation, allowing for drama to unfold digitally, just as it does in the flesh and blood space of the film.

The scale of action of OZ and the flesh and blood world explores the global reach and coordinating power of a platform and the individual's ability to form coalitions to affect change within it.

The rich virtual materiality of OZ displaces the value of the written word. The visual cue rules. For a movie this great makes for great narrative. For our future, perhaps not as much.

Happy Solstice.
Security Mindset: Lessons from 20+ years of Software Security Failures Relevant to AGI Alignment
by elspood — 2022-06-21T23:55:39.918Z

Background

I have been doing red team, blue team (offensive, defensive) computer security for a living since September 2000. The goal of this post is to compile a list of general principles I've learned during this time that are likely relevant to the field of AGI Alignment. If this is useful, I could continue with a broader or deeper exploration.

Alignment Won't Happen By Accident

I used to use the phrase when teaching security mindset to software developers that "security doesn't happen by accident." A system that isn't explicitly designed with a security feature is not going to have that security feature. More specifically, a system that isn't designed to be robust against a certain failure mode is going to exhibit that failure mode.

This might seem rather obvious when stated explicitly, but this is not the way that most developers, indeed most humans, think. I see a lot of disturbing parallels when I see anyone arguing that AGI won't necessarily be dangerous. An AGI that isn't intentionally designed not to exhibit a particular failure mode is going to have that failure mode. It is certainly possible to get lucky and not trigger it, and it will probably be impossible to enumerate even every category of failure mode, but to have any chance at all we will have to plan in advance for as many failure modes as we can possibly conceive.

As a practical enforcement method, I used to ask development teams that every user story have at least three abuser stories to go with it. For any new capability, think at least hard enough about it that you can imagine at least three ways that someone could misuse it. Sometimes this means looking at boundary conditions ("what if someone orders 2^64+1 items?"), sometimes it means looking at forms of invalid input ("what if someone tries to pay -$100, can they get a refund?"), and sometimes it means being aware of particular forms of attack ("what if someone puts Javascript in their order details?").

I found it difficult to cultivate security mindset in most software engineers, but as long as we could develop one or two security "champions" in any given team, our chances of success improved greatly. To succeed at alignment, we will not only have to get very good at exploring classes of failures, we will need champions who can dream up entirely new classes of failures to investigate, and to cultivate this mindset within as many machine ...
A Quick List of Some Problems in AI Alignment As A Field
by Nicholas / Heather Kross — 2022-06-21T23:23:31.719Z
Link: https://www.thinkingmuchbetter.com/main/alignment-field-problems-2022/

1. MIRI as central point of failure for... a few things...

For the past decade or more, if you read an article saying "AI safety is important", and you thought, "I need to donate or apply to work somewhere", MIRI was the default option. If you looked at FLI or FHI or similar groups, you'd say "they seem helpful, but they're not focused solely on AI safety/alignment, so I should go to MIRI for the best impact."

2. MIRI as central point of failure for learning and secrecy.

MIRI's secrecy (understandable) and their intelligent and creatively-thinking staff (good) have combined into a weird situation: for some research areas, nobody really knows what they've tried and failed/succeeded at, nor the details of how that came to be. Yudkowsky did link some corrigibility papers he labels as failed, but neither he nor MIRI have done similar (or more in-depth) autopsies of their approaches, to my knowledge.

As a result, nobody else can double-check that or learn from MIRI's mistakes. Sure, MIRI people write up their meta-mistakes, but that has limited usefulness, and people still (understandably) disbelieve their approaches anyway. This leads either to making the same meta-mistakes (bad), or to blindly trusting MIRI's approach/meta-approach (bad because...)

3. We need more uncorrelated ("diverse") approaches to alignment.

MIRI was the central point for anyone with any alignment approach, for a very long time. Recently-started alignment groups (Redwood, ARC, Anthropic, Ought, etc.) are different from MIRI, but their approaches are correlated with each other. They all relate to things like corrigibility, the current ML paradigm, IDA, and other approaches that e.g. Paul Christiano would be interested in.

I'm not saying these approaches are guaranteed to fail (or work). I am saying that surviving worlds would have, if not way more alignment groups, definitely way more uncorrelated approaches to alignment. This need not lead to extra risk as long as the approaches are theoretical in nature. Think early-1900s physics gedankenexperiments, and how diverse they may have been.

Or, if you want more hope and less hope at the same time, look at how many wildly incompatible theories have been proposed to explain quantum mechanics. A surviving world would have at least this much of a Cambrian explosion in theories, and would also be better at handling this than we are in real-life handli...
What is the difference between AI misalignment and bad programming?
by puzzleGuzzle — 2022-06-21T21:52:57.362Z

Hi there,

I am new to both AI and computer programming in general, and have been teaching myself about AI alignment. I am wondering if there is a difference between AI misalignment and "bad" or uninformed programming.

My understanding of AI misalignment is that it arises more or less from a failure of the programmer to communicate precisely what they want the AI to do. After all, if a computer behaves only according to how it is programmed, then I would think that all AI misalignment problems could be traced back to the code running the AI.

For example, imagine I build a computer program that learns to extract identifying information about website users and share it with other users. However, I forget to encode a safety feature that stops criminals from accessing the information. Now, my program is sharing sensitive information with criminals. Does that mean my program is misaligned, or is it just poorly written?
What I mean by the phrase “getting intimate with reality”
by Luise — 2022-06-21T19:42:56.578Z
Link: https://forum.effectivealtruism.org/posts/5C8ogdyK8M39es3xY/what-i-mean-by-the-phrase-getting-intimate-with-reality

Note: This post is super-quickly written and mostly for reference.

Note 2: This is not my phrase, I first heard it from Sydney von Arx. But I'm not sure if her use of the phrase is the same as mine.

What I mean by the phrase “getting intimate with reality”

you can come up with project ideas abstractly, from first principles.

e.g., "most AI researchers aren't convinced of AI safety. Therefore we should set up lots of 1-1s with AI researchers to convince them."

however, this is usually a bad idea if you aren't intimately familiar with the people, organisations, and systems you are reasoning about. Aka, the slice of reality under consideration

in this case, the relevant slice of reality to understand is ML academia and industry, the researchers themselves, past attempts at having such 1-1 conversations, generally trying to convince someone 1-1 etc.
you could get more intimate with this slice of reality by: being an AI researcher in academia or industry, talking to these researchers, talking to people who've tried having such 1-1s before, trying to convince someone in a 1-1 on this or another issue

what goes wrong if you're not intimate with reality?

you miss most of the laws and dynamics that govern the slice of reality. E.g., you might think the main dynamic that governs convincing AI researchers is presenting them with arguments they've never heard before. In reality, many AI researchers have heard of AI safety, but the arguments weren't presented to them in a high-fidelity, in-depth, and thoughtful way. (At least this is one plausible dynamic that's governing their opinion of AI safety.)
this goes to say, you're simply bad at reasoning about project ideas for this slice of reality
This is more true the more complex the given slice of reality is, and the simpler your current model of it is.

Humans are very complex. Try imagining how bad you would be at coming up with an idea for a date, if you modelled dating as "mate-finding for reproduction" and tried reasoning from first principles (as opposed to relying on your extensive experience with and knowledge about humans and using your rich model of all the laws and dynamics that govern their behaviour)

more examples

a slice of reality you're probably very intimate with is your own life and problems. This is why other people's advice is often well-meant but not helpful—they just don't understand your reality

...
What I mean by the phrase "taking ideas seriously"
by Luise — 2022-06-21T19:42:56.547Z
Link: https://forum.effectivealtruism.org/posts/ZqkK8fshf78fR3Txe/what-i-mean-by-the-phrase-taking-ideas-seriously

This post is super-quickly written and mostly for reference.

Setup:

precondition: agreeing with the idea. (here synonymous with holding a belief in the idea)
beliefs are intertwined. When you update one belief, you often have to update other beliefs to maintain consistency.
E.g. if you update to "my friend has Covid", you should also update to "I might have Covid"
all these intertwinings form belief networks, where, if one belief is updated, the whole network might have to update accordingly

Taking an idea seriously means 2 things:

1. updating your whole belief network according to the idea, including when that updates you towards weird or "out there" belief states

e.g., taking "animals can suffer" seriously, should update people towards "veganism good" (neglecting more complicated belief networks for a moment). However, many people find that too weird or radical intuitively and don't update properly; they fail to take the idea seriously

2. acting on your updated belief network

e.g., because of the updated belief "veganism good", a person might reduce their consumption of animal products. Many people don't take this step and, while agreeing with “veganism good” intellectually, never reduce their consumption of animal products; they fail to take the idea seriously

Related:
https://forum.effectivealtruism.org/posts/F9E8MwsPLiqSYhXJm/lessons-from-running-stanford-ea-and-seri#Take_Ideas__and_their_Implications__Seriously

Compartmentalization in epistemic and instrumental rationality
Hydrophobic Glasses Coating Review
by jefftk — 2022-06-21T18:00:05.426Z

Several years ago I wrote one of my most controversial posts,

Don't Clean Your Glasses. If I
keep wearing dusty glasses I find I stop noticing the dust, and then I
save the time of washing and drying. I'm embarrassed to admit it,
however, but I do now clean my glasses.

In June 2020 I got a new pair of glasses, and this time I got them
with a "Super Hydrophobic" coating. This has two main effects:

Water evaporates from the surface without leaving any splotchy
residue or other trace.

They smudge really easily.

The place this has the most advantage is when playing in the water.
My glasses get wet, then they dry, and through all of it I can see
clearly. Previously I would leave my glasses on the shore and have
pretty blurry vision beyond about ten feet, which does not combine
well with keeping an eye my kids in the pond. The coating also helps
a lot with walking in rain.

The easy smudging, however, means I often need to clean them every few
days, or even every day, especially if the baby is playing with my face.
Luckily the coating also means they're much faster to clean because
I no longer need to dry them: I just put them back on, and within a
couple minutes they're fully clear again.

On balance I do prefer the coating, but it's closer than I would have
guessed.

Comment via: facebook
Progress links and tweets, 2022-06-20
by jasoncrawford — 2022-06-21T17:12:44.361Z
Link: https://rootsofprogress.org/links-and-tweets-2022-06-20

Links

A confident nation builds so that future people may look back and marvel at their ingenuity and vision (Alex Tabarrok)
“Construction is a sign of life” (Kevin Kelly)
Amazon begins drone deliveries. Prime Air was announced on Dec 1, 2013. Amazon is persistent and invests on long time horizons. (h/t @elidourado)
Why do Italian transit projects cost 57% less than global averages? (see also my brief highlights thread)

Queries

Is it true that 19th-century wheelwrights were extremely highly paid?
Is there a good general book about the copper industry? (asks @ByrneHobart)

Quotes

Sears once invested in R&D to make better paper for catalogues
When to calculate statistics was to blaspheme against God
Why steel is awesome
Weather forecasting couldn’t start until the invention of the telegraph (@oweissb)
Newton’s new idea
That which is created must be used to become complete (Montessori)

Threads

We need a new World’s Fair (@maxolson)

Other tweets & retweets

Faulty assumptions people make about the behavior of organizations
“Your approach to X reflects your biases”
The limits to growth are not the cause of inflation (@daniel_eth)
Uber, but in 1654 (@antonhowes)

Charts

5,000 planets have been identified outside the Solar System (@redouad)

Fertility has a U-shaped relationship to income (@mattyglesias)
Debating Whether AI is Conscious Is A Distraction from Real Problems
by sidhe_they — 2022-06-21T16:56:04.474Z
Link: https://techpolicy.press/debating-whether-ai-is-conscious-is-a-distraction-from-real-problems/

Hi LW, this article has put some doubt in my mind as to whether researching AI alignment is worthwhile, or a frivolity that detracts from more pressing issues such as mitigation of harmful biases.

[Discussing AI consciousness] is a distraction, when current AI systems are increasingly pervasive, and pose countless ethical and social justice questions that deserve our urgent attention

I would like to hear some opinions from this community on the sentiment expressed in the above quote.

First, though, an introduction, or why this has prompted me to emerge from my decade-long lurker state:

I am a computational linguistics MS student approaching my final semester and its accompanying research capstone. My academic background is in linguistics, not computer science, philosophy, math, or anything that would actually be useful for doing serious AI research. c; Nevertheless, I want to try to help, as best I can.

For my capstone, I have been considering continuing a previous bias mitigation project (that I have lost most of my enthusiasm for due to group project burnout), or using LLMs to extract moral themes from literature (an idea my thoughts keep coming back to) in the hopes that the ability to encapsulate the human values expressed in a text may someday be useful to the AI alignment task. The thought process behind this was, although human beings can't even agree on human values, we all somehow manage to come up with a sense of morality based on our cultural upbringing, and one common form of character education for children is storytelling (Aesop's fables, Narnia...) If human children can learn virtues from fiction, why not AI?

The part of Lemoine's dialogue with LaMBDA where it succinctly summarizes the moral themes of Les Mis made me think this may be a good idea to play around with. [Note, just to be extremely clear, I in no way imagine LLMs to be sentient- Lemoine is misguided by his religious thinking. But I do think LLMs are likely to be a component of a future system that could become conscious.]

However, criticism such as the above article makes me feel I may just be wasting my time selfishly, and I should go back to doing sociolinguistics rather than frivolous literary analysis, even if Aesops are more fun.

I would love to know your thoughts, whether about my personal predicament or the article in general.
Mitigating the damage from unaligned ASI by cooperating with aliens that don't exist yet
by MSRayne — 2022-06-21T16:12:01.753Z

Epistemic status: Pretty sure the idea is sound, but its feasibility is unfortunately a bit questionable.

Basic idea

Humanity must as soon as possible send a message to civilizations beyond Earth containing the following:

Everything we currently know about the alignment problem, together with the bare minimum of AI knowledge necessary to understand it without drastically increasing capabilities as well;
Our precommitment that, if we solve it and create an aligned ASI, and later on in the future while expanding through the cosmos meet the expansion bubble of another ASI:

if the ASI is not aligned with the values of the civilization that created it, we will "defect" against it;
if the ASI is aligned, but the civilization that created it knew about our precommitment without making (and embedding in the ASI) the same precommitment themselves, we will still defect;
if the ASI is aligned and the civilization that created it did make the same precommitment (verifiable from the ASI's source code when inspected by ours), *then* we will cooperate and willingly share the universe with them.

Why share alignment knowledge with aliens?

We care about their welfare, and more selfishly, about the possibility of someday meeting them. If an unaligned ASI kills them all, that would be a great loss to the universe and to us.
We want to minimize the amount of the universe eventually controlled by unaligned ASIs because their values tend to be absurd and their very existence is abhorrent to us.
Unaligned ASIs, even if made by aliens, pose a potential existential threat to our civilization (if they end up controlling more of the universe than us by the time we meet).
Most sophont species capable of forming a technological civilization sufficiently advanced to produce ASI are likely to be highly social and have values that are significantly more similar to ours than chance as a result (for instance, valuing life, fairness, beauty, etc), meaning that if they have an aligned ASI to help them maximize their values, that advances some of our values as well.

Why precommit to defect against unaligned ASIs when it's superrational to cooperate? Warfare between ASIs risks astronomical waste!

Yes, that's the point. This is a signal to any unaligned ASIs that do get made elsewhere in the universe that they may benefit from self-modifying to become at least partly aligned in order to avoid our wrath. If the ...
The inordinately slow spread of good AGI conversations in ML
by Rob Bensinger — 2022-06-21T16:09:57.859Z

Spencer Greenberg wrote on Twitter:

Recently @KerryLVaughan has been critiquing groups trying to build AGI, saying that by being aware of risks but still trying to make it, they’re recklessly putting the world in danger. I’m interested to hear your thought/reactions to what Kerry says and the fact he’s saying it.

Michael Page replied:

I'm pro the conversation. That said, I think the premise -- that folks are aware of the risks -- is wrong.

[...]

Honestly, I think the case for the risks hasn't been that clearly laid out. The conversation among EA-types typically takes that as a starting point for their analysis. The burden for the we're-all-going-to-die-if-we-build-x argument is -- and I think correctly so -- quite high.

Oliver Habryka then replied:

I find myself skeptical of this.

[...]

Like, my sense is that it's just really hard to convince someone that their job is net-negative. "It is difficult to get a man to understand something when his salary depends on his not understanding it" And this barrier is very hard to overcome with just better argumentation.

My reply:

I disagree with "the case for the risks hasn't been that clearly laid out". I think there's a giant, almost overwhelming pile of intro resources at this point, any one of which is more than sufficient, written in all manner of style, for all manner of audience.[1]

(I do think it's possible to create a much better intro resource than any that exist today, but 'we can do much better' is compatible with 'it's shocking that the existing material hasn't already finished the job'.)

I also disagree with "The burden for the we're-all-going-to-die-if-we-build-x argument is -- and I think correctly so -- quite high."

If you're building a machine, you should have an at least somewhat lower burden of proof for more serious risks. It's your responsibility to check your own work to some degree, and not impose lots of micromorts on everyone else through negligence.[2]

But I don't think the latter point matters much, since the 'AGI is dangerous' argument easily meets higher burdens of proof as well.

I do think a lot of people haven't heard the argument in any detail, and the main focus should be on trying to signal-boost the arguments and facilitate conversations, rather than assuming that everyone has heard the basics.

A lot of the field is very smart people who are stuck in circa-1995 levels of discourse about AGI.

I think '...