trentmkelly/LessWrong-43k huggingface dataset:
The Adventure: a new Utopia story
by Stuart_Armstrong — 2020-02-05T16:50:42.909Z

For an introduction to why writing utopias is hard, see here. For a previous utopian attempt, see here. This story only explores a tiny part of this utopia.

The Adventure

The cold cut him off from his toes, then fingers, then feet, then hands. Clutched in a grip he could not unclench, his phone beeped once. He tried to lift a head too weak to rise, to point ruined eyes too weak to see. Then he gave up.

So he never saw the last message from his daughter, reporting how she’d been delayed at the airport but would be the soon, promise, and did he need anything, lots of love, Emily. Instead he saw the orange of the ceiling become blurry, that particularly hateful colour filling what was left of his sight.

His world reduced to that orange blur, the eternally throbbing sore on his butt, and the crisp tick of a faraway clock. Orange. Pain. Tick. Orange. Pain. Tick.

He tried to focus on his life, gather some thoughts for eternity. His dry throat rasped - another flash of pain to mingle with the rest - so he certainly couldn’t speak words aloud to the absent witnesses. But he hoped that, facing death, he could at least put together some mental last words, some summary of the wisdom and experience of years of living.

But his memories were denied him. He couldn’t remember who he was - a name, Grant, was that it? How old was he? He’d loved and been loved, of course - but what were the details? The only thought he could call up, the only memory that sometimes displaced the pain, was of him being persistently sick in a broken toilet. Was that yesterday or seventy years ago?

Though his skin hung loose on nearly muscle-free bones, he felt it as if it grew suddenly tight, and sweat and piss poured from him. Orange. Pain. Tick. Broken toilet. Skin. Orange. Pain...

The last few living parts of Grant started dying at different rates.

*~*~*

Much later:

“What have you learnt so far?”

“That talking to myself is barely half-helpful.”

“Then let’s hallf-stop it, then.”

*~*~*

Half a second later than that:

“Quick! Get the bognor now!”

Grant fell out of the tube, blinking against the warm yellow light. “What...”, he said, his voice coming clear, as a tiny pink goblin carrying a large blue flag raced in front of him. It stopped to bite him lightly but viciously on the knee as it passed, then g

...
"But that's your job": why organisations can work
by Stuart_Armstrong — 2020-02-05T12:25:59.636Z

It's no secret that corporations, bureaucracies, and governments don't operate at peak efficiency for ideal goals. Economics has literature on its own version of the problem; on this very site, Zvi has been presenting a terrifying tale of Immoral Mazes, or how politics can eat all the productivity of an organisation. Eliezer has explored similar themes in Inadequate Equilibria.

But reading these various works, especially Zvi's, has left me with a puzzle: why do most organisations kinda work? Yes, far from maximal efficiency and with many political and mismeasurement issues. But still:

The police spend some of their time pursuing criminals, and enjoy some measure of success.

Mail is generally delivered to the right people, mostly on time, by both governments and private firms.

Infrastructure gets built, repaired, and often maintained.

Garbage is regularly removed, and public places are often cleaned.

So some organisations do something along the lines of what they were supposed to, instead of, I don't know, spending their time doing interpretive dance seminars. You might say I've selected examples where the outcome is clearly measurable; yet even in situations where measuring is difficult, or there is no pressure to measure, we see:

Central banks that set monetary policy a bit too loose or a bit too tight - as opposed to pegging it to random numbers from the expansion of pi.

Education or health systems that might have low or zero marginal impact, but that seem to have a high overall impact - as in, the country is better off with them than completely without them.

A lot of academic research actually uncovers new knowledge.

Many charities spend some of their efforts doing some of the good they claim to do.

For that last example, inefficient charities are particularly fascinating. Efficient charities are easy to understand; so are outright scams. But ones in the middle - how do they happen? To pick one example almost at random, consider Heifer Project International, which claims to give livestock to people in the developing world. In no way is this an efficient use of money, but it seems that Heifer is indeed gifting some animals[1] and also giving some less publicised agricultural advice (that may be more useful than the animals). So, Heifer is inefficient, poorly assessed, different from the image it presents to donors, and possibly counter-productive overal

...
Training a tiny SupAmp model on easy tasks. The influence of failure rate on learning curves
by rmoehn — 2020-02-05T07:22:49.624Z

Link: https://github.com/rmoehn/farlamp/blob/master/build/tiny-supfail.pdf

This is the first experiment report of my AI alignment research project. A
reminder what it is about:

I'm studying the impact of overseer failure on RL-based IDA,
because I want to know under what conditions
amplification and distillation increase or decrease the
failure rate,
in order to help my reader understand whether
explicit reliability amplification is necessary for
IDA to work in practice.

And an excerpt from the introduction of the report:

I ran experiments with a tiny SupAmp model on easy tasks. Small data, short
training times and local execution allow for a tight feedback loop with few
moving parts. Thus I could understand and adapt the code, and fix mistakes
quickly.

The drawback is that it doesn't yield many insights. One result […], however,
was encouraging: XPA does learn to be more reliable than
its overseer
AmplifyH′(XPA).
The rest of the results fall into two categories: Results confirming my
understanding of the code and learning behaviour. And results made useless by
mistakes and misunderstandings of mine. Although the latter sounds bad, it is
actually useful to have exposed these mistakes before running more expensive
experiments.

I welcome feedback of any kind.
Physical alignment - do you have it? Take a minute & check.
by leggi — 2020-02-05T04:02:49.886Z

The first virtue of rationality is listed as curiosity - have you got it?

A minute of your time to increase self-awareness.

Find the following 5 markers on your body and consider their relative positioning.

Can you move your body in order to increase your physical alignment?

1. Pubic symphysis.

Midline, where the pubic bones meet at the front of the pelvis.

2. Navel

3. Xiphoid process of sternum

4. Jugular notch of sternum

5. External occipital protuberance

Keep thinking about these five midline markers.

Increase your awareness of your physical state, see how it feels.

Breathing technique.

Are you corporeally aligned?

Feeling balanced,

body & mind?
Open & Welcome Thread - February 2020
by ryan_b — 2020-02-04T20:49:54.924Z

If it’s worth saying, but not worth its own post, here's a place to put it. (You can also make a shortform post)

And, if you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are welcome.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ.

The Open Thread sequence is here.
Meta-Preference Utilitarianism
by B Jacobs — 2020-02-04T20:24:36.814Z

There has been a long standing debate between utilitarians about what should be maximized. Most fall on either the side of ‘average utility’ or ‘total utility’. A select group even chooses ‘minimum utility’ or other lesser known methods.

Previously I have tried to solve this by charting different moral-theories on different axis and by prioritizing those actions that achieve success in most moral theories (see Q-balance utilitarianism).

I have come to the conclusion that this is just a bandaid on a more fundamental problem. Whether we should choose total, average or even median utility isn’t something we could objectively decide. So I suggest that we go up one level, and maximize what most people want to maximize.

Let’s say we were able to gauge everyone’s (underlying) preferences about how much they like certain methods of maximizing by holding a so called utilitarian vote.

A utilitarian method that would violate your preferences completely would get a score of 0, one that would encapsulate your preferences perfectly would get a score of 1 and one that you like to a certain extent but not completely gets e.g 0,732.

If there is such a thing as ‘meta-preference ambivalence’ we could gauge that too: “People who do not have any meta-preferences in their utility-function get a score of 0, people for whom the entire purpose in life is the promotion of average utilitarianism will get a score of 1 etc.

Just multiply the ambivalence with the meta-preference and then add all the scores of the individual methods together (add all the scores of the preferences for “median utility” together, add all the scores for “total utility” together etc) and compare.

Now one way of maximizing comes out on top, but should we pursue it absolutely or proportionally? If “total utilitarianism” wins with 70% and “average utilitarianism” loses with 30% of the vote, should we act as “total utilitarians” 100% of the time or 70% of the time? (with average utilitarianism 30% of the time at random intervals)

Well, we could solve that with a vote too: “Would you prefer pursuing the winning method 100% of the time or would you prefer pursuing it proportionally to it’s victory?”

And any other problem we’ll face after this wil...
Philosophical self-ratification
by jessicata — 2020-02-03T22:48:46.985Z
Link: https://unstableontology.com/2020/02/03/philosophical-self-ratification/

"Ratification" is defined as "the act or process of ratifying something (such as a treaty or amendment) : formal confirmation or sanction". Self-ratification, then, is assigning validity to one's self. (My use of the term "self-ratification" follows philosophical usage in analysis of causal decision theory)

At first this seems like a trivial condition. It is, indeed, easy to write silly sentences such as "This sentence is true and also the sky is green", which are self-ratifying. However, self-ratification combined with other ontological and epistemic coherence conditions is a much less trivial condition, which I believe to be quite important for philosophical theory-development and criticism.

I will walk through some examples.

Causal decision theory

Formal studies of causal decision theory run into a problem with self-ratification. Suppose some agent A is deciding between two actions, L and R. Suppose the agent may randomize their action, and that their payoff equals their believed probability that they take the action other than the one they actually take. (For example, if the agent takes action L with 40% probability and actually takes action R, the agent's payoff is 0.4)

If the agent believes they will take action L with 30% probability, then, if they are a causal decision theorist, they will take action L with 100% probability, because that leads to 0.7 payoff instead of 0.3 payoff. But, if they do so, this invalidates their original belief that they will take action L with 30% probability. Thus, the agent's belief that they will take action L with 30% probability is not self-ratifying: the fact of the agent having this belief leads to the conclusion that they take action L with 100% probability, not 30%, which contradicts the original belief.

The only self-ratifying belief is that the agent will take each action with 50% probability; this way, both actions yield equal expected utility, and so a policy 50/50 randomization is compatible with causal decision theory, and this policy ratifies the original belief.

Genetic optimism

(This example is due to Robin Hanson's "Uncommon Priors Require Origin Disputes".)

Suppose Oscar and Peter are brothers. Oscar is more optimistic than Peter. Oscar comes to believe that the reason he is more optimistic is due to inheriting a gene that inflates beliefs about positive outcomes, whereas Peter did not inherit this same gene

...
Twenty-three AI alignment research project definitions
by rmoehn — 2020-02-03T22:21:49.089Z

I came up with these research project definitions when I read the iterated
amplification sequence. Last
year I put five of them up for voting (see Which of these five AI alignment
research projects ideas are no
good?)
and chose no. 23 to work on (see IDA with RL and overseer
failures). But I didn't think of publishing
all of them until JJ Hepburn gave me the idea that they might be useful to
others.

The definitions follow the format recommended in The Craft of
Research:
‘I'm studying X, because I want to find out Y, so that I (and you) can better
understand Z.’ The quality of language and content varies.

An unaligned benchmark
----------------------

1
I'm studying mild optimization,
because I want to understand what its problems and limitations
are,
because I want to decide whether or not to work on one of
them,
because well-functioning mild optimization appears to
be close to what humans and organisations safely do
today, so basing an AI on it might have useful
outcomes.
- This is more a study project than a research project.

2
I'm studying how image classifiers can recognize new types of
images,
because I want to find out how to detect out-of-distribution
inputs,
in order to help my reader understand how to make ML
systems that don't confidently make wrong predictions.

Approval-directed agents/bootstrapping
--------------------------------------

3
I'm studying Bayesian machine learning,
because I want to understand how to make ML systems that
notice when they are confused
in order to help my reader understand how to make ML
systems that will ask the overseer for input when doing
otherwise would lead to failure.

4
I'm studying possible structures of approval-directed agents
because I want to understand how much human thought and input
they would require
in order to help my reader understand whether
approval-directed agents are feasible.

Humans Consulting HCH
---------------------

5
I'm studying the articles linked from Humans Consulting HCH,
because I want to understand the section ‘Hope’ from Humans
Consulting HCH
in order be able to think about the tension between the
system being capable and reflecting the human's judgment.

Corrigibility (Christiano 2017)
----------------

...
Absent coordination, future technology will cause human extinction
by Jeffrey Ladish — 2020-02-03T21:52:55.764Z

(Crossposted from on Medium and my blog)

Nick Bostrom, of the Future of Humanity Institute, uses an evocative metaphor to describe the future of humanity’s technological development:

One way of looking at human creativity is as a process of pulling balls out of a giant urn. The balls represent possible ideas, discoveries, technological inventions. Over the course of history, we have extracted a great many balls — mostly white (beneficial) but also various shades of grey (moderately harmful ones and mixed blessings).

What we haven’t extracted, so far, is a black ball: a technology that invariably or by default destroys the civilization that invents it. The reason is not that we have been particularly careful or wise in our technology policy. We have just been lucky.

The atom bomb, together with the long range bomber, marked the first time a small group of people could destroy dozens of cities in a matter of hours. The physicists who worked on the bomb knew that this invention had the capacity to threaten human civilization with unprecedented destruction. They built it out of fear that if they did not, an enemy state like Nazi Germany would build it first. With this development, the destructive power of humanity increased by several orders of magnitude.

History barely had time to catch its breath before many of these same physicists created a new type of nuclear bomb, the hydrogen bomb, that was hundreds of times more powerful than the atom bomb. They did it for the same reasons, out of fear that a rival state would build it first. The Soviet Union used the same justification to build their biological weapons program during the Cold War, producing large quantities of anthrax, plague, smallpox, and other biological weaponry. As far as we know, the United States did not have a comparably large program, but the fear that the US might have one was sufficient to motivate the Soviet leadership. Examples like this are not exceptions; they are the norm.

It’s clear from the history of warfare that the fear of a rival getting a technology first is sufficient to motivate the creation of purely destructive technology, including those that risk massive blowback from radiation, disease, or direct retaliation. This desire to get there first is not the only incentive to develop civilization-threatening technology, but it is the one that seems to drive pe...
Long Now, and Culture vs Artifacts
by Raemon — 2020-02-03T21:49:25.367Z

I've recently gotten re-interested in The Long Now Foundation, and had a conversation about it that seemed worth writing down.

Longtermism vs Pivotal Generationism

I love the Long Now's aesthetic. I'm super looking forward to one day taking the pilgrimage to the Giant Forever Clock In the Mountain that they're building. But I've felt sort of frustrated by them not seeming to 'get x-risk' or things like that.

A recent conversation reframed that slightly: their actions might make more sense if you assume that they are Longtermist, without being Pivotal Generation-ist.

(Separately, I think they don't think the future will get "So Weird As To Invalidate Everything", i.e. converting the world to computronium. Although they might just not think that's tractable to plan for.)

Culture is Medium Term, Artifacts are Long Term.

The conversation explored Longtermism vs Pivotal Generationism a bit. But, the most interesting takeaway for me was: If you are acting on 10,000 year timescales, and you don't think your generation is particularly special, culture is not a very effective way to steer the future. Cultures usually last hundreds of years. Physical artifacts are much more reliable ways to affect people 10,000 years from now.

I had been thinking of "shaping culture" as a way to interact with the longterm future. But, at least historically (setting aside for now Very Weird Futures), cultures tend to act on the medium term, a few hundred years at most.

The Long Now foundation does also do cultural work (in addition to their meetups and Ted Talk-esque presentations, they push Long Bets and Predictions which seem pretty valuable to me). So I don't know that the "culture is medium term" hypothesis is that salient to them. But, it still was an interesting update for me.

When is culture enduring?

Some cultures last for thousands of years. Others do not. Can you predict ahead of time which is which? If your goal is affecting things 1,000 or 10,000 years from now, is culture a viable way of doing that? Or is endurance just selection effect?

Lots of cultures aim to be "generally enduring" but still seem to have changed radically.

My impression is that Christianity succeeded somewhat intentionally, but it wasn't doing things that were strategically novel (beyond what many religions/cultures/civilizations do re: indoctrination and institution-building). My off-

...
Looking for books about software engineering as a field 
by mingyuan — 2020-02-03T21:49:05.926Z

I work in the software industry but am not a software developer. My job is to write about software development, and I've learned a whole bucketload of terms: stuff like 'linked lists', 'CI/CD', 'performance optimization', 'deploy to AWS', 'dockerize', 'microservices', 'SQL injection', 'multithreaded program', 'vectorized code', and on and on and on. However, a lot of the time I'm basically just Chinese-rooming – I can write about these things, but I don't actually understand how any of them fit together. For example, I've had three people try to explain exactly what an API is to me, for more than two hours total, but I just can't internalize it. I feel that there's some impossible-to-articulate piece I'm missing, and none of the words people say to me about software stuff stick because I'm lacking a foundation on which to build up my understanding.

So my question is, are there any books (or other resources) that explain the field of software engineering as a cohesive whole? I'm not looking for books that will teach me to code, because I don't think that's the thing I want. Feel free to ask clarifying questions. Thanks!

-

EDIT: I realized I should include more context on my work and my background, so here it is:

I have an undergrad degree in physics, which gave me extremely minimal exposure to Python. I also took two quarters of intro CS, one in C and one in Racket. As a result I know how to write a for loop and a bit about very basic algorithms; that's about it. I've been in my current job for nearly a year, and my primary task is to write about the skillsets of individual software engineers. This entails things like connecting someone's verbal knowledge of back-end web development to their experience creating microservices; I can do this quite competently and don't make many technical mistakes. I have also learned a bit on the job regarding a couple data structures, some web stuff, and smatterings of info about ML, data science, DevOps, front-end/UI, and mobile development.