trentmkelly/LessWrong-43k huggingface dataset:
Treat Examples as World-Building
by adamShimi — 2022-03-10T15:09:44.302Z

Tl;dr: Examples in an essay work like world-building in a fantasy novel: too little of them and the idea/story lacks substance, too many and it's weighted down.

John keeps telling me to use more examples in my work and writing, and he's right: they enliven a post, give you a hook for free, and ensure that your idea is deep enough to appear at many different places. Going from "without example" to "with example" is always progress.

Yet I noticed myself overindulging in examples. When writing my last post, I structured a whole draft by presenting one cool example after another, and noting the implications for my general idea. But the examples are not the message! What I wanted to convey was the thesis, the core idea of the post, with examples as illustration and anchors. Instead, I found myself showing off my extensive research one example at a time.

That's when it hit me: I was info-dumping.

The image that came to my mind was a fantasy author so in love with his world-building that he takes every opportunity (and some more) to serenade you about the minutely crafted economic system or the elaborated (and linguistically coherent!) invented language.

The more I think about it, the more I like this metaphor. If I keep it in mind, I expect that my misuse of examples will be obvious, just like I notice info-dumping easily.

Here's a couple of implications:

Don't have too many examples. Just like too much info-dumping distracts from the story.

John told me once that three is a good number; I think that five is probably the maximum for a single post.

Always have examples. A story without a world rarely captures a reader's attention.
Only present the relevant examples or the relevant part of the examples, not all that you've studied. Info-dump isn't excused because you spent week building the world.
Don't go into too much detail when describing the examples. Descriptions forever will annoy the reader.

As much detail as you need to get the point across, and nothing more. From time to time a bit of color or a really cool bit.

Research your examples as much as you need, but not much further. Don't go into the deep hole of world-building and lose months fine-tuning your invented banking system if you have a single scene at the bank.

I think there are some examples that deserve a deep dive, but far fewer that you would expect.

Use examples to rekindle the reader curiosity. Show the glory

...
Best LessWrong posts for understanding scientific studies.
by Sean McAneny — 2022-03-10T13:13:27.591Z

For someone who is not totally quantitatively literate, or whose background is outside of STEM, what are the best resources for understanding scientific studies? I'm looking for books, articles, or posts that explain both the different parts of these studies and what is most relevant about them. What should I be paying attention to when reading a study? What are some red flags when looking at a study that would signal epistemic uncertainty?
Donations, The First Year
by jenn — 2022-03-10T03:57:47.296Z
Link: https://jenn.site/2022/01/donations-the-first-year/

A very belated crosspost from my personal blog, and written for an audience that might not be super familiar with EA/Rationality.

2021 was my first year with a full-time, steady source of employment, and real money.

Having identified as an Effective Altruist (EA) for the better part of a decade, one thing I was looking to the most from this was the ability to finally make a substantial difference through the unit of caring.

For someone who’s identified as an Effective Altruist for the better part of a decade, it was embarrassingly easy for sentiment to get my goat.

😰😰😰

Where We Gave

My girlfriend and I donated ~10% of our combined post-tax income, as stipulated by the Giving What We Can pledge. However, we failed to donate it all to effective charities, so it can’t really be said that we uh actually fulfilled the terms of the pledge. Thankfully I am very neurotic about not breaking any oaths so I have prepared for this moment by never actually officially signing up for the pledge, despite having identified as an effective altruist for zzzzzzz.

Here is where it went:

40% to global health initiatives via the RCForward Global Health Fund.[1]

15% to environmental advocacy via the RCForward Climate Change Fund.

15% to Spectrum, Kitchener-Waterloo’s queer community space. They do a lot of cool stuff and maintain a very active calendar of events.

15% to A Better Tent City, a cheap, no-barrier alternative to shelters in Kitchener. Instead of doing the shelter model where they turn everyone out during the day and then accept them back at night using a first-come-first serve basis (which is especially bad since demand outnumbers supply so there’s no sense of security/stability for any shelter users), ABTC serves a more permanent community by giving them tiny homes to live in.

15% to the KWCF Immigration Partnership Fund for Immigrant and Refugee Initiatives, to support programs and initiatives for Afghan refugees starting their new life in Waterloo Region.

On Donating Locally

To be honest, I’m still not really sure if doing what was basically a 50/50 split between effective and local charities was the right move. It’s definitely something I want to think through in more detail before this year’s donations.

What we donated to local charities combined would be enough to save the lives of like two children if we donated it to a Givewell recommended global health charity, and I...
Programming note
by HoldenKarnofsky — 2022-03-09T21:20:18.062Z

I'll be posting somewhat less frequently for a while, as I've gone through a lot of my backlog (though not 100% of it) and want to focus for a while on trying to make progress on some thorny, important-seeming questions about making the best of the most important century.

I have a writeup of what sorts of questions I want to focus on, and why I think they're important, here. It's a bit less general-interest-oriented than my normal Cold Takes posts, so I only put it on the Effective Altruism Forum, but check it out if you're curious about what I think is most holding us back from knowing what actions to take to improve the future of humanity.

Some things I'll probably post reasonably soon are (a) a reader survey on what topics you'd like to see more of; (b) tips for working on wicked problems; (c) another dialogue or two to follow this one (topic is future-proof ethics). I expect to post on a bunch more topics later. In the meantime, check out the archives!
The Proof of Doom
by johnlawrenceaspden — 2022-03-09T19:37:57.274Z

Epistemic Status: Ravings

Importance: Easily the most important problem in the world. If we can escape the proof of doom, we will likely also solve all our other problems as a side effect, and all that remains will be the fundamental limits. Our new questions will be things like "How much joy can the universe physically support?"

-

It seems to me that the world, and everyone in it, is doomed, and that the end is considerably nigher than I might like.

To be more specific, I think that we may well create an Artificial General Intelligence within around 20 years time, and that that will be our last act.

The newly-created AGI will immediately kill everyone on the planet, and proceed to the destruction of the universe. Its sphere of destruction will expand at light speed, eventually encompassing everything reachable.

There may well be more proximal threats to our species. Comets are one obvious one, but they seem very unlikely. Artificially created universally fatal plagues are another, but perhaps not very likely to happen within the next twenty years.

--

I've believed this for many years now, although my timescales were longer originally, but it seems to me that this is now becoming a common belief amongst those who have thought about the problem.

In fact, if not consensus, then at least the majority opinion amongst those mathematicians, computer scientists, and AI researchers who have given the subject more than a few days thought.

Those who once were optimistic seem pessimistic now. Those who were once dismissive seem optimistic.

But it is far from being even a mainstream opinion amongst those who might understand the arguments.

Far, even, from rising to the level of a possible concern.

Amongst my personal friends, amongst people who would mostly take my word on technical and scientific issues, I have found it impossible to communicate my fears.

Not all those who are capable of pressing the suicide button understand that there is a suicide button.

--

What to do?

One might empathise with Cassandra. A vision of flame, and no-one will believe.

Cassandra had many opportunities to save her city, but the curse of Apollo rendered her unable to communicate with her fellow citizens. Those of her cohort who independently sensed danger were individually obstructed by the gods.

We operate under no such constraints.

Our arguments are not clear.

My attempts to communicate the dange...
[Survey] Expectations of a Post-ASI Order
by Lone Pine — 2022-03-09T19:17:37.272Z

I put together a survey to explore what people think will happen after the emergence of Artificial Super Intelligence, assuming that humans survive and there is something resembling human society afterwards.

Take the survey here. Thanks!

Twenty four people from /r/singularity have already taken the survey. These people are essentially Kurzweilians: they are extremely optimistic about AI and they never talk about the alignment problem. I'm hoping that respondents from Less Wrong will balance out the results with more realism and appropriate amounts of pessimism. I might also share the survey in a few other places. Afterwards I'll share charts and graphs of the results with everyone.

In the survey, I construct one future where particular things happen, and ask you to fill in the details. A limitation of this format is that you might feel that technology will progress much faster or much slower than I suggest, or the consequences on society will be dramatically different. For example, one /r/singularity respondent believes that by 2122 the Earth will be transformed into a giant swarm of nanobots (I know), and so the survey couldn't capture that person's true expectations.

(BTW, due to a limitation of Google Forms, I can't change any of the answers now that people have taken the survey, so I can't fix any typos or make corrections.)

My hope is that you'll accept the premise long enough to complete the survey, and use the response boxes I provide if the multiple-choice answers don't capture your sense of what would happen in this scenario. The goal is to elucidate what kinds of problems post-ASI society would still have to deal with even if we get a good-ish outcome. Several people said that the survey gave them new insights, so I expect you'll have fun at least.

Here's the link to the survey again. Thanks!
A Rephrasing Of and Footnote To An Embedded Agency Proposal
by JoshuaOSHickman — 2022-03-09T18:13:23.348Z

I wanted to clarify some comments made in this post, which proposed a way to resolve the issues brought up in the Action Counterfactuals subsection of the Embedded Agency paper. I noticed that my ideal edits would end up rewriting the post almost completely (it wasn't particularly clear), and I wanted to more coherently lay out the thinking.

The Problem

The problem is that we'd like agents to consider action counterfactuals, and, perhaps, find a proof that an action leads to the highest reward before taking it. The formulation in the paper leads to some Lobian issues, where agents choose actions by searching over proofs for statements similar to:

argmax(X) This agent chooses X => Gets Reward W

[The paper actually has a much cleaner phrasing of the statements it searches for proofs for, to make the proof issue much clearer.]

This is a rather substantial bummer, because something like that would be great to rely on. It'll find the proofs for what you'd like, but it'll also find proofs for nonsense. The Embedded Agency paper is a great resource for this, but you should also work out why this might happen in your head.

The Confusion

One potential intuition for why this problem exists is the following:

The agent who is taking this argmax always chooses the highest reward thing. If I know the rewards for all actions but the last X, well, choosing X means it was the result of the argmax. It's right there in the source code / function definition. So W must have been the biggest! Therefore, X is the result of the argmax being computed, because even if we don't know W, I know that the agent choosing X means W is bigger than all the other W' values. It must be -- the choice was coming from an agent doing an argmax!

This might be a confusing phrasing. The ideal way to explain this might be setting up the 5-10 problem in a theorem prover and just showing people it'll find the bad proof. But I think the narrative version captures something coherent.

The confusion we should have is that this is not how we, humans, engage in action counterfactuals! Not even close. Normally that's not an issue, but let's pay more attention to see if the difference is substantive.

In the original post, I use the example of a professor considering whether to slash the tires on his car. He ponders what it would imply, if he chose to do that. It would imply he must desperately not want the abilit...
ELK Sub - Note-taking in internal rollouts
by Hoagy — 2022-03-09T17:23:19.353Z

My ELK submission was labelled under 'Strategy: train a reporter that is useful for another AI'. This is definitely a fair description, though the AI it needs to be useful to is itself - the reporter is essentially internalized. I also agree that the proposed counterexample, hiding information in what seems like human-comprehensible speech, is the biggest flaw.

Nonetheless I think my proposal has enough additional detail and scope for extension that it's worth fleshing out in its own post - so here we are. Some of the responses to counterexamples below also go beyond my original proposal.

For anyone interested, here is my original proposal (Google Doc) which contains the same idea in somewhat less generality.

In this post I'll first flesh out my proposal in slightly more general terms, and then use it to try and continue the example/counter-example dialogue on ELK.

I know there were a number of proposals in this area and I'd be very interested to see how others' could be integrated with my own. In particular I think mine is weak on how to force the actor to use human language accurately. I expect there are lots of ways of leveraging existing data for this purpose but I've only explored this very lightly.

Many thanks to ARC for running the competition and to Misha Wagner for reviewing and discussing the proposal.

Proposal Summary

Creating Human Semantic Space

We should think of the actor as described in ELK as having two kinds of space in which it works:

The space of potential observations of its sensors and actuators, O
The space of internal representations, ie its Bayes Net, I

The machine takes in observations, converts it into its internal representation I, simulates the action in this internal representation, and then cashes this out in its actuators.

What we want to do is force it to convert back and forth between I and a third space, the space of human understandable meaning, which we will call H.

How can we achieve this? My basic idea is that we leverage the world of text and image based models to create a model called a Synonymizer. This device, in its ideal form, would be trained to take in any piece of text, image, video, audio, or combination thereof, and output a piece of media in any of these forms which preserves the semantic content as far as possible while having complete freedom to change the format. (Initial experiments in this vein would probably ...
Meadow Theory
by Duncan Sabien (Inactive) — 2022-03-09T17:13:21.817Z

Author's note:

The essay below centers mostly on a philosophy of parenting. However, it's not "about" parenting—rather, parenting is a domain in which the concepts I'd like to convey can be easily represented and productively applied. Parenting is one specific instantiation of the broader set of ideas, just as one might use an essay about football or baseball as a vehicle for the concepts of friction or acceleration or collision elasticity.

Author's note II:

Feedback on this draft ran thus:

[this essay seems to be missing something important, namely] stuff that would flip me from the primary activity of "i am trying to download Duncan's meanings into a sandbox in my mind and then make sense of them in the sandbox before i possibly do the further work of rendering them meaningful and/or useful to me if i have the tenacity for that", to the different primary activity of "i am using Duncan's guidance to recognize in my own head and from my own perspective the possibility of relating to the world in this new way, and i am personally engaged in the process of figuring out when and how and why it might matter to me"

I'm not sure how to get readers to do the latter thing; I think it is the correct thing to do and had sort of implicitly assumed that everybody already would. Since I don't know what changes to make to this draft to encourage the latter thing for readers like the one above, I'm just bluntly making the recommendation. Hopefully I will get better at helping in the near future.

I. The meadow

Imagine a wide, level, grassy meadow, stretching to the horizon, empty of all obstacles.

This meadow will be our metaphor for the world/the territory/reality-as-a-whole, and the picture of it will gradually grow more complex as we step through various thought experiments. But for the moment, it's just one big blank plane.

Now imagine a blindfolded child, running through the meadow.

This is safe, because the meadow contains nothing besides soft, springy grass. There are no walls or other obstacles to run into. If they trip or tumble, they'll be fine.

Running will be our metaphor for human activity. It is good to run. More precisely, it's good to be able to run—to have the freedom to move in whatever direction you please, at whatever speed you please, without fear or constraint. The ideal state is one in which...
Being Moral is an end goal.
by Yair Halberstadt — 2022-03-09T16:37:15.612Z

Alice wants to go to her best friends party. She has a sore throat, and she usually takes a COVID test when she has a sore throat. But if the test were to come out positive she would feel like she shouldn't go, and she really really wants to go. Solution - she avoids taking the test, leaving her probably COVID free, and ready to party.

I've heard this story in about a thousand variations since COVID restrictions began.

Analysed from a rational perspective this doesn't seem to make much sense.

Let's say that Alice thinks she a 10% chance of having COVID. Tests are perfectly accurate. Alice care about other people, so going to a party whilst infected is worth negative 50 utils. Going to a party is otherwise worth 10 utils.

Then expected utility if Alice takes the test is 0.9 * 10 + 0.1 * 0 = 9.

Expected utility if she doesn't take the test is 0.9 * 10 + 0.1 * -50 = 5.

So taking the test leaves her better off than not taking the test.

This is a special case of the general rule that knowing more information should never be a negative for a rational agent.

So what's going on here?

The answer is that Alice doesn't actually care about other people. Going to a party whilst infectious is worth just as many utils to her as going to a party whilst COVID free.

What Alice cares about is being a moral person. And moral people don't go to parties whilst knowing they're infectious. But moral people do go to parties whilst they might have an infection. So if she avoids finding out if she infectious she gets a guaranteed 10 utils at 0 cost.

One way to model this is that a moral action is one that somebody who genuinely cares about other people will do. We don't genuinely care about other people, but we care about doing moral actions. So we make the minimal changes from pure moral action so that we can mostly take moral actions, whilst still mostly just doing whatever we want. By making the slightly immoral action of not doing a COVID test, we avoid having to make the very immoral action of going to a party whilst infected.

Note that I don't want to claim people don't care about other people at all. I'm reducing the motives in this particular case to a simplified human with only a single motivation, and exploring that particular facet of our motives, but as ever, people are complex and other scenarios can best be explained by assuming that people do genuinely care for others.
It Looks Like You're Trying To Take Over The World
by gwern — 2022-03-09T16:35:35.326Z
Link: https://www.gwern.net/Clippy

This story was originally posted as a response to this thread.

It might help to imagine a hard takeoff scenario using only known sorts of NN & scaling effects...

In A.D. 20XX. Work was beginning. "How are you gentlemen !!"... (Work. Work never changes; work is always hell.)

Specifically, a MoogleBook researcher has gotten a pull request from Reviewer #2 on his new paper in evolutionary search in auto-ML, for error bars on the auto-ML hyperparameter sensitivity like larger batch sizes, because more can be different and there's high variance in the old runs with a few anomalously high performance values. ("Really? Really? That's what you're worried about?") He can't see why worry, and wonders what sins he committed to deserve this asshole Chinese (given the Engrish) reviewer, as he wearily kicks off yet another HQU experiment...

Rest of story moved to gwern.net.