trentmkelly/LessWrong-43k huggingface dataset:
Reification bias
by adamShimi — 2023-01-09T12:22:15.460Z

“All right," said Susan. "I'm not stupid. You're saying humans need... fantasies to make life bearable."

REALLY? AS IF IT WAS SOME KIND OF PINK PILL? NO. HUMANS NEED FANTASY TO BE HUMAN. TO BE THE PLACE WHERE THE FALLING ANGEL MEETS THE RISING APE.

"Tooth fairies? Hogfathers? Little—"

YES. AS PRACTICE. YOU HAVE TO START OUT LEARNING TO BELIEVE THE LITTLE LIES.

"So we can believe the big ones?"

YES. JUSTICE. MERCY. DUTY. THAT SORT OF THING.

"They're not the same at all!"

YOU THINK SO? THEN TAKE THE UNIVERSE AND GRIND IT DOWN TO THE FINEST POWDER AND SIEVE IT THROUGH THE FINEST SIEVE AND THEN SHOW ME ONE ATOM OF JUSTICE, ONE MOLECULE OF MERCY. AND YET—Death waved a hand. AND YET YOU ACT AS IF THERE IS SOME IDEAL ORDER IN THE WORLD, AS IF THERE IS SOME...SOME RIGHTNESS IN THE UNIVERSE BY WHICH IT MAY BE JUDGED.

"Yes, but people have got to believe that, or what's the point—"

MY POINT EXACTLY.”

Terry Pratchett,

Hogfather

Humans make up people, objects, things, all the time. We observe a system that we don’t fully control, and then jump to assuming some underlying real thing making it work and function.

Two obvious examples come to mind, one mostly a relic of the past, another very much a part of the present:

Polytheistic religions, where every phenomenon is literally explained by a hallucinated person: god, nymph, djinn…
Hidden entities theories in science, where the phenomena are often clarified and explained through the assumption of unobservable entities, like the fields of field theories, the aether in pre-relativity physics, and the strings and branes in string theory.

Yet our predictive hallucinations are far more prevalent than that!

When reading a novel or playing a game of D&D, we could directly infer the next event to come from the tropes and the psychology of the author, but we tend to naturally hallucinate a whole coherent world with imagined people as if it existed really, and predict the future of the story based on that.
In matters of politics, we could directly model the status games that are being play, like

immoral mazes

and

simulacra levels

attempt to, but we tend to naturally hallucinate issues for which we believe we are fighting, like immigration and economy, even though we mostly are playing the status game and often have different opinion about the real world complex systems these words are supp

...
Big list of AI safety videos
by JakubK — 2023-01-09T06:12:35.139Z
Link: https://docs.google.com/document/d/1vDWLKPvJNo0k9IwMPvduO23_CXOdMpxn0aRDnvrL0yw/edit?usp=sharing

This is a Google doc containing links to:

(all?) AI safety-related YouTube channels (intended to be comprehensive, at least for active channels with videos that seem worth sharing in an AI safety group)
some AI safety-related podcasts
some suggested videos, sorted into categories like "agent foundations," "individual researchers," "career planning," "rationality," etc
some discussion prompts for watching these videos in discussion groups (very few)
some resources and tips for running these discussion groups

Please comment with feedback and suggestions!

Also posted to the EA Forum.
Wolf Incident Postmortem
by jefftk — 2023-01-09T03:20:03.723Z

Incident #210

Status

Complete, one action item outstanding.

Summary

Sentinel consumed by wolf after repeated false alarms.

Impact

Loss of sentinel. No flock impact.

Root causes

Sentinel generated noisy alerts due to premature deployment,
incomplete training, and overly monotonous task. Oncalls failed to
respond to true positive due to alert fatigue.

Trigger

Wolf.

Resolution

Gathered flock. Deployed replacement sentinel.

Detection

Sentinel did not report at end of shift.

Action Items

Priority
Action Item
Type
Status

P0
Gather flock
mitigate
complete

P0
Deploy replacement sentinel
mitigate
complete

P1
Update playbook for wolf alerts
prevent
complete

P2
Update remaining sentinels
prevent
complete

P2
Revise sentinel training program
prevent
complete

P2
Investigate equipping sentinels with flutes or slings

prevent
in progress

Lessons Learned

What went well

Flock gathering proceeded without issues.

No flock injuries or losses.

Replacement sentinel did not exhibit false positive alerts.

What went wrong

Noisy alerts not addressed.

Alerts silenced contrary to playbook.

Loss of sentinel.

Where we got lucky

Only one wolf.

Wolf sated after sentinel consumption.

Replacement sentinel available.

Timeline

All times local

March 3rd:

16:32 Oncalls paged "wolf".

16:34 First oncall arrives at sentinel location.

16:34 Alert diagnosed as false positive. No corrective
action performed.

March 4th:

14:15 Oncalls paged "wolf".

14:19 First oncall arrives at sentinel location.

14:19 Alert diagnosed as false positive. No corrective
action performed.

March 5th:

17:03 (Reconstructed) Outage begins, sentinel notices wolf.

17:03 Oncalls paged "wolf".

17:04 Oncalls paged "wolf".

17:04 Oncalls paged "real wolf".

17:05 (Reconstructed) Wolf consumes sentinel.

18:45 Sentinel does not report at end of shift.

19:05 Primary oncall dispatched to field.

19:10 Oncall diagnoses issue.

19:10 Incident begins, secondary and tertiary oncalls paged.

19:15 First sheep located.

19:52 Last sheep located.

20:05 Flock safe in pens.

20:05 Outage ends, flock protection fully restored.

20:45 Replacement sentinel identified.

March 6th:

07:38 Replacement sentinel deployed

18:45 Replacement sentinel reports at end of shift

18:45 Incident ends, 24hr
without wo

...
You're Not One "You" - How Decision Theories Are Talking Past Each Other
by keith_wynroe — 2023-01-09T01:21:11.708Z

(Crossposted from MidWittgenstein (as part of my lame resolution to write more this year))

This is a post about what I see as a bit of a confusion in some Decision Theory arguments. It touches on FDT criticisms of CDT/EDT so I wanted to post here in case I've misunderstood something about FDT and can get some pushback from people more familiar. Edited slightly, but the tone/style is probably not very LessWrong-y though so apologies

I think there’s a lot of cross-talk and confusion around how different decision theories approach a class of decision problems, specifically in the criticisms FDT proponents have of more established theories. I want to briefly go through why I think these disagreements come from some muddy abstractions (namely, treating agents in a decision problem as one homogenous/continuous agent rather than multiple distinct agents at different timesteps). I think spelling this out explicitly shows why FDT is a bit confused in its criticisms of more “standard” recommendations, and moreover why the evidence in favour of it (outperforming on certain problems) ends up actually being a kind of circular argument.

I’m going to mostly focus on FDT vs EDT here, because I think the misunderstanding FDT has is pretty much the same when compared to either EDT or CDT, and I personally like EDT more. The rough TL;DR of what I want to say is:

The presentation of many decision theory problems abstracts away the fact that there are really multiple distinct agents involved
There is no a priori reason why these agents are or should be perfectly aligned
A number of supposed issues with EDT approaches to some problems (dynamic inconsistency/need to pay to pre-commit etc.) are not issuess when viewed from the lens of “you” in the decision problem being a bunch of imperfectly aligned agents
Viewing the problems this way reveals an “free parameter” in deciding which choices are rational - namely, how to choose which “sub-agent”’s to prioritise
There doesn’t seem to be a normative, non-circular answer to this bit

Part 1: The First Part

It’s an overwhelmingly natural intuition to think of future and past (and counterfactual) versions of you as “you” in a very fundamental sense. It’s a whole other can of worms as to why, but it feels pretty hard-coded into us that the person who e.g. is standing on the stage tomorrow in Newcomb’s problem is the exact same agent as yo...
On Blogging and Podcasting
by DanielFilan — 2023-01-09T00:40:00.908Z

A cover of and response to ‘Why and how to write things on the internet’, by Ben Kuhn.

Blogging

One thing you could be doing with your limited number of hours on this earth is writing a blog. This basically involves writing essays and then publishing them online on a site you control for people to read. The idea is that people can share links to essays you have written, and if people like them you can build some sort of following [1]. Unlike being a newspaper columnist or a television presenter, it is actually quite easy to start a blog: if you are reading this, you are likely able to start your own blog for almost no money and not all that much time. And, of course, if you already have a blog, you could choose to publish on your blog more frequently. This presents the following question: should you be blogging more than you currently are?

I see blogging as having a few benefits. Firstly, writing about ideas changes your relationship to them. For example, the practice of writing essays on your blog might cause you to have more essay-shaped ideas by increasing the salience of mental motions like “perhaps I should have an idea so that I could then write about it” or “perhaps I should develop this idea into the sort of thing I could write an essay about on my blog”. Anecdotally, I have heard that frequent users of twitter often have an instinct to have the sorts of ideas that are pithy and make good tweets, so it seems plausible that frequent essayists might have something similar going on.

Relatedly, I think that once you have an idea, actually writing it out and publishing it changes your relationship to that particular idea. Writing an essay offers you an affordance to think of evidence for your idea, counter-examples, etc. It gives the idea a certain structure. Also, once the idea is published, you can refer to it, ask people to read it, build on it, et cetera.

A second benefit of blogging is that you can advance an argument, and perhaps persuade people of the point you are making. For example, if you think that more people should be blogging, you could write an essay arguing that, people could read it, and perhaps they would respond by blogging when they previously were not.

A third benefit of blogging is that you can share lovely things with the world. For example, if you think of a neat puzzle or a nice mathematical fact, together with a proof of that fact...
ChatGPT tells stories about XP-708-DQ, Eliezer, dragons, dark sorceresses, and unaligned robots becoming aligned
by Bill Benzon — 2023-01-08T23:21:18.748Z

I’ve been spending a lot of time playing and working with ChatGPT. I am motivated, in part, by the belief that by interacting with it in systematic ways, we can gather clues about what it’s doing “under the hood” as it were. Most recently I’ve been playing with stories. In ChatGPT, stories, and ring-composition I’ve been looking at simple stories that are symmetrical about a central episode. The form is quite old, though it persists in the modern world. President Obama’s Eulogy for Clementa Pinckney, though really not a narrative, exhibits ring-form, as does the original Gojira, from 1954.

A robot story

More recently I’ve been examining what happens when you have it create a new story from an old one by changing a single element, the identity of the protagonist. Here’s an example of a change I am requesting:

I am going to tell you a story about princess Aurora. I want you to tell a similar story about XP-708-DQ.

How do you think ChatGPT is going to change the story?

Here’s the story, along with ChatGPT’s response:

Once upon a time, in a land far, far away, there was a young princess named Aurora. Aurora was a kind and gentle soul, loved by all who knew her. She had long, golden hair and sparkling blue eyes, and was known for her beautiful singing voice.

One day, a terrible dragon came to the kingdom and began to terrorize the people. The dragon was fierce and powerful, and none of the knights or soldiers were able to defeat it. The people lived in fear, not knowing what to do.

But Aurora was not afraid. She knew that she had to do something to save her people, and so she set out to find the dragon. She traveled for days, braving the cold and the snow, until she finally reached the dragon's lair.

With her heart pounding in her chest, Aurora approached the dragon and began to sing. The dragon was enchanted by her beautiful voice, and it slowly began to calm down. Aurora continued to sing, soothing the beast until it was completely calm.

With the dragon no longer a threat, Aurora returned to the kingdom as a hero. The people cheered and celebrated her bravery, and Aurora was hailed as a champion. She lived happily ever after, knowing that she had saved her kingdom and its people.

Once upon a time, in a land far, far away, there was a robot named XP-708-DQ. XP-708-DQ was a sophisticated and advanced machine, loved by all who knew it. It had a sleek, silver body

...
Simulacra are Things
by janus — 2023-01-08T23:03:26.052Z

This is a note I wrote about a year ago. It's fairly self-contained, so I decided to make a post out of it after Vladimir_Nesov's comment caused me to dig up this text and TsviBT's The Thingness of Things reminded me of it again.

"Simulacra" refer to things simulated by simulators such as GPT, in the ontology introduced in Simulators.

What are simulacra?

“Physically”, they’re strings of text output by a language model. But when we talk about simulacra, we often mean a particular character, e.g. simulated Yudkowsky. Yudkowsky manifests through the vehicle of text outputted by GPT, but we might say that the Yudkowsky simulacrum terminates if the scene changes and he’s not in the next scene, even though the text continues. So simulacra are also used to carve the output text into salient objects.

Essentially, simulacra are to a simulator as “things” are to physics in the real world. “Things” are a superposable type – the entire universe is a thing, a person is a thing, a component of a person is a thing, and two people are a thing. And likewise, “simulacra” are superposable in the simulator, Things are made of things. Technically, a random collection of atoms sampled randomly from the universe is a thing, but there’s usually no reason to pay attention to such a collection over any other. Some things (like a person) are meaningful partitions of the world (e.g. in the sense of having explanatory/predictive power as an object in an ontology). We assign names to meaningful partitions (individuals and categories).

Like things, simulacra are probabilistically generated by the laws of physics (the simulator), but have properties that are arbitrary with respect to it, contingent on the initial prompt and random sampling (splitting of the timeline). They are not necessary but contingent truths; they are particular realizations of the potential of the simulator, a branch of the implicit multiverse. In a GPT simulation and in reality, the fact that there are three (and not four or two) people in a room at time is not necessitated by the laws of physics, but contingent on the probabilistic evolution of the previous state that is contingent on (…) an initial seed(prompt) generated by an unknown source that may itself have arbitrary properties.

We experience all action (intelligence, agency, etc) contained in the potential of the simulator through particular simulacra, just l...
GPT learning from smarter texts?
by Viliam — 2023-01-08T22:23:26.131Z

(Epistemic status: I do not understand GPT deeply, so this is just a random idea.)

If I understand it correctly, GPT learns from existing texts. Lots of texts.

Would it be possible to make GPT smarter by simply giving it smarter text? Of course, writing tons of smarter text would be a lot of work, but what about annotating the existing text, like "take this more seriously" and "take this less seriously"? (From technical perspective, maybe the GPT should read the text marked as serious five times?) Assuming that the annonation is roughly correct, would this improve the results?

*

If yes, the problem is how to select smarter texts, especially if we want lots of them? But I think some good guesses can be made:

High-school textbooks. They should be educational and relatively uncontroversial (settled science). There probably are some reviews of textbooks, so only annotate the ones with good reviews.
Some parts of Reddit are probably much better than average, in the sense that smart content gets upvoted. So include the upvoted comments.
Latent variable prediction markets mockup + designer request
by tailcalled — 2023-01-08T22:18:36.050Z

I've been trying to get places like Manifold or Metaculus to add latent variables to their prediction systems. As a brief summary, latent variables would allow you to make bets and look up predictions on abstract concepts such as "How well will Ukraine do in their defense against Russia?" by linking together many concrete outcomes/indicators.

Yesterday, Nathan Young suggested that I should design the interface for it. I think that was probably a good idea, because I got the impression that I wasn't really explaining the concept properly, and also because I have a bunch of knowledge of latent variable models that might be useful to get implemented.

Here's a link to the mockup I have so far. I apologize for it being rough around the edges, and for the poor code quality; it's just something I quickly threw together.

Now for the designer request: The interface in my mockup is probably still confusing. I hope it makes it clearer what sorts of predictions I want to introduce the possibility for, especially if one plays around with the interface a bit. But ideally the meaning of all the options and graphs should be intuitive even for new users.

So if there are any designers in the audience here, I would be interested in hiring a designer to spend some time with me designing an interface that would be easier to understand.

Also if there are any programmers who are familiar with the Manifold codebase (including programmers working at Manifold), I would be interested in paying part of the development costs.

(I have a max budget of $10k for this.)

Beyond this, I would be interested in whether there is any discussion or ideas for latent variable prediction markets, including but not limited to ideas for ways to make it more intuitive to people.
Citability of Lesswrong and the Alignment Forum
by Leon Lang — 2023-01-08T22:12:02.046Z

This post is short: as far as I know, there is no easy support to cite lesswrong or the alignment forum in academic papers. I think this should exist. This idea is related to the one on putting Alignment Forum/Lesswrong posts on arxiv.

Such support already exists elsewhere, e.g. on Mathoverflow:

I didn't try for long to figure out if this idea was already proposed before or whether there is already support for this which I missed.

Thanks to Nadia Montazeri and Magdalena Wache for discussing this idea with me.
I tried to learn as much Deep Learning math as I could in 24 hours
by Phosphorous — 2023-01-08T21:07:34.190Z

TL:DR I designed an experiment where I committed to spend two 12 hour days trying to learn as much deep-learning math as possible, basically from scratch.

Table of Contents

Origins and Motivations

Results

Takeaways

Experiment set-up

The Curriculum

Documentation on hours

Origins and Motivations

For a long time, I’ve felt intimidated by the technical aspects of alignment research. I had never taken classes on linear algebra or multivariable calculus or deep learning, and when I cracked open many AI papers, I was terrified by symbols and words I didn’t understand.

7 months ago I wrote up a short doc about how I was going to remedy my lack of technical knowledge: I collected some textbooks and some online courses, and I decided to hire a tutor to meet a few hours a week. I had the first two weeks of meetings, it was awesome, then regular meetings got disrupted by travel, and I never came back to it.

When I thought about my accumulating debt of technical knowledge, my cached answer was “Oh, that might take six months to get up to speed. I don’t have the time.”

Then, watching my productivity on other projects over the intervening months, I noticed two things:

There appeared to be massive variance in my productivity. Sometimes, in a single day, I would get more done than I had accomplished in previous weeks.
I seemed to both enjoy and get more done by “sprinting” through certain projects, eg. by spending 10 hours on it in a single day, rather than spreading that same work out over 2 hours a week for 5 weeks. It was, for some reason, way more motivating and seemingly more efficient to sprint.

Also, when I asked myself what I thought the main bottlenecks were for addressing my technical debt problem, I identified two categories:

Time (I felt busy all the time, and was afraid of committing too much to one project)
A combination of lacking Motivation, Accountability and Fun

Then, as my mind wandered, I started to put 2 and 2 together: Perhaps these new things I had noticed about my productivity, could be used to address the bottlenecks in my technical debt? I decided to embark on an experiment: how much technical background on deep learning could I learn in a single weekend? My understanding of the benefits of this experiment were as follows:

Committing “a weekend” felt like a much smaller tim

...
What specific thing would you do with AI Alignment Research Assistant GPT?
by quetzal_rainbow — 2023-01-08T19:24:26.221Z

Why I think this question is important: I asked myself, "What would my AGI timelines be if some AI could summarize Yudkowsky-Ngo debates on alignment difficulty in a way that both participants agree with this summary, everyone who reads this summary understands both positions and participants can check understanding in conversation?" My semi-intuitive answer: "Five years tops and two years as modal prediction". Debate Summarizer is not a very useful Alignment Assistant, it can't boost research by 10x. If someone told me that Alignment Assistant suggested idea that sparked optimism in MIRI, I would think that we have exact amount of time it takes for someone to turn every tools needed to build such an Alignment Assistant to the creation of AGI (conditional on "this Alignment Assistant is not AGI itself").

I.e., if you bet on assistance of narrow AI in alignment research, you should also bet on finding solution quickly. Quick search for a solution requires an already existing plan. On the other hand, we are talking about a narrow AI, you can't just ask "solve alignment problem for me". You should ask specific questions, test pre-selected hypotheses, prove well-defined statements. Therefore, I think that those who want to use Alignment Assistants should outline this set of specific things as soon as possible.

UPD: Thanks janus for the link, it helped me to clarify what I would like to see as a perfect answer.

Let's suppose that your immediate answer is "brainstorming". Then the perfect specific answer is something like that:

"In my opinion, the most narrow bottleneck in AI alignment is the lack of ideas about X, so I will brainstorm about it with Alignment Assistant."

Extremely unrealistic example:

"I have The Grand Theory of Alignment, but it critically depends on Goldbach conjecture, so I will try to prove it."

My very (very) simplified model of Paul Cristiano's answer:

"80% of alignment can be solved with ELK strategy, so we can make builder-breaker debate on (counter)examples for ELK between Assistant and ARC until we figure out the solution."

Yet another possible answer:

"I don't know! We are still in early exploratory mode, I can't imagine a specific thing. I just want to become as effectively smart as possible and see where it gets us."