trentmkelly/LessWrong-43k huggingface dataset:
Can We Align AI by Having It Learn Human Preferences? I’m Scared (summary  of last third of Human Compatible)
by apollonianblues — 2022-06-29T04:09:06.213Z

Epistemic status: I am just learning about alignment and just read Human Compatible. Below is a summary of the paradigm he outlines for aligning AI in the last third of his book, and the questions I have of this project as a new reader.

AI researcher Stuart Russell’s 2019 book Human Compatible is one of the most popular/widely-circulated books on AI alignment right now. It argues that we need to change the paradigm of AI development in general from what he calls goal-directed behavior, where a machine optimizes on a reward function written by humans, to behavior that attempts to learn and follow human objectives.

Russell provides a set of general design principles that someone building this kind of AI would have to follow, quoted directly below:

The machine’s only objective is to maximize the realization of human preferences.
The machine is initially uncertain about what those preferences are.
The ultimate source of information about human preferences is human behavior.

He makes sure to state that he means “preferences” in the way that an economist would define them. Under this definition, a “preference” refers to whatever, given a set of options, a person would rather choose. He attempts to distinguish this from optimizing for “happiness” in a utilitarian framework, which might involve somehow getting a metric of how happy people are that doesn’t just depend on people’s choices, or for some other ethical system encoded by a human. So, instead of getting an objective right the first time, which is insanely hard, you have an AI that does machine learning to figure out what a person, or people, actually want, from observing their preferences.

Russell connects this to work he and others have done in Inverse Reinforcement Learning (IRL), which is machine learning that observes an agent’s behavior and determines the reward function the agent is optimizing for. For example, Russell describes observing a cricket walking on various terrains and wanting to understand its behavior and how to emulate it. In order to understand this, he first needed to determine the cricket’s reward function when walking; what feedback did the cricket seem to be getting to determine it was walking properly? This process basically just starts with a bunch of priors for the agent’s reward function and adjusts until the reward function properly predicts the agent’s behavior. The reward func...
Kurzgesagt – The Last Human (Youtube)
by habryka — 2022-06-29T03:28:44.213Z
Link: https://www.youtube.com/watch?v=LEENEFaVUzU&feature=youtu.be

TLDW: Big Youtube channel Kurzgesagt released a video on the potential of humanity's future that's I think my favorite videos of theirs yet.

I have a pretty high bar for posting video content to LessWrong, but this video seemed better than most other videos in terms of capturing some of the things that make me interested in a lot of things on LessWrong.
Literature on How to Maximize Preferences
by josh — 2022-06-28T22:41:38.152Z

What mainstream academic literature should I look at in order to better think through questions like "In order to maximize my preferences to what extent should I invest time in better understanding my goals?", "To the extent that question 1 indicates that this is important, how can I better understand my goals?", "In order to maximize my preferences to what extent should I invest time into better understand different parts of the world?", and "What is the best way to better understand different parts of the world?" I would imagine something related to epistemics or psychology would answer "to what extent should I invest time in better understanding my goals" and statistics/Tetlock's research would would answer questions related to "what is the best way to better understand different parts of the world but are there any textbooks people could suggest on these topics? These seem like some of the most basic questions one can ask, but I notice my thinking on them is particularly fuzzy.
Challenge: A Much More Alien Message
by kman — 2022-06-28T21:50:59.877Z

Since this challenge was solved pretty quickly, I thought I'd try to make a substantially more difficult program induction challenge.

The Challenge

Find a short program which generates this binary file.

Hints (because I expect this to be extremely hard)

For a sense of how simple the program is, I generated the file with a python script of a similar length and complexity to the solution to the prior challenge.

Other than writing out to the file at the end, the generator uses only basic primitives of the python language (integer arithmetic, logic and flow control, arrays)

The only numeric constants used in the generator script are 0,1,2,2222.
The generator script takes nearly 10 minutes to run on my laptop!

So if you have any good hypotheses, you might want to test them with a compiled language for a 100-fold speedup over python.
It’s Probably Not Lithium
by Natália — 2022-06-28T21:24:10.246Z

This post has been recorded as part of the LessWrong Curated Podcast, and can be listened to on Spotify, Apple Podcasts, Libsyn, and more.

A Chemical Hunger

(

a

), a series by the authors of the blog Slime Mold Time Mold (SMTM) that

has been

received positively

on LessWrong

, argues that the obesity epidemic is

entirely caused

(

a

) by environmental contaminants. The authors’ top suspect

is lithium

(

a

)[1], primarily because it is known to cause weight gain at the doses used to treat bipolar disorder.

After doing some research, however, I found that it is not plausible that lithium plays a major role in the obesity epidemic, and that a lot of the claims the SMTM authors make about the topic are misleading, flat-out wrong, or based on extremely cherry-picked evidence. I have the impression that reading what they have to say about this often leaves the reader with a worse model of reality than they started with, and I’ll explain why I have that impression in this post.

(Preamble) A brief summary of their hypotheses

The SMTM authors have

recently

(

a

) summarized their hypotheses on how lithium exposure could explain the obesity epidemic. The first hypothesis is that trace exposure is responsible:

One possibility is that small amounts of lithium are enough to cause obesity, at least with daily exposure.

And the second one is that people are intermittently exposed to therapeutic doses:

[E]ven if people aren’t getting that much lithium on average, if they sometimes get huge doses, that could be enough to drive their lipostat upward.

I am going to argue that neither of those is plausible. I address the plausibility of the second hypothesis in the next section, and the plausibility of the first one in the rest of the post.

Lithium exposure in the general population is extremely low, even at the tails, in the majority of countries for which we have data

A few days ago, the SMTM authors published

a literature review

(

a

) on the lithium content of food. They conclude that, whereas the existing literature isn’t great, “[i]t seems like most people get at least 1 mg [of lithium] a day from their food, and on many days, there’s a good chance you’ll get more.” They also say it seems plausible that people are intermittently exposed to doses of lithium within the therapeutic range through their diet.

However, their literature review prett...
Reflections on Living in "Guess Culture"
by Dalton Mabery — 2022-06-28T21:00:39.680Z

This is cross-posted from my blog.

There's been two articles I've seen recently on the difference between people who ask for what they want (ask culture) vs. people who drop subtle hints at what they want (guess culture). I'm not afraid to admit I fall in the latter group. Does this mean some emotional aspect of my social life is immature or not developed enough?

Some say I'm "easy going" or very "go with the flow" and I'd agree with both of those statements. But it's not because I have a big soft spot in my heart and I want to be selfless. Rather, it comes from a deep sense of insecurity of not wanting to inconvenience other people. Most of my approach with groups of people, especially groups of people I don't know well, is to ensure other people aren't inconvenienced.

The reason I do that is simple: I know I can handle being inconvenienced. If I really want to go somewhere on a vacation and am unable to make it by the end, sure it's a bit sad, but life's still great. I just keep on keepin' on. Others are different. Some people really value seeing certain areas when they're in a new city and would be crushed to not see that thing. Sadness comes first followed me resentment at the people/reasons why they weren't able to see that thing. It's just much easier if I inconvenience myself because I know I can handle it. Is that egotistical? Maybe. A bit stupid? Probably? But does it make sense? In my head, yes.

Part of managing other people's inconvenience's has instilled in me a habit of practicing "guess culture." Guess/Ask Culture is an interesting social phenomenon originally reported on by The Atlantic, but recently covered by Maybe Baby in an essay. A user named Andrea Donderi posted the following response to someone asking how to get out out of hosting someone at their house for a vacation:

The ideas are really amazing and surprisingly not very well known. I will say here and now that I am an addicted guesser. I rarely, if ever, put a direct ask into words except to my wife or my best friends. If I get a burger from a restaurant that has mustard on it even though I specifically asked for no mustard, I will not say a thing. I will scrape the mustard off with a knife and enjoy it the best that I can. But if the waiter catches me scrapping off the mustard and asks, "Would you like us to remake it?" I will respond with something along the lines of "If you don't mind...
What Are You Tracking In Your Head?
by johnswentworth — 2022-06-28T19:30:06.164Z

A large chunk - plausibly the majority - of real-world expertise seems to be in the form of illegible skills: skills/knowledge which are hard to transmit by direct explanation. They’re not necessarily things which a teacher would even notice enough to consider important - just background skills or knowledge which is so ingrained that it becomes invisible.

I’ve recently noticed a certain common type of illegible skill which I think might account for the majority of illegible-skill-value across a wide variety of domains.

Here are a few examples of the type of skill I have in mind:

While operating a machine, track an estimate of its internal state.
While talking to a person, track an estimate of their internal mental state - emotions, engagement, thoughts/worries,

true motivations

, etc.
While writing an algorithm, track a Fermi estimate of runtime.
While reading or writing math,

track a prototypical example

of what the math is talking about.
While playing a competitive game, track an estimate of the other players’ plans, intentions and private information
While writing, track an estimate of the mental state of a future reader - confusion, excitement, eyes glossing over, etc.
While reasoning through a difficult search/optimization problem, track an estimate of which constraints are most taut.
While working on math, physics, or a program, track types/units
While working on math, physics, or a program, track asymptotic behavior
While in conversation, track ambiguous tokenization for potential jokes.
While presenting to a crowd, track engagement level.
While absorbing claims/information, track an estimate of the physical process which produced the information, and how that process entangles the information with physical reality.

The common pattern among all these is that, while performing a task, the expert tracks some extra information/estimate in their head. Usually the extra information is an estimate of some not-directly-observed aspect of the system of interest. From outside, watching the expert work, that extra tracking is largely invisible; the expert may not even be aware of it themselves. Rarely are these mental tracking skills explicitly taught. And yet, based on personal experience, each of these is a central piece of performing the task well - arguably the central piece, in most cases.

Let’s assume that this sort of extra-information-tracking ...
Why is so much political commentary misleading?
by contrarianbrit — 2022-06-28T17:10:58.743Z
Link: https://thomasprosser.substack.com/p/why-is-so-much-political-commentary

Much political commentary is highly misleading. Though media evaluation of policy agendas and the conduct of politicians tends to be better, analysis of policy effects and the role of parties is very poor, even in the ‘quality’ press. When assessing policy records, commentators tend to mistake noise for the effects of policy interventions, research

suggesting

that performance metrics are mostly outside the control of politicians.

Why would this be? If societies are to function optimally, should they not use information which conforms tightly with reality? As cultural evolution theory emphasizes, the most efficient societies tend to outcompete alternatives, good information being crucial to this.

In the case of political commentary, better information is available, academic research being more reliable than media analysis. The reasons why this phenomenon occurs are fascinating, demonstrating the limits of rationality. Misleading commentary boosts confidence in liberal democracy; it may be better not to challenge it.

***

Political commentators make assumptions about performance metrics and party differences. On performance metrics, most analysts presume that one can evaluate the effects of distinct policies. For example, comment pages are filled with assessment of the effects of government policy in areas such as crime and healthcare. Typically, these analyses will be confident, passing judgement on the efficacy of distinct policies, even in quality newspapers. In the UK, there is lively debate about the government’s levelling-up agenda – the Johnson government is trying to improve conditions in the north of England - many commentators concluding that it has been fruitless. All countries have their equivalent, analysts dissecting the effects of political interventions.

But there are good reasons for doubting such judgements. A recent study is damning. Writing in the prestigious American Political Science Review and using quasi-experimental methods, Dynes and Holbein

demonstrate

that voters struggle to hold governments accountable. Performance...
CFAR Handbook: Introduction
by CFAR!Duncan — 2022-06-28T16:53:53.312Z

The Center for Applied Rationality is a Bay Area non-profit that, among other things, ran lots of workshops to offer people tools and techniques for solving problems and improving their thinking. Those workshops were accompanied by a reference handbook, which has been available as a PDF since 2020.

The handbook hasn't been substantially updated since it was written in 2016, but it remains a fairly straightforward primer for a lot of core rationality content. The LW team, working with the handbook's author Duncan Sabien, have decided to republish it as a lightly-edited sequence, so that each section can be linked on its own.

In the workshop context, the handbook was a supplement to lectures, activities, and conversations taking place between participants and staff. Care was taken to emphasize the fact that each tool or technique or perspective was only as good as it was effectively applied to one's problems, plans, and goals. The workshop was intentionally structured to cause participants to actually try things (including iterating on or developing their own versions of what they were being shown), rather than simply passively absorb content. Keep this in mind as you read—mere knowledge of how to exercise does not confer the benefits of exercise!

Discussion is strongly encouraged, and disagreement and debate are explicitly welcomed. Many LWers (including the staff of CFAR itself) have been tinkering with these concepts for years, and will have developed new perspectives on them, or interesting objections to them, or thoughts about how they work or break in practice. What follows is a historical artifact—the rough state-of-the-art at the time the handbook was written, circa 2017. That's an excellent jumping-off point, especially for newcomers, but there's been a lot of scattered progress since then, and we hope some of it will make its way into the comments.
Units of Exchange
by CFAR!Duncan — 2022-06-28T16:53:53.069Z

Epistemic status: Established and confirmed

The lessons taught in Units of Exchange are straightforward applications of extremely well-established principles from economics and sociology, such as supply and demand, Pareto curves, value of information, the sunk cost fallacy, and arbitrage. The causal relationships underlying by each of these principles have been robustly confirmed in a wide variety of domains, and the recommended actions we’ve derived from them are simple and fairly conventional.

A version of this course is taught near the beginning of every CFAR workshop, often as the very first class. That’s not because the concepts it covers are revelational or groundbreaking, but rather the opposite—they’re core concepts, fundamental prerequisites that underlie and inform much of the rest of our content.

If you’re already familiar with them—great! This is a quick-and-dirty overview—we don’t mean to condescend to people who’ve already had specific training in these fields, only to provide those same tools to all of our participants. If these are not the droids you’re looking for, feel free to skip ahead to Inner Simulator.

The Lego Principle: Bricks and bargains

If you’ve ever done proofs in math or logic, then you know that it’s possible to reach complex and interesting results by starting from a limited number of assumptions. The conclusions in Units of Exchange aren’t mathematically rigorous, but they do emerge from two key premises, which combine to form what we call the Lego Principle.

1. Things are made of parts

The first half of the Lego Principle is reductionism, or the idea that, having fully explained all of the components of a thing and how those components interrelate, there’s nothing left to say. Metaphorically, if one has described the trees, shrubs, and fauna in all of their relevant detail, one has fully explained the forest; there is no ephemeral “missing” property that is forest-ness.

Reductionism is a powerful concept, because it allows us to interface with complex phenomena by dealing with smaller, simpler sub-phenomena. Brains, emotions, societies, financial markets—these are all large, and sometimes daunting to engage with. But neurons, cognitive if-then patterns, social norms, and individual commodities are all at least relatively more tractable.

2. Parts may be exchanged

The technical term here is currencies, and the idea is that, just as we ...
Scott Aaronson and Steven Pinker Debate AI Scaling
by Liron — 2022-06-28T16:04:58.515Z
Link: https://scottaaronson.blog/?p=6524

Before June 2022 was the month of the possible start of the Second American Civil War, it was the month of a lively debate between Scott Alexander and Gary Marcus about the scaling of large language models, such as GPT-3. Will GPT-n be able to do all the intellectual work that humans do, in the limit of large n? If so, should we be impressed? Terrified? Should we dismiss these language models as mere “stochastic parrots”?

I was privileged to be part of various email exchanges about those same questions with Steven Pinker, Ernest Davis, Gary Marcus, Douglas Hofstadter, and Scott Alexander. It’s fair to say that, overall, Pinker, Davis, Marcus, and Hofstadter were more impressed by GPT-3’s blunders, while we Scotts were more impressed by its abilities. (On the other hand, Hofstadter, more so than Pinker, Davis, or Marcus, said that he’s terrified about how powerful GPT-like systems will become in the future.)

Anyway, at some point Pinker produced an essay setting out his thoughts, and asked whether “either of the Scotts” wanted to share it on our blogs. Knowing an intellectual scoop when I see one, I answered that I’d be honored to host Steve’s essay—along with my response, along with Steve’s response to that. To my delight, Steve immediately agreed. Enjoy! –SA
A physicist's approach to Origins of Life
by pchvykov — 2022-06-28T15:23:23.310Z

Note: this is cross-posted from the post here, where I had a few videos that I could not add in here - see the original post for the full experience

I spent the 6 years in my Ph.D. thinking of the broad topic of origins of life, or “abiogenesis”: how does non-living matter turn into living matter? This was a lot of fun, big ideas, wild speculations, and crazy discussions with friends and colleagues – along with heart-wrenching frustration of trying to actually produce some concrete novel scientific results in a topic this big and old. When the smoke cleared, the few concrete results I managed to publish were annoyingly narrow and specific, or abstract and hard to follow. All in all, the relationship between the big exciting question and my published work is probably clear only to me, since journals don’t like publishing a 10 page philosophical discussion in the intro. So I decided to take the matter in my own hands and publish said discussion in this post. I’ll try to keep it shorter than 10 pages, will avoid getting too technical, and will illustrate the ideas on a concrete example system that I studied in my research, but that was too simple to warrant its own publication. A word of caution right from the start: this article will open a lot more questions than it will give answers, so rather than explaining how life originated on Earth, my intention here is to set some groundwork for future research on this.

So where does one start to think about how life came up here? Most people start by looking into the chemistry of the primordial molecular soup, and then ask questions like: “how could some simple self-replicating molecule first come up and transmit genetic information?” (termed “RNA-first” approach), or perhaps “could a group of molecules learn to stick together in a little puddle that could eat, self-heal and adapt?” (termed “metabolism-first” approach). Being a physicist, and worse yet a theorist, I started from a totally different point: what is the simplest possible dynamical system I could find that could show some signs we might typically associate with being ‘alive’? The hope was that if found, I could then study such system in detail, and understand what caused it to cross the line from apparently “non-living” to “living.”

The characteristics of "life" - from Wikipedia

This formulation already hints at one unique aspect of origins-of-life research:...