trentmkelly/LessWrong-43k huggingface dataset:
Arguments for the existence of qualia
by Chris_Leong — 2019-07-28T10:52:42.997Z

One trap that we must be wary of is adopting beliefs because they are popular among people who strive to think critically or scientifically, as opposed to being the result of critical or scientific thinking. One good example of this is RationalWiki which purports to report the political beliefs which all Rational(TM) people should hold. Similarly, I believe that most people who believe in materialism do so on the basis of extremely poor reasons and without knowledge of some of the stronger arguments for qualia existing. Maybe we should ultimately support materialism, but I get the impression that many people jump to this conclusion too quickly. Here are some points I believe people should consider first

(I know someone else in the rationality community wrote a post arguing for consciousness recently and I was meaning to read it, but I lost the link before I had a chance)

Failure to bite the bullet argument

If qualia don't exist, why is anything that you experience good or bad? In this case, things like pleasure, pain and meaning are nothing more than ways that particles can combine. But if this is the case, why are these combinations special or more important than other combinations? Why should we try to make certain combinations happen and certain combinations not?

It's fairly common to deny the existence of the objectively good or bad, but denying the existence of the subjectively good or bad is a much stronger claim. But everyone acts like this matters and so it appears somewhat hypocritical. And there is an argument that we should continue to act normally on the basis of meta-theoretic uncertainty, but no-one makes that argument.

Pascal's Wager Argument

This last point is actually a pretty strong argument for believing in qualia. If they don't exist, nothing matters, but if they do exist then we benefit from acting as though they do exist. Therefore, we should assume the later.

Expected Evidence Argument

Claiming the existence of qualia is often seen as anti-scientific, some people would even go as far to say that they don't see much difference between claiming the existence of ghosts or qualia. One key difference is that if ghosts existed we would expect objective evidence of them, even if the experiments would be hard to run. For example, we would expect a greater rate of howling in houses where someone was murdered. Even if they could...
Applying Overoptimization to Selection vs. Control (Optimizing and Goodhart Effects - Clarifying Thoughts, Part 3)
by Davidmanheim — 2019-07-28T09:32:25.878Z

Clarifying Thoughts on Optimizing and Goodhart Effects - Part 3

Previous Posts: Re-introducing Selection vs Control for Optimization, What does Optimization Mean, Again? -

Following the previous two posts, I'm going to try to first lay out the way Goodhart's Law applies in the earlier example of rockets, then try to explain why this differs between selection and control. (Note: Adversarial Goodhart isn't explored, because we want to keep the setting sufficiently simple.) This sets up the next post, which will discuss Mesa-Optimizers.

Revisting Selection vs. Control Systems

Basically everything in the earlier post that used the example process of rocket design and launching is susceptible to some form of overoptimization, in different ways. Interestingly, there seem to be clear places where different types of overoptimization is important. Before looking at this, I want to revisit the selection-control dichotomy from a new angle.

In a (pure) control system, we cannot sample datapoints without navigating to them. If the agent is an embedded agent, and has sufficient span of control to cause changes in the environment, we cannot necessarily reset and try over. In a selection system, we only sample points in ways that do not affect the larger system. Even when designing a rocket, our very expensive testing has approximately no longer term effects. (We'll leave space debris from failures aside, but get back to it below.)

This explains why we potentially care about control systems more than selection systems. It also points to why Oracles are supposed to be safer than other AIs - they can't directly impact anything, so their output is done in a pure selection framework. Of course, if they are sufficiently powerful, and are relied on, the changes made become irreversible, which is why Oracles are not a clear solution to AI safety.

Goodhart in Selection vs. Control Systems

Regressional and Extremal Goodhart are particularly pernicious for selection, and potentially less worrying for control. Regressional Goodhart is always present if we are insufficiently aware of our goals, but in general Causal Goodhart failures seems more critical in control, because it is often narrower. To keep this concrete, I'll go through the classes of failure, and note how they could occur at each stage of rocket design. To do so, we need to clarify goals at each sta...
What does Optimization Mean, Again?  (Optimizing and Goodhart Effects - Clarifying Thoughts, Part 2)
by Davidmanheim — 2019-07-28T09:30:29.792Z

Clarifying Thoughts on Optimizing and Goodhart Effects - Part 2

Previous Post: Re-introducing Selection vs Control for Optimization In the post, I reviewed Abram's selection/control distinction, and suggested how it relates to actual design. I then argue that there is a bit of a continuum between the two cases, and that we should add an addition extreme case to the typology, direct solution.

Here, I will revisit the question of what optimization means.

NOTE: This is not completely new content, and is instead split off from the previous version and rewritten to include an (Added) discussion of Eliezer's definition for measuring optimization power, from 2008. Hopefully this will make the sequence clearer for future readers.

In the next post, Applying over-Optimization in Selection and Control, I apply these ideas, and concretize the discussion a bit more before moving on to discussing Mesa-Optimizers in Part 4.

What does Optimization Mean, Again?

This question has been discussed a bit, but I still don't think its clear. So I want to start by revisiting a post Eliezer wrote in 2008, where he suggested that optimization power was ability to select states from a preference ordering over different states, and could be measured with entropy. He notes that this is not computable, but gives us insight. I agree, except that I think that the notion of the state space is difficult, for some of the reasons Scott discussed when he mentioned that he was confused about the relationship between gradient descent and Goodhart's law. In doing so, Scott proposed a naive model that looks very similar to Eliezer's;

simple proxy of "sample points until I get one with a large U value" or "sample n points, and [select] the one with the largest U value" when I think about what it means to optimize something for U. I might even say something like "n bits of optimization" to refer to sampling 2n points. I think this is not a very good proxy for what most forms of optimization look like."

I want to start by noting that this is absolutely and completely a "selection" type of optimization, in Abram's terms. As Scott noted, however, it's not a good model for what most optimization looks like, and that's part of why I think Eliezer's model is less helpful than I did when I originally read it.

There's a ...
Keep Your Beliefs Cruxy
by Raemon — 2019-07-28T01:18:13.611Z

Previously in sequence: Doublecrux is for building products.

To recap – you might want to doublecrux if either:

You're building a product, metaphorical or literal, and you disagree about how to proceed.
You want to make your beliefs more accurate, and you think a particular person you disagree with is likely to have useful information for you.
You just... enjoy resolving disagreements in a way that mutually pursues truth for whatever reason.

Regardless, you might find yourself with the problem:

Doublecruxing takes a lot of time.

For a 'serious' disagreement, it frequently takes a least an hour, and often much longer. Habryka and I once took 12 hours over the course of 3 days to make any kind of progress on a particularly gnarly disagreement. And sometimes disagreements can persist for years despite significant mutual effort.

Now, doublecruxing is faster than many other forms of truth-aligned-disagreement resolution. I actually it's helpful to think of doublecrux as "the fastest way for two disagreeing-but-honest-people to converge locally towards the truth", and if someone came up with a faster method, I'd recommend deprecating doublecrux in favor it that. (Meanwhile, doublecrux is not guaranteed to be faster for 3+ people to converge but I still expect it to be faster for smallish groups with particularly confusing disagreements)

Regardless, multiple hours is a long time. Can we do better?

I think the answer is yes, and it basically comes in the form of:

Practice finding your own cruxes
Practice helping other people find their cruxes
Develop metacognitive skills that make cruxfinding natural and intuitive
Caching the results into a clearer belief-network

Those are all things you can do unilaterally. If you get buy-in from your colleagues, you might also try something like "develop culture that encourages people to do those four things, and help each other to do so."

I'd summarize all of that as "develop the skill and practice of keeping your beliefs cruxy."

By default, humans form beliefs for all kinds of reasons, without regard for how falsifiable they are. The result is a tangled, impenetrable web. Productive disagreement takes a long time because people are starting from the position of "impenetrable web."

If you make a habit of asking yourself "what observations would change my mind about this?", then you gain a few benefits.

First, your beliefs should (hopefully?) b...
The wisdom of the Crowd
by Jason Geringer — 2019-07-27T23:14:29.694Z

I'm looking for people to be part of my facebook group named The wisdom of the of the crowd to help with the initial structure building of the group. developers would be awesome, but if you would like to just be part of the experiment you're welcome too. If there are any of you who are familiar with the theory of the wisdom of the crowd, and may have insights on how to set up the parameters to conduct some social experiments, well some of those people would be awesome too. if interested let me know. i'm not sure if i should put a link on here.
Is this info on zinc lozenges accurate?
by philh — 2019-07-27T22:05:11.318Z

Podcast: Zinc Definitely Fights Colds, But You’re Probably Using the Wrong Kind

This podcast claims that zinc lozenges are "probably almost essentially a cure for the common cold". But there are many caveats:

must be zinc acetate or zinc gluconate (but gluconate is strictly worse)

use immediately on getting a cold

18mg zinc per lozenge

dissolve in mouth 20-30 minutes

take every 2 hours

must have metallic taste, astringency

must free of anything ending -ate (except stearate) or -ic acid; free of magnesium except magnesium stearate (it's insoluble)

only one product on the market satisfies these requirements

The guy sounds to me like he knows what he's talking about. But I don't have the technical expertise to really know. (I think I could detect a mediocre bullshitter, but not necessarily a high level one.) If true, it seems like the sort of information that would be good for more people to know; but also like the sort of information that would be more widely known if it were true. (But I can sketch an argument for why it might not.) The research section at the linked page cites three journal articles of which two are open access, but I haven't looked closely at them.

My own experience is that I got some of these lozenges about a year ago, after reading a transcript of the episode. I thought I'd gotten them too late, but my cold cleared up much faster than I would have expected otherwise. Since then I've been trying to collect more anecdotal data, but my body is stubbornly refusing to even start coming down with a cold. Twice I thought it might be, and I took lozenges and didn't; but I think I took one lozenge on one occasion and three (spaced out) on another, and he thinks they shouldn't be effective enough for that to have worked. I'm not sure what to make of this, except that it shouldn't be much given the sample size.

Unfortunately the transcript has now been removed, and I can't find it on archive.org. I've made notes of the first ~35 minutes (of ~70). If someone could take a look (or listen), and say whether it all seems basically accurate, that would be fantastic. Almost all of it seems consistent with what I think I know, with one surprise that I've bolded. Apologies for the poor formatting.

zinc is important to the immune system in ways that are irrelevant to this. If you aren't getting enough, you'll probably benefit from getting more. Good sources

...
Shortform Beta Launch
by Raemon — 2019-07-27T20:09:11.599Z

We've had unofficial experiments with shortform for over a year. More and more people have been trying it out and finding it useful. Now, we're pushing shortform into an officially supported feature.

My description of shortform, inspired by pattern's comment, is:

Writing that is short in length, or written in a short amount of time. Includes off-the-cuff thoughts and brainstorming.

Why shortform?

Sometimes shorter is better

I've noticed when I write a Facebook post... it ends up exactly as long as it's supposed to be. I write 3-5 paragraphs that nicely encapsulate my idea, and then it looks about the right length, and I click submit and have a nice, clear discussion.

When I start writing a LessWrong post, sometimes I look at the beautiful serif text on the nice blank white page and... I dunno, it feels like I'm supposed to write a 3 page essay, so I do. But my idea would have been better if I expressed it in 3 paragraphs.

Sometimes off-the-cuff is better

I also often want to brainstorm early stage ideas in way that isn't (necessarily) optimized for others to read – figuring out how to explain something well might be hard, and I'm not even sure the idea is good yet. But, people who've been following along with my thought process and understand what I'm gesturing at can still chime in with ideas.

Sometimes I just wanna start writing without worrying about what sort of thing I'm writing yet.

There's also an important in-between case, where maybe I'm writing something off-the-cuff and brainstormy, and maybe I'm actually writing a full treatise on something important. And I just... don't want to spend cycles figuring that out. I want my editor to feel unopinionated, and I want to be able to click 'submit' at the end without stressing out about whether I'm submitting 'good enough content'.

Sometimes, this results in an initial shortform comment eventually getting revamped into a major post.

For all of these reasons, and more, it seems useful to have a part of lesswrong optimized for shorter writing.

New Features, focusing on Visibility

Shortform is created in the form of comments (attached to an automatically generated shortform post). The new features mostly aim to:

Automate a lot of work. Instead of having to manually create a post called "So-and-so's shor

...
The Artificial Intentional Stance
by Charlie Steiner — 2019-07-27T07:00:47.710Z

Another post in the same incremental vein. Still hoarding the wild speculation for later.

I

The idea of the "intentional stance" comes from Dan Dennett, who wanted to explain why it makes sense that we should think humans have such things as "beliefs" or "desires." The intentional stance is just a fancy name for how humans usually think in order to predict the normal human-scale world - we reason in terms of people, desires, right and wrong, etc. Even if you could be prodded into admitting that subatomic particles are "what there really are," you don't try to predict people by thinking about subatomic particles, you think about their desires and abilities.

We want to design AIs that can copy this way of thinking. Thus, the problem of the artificial intentional stance. Value learning is the project of getting the AI to know what humans want, and the intentional stance is the framework that goes between raw sensory data and reasoning as if there are things called "humans" out there that "want" things.

II

Suppose you want to eat a strawberry, and you are trying to program an AI that can learn that you want to eat a strawberry. A naive approach would be to train the AI to model the world as best it can (I call this best-fit model the AI's "native ontology"), and then bolt on some rules telling it that you are a special object who should be modeled as an agent with desires.

The reason this doesn't work is because the intentional stance is sort of infectious. When I think about you wanting to eat a strawberry using my intentional stance, I don't think about "you" as a special case and then use my best understanding of physiology, biochemistry, and particle physics to model the strawberry. Instead, I think of the verb "to eat" in terms of human desires and abilities, and I think of the strawberry in terms of how humans might acquire or eat one.

This is related to the concept of "affordances" introduced by James J. Gibson. Affordances are the building blocks for how we make plans in the environment. If I see a door, I intuitively think of opening or locking it - it "affords opening." But maybe an experienced thief will intuitively think of how to bypass the door - they'll have a different intuitive model of reality, in which different affo...
Results of LW Technical Background Survey
by johnswentworth — 2019-07-26T17:33:01.999Z

See results here.

The main goal of the survey was to provide info for authors about their target audience, so here's a high-level overview toward that end:

The average respondent is some kind of professional programmer, with an undergrad degree (or equivalent) in CS.
Most people have seen at least some economics and probability, but not at the level of a undergrad degree.
Almost everyone knows calculus, but linear algebra or differential equations will likely be lost on at least ~25% of respondents.
There are substantial zero-knowledge and high-knowledge counts for most areas.
About half of respondents had read the sequences in their entirety.

Here are charts of the responses to each question. I strongly recommend looking at them directly rather than just taking my summary at face value. As always, remember this is an opt-in survey without any sort of verification of responses, so take everything with a grain of salt.

One interesting note: we had a handful of respondents declaring very high skill levels (Nobel-level economists, Turing-level computer scientists, or primary developers of popular software). I'd personally be interested to hear what exactly those people work on, especially if they're willing to occasionally field questions on their area of expertise. All y'all should leave a comment or something.

Actually, I'm curious what everyone works on, especially specialties for all the researchers. Feel free to leave a quick comment, especially if you're able and willing to occasionally field questions in your area of expertise.
How many researchers does a new paper reach?
by Slider — 2019-07-26T04:14:06.930Z

How much of a field of research papers you must read to be "read" in your specialty in your concurrency? What contributing factors beside novelty does a paper have to have to get read? Are there some research areas where an idea needs to be "repapered" like a retweet multiple times before it registers in the collective conciouness?
Old Man Jevons Can’t Save You Now (Part 2/2)
by Closed Limelike Curves — 2019-07-26T03:51:34.447Z
Link: https://thelimelike.wordpress.com/2019/07/26/old-man-jevons-cant-save-you-now-part-2-2/

Followup to "Thank You, Old Man Jevons, for My Job"
CW: Robots, deadly to drastically oversimplified models presented in previous posts

So, I think it’s about time I crush your misplaced optimism and onto the reasons for doom and gloom and why everyone’s going to lose their jobs.

There’s really one big reason why the model of labor markets in my last post can break down: market segmentation.

I kinda lied when I said a 20% increase in productivity means a 20% increase in per-worker profit. The naive mistake that people make when thinking about automation as decreasing employment within an industry implicitly holds the quantity of goods being produced constant, such that a 20% increase in productivity will result in a 20% decrease in employment within a sector. Of course, that’s not true, or even close to true in most cases — when productivity increases, the quantity of goods produced increases as well, since it becomes more profitable to produce this good. But the analysis I put forward made a different assumption: That prices are constant. This is roughly correct for small changes in productivity: a 1% increase in workers’ efficiency is unlikely to have major effects on the market. But it breaks down completely when you talk about something like a 20% increase in productivity, which can significantly affect prices.

At the same time, though, it’s important to realize that the total price of all goods in the economy are mostly controlled by macroeconomic policy. If the Federal Reserve wants to keep prices constant, it can damn well keep prices constant. To quote Ben Bernanke, “We have the keys to the printing presses, and we are not afraid to use them.”

But what if the labor market isn’t actually a labor market? There’s obviously a big difference between someone with a doctorate and a high school dropout — they’re not going to be competing for the same jobs, even slightly. So we still have to consider the effects on prices to see how output for a particular market (The one receiving the productivity shock) will be affected. The Federal Reserve can keep the average price level constant, but it can’t keep the price of every single good constant — if productivity improvements are concentrated in a single part of the market (e.g. goods and not the service sector), then prices in that market are going to take a beating

So let’s make all of this a bit more rigor...
Can you summarize highlights from Vernon's Creativity?
by Raemon — 2019-07-26T01:12:31.724Z

Gwern had responded to Elizabeth's question about creativity with the following:

A good anthology to read is Creativity, ed Vernon 1970 - it's old but it shows you what people were thinking back when Torrance was trying to come up with creativity tests, and the many psychometric criticisms back then which I'm not sure have been convincingly resolved.

I'm not sure whether Elizabeth has already read it, but I'd be interested in reading the highlights from that if anyone was up for distilling it down into something more manageable.
How often are new ideas discovered in old papers?
by TurnTrout — 2019-07-26T01:00:34.684Z

Suppose someone wrote a paper about X two decades ago. A modern reader realizes the X paper sheds light on an unrelated idea Y. Do we have any information on how often this happens? How often is this just "I figured out Y for a different reason, and while doing my lit review I realized that the X paper is also relevant for Y"?