trentmkelly/LessWrong-43k huggingface dataset:
AMA: I was unschooled for most of my childhood, then voluntarily chose to leave to go to a large public high school.
by iceplant — 2022-01-07T07:48:27.335Z

I've seen a lot of discussion in the LW community about alternatives to traditional school for kids. Given how few kids actually get to experience the alternatives (<4% of kids in the US are homeschooled, <10% of those are unschooled), I think the success and failure modes are less well understood than for traditional schools. I want to offer myself as a datapoint to anyone interested in my subjective experience. Eventually, I want this to be a standalone blog post, but my thoughts still feel pretty disparate, so I'm hoping this will help me find a more cohesive narrative.

About me: I'm now in my mid 20s, went to a "good college" and now have a "good job" in tech (despite a brief gap derping around as an artist...what can I say I was unschooled). I'm not a teacher. I don't have kids. I do feel that unschooling had a big impact on my life, and I'm hoping this will help me understand it better, and how it can empirically affect others.
Signaling isn't about signaling, it's about Goodhart
by Valentine — 2022-01-06T18:49:48.534Z

Epistemic status: Fuzzy conjecture in a faintly mathematically flavored way. Clear intuitions about Gears and a conclusion, but nothing like a formal proof or even formal definitions. Anecdotes offered to clarify the intuition rather than as an attempt at data. Plenty of room for development and increased rigor if so desired.

Suppose that for whatever reason, you want to convince someone (let's call them "Bob") that they can trust you.

I'd like to sketch two different strategy types for doing this:

You can try to figure out how Bob reads trust signals. Maybe you recognize that Bob is more likely to trust someone who brings a bottle of his favorite wine to the meeting because it signals thoughtfulness and attention. Maybe revealing something vulnerably helps Bob to relax. You're not really trying to deceive Bob per se here, but you recognize that in order for him to trust you you need to put some energy into showing him that he can trust you.
You make a point within yourself to be in fact worthy of Bob's trust. Then, without knowing how Bob will take it, you drop all attempts to signal anything about your trustworthiness or lack thereof. Instead you just let Bob come to whatever conclusion he's going to come to.

That second strategy might sound nuts.

Despite that, I claim it's actually almost strictly more effective.

If you see why, you probably have the bulk of my point.

I'll say a few more things to spell this out, together with some Gears I see and some implications.

A rephrasing of Goodhart's Law goes something like this:

The more explicit attention a signal gets, the more pressure there is to decouple it from what it's a signal of.

The mechanism is basically analogous to wireheading. If you get a reward for a signal happening, you're incentivized to find the cheapest way to make that signal happen.

Like when someone's trying to lose weight, so they make a point of weighing themselves first thing in the morning before drinking water and after using the toilet.

This might accidentally create some kind of standard baseline, but that isn't what's motivating the person to do this. They're trying to make the scale's numbers be lower.

Even weirder is when they stop drinking as much water because the scales reward them for that.

An often missed corollary of Goodhart — and basically the center of what I want to point at here — is this:

If you want a signal to retain its integrit

...
Covid 1/6/22: The Blip
by Zvi — 2022-01-06T18:30:00.964Z

Cases are hitting record highs around the world. That’s going to continue for another few weeks. While it does, and for a bit after it stops, there are disruptions all around as too many people are sick at one time. Thus, the CDC guideline adjustments, the hospitals under pressure, many schools forced to not meet in person (and then deciding to torture kids with ‘remote learning’ rather than giving them a vacation.)

No question it’s annoying, and for those in health care it’s overwhelming and terrible. I don’t want to minimize it. And if you care about avoiding this it’s going to be a lonely few months.

But compared to March 2020, this is nothing. We got this.

There are dissenting voices, but mostly the consensus seems to be growing that Omicron is milder, the center will hold, and our lives beckon.

Executive Summary

Record high case numbers likely to continue for a few weeks.
Hospitals under pressure but holding, deaths strangely low.
If you get sick you should still do your best to get a negative test before ending your isolation no matter what the CDC says.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 3.5mm cases (+94%) and 10,800 deaths (+25%).

Results: 3.57mm cases (+96%) and 8,814 deaths (+2%).

Prediction for next week: 6mm cases (+71%) and 9,700 deaths (+10%).

On cases that’s another random bullseye. On deaths it’s a big miss, and given how little of the past cases were Omicron I don’t understand how the number came in this low, still below the level 2-3 weeks ago despite much higher case counts.

This is the dog that did not bark. The number damn well should have gone up a bunch, and it didn’t, and it makes me think that Omicron took over faster than I previously thought. I can’t think of another way to explain it, and the fact that I’m using that to explain it seems like very good news going forward.

I do expect deaths to start climbing, how could they not, but we keep not seeing big jumps there, so I’m going to downshift my expectations there even as lagged cases start rising faster.

Deaths

Report that deaths among young people are up 40% from pre-pandemic levels, in the context of life insurance. This sounds terrible and is terrible, but in terms of how one should live one’s life young people are still very unlikely to die and shouldn’t take minimizing death risk as a major life task except whe...
The Problem of the Criterion is NOT an Open Problem
by Gordon Seidoh Worley — 2022-01-06T16:31:09.183Z

Not long ago, I added a tag to LessWrong for the problem of the criterion. Shortly after that its text received an edit claiming it is an open problem. However, I think the problem of the criterion is not really an open problem. Here's why.

Here on LessWrong we say open problems are "the things in a field that haven't yet been figured out". On Wikipedia, an open problem is described as "a known problem which can be accurately stated, and which is assumed to have an objective and verifiable solution, but which has not yet been solved (i.e., no solution for it is known)." By either of these standards, does the problem of the criterion qualify as an open problem?

If our standard is "has it been figured out?", I'd say yes. I think not everyone likes what was figured out, which is a reason to make a bid to claim the problem is open, but not liking the answer should be insufficient reason to make such a claim. After all, I don't exactly like that the solution to the halting problem is that we can't create a general program that can determine if an arbitrary program halts, but so be it, that's the universe we find ourselves in, let's move on. So I think it is with the problem of the criterion: yeah, kinda sucks the kind of truth we can achieve when we restrict ourselves to mathematical thinking can't be achieved everywhere, but that's how the world is, so I guess we'll figure out how to live with it because we already are.

What about if we use Wikipedia's definition of an open problem? The problem of the criterion is certainly "a known problem which can be accurately stated". Is it "assumed to have an objective and verifiable solution"? The whole point of my and other's analysis of the problem is to show that this assumption is unfounded in some way and that no objective and verifiable solution can be found because the question is fundamentally flawed by asking for something impossible. In this way it seems similar to the "complete and consistent problem" or the "momentum and position problem": Gödel's incompleteness theorems and Heisenberg's uncertainty principle respectively show these are problems that don't have "objective and verifiable solutions". In this same way, I and others claim the problem of the criterion is "solved" because we've shown that solving it is impossible.

This is small issue, so why have I bothered to take the time to talk about it? Because I t...
Importance of foresight evaluations within ELK
by Jonathan Uesato — 2022-01-06T15:34:00.807Z

This post makes a few basic observations regarding the importance of foresight-based evaluations within ELK [

Christiano et al 2021

]:

Without foresight-based evaluations, (narrow) ELK is insufficient to avoid subtle manipulation (we’d require a more ambitious version)
Hindsight creep can occur even with seemingly foresight-based evaluations. One way this could happen is if human evaluators primarily defer to evaluations made by future humans in the predicted outcomes.
Because of both points, even if we are counting on a technique like narrow ELK, improving the quality of foresight-based evaluations is very valuable.

The points here are largely covered in ELK appendices on

narrow elicitation

,

defining a utility function

, and

subtle manipulation

. The aim here is to call out some of these points explicitly, and facilitate further focused discussion.

Subtle manipulation

The ELK report focuses on what it calls narrow ELK, which does not aim to address subtle manipulation via ELK:

[In narrow ELK,] We won’t ask our AI to tell us anything at all about subtle manipulation.
…
A mundane [example of subtle manipulation] is that they may have talked with someone who cleverly manipulates them into a position that they wouldn’t really endorse, or emotionally manipulated them in a way that will change their future conclusions.

Subtle manipulation is clearly problematic. Within the overall RL training approach of the ELK report, foresight evaluations are necessary to avoid subtle manipulation. That is, even if we solve narrow ELK, we still need foresight-based evaluations to avoid subtle manipulation.

With hindsight-based policy evaluations, narrow ELK is insufficient

The ELK report provides a simple policy training algorithm based on approval-in-foresight combined with hindsight-based outcomes predictors:

Still, even though humans can’t directly follow along with the actions, they can evaluate the predicted consequence of an action sequence:
[image]
We can then train a model to predict these human evaluations, and search for actions that lead to predicted futures that look good.

(See also: Appendix: Policy training pseudocode.)

This uses a hybrid of foresight and hindsight feedback.[1] The outcomes predictor is trained in hindsight: trying actions, observing what outcomes result, and training a model to predict those outcomes. Conditioned on these predictions, the policy fee...
How reasonable are concerns about asking for patents to be lifted for COVID-19 vaccines?
by ChristianKl — 2022-01-06T13:09:07.414Z

I have intuitions that lifting patents for the approved COVID-19 vaccines would do little as the technology is a higher barrier then just licensing the patents. It feels to me like calls for opening up patents mostly come out of political reflexes without any underlying analysis of the impacts. I'm however open to updating to a more informed view if someone looked into the issue deeper. Has anyone a good grasp on the issue?
A common misconception about the evolution of viruses
by mukashi — 2022-01-06T10:00:36.706Z

This might be obvious for most people here but I haven't found yet a good blog post that I can point out to people repeating something that I have been hearing from the beginning of the pandemic many times, and is false.

Viruses have a natural tendency to evolve towards less deadly forms.

Deadliness is mostly orthogonal to reproducibility. Viruses do not have a tendency to become less deadly unless becoming less deadly makes them also more capable to leave copies of themselves.

In what cases would we see the virus evolving towards less dangerous? In cases where the virus is so deadly that as soon as you are infected, the virus kills you before you had the chance of disseminating copies of the virus to other hosts. In this case, being too deadly is clearly disadvantageous to the virus, so the strains that cause instant death disappear because they cannot propagate to others. It can also happen that a new variant of the virus simply happens to be less deadly and more contagious (which might be the case with the omicron variant), but this is not part of any magical trend where viruses mysteriously evolve until they become the common cold (has HIV evolved towards a less severe disease?). The opposite can happen too. A new strain might appear that happens to be also more contagious and more deadly.
What I'm about
by Flaglandbase — 2022-01-06T09:20:47.444Z

Death is horrible, or at least the things that happen as it approaches. I've worked in nursing homes and can't understand what normal people will tolerate. Every day is filled with undiscussed horrors. It's unacceptable, too dreadful to face.
Clearly, the mainstream has unimaginably different opinions than me. But still, there should be SOME effort to do something about death, like a scientific religion or social movement.
I'm certainly not afraid of death because my life is so good, but more afraid of losing something I've never had. I have a highly specific vision of a virtual reality heaven. Basically, I would be left alone for all eternity on my personal island designed in such a way nothing could go wrong. After countless eons I would begin to relive days I lived before.
The first step to achieve such a thing would be a device to accurately scan the contents of a human brain. That data would then be used to recreate the brain's software "continuation".
However, this process would require science fiction-like devices even more complicated than human brains. No one has managed to scan the connections between even a few neurons. I suspect Kurzweil was too optimistic about the rate of technological progress, and we will all die before that technology is invented.
Those of us who fear death enough, will have to invent a way to preserve our mind contents using today's highly inadequate technology. The best hope is obviously cryonics. Let's freeze our brains until they can be scanned in the future, ideally before they are decayed by old age or other horrible conditions.
Unfortunately, I have no money to afford cryonics. There's just no way, unless the government makes it available for free like emergency healthcare. There is no political will for that - but why doesn't every millionaire have their head frozen after death?
There may be one other alternative.
It started for me in the early 1980s with a vague interest on how to record fleeting memory states in an effort to hold on to the past. This eventually led to research on Rudy Rucker's Lifebox idea (https://www.rudyrucker.com/pdf/rucker_marvell_lifebox_immortality.pdf). Basically, his method involves recording everything you have done and learned in as much detail as possible.
I have scanned about 40,000 documents sorted by approximate date, plus under a million digital files. I did it all myself, but co...
You can't understand human agency without understanding amoeba agency
by Shmi — 2022-01-06T04:42:51.887Z

This is supposed to be a trivially true statement, and yet it sounds controversial somehow, doesn't it?

The difference between the two is quantitative, not qualitative, unless you explicitly or implicitly subscribe to the "humans are the only real agents" idea, which is not even one step removed from "unlike other animals, humans possess souls". And yet when I browse through research on (embedded) agency, most of it is about humans, not anything less complicated. Sure, studying humans has an apparent advantage over studying other animals because we, as humans, have an inside view not readily accessible in other species. Whether it is a real advantage depends on how misleading the inside view is. After all, it comes with a dangling node of free will, with the feeling of qualia and ineffable redness of red, and with a host of other evolutionary artifacts that once may have been useful for survival in a small tribe, but now are a just along for the ride. And in any case, evolution satisfices for survival, it does not optimize the inside view for accuracy.

I understand why philosophy focuses almost exclusively on humans, but I don't understand why, say, AI research does not focus on simple creatures first, and maybe even on non-living agents before that. Well, that's not entirely true, posts like this one do appear:

https://www.lesswrong.com/posts/3SG4WbNPoP8fsuZgs/agency-in-conway-s-game-of-life

but they seem to be a small minority, as far as I can tell. What am I missing here?
Brain Efficiency: Much More than You Wanted to Know
by jacob_cannell — 2022-01-06T03:38:00.320Z

What if the brain is highly efficient? To be more specific, there are several interconnected key measures of efficiency for physical learning machines:

energy efficiency in ops/J

spatial efficiency in ops/mm^2 or ops/mm^3

speed efficiency in time/delay for key learned tasks

circuit/compute efficiency in size and steps for key low level algorithmic tasks [1]

learning/data efficiency in samples/observations/bits required to achieve a level of circuit efficiency, or per unit thereof

software efficiency in suitability of learned algorithms to important tasks, is not directly addressed in this article[2]

Why should we care?
Brain efficiency matters a great deal for AGI timelines and takeoff speeds, as AGI is implicitly/explicitly defined in terms of brain parity. If the brain is about 6 OOM away from the practical physical limits of energy efficiency, then roughly speaking we should expect about 6 OOM of further Moore's Law hardware improvement past the point of brain parity: perhaps two decades of progress at current rates, which could be compressed into a much shorter time period by an intelligence explosion - a hard takeoff.

But if the brain is already near said practical physical limits, then merely achieving brain parity in AGI at all will already require using up most of the optimizational slack, leaving not much left for a hard takeoff - thus a slower takeoff.

In worlds where brains are efficient, AGI is first feasible only near the end of Moore's Law (for non-exotic, irreversible computers), whereas in worlds where brains are highly inefficient, AGI's arrival is more decorrelated, but would probably come well before any Moore's Law slowdown.

In worlds where brains are ultra-efficient, AGI necessarily becomes neuromorphic or brain-like, as brains are then simply what economically efficient intelligence looks like in practice, as constrained by physics. This has important implications for AI-safety: it predicts/postdicts the success of AI approaches based on brain reverse engineering (such as DL) and the failure of non-brain like approaches, it predicts that AGI will consume compute & data in predictable brain like ways, and it suggests that AGI will be far more like human simulations/emulations than you'd otherwise expect and will require training/education/raising vaguely like humans, and thus that neuroscience and psychology are perhaps more useful fo...
Satisf-AI: A Route to Reducing Risks From AI
by harsimony — 2022-01-06T02:34:46.033Z

(Ok so I admit the word “Satisf-AI” is weird, I was intending for it to be a play off of “Satisfy” and “AI” and for it to be pronounced like “Satisfy”)

I think Holden Karenofsky’s PASTA is a good way to frame the discussion of transformative AI. It resonates with something I have been turning over for a while now. The AI safety community is typically focused on the question “how do we build AI that is safe?”. But I think a better question is “how do we get what we want from AI in a safe way?”.

In other words, rather than focus on building a superintelligence which is safe, can we find a way to get the benefits of a superintelligence without actually building one?

To make this more concrete, lets consider the major benefits we expect from an AI:

Innovation: We want AI’s to generate new inventions and insights that we can use.

Complete Tasks: We want AI’s to perform specific jobs at scale which are too menial, dangerous, or complex for people to do.

Make Decisions: We want AI’s to advise our decisions and to make decisions on our behalf.

Right now there is a strong push to create AI that can meet all of these goals. People worry that the competitive pressures associated with this effort might create a harmful race-to-the-bottom on AI safety.

But what if we could remove the draw of AI? What if we could find a way to get everything we want on the list without having to use superintelligence?

This would significantly reduce the competitive pressures that might lead to unsafe AI and it would give AI safety researchers a crucial resource: time.

But how do we get the benefits of AI, without AI?

I think many of the hopes we have for AI are achievable with much safer approaches. To be clear, these approaches might still involve modern-day machine learning, but would not constitute a general intelligence. These more narrowly focused artificial intelligence systems could satisfy our desire for a superintelligence, hence the name “satisf-AI”.

This “narrow AI” idea isn’t new. Drexler outlines something similar in his CAIS model while Gwern warns against a false sense of security that these Tool AI’s might bring. I will return to this last concern in a later section.

Satisfactory AI in Each Domain

Let’s consider what it would take to satisfy our needs in each domain without resorting to superintelligence.

Innovation: One could imagine automating much of science, performing experi...
How To Raise Others’ Aspirations in 17 Easy Steps
by chanamessinger — 2022-01-06T00:14:15.651Z

Title taken from Tyler Cowen: The high-return activity of raising others’ aspirations

This is taken from a twitter thread (which I also wrote up on my website) that I was - aptly for the content - encouraged to put in non-Twitter places. Thanks for the encouragement!

Things you can say to people in ten seconds that sometimes produce insanely outsized effects:

Yeah, someone *should* do that. Why not you?
Is there something you could do about that problem in the next five minutes?
That's a great thought - have you written it up somewhere? I'd be excited to share it if so.
Should you write a book / blog?
You want to do that but don't think you will? Do you want to make a concrete plan now?
Do you want me to ask in a week / month if you've done that or how it's going?
Feeling stuck sucks. Want to brainstorm together?
Feeling stuck sucks. Have you spent a five minute timer generating options?
What's the twenty minute / minimum viable product version of this overwhelming-feeling thing?
Do you want me to sit with you while you fill out that application/write that email? Too bad, it's happening.
Is it worth just asking / cold emailing and seeing?
Is there anyone I know who you'd like to be introduced to?
Do you want to just set an alarm on your phone now as a reminder? (from Damon Sasi)
Do you know anyone else who might have struggled with or succeeded at that? Have you talked to them about it? (from Damon Sasi)
Who do you know who you could ask for help from?
You are Allowed (to try, to do, to build, to ask). I strongly recommend this link of things you are allowed to do.

(Or do you not need permission in the first place?)

Do you sort of already know what you're going to do / have your mind made up about this?