trentmkelly/LessWrong-43k huggingface dataset:
Kids Learn by Copying
by lsusr — 2021-09-05T23:09:06.704Z

How kids learn fascinates me. They don't do what you tell them to. Obeying orders is an abstract non-instinctual skill. Its simpler to just copy other people.

Plagiarism has advantages over obedience. First of all, it can be bootstrapped. Children are born without understanding language. Nobody can tell you how, in English, to speak English if you don't already understand English. More importantly, copying others is robust against deception. If everyone says you should obey the government while actually subverting it you should probably be a criminal too.

There is a teenage girl in my family who likes hanging out at my house while I work from home. She takes the initiative to do chores wherever she can be useful. This makes sense from an evolutionary perspective. She says she learns more per hour hanging out with me than going to school. (She complains high school doesn't teach how to do one's taxes. I should have her do my taxes.) It's more fun and useful too.

From what I know about primitive societies, they used little abstract verbal instruction. Most of the time spent learning to forage is spent foraging. Most of the time spent learning to cook is spent cooking. Elephants and chimpanzees spend no time on dialectic instructions at all.

What separates human pedagogy from other animals is our stories. Kids love stories, especially stories about people. Kids like stories about people so much the word "story" usually implies a story about people. I used to think the primary purpose of stories is to learn vicariously. But stories are often fantastical. I think the more important purpose is to provide examples of admirable behavior. It's no coincidence the most popular stories are all revolve around sympathetic capable hero protagonists.

Another fascinating attribute of kids is how immutable their preferences are. The kids in my family are all girls. They have little interest in how machines work[1]. They have no interest in weapons or dominance. They care about animals, medicine and relationships. The teenage girl wants to be a marketer so much she took the initiative to volunteer to market my business for free. (A job all three of our male co-founders loathed.) I learned marketing reluctantly after I accepted it was mandatory to achieving my other ambitions.

Your interests reflect your values. I frequently modify others' behavior by showing them how to manifest the...
Obstacles to gradient hacking
by leogao — 2021-09-05T22:42:22.876Z

[Epistemic status: brainstorming, still somewhat confused, with a bunch of loose threads left.]

This post is essentially the summary of a long discussion on the EleutherAI discord about trying to exhibit gradient hacking in real models by hand crafting an example. The discussion was sparked by this post. We didn't end up coming up with any good examples (or proofs of nonexistence) but hopefully this post is helpful for anyone else trying to construct gradient hacking examples.

Note that because our goal is to construct a concrete example of gradient hacking, when I write about "what we want'' and "unfortunate" roadblocks, those are from the perspective of a mesaoptimizer (or a researcher trying to construct an example of a mesaoptimizer to study), not from the perspective of a researcher attempting to build aligned AI.

Thanks to Adam Shimi, AI_WAIFU, Charles Foster, Evan Hubinger, Kyle and Laria, Stella Biderman, and alstroemeria313 for their contributions to the discussion.

Formulation

We mostly considered the following specific formulation since it seemed like a reasonably general framework.

There is some base model being trained, which contains an honest subnetwork h that attempts to solve for the base objective, a mesaobjective embedded somewhere in the model’s weights, and a malicious subnetwork (or mesaoptimizer, though I prefer the former term because it makes it clear that we're talking about the subnetwork in particular) that attempts to use the gradient to optimize itself for the mesaobjective.

In particular, we assume the mesaobjective looks a lot like a supervised learning objective, since it seems to be intuitively the most useful kind of mesaobjective. In other words, the objective looks like minimizing some loss over a dataset. To solve the problem of getting this training data to the malicious subnetwork, the mesaobjective is implicitly embedded in the form of a mesa loss function L and a pair of functions f and g that take training data given to the model from the base optimization process and creates training pairs to give to the malicious model. To be more concrete, the mesaloss is L(m(f(x)),g(x)) for any training data x. This loss is then used in an enforcement mechanism that perturbs the output from the honest model in some way before outputting it.

While it may initially s...
Cliffs Notes: How much should fully vaccinated people care about Long COVID?
by Sameerishere — 2021-09-05T22:01:54.538Z

Updates

12/29/21: I am not making anything remotely resembling an attempt to document a comprehensive update of this for Omicron, but capturing a few further updates here, mostly for my own reference; maybe you will find them helpful too. If you have further thoughts please chime in!

Relevant snippets from Zvi's 12/28/21 post, "Omicron: My Current Model":

Summary: "Long Covid risk small but real for all age groups, vaccination likely helps a lot."
Details: "Long Covid is real but rare and risk scales with severity. This is not something we can be confident in, and there are big unknowns to be sure, but my baseline continues to be that Long Covid risks are mostly proportional to short-term serious Covid risks aside from not scaling as much with age, and other things that reduce one reduce the other. Long Covid is still the biggest downside to getting Covid if one is young. I wish I could put a magnitude on this risk, but my best guess continues to be that this is not that much worse or different than e.g. Long Flu or Long Lyme, sometimes getting diseases does longer term damage than we realize and curing and preventing disease is therefore even more valuable than we think. But to extent you worry, Paxlovid probably does a lot to prevent this, so holding out until it is available would help you here."

Reasons to think the situation may be worse than Zvi's assessment above (I don't know if Zvi has incorporated these findings into his model or missed them... posted a comment here to see if he can chime in on that):

9/29/21 NPR article recapping a study from around that time

Affects over 1/3 of COVID sufferers, 2x as high as for flu:

"In the study published Tuesday in the journal PLOS Medicine, researchers found that about 36% of those studied still reported COVID-like symptoms three and six months after diagnosis. Most previous studies have estimated lingering post-COVID symptoms in 10% to 30% of patients.
...Although long COVID is poorly defined, the researchers looked at such symptoms as chest/throat pain, abnormal breathing, abdominal symptoms, fatigue, depression, headaches, cognitive dysfunction and muscle pain.
... the new study concludes that the chances of getting COVID-19 symptoms months after the acute stage of the illness was more than twice as high as for influenza."

Other points to consider:

"The Oxford-led team also found that people who had more severe

...
Are there any websites that collate useful psychological study results?
by Phoenix Eliot — 2021-09-05T21:45:44.032Z

I recently learned there was a study that found that BPD anger behavior is largely mediated by this chain: [shame → anger → anger rumination → BPD behaviors]. The takeaway for me is: notice if there is shame under my anger and try to attend to that if I can, and be mindful about anger rumination and try to do it less.

This is pretty useful, and it's especially noteworthy that the useful part of this is expressible in just 1-2 sentences.

Are there any sites that collate scientific results like this in concise lists of useful/actionable findings? Surely there are many such findings, but combing through research papers is very slow and likely to miss things if one is not seriously dedicated to being thorough, since many papers one finds by searching do not contain particularly actionable information for laypeople with the disorder being studied. And pop-sci articles are full of fluff text that often does not say much more than the 1-2 sentence summary.
Why there are no online CFAR workshops?
by Valentin2026 — 2021-09-05T15:02:54.192Z

As there are many people here here who attended CFAR workshops, I wanted to ask, if you understand why there are no online workshops, even when the in-person workshops were not possible due to the pandemic. In my experience, everything that does not require physical contact (like martial arts) can be taught online. There can be some inconveniences, of course, but if planned well, it can work quite smoothly. I can think about very small amount of reasons, and none of them seems persuasive to me.
1. Maybe CFAR workshop indeed requires physical contact? Sounds quite strange.
2. Maybe organizers are not good with technologies and don't know how to do different Zoom rooms, how to make parties in Mozilla Hubs etc.? Maybe they have never heard about properly organized online schools? Sounds even less likely.
3. Maybe CFAR workshop requires full embedding with no distraction? Well, it can be clarified in the announcement as a requirement for the participants, and for many people, it is certainly a possible thing to do.
4. Maybe CFAR somehow got into a death spiral and turned into something like Lifespring? I would hope it is unlikely in the rationalistic community, but here is more hope than actual belief.
Russian x-risks newsletter summer 2021
by avturchin — 2021-09-05T08:23:11.818Z

There were not many x-risk-related events in this period in Russia.

Covid mortality reached all times high in Russia, despite obvious lies about the number of deaths: they are almost always 799 a day. This is mostly due to the delta variant and failure to vaccinate most people. As a friend said: “My organisation is making up all numbers. Why I should believe the numbers from the other Russian organisation about vaccine efficiency?” However, citizen scientists showed that only the Sputnik vaccine actually works, and two other Russian vaccines, Covivak and Epivak, have near-zero efficiency. No foreign vaccines are allowed in Russia.

I have revaccinated with Astrazeneca in Montenegro and had some small peripheral neuropathy in the leg as a side effect. Revaccination with Sputnik is also open in Russia, but Russian vaccination certificates are not accepted in most countries.

Smoke from large forest fires in Yakutia region reached the North Pole. Russia has one of the largest forest reserves in the world and it was suggested decades ago that artificial fires in them could cause an effect similar to nuclear winter and be used as a climatic weapon.

As it typically happens, aeroplane accidents peaked in August, with 8 different aircraft crashed, all Russian made, but no large ones. This is due to the peak in tourism and tests as well as the peak of hot weather.
Should I treat pain differently if it’s “all in my head?”
by jp — 2021-09-05T08:06:59.962Z

I sometimes suffer from RSI. I’ve heard several people I respect say that they think most RSI is a psychological phenomenon. My understanding of their view is that they think there’s a feedback loop where you notice your wrists hurting and then you are anxious about the pain, causing you to pay more attention to it, which causes more pain though some mechanism we don’t yet understand. For the purposes of this post, I’m interested in the hypothetical of if something like that story is true.

I have some stories for what I’d do differently:

Ask a doctor to prescribe some psychiatric drugs, like SSRIs if I thought it was related to anxiety

And some things I’d do the same:

Continue to rest my wrists and do voice work when things get bad

But some things I’m uncertain about:

Do ergnomics matter?

I know several others who also suffer from RSI, and honestly I think the question is even more complicated there. Am I making my friends’ conditions worse if I talk about my condition?

(As a side note: I’m super glad that I can do voice work and give my wrists a rest. I honestly enjoy it, which is for sure not the typical experience. I am very grateful that I drew ”enjoys voice input” in the lottery of fascinations. While I admittedly haven’t seen many people get the benefit I have, if you’re reading this and would like my help getting set up using Talon Voice, send me a DM.)
Acausal Trade and the Ultimatum Game
by Yair Halberstadt — 2021-09-05T05:36:28.171Z

In the ultimatum game you have two participants. Let's call them Adam and Becky. You offer Adam 100 dollars and tell him he has to split it with Becky. Adam makes an offer to Becky, and Becky can either accept or reject the offer. If she rejects the offer they both get nothing, otherwise they split the cash in accordance with Adam's offer.

For example Adam might offer Becky $20. If Becky rejects that they both end up with 0. If Becky accepts it, Becky get's $20, and Adam gets $80.

What is the best strategy for both Becky and Adam?

Under causal decision theory the answer is simple. Whatever amount Adam offers Becky, Becky should accept, as the alternative is getting nothing. Therefore Adam should offer Becky a single cent, which Becky will accept, leaving Adam with $99.99, and Becky with 1 cent.

Surprise surprise, when you offer real humans this game they don't accept the single cent, and usually end up rejecting any offers below about 30%. In most cases Adam will offer Becky a 50/50 split, which she will accept.

Wikipedia has a large section dedicated to explaining why humans don't act like we would expect based on causal decision theory here. Perhaps humans are just irrational, or maybe they're being rational when you take into account other factors, like maintaining their image as a fair person.

For some reason nobody points out that what the humans are doing is absolutely 100% the correct thing to do! Our theoretical Homo Economicus who accepts 1 cent is clearly beaten by real life Becky who refuses to settle for less than 50/50, and so gets $50 every time! If your strategy loses, that's a problem with your decision theory not the world!

That's the main point of this post - I was shocked to see that nobody pointed that out in the article, so felt the need to shout it from the rooftops here.

The rest of this post will be dedicated to explaining how a rational agent might arrive at a 50/50 split.

Under causal decision theory, even though Becky should accept 1 cent when it's offered, if she can precommit to only accepting $99.99 up front, she should. For example, she could offer a friend $10,000 if she accepts any offer less than $99.99. Adam, knowing that Becky will be forced to reject any offer for under $99.99 will be forced to make such an offer, so that at least he gains 1 cent.

Of course Adam could counter by precommitting before Becky does to not make an offer ...
The Coordination Frontier: Sequence Intro
by Raemon — 2021-09-04T22:11:00.122Z

Sometimes, groups of humans disagree about what to do.

We also sometimes disagree about how to decide what to do.

Sometimes we even disagree about how to decide how to decide.

Among the philosophically unsophisticated, there is a sad, frustrating way this can play out: People resolve "how to decide" with yelling, or bloodshed, or, (if you’re lucky), charismatic leaders assembling coalitions. This can leave lots of value on the table, or actively destroy value.

Among the extremely philosophically sophisticated, there are different sad, frustrating ways this can play out: People have very well thought out principles informing their sense of "how to coordinate well." But, their principles are not the same, and they don’t have good meta-principles on when/how to compromise. They spend hours (or

years

) arguing about how to decide. Or they burn a lot of energy in conflict. Or they end up walking away from what could have been a good deal, if only people were a bit better at communicating.

I’ve gone through multiple iterations on this sequence intro, some optimistic, some pessimistic.

Optimistic takes include: “I think rationalists are in a rare position to actually figure out good coordination meta-principles, because we are smart, and care, and are in positions where good coordination actually matters. This is exciting, because coordination is basically the most important thing [citation needed]. Anyone with a shot at pushing humanity’s coordination theory and capacity forward should do that.”

Pessimistic takes include: “Geez louise, rationalists are all philosophical contrarians with weird, extreme, self-architected psychology who are a pain to work with”, as well as “Actually, the most important facets of coordination to improve are maybe more like ‘slightly better markets’ than like ‘figuring out how to help oddly specific rationalists get along’.”

I started writing this post several years ago because I was annoyed at, like, 6 particular people, many of them smarter and more competent than me, many of whom were explicitly interested in coordination theory, who nonetheless seemed to despair at coordinating with rationalists-in-particular (including each other). The post grew into a sequence. The sequence grew into a sprawling research project. My goal was “provide a good foundation to get rationalists through the Valley of Bad Coordination”. I f...
Quantum particles and classical filaments
by From_Branes_to_Brains — 2021-09-04T21:45:42.807Z

The first post in the "From Branes to Brains" series. For the introduction to the series see this post.

In this post, I will mostly follow the ideas from the classical book by Polyakov, as well as from this paper by Kleinert.

First of all, consider a classical particle with mass m in the one-dimensional potential V(x,t). If we know, that at the moment t1 the particle was in the point x1, and in the moment t2 the particle was in the point x2, the particle should have travelled between these two points along the path that minimizes the action

S[x(t),x1,x2,t1,t2]=∫t2t1dt(m2(dx(t)dt)2−V(x(t),t))

This is known as the minimal action principle. In quantum mechanics, instead, the particle travels along all the possible trajectories, according to the Feynman principle. Namely, the amplitude of the probability of the transition from point x1 at the moment t1 to the point x2 at the moment t2 is given by:

⟨x1,t1|x2,t2⟩=∫x(t1)=x1,x(t2)=x2Dx(t)exp(iℏS[x(t),x1,x2,t1,t2])

The Feynman-Hibbs book is quite a good introduction to this topic. Now let us consider a system, that seems to be totally different: classical inextensible flexible filament (like a guitar string or long polymer molecule) in the two-dimensional potential V(x,y) under tension F. The string is pinned in points x1,y1 and x2,y2, and there are length reservoirs at this point. To visualize, you can think about string on the table that passes through the holes in the table in points x1,y1 and x2,y2, and there is tension F applied to the ends. The energy of such string is total potential energy plus τL, where L is the length of the string on the table.
If the angle of the string tangential with Y axis is everywhere small, the length is approximately given by the formula

L=∫y2y1dy[1+12(dx(y)dy)2]

where x(y) controls the shape of the filament.
The total energy of the string is then

E[x(y),x1,x2,y1,y2]=∫y2y1dy[F2(dx(y)dy)2+V(x(y),y)]

where we ignored the term 1 since it just gives a constant contribution to the energy, without changing anything.
And the partition function is then

Z=∫x(y1)=x1,x(y2)=x2Dx(y)exp[−1TE[x(y),x1,x2,y1,y2]]

where each term in the sum corresponds to the pro...
Are there substantial research efforts towards aligning narrow AIs?
by Rossin — 2021-09-04T18:40:18.181Z

I am certainly not an expert, but what I have read in the field of AI alignment is almost solely targeted at handling the question of how to make sure that a general AI does not start optimizing for outcomes that humans do not want to happen.

This is, to me, an obviously important problem, in that it seems like we only get one chance at it and if we fail we are all Doomed.

However, it also seems like a very difficult problem to get good feedback on since there are no AGIs to test different strategies on, which can make estimating the promise of different avenues of research very difficult.

A related problem in which real world feedback can be more readily attained might be alignment of narrow AIs. It seems to me that existing AIs already demonstrate properties that appear to be misaligned with the intentions of their programmers. For example, websites such as Facebook and YouTube employ algorithms to personalize the recommended content a user sees to keep them engaged and thus generating the companies more ad revenue. On the face of it, there seems to be nothing wrong with optimizing to show people what they want to see. Unfortunately, a very effective strategy for accomplishing this goal in many cases appears to be showing people extremely misleading or fabricated stories about reality which both support what the person already believes and creates a large emotional response. In this way these platforms support the spread of misinformation and political polarization, which was almost certainly not the intent of the programmers who probably imagined themselves to be creating a service that would be uniquely tailored to be enjoyable for everyone.

This problem does not seem to be something that can be simply patched, much in the way deficient friendly AI designs do not seem patchable. It seems, at least when I try to consider a possible solution, impossible, though not as impossible an impossibility as aligned AGI. The key immediate problems in my mind are “how do you decide what counts as misinformation in a relatively unbiased manner?” and “if you don’t fulfill people’s desire for confirmation of their poorly supported beliefs, how do you make your platform as attractive as a competitor who will fill that confirmation bias itch?”

Tackling the challenge of “build an AI system that recommends content to users without spreading misinformation” or somethi...