trentmkelly/LessWrong-43k huggingface dataset:
Fixing science via a basic income
by Kaj_Sotala — 2018-08-02T08:30:04.380Z

I ran across Ed Hagen’s article “Academic success is either a crapshoot or a scam”, which pointed out that all the methodological discussion about science’s replication crisis is kinda missing the point: yes, all of the methodological stuff like p-hacking is something that would be valuable to fix, but the real problem is in the incentives created by the crazy publish-or-perish culture:

In my field of anthropology, the minimum acceptable number of pubs per year for a researcher with aspirations for tenure and promotion is about three. This means that, each year, I must discover three important new things about the world. […]

Let’s say I choose to run 3 studies that each has a 50% chance of getting a sexy result. If I run 3 great studies, mother nature will reward me with 3 sexy results only 12.5% of the time. I would have to run 9 studies to have about a 90% chance that at least 3 would be sexy enough to publish in a prestigious journal.

I do not have the time or money to run 9 new studies every year.

I could instead choose to investigate phenomena that are more likely to yield strong positive results. If I choose to investigate phenomena that are 75% likely to yield such results, for instance, I would only have to run about 5 studies (still too many) for mother nature to usually grace me with at least 3 positive results. But then I run the risk that these results will seem obvious, and not sexy enough to publish in prestigious journals.

To put things in deliberately provocative terms, empirical social scientists with lots of pubs in prestigious journals are either very lucky, or they are p-hacking.

I don’t really blame the p-hackers. By tying academic success to high-profile publications, which, in turn, require sexy results, we academic researchers have put our fates in the hands of a fickle mother nature. Academic success is therefore either a crapshoot or, since few of us are willing to subject the success or failure of our careers to the roll of the dice, a scam.

The article then suggests that the solution would be to have better standards for research, and also blames prestigious journal publishers for exploiting their monopoly on the field. I think that looking at the researcher incentives is indeed the correct thing to do here, but I’m not sure the article goes deep enough with it. Mainly, it doesn’t ask the o...
Emotional Training Model
by Elo — 2018-08-01T06:41:09.832Z

Bearlamp: Previous, First

Greaterwrong: Previous, First

Lesswrong: Previous, First

Life is propagated by two main clusters of emotions. (*Yes I know it's more like a spectrum but this is the poor simplification I am using for now)

The "good" feelings that we move towards, and the "bad" feelings that we move away from. (then there's the neutral ones we hang around in sometimes but that's for another time).

If you spend your life always running from the bad ones and always running towards the good ones, you may have a good life. You may have a life that just gets lucky and has more good than bad. Alternatively you may have hard things to do that involve feeling uncomfortable for short or long periods of time. If you are forever running away from the bad emotions, and forever addicted to running towards the good emotions, you are severely limited in your agency compared to if you have even a little bit of freedom to do something like, "avoid short term rewards", or "put up with scary moments" on the way to other experiences. (see also The Trauma model of mental health)

The carnival ride example

Charlie the five year old goes on a carnival ride. Charlie the five year old throws up. Charlie the five year old learns that carnival rides make you feel terrible.

In (one of many) healthy worlds: When charlie turns ten, his friends ask him to go to the carnival again. Charlie realises that the carnival ride might be different now, he fuels himself with a bit of peer pressure and he runs a new experiment, pushing back on the terrible feeling that he would usually avoid and rewrites his inclination to avoid terrible things. Consequently charlie relearns that a carnival ride is only sometimes terrible and with the support of friends it can be good.

In (one of many) unhealthy worlds: When charlie turns ten, his friends ask him to go to the carnival again. Charlie remembers that the carnival rides feel terrible and decides not to go to the carnival. This reinforces the terrible feeling. Charlie feels entirely justified in avoiding a terrible thing, his friends don't really care either way and life goes on. Charlie keeping a tiny reinforced experience that he should avoid terrible things.

Objectively speaking, a carnival ride is not terrible or good. Subjectively, the feelings we attach to such experiences are what ...
Open Thread August 2018
by Elo — 2018-08-01T01:49:18.598Z

If it’s worth saying, but not worth its own post, then it goes here.

Notes for future OT posters:

Check if there is an active Open Thread before posting a new one (use search for Open Thread ).
What accomplishments are you celebrating from the last month?
What are you reading?
What reflections do you have for yourself or others from the last month?
Sandboxing by Physical Simulation?
by moridinamael — 2018-08-01T00:36:32.374Z

This is a simple idea. I do not remember if I've seen it anywhere. It is probably not original, but I am mildly surprised that I haven't come across it, even if only to see it refuted. If this is an old/dumb idea, please let me know and I'll delete it.

People have built universal Turing machines in Minecraft. It is straightforward to build a virtualized Turing machine by simulating a physical system which carries out the mechanical actions needed to instantiate one. You could obviously build a computer in a simulated physics simpler than Minecraft, but the Minecraft example is a more vivid one.

I don't even want to guess how much more computationally expensive it would be to run an AI, and the AI's simulated environment, on a Turing machine being run on a simulated physical computer, itself being simulated on mundane hardware. But it does strike me that an AI should have significantly more trouble hacking its way out of this kind of sandboxing.

It would likely have more trouble noticing that it's in a simulation in the first place, and if it did notice, it would likely have a much harder time doing anything about it. Our hardware relies on physical laws that we don't full understand. It makes errors that we sometimes can't really notice or stop, because fundamentally it's made of atoms and fields. One can imagine ways in which a clever being might intentionally strain physical hardware to see how it reacts. We already know that an attacker can induce a bit flip in neighboring memory through physics trickery. In contrast, the underlying simulated physics of the AI world/brain could be extremely simple and designed to be as free of surprises as possible. Within the doubly-simulated world, and within its own simulated brain, there would be no stray EM fields, no Internet ports accidentally left open. The AI could bang on the strata of its physics all it wanted, and all "we" would see would be the flickering light of the Redstone machine.

I'm not one to underestimate a superintelligence, but the baseline security of this kind of double-Sandboxing feels qualitatively different than that of physical hardware.
Safely and usefully spectating on AIs optimizing over toy worlds
by AlexMennen — 2018-07-31T18:30:38.392Z

Consider an AI that is trying to achieve a certain result in a toy world running on a computer. Compare two models of what the AI is and what it's trying to do: first, you could say the AI is a physical program on a computer, which is trying to cause the physical computer that the toy world is running on to enter a certain state. Alternatively, you could say that the AI is an abstract computational process which is trying achieve certain results in another abstract computational process (the toy world) that it is interfacing with.

On the first view, if the AI is clever enough, it might figure out how to manipulate the outside world, by, for instance, hacking into other computers to gain more computing power. On the second view, the outside world is irrelevant to the AI's interests, since changing what's running on certain physical computers in the real world would have no effect on the idealized computational model that the AI is optimizing over, so the AI has no incentive to optimize over our world.

AIs for which the second model is more accurate seem generally safer than AIs for which the first model is more accurate. So trying to encourage AI development to follow the second model could help delay the development of dangerous AGI.

AIs following this model are limited in some ways. For instance, they could not be used to figure out how to prevent the development of other dangerous AGI, since this requires reasoning about what happens in the real world.

But such AIs could still be quite useful for many things, such as engineering. In order to use AIs optimizing over toy worlds to design things that are useful in the real world, we could make the toy world have physics and materials similar enough to our world that designs that work well in the toy world should be expected to also work well in the real world. We then take the designs the AI builds in the toy world, and replicate them in the real world. If they don't work in the real world, then we try to find the discrepancy between real-world physics and toy-world physics that accounts for it, fix the discrepancy, and try again.

One possible thing that could go catastrophically wrong with this strategy is if the design the AI comes up with has an agent in it. If the AI designs this agent to figure out what sort of world it's in, rather than hard-coding the agent to believe it’s in the to...
Counterfactuals, thick and thin
by Nisan — 2018-07-31T15:43:59.187Z

Summary: There's a "thin" concept of counterfactual that's easy to formalize and a "thick" concept that's harder to formalize.

Suppose you're trying to guess the outcome of a coinflip. You guess heads, and the coin lands tails. Now you can ask how the coin would have landed if you had guessed tails. The obvious answer is that it would still have landed tails. One way to think about this is that we have two variables, your guess A and the coin C, that are independent in some sense; so we can counterfactually vary A while keeping C constant.

But consider the variable X=A XOR C. If we change A to tails and keep X the same, we conclude that if we had guessed tails, the coin would have landed heads!

Now this is clearly silly. In real life, we have a causal model of the world that tells us that the first counterfactual is correct. But we don't have anything like that for logical uncertainty; the best we have is logical induction, which just give us a joint distribution. Given a joint distribution over A×C, there's no reason to prefer holding C constant rather than holding A XOR C constant. I want a thin concept of counterfactuals that includes both choices. Here are a few definitions, in increasing generality:

1. Given independent discrete random variables A and C, such that C is uniform, a thin counterfactual is a choice of permutation ϕ(a) of C for every a∈A.

2. Given a joint distribution over A and Y, a thin counterfactual is a random variable Z independent of A and an isomorphism of probability spaces A×Z≈A×Y that commutes with the projection to A.

3. Given a probability space A and a probability kernel κ:A→Y, a thin counterfactual is a probability space Z and a kernel λ:A×Z→Y such that ∫Zλdz=κ.

There are often multiple choices of thin counterfactual. When we say that one of the thin counterfactuals is more natural or better than the others, we are using a thick concept of counterfactuals. Pearl's concept of counterfactuals is a thick one. No one has yet formalized a thick concept of counterfactuals in the setting of logical uncertainty.
Alignment Newsletter #17
by Rohin Shah — 2018-07-30T16:10:02.008Z

Highlights

Differentiable Image Parameterizations

(Alexander Mordvintsev et al): There are lots of techniques for generating images using neural nets. A common approach is to take a neural net trained to classify images, and then use gradient descent to optimize the input image instead of the weights of the neural net. You might think that the only way to affect the generated input image would be to change the loss function on which you run gradient descent, but in reality the way in which you represent the image makes a huge difference. They describe why this might be the case, and go through several examples:

1. Suppose you want to see how two neurons interact. You could optimize an image to maximize the sum of the activations of the neurons. Even better, you could create an animation of how the image changes as you trade off how much you care about each neuron. Done naively, this doesn't look good, because there's a lot of randomness that changes between each image in the animation, which swamps out the differences we actually care about. To fix this, we can generate each frame in the animation as the sum of two images, one shared across all frames, and one that is frame-specific. Despite changing neither the loss function nor the space of input images, this is sufficient to remove the randomness between frames.

2. You've probably seen

style transfer

before, but did you know it only works with the VGG architecture? We can get it to work with other architectures by representing images in Fourier-space instead of pixel-space, again without any change in the loss function or expressible space of images.

3. If you generate the pixel-space representation of an image from a lower-dimensional representation using a Compositional Pattern Producing Network (CPPN), then gradient descent will optimize the lower-dimensional representation. It turns out that this produces images vaguely reminiscent of light-paintings. (I believe in this case, while the loss function doesn't change, the space of expressible images does change.)

4. Often when we see the feature visualization for a neuron, there are a lot of areas of the image that don't actually matter for the neuron's activation. So, we can add transparency, and add a term in the loss function that encourages transparency. We also have to change the representation of the image to include a tran...
The Feedback Problem
by Elo — 2018-07-29T23:54:13.059Z

My Blog Bearlamp: First, Previous

Lesswrong: First, Previous -- Greaterwrong: First, Previous

“Let me practice my free throw from different distances so that I can throw well wherever I am in the game”. Anders Ericsson (10,000 hours guy) in Peak, talks about deliberate practice. He also talks about the difficulty in getting feedback.

Feedback is the hardest part of learning anything. Good feedback and you can go from chopsticks to beethoven in simple steps. Technically speaking, the “hard part” of the skill is not the part which requires you to physically press the buttons on a piano. Or the part that requires you to work out how to move the piece in tetris to where you want it to go. The part of the chess game that is moving the piece to the next location on the board. The part of poker that is reading the cards and knowing which ones you have.

Most games have a basic level of skill that isn’t that hard. Anyone can play tennis provided they can hold a racket and swing.. Okay maybe you need eyeballs and the ability to move around a court too but the barrier isn’t much higher than that. Some skills require more, unicycle actually takes balance, that might take longer to learn, and some games are complicated like this too.

Bad feedback is also useful. From the books, How to Measure Anything, Superforecasters, and everyone in the quantified self movement… Even a shitty piece of feedback has a Value of Information that can be valuable. One of my favourite poor pieces of feedback is when I added to my Self Form, “did I stick to my diet today? yes/no/maybe”. Like magic for a month I stuck to my diet and I lost 2kg. One good clean feedback measure and I made leaps. (Sure enough, eventually other problems got in my way, but it was a good start.)

The feedback problem asks, “How would I know if I am improving?”. For a musician, that might be to recording yourself playing then listen back to what you sound like. For a farmer, that might be to weigh or count the crop and compare that to last year. For a scientist that might be repeated tests for reliability, and for someone with an emotionally trauma history that might look like “I don’t feel terrible”.

Next: Emotional training model
Applying Bayes to an incompletely specified sample space
by abstractapplic — 2018-07-29T17:33:53.978Z

Epistemic Status: Novel mathematics from an amateur, presented with unjustified confidence. Submitted here and not in a mathematical journal because I think LW will be better and faster at explaining where I’ve gone wrong and/or what common approach I’ve reinvented.

Introduction

Bayesian reasoning is pretty great, but it has its limits. One of these is that it cannot be applied to an incomplete sample space; in other words, if you assigned 66% of your prior to possibility A, 33% of your prior to possibility B, and the remaining 1% to ??? (i.e. Everything Else), the techniques we know and love fundamentally fail to function.

Or so people seem to think. I think otherwise: in this post, I will present a new (as far as I know) method for applying Bayes, which can be made to work even if you haven’t fully specified the distribution you’re applying it to.

Unworkable worked example

Let’s say you assign your prior as above: 66% to A, 33% to B, 1% to ???. Then, you get some evidence: an event occurs which would have had a 1/11 chance of happening if A were true, and a 6/11 chance of happening if B were true.

So you multiply out your probabilities:

For A: 0.66*1/11 = 0.06

For B: 0.33*6/11 = 0.18

and for ??? . . .

Well, since we don’t know what ??? is, we can’t say exactly how it updates. But we can impose limits. The probability of the event given ??? cannot exceed 100%, and cannot be lower than 0%. As a result, we can produce upper and lower bounds.

For ???: 0.01*[0 to 1] = [0 to 0.01]

Then, we renormalise for upper and lower bounds on ???:

P(A): 0.06 / (0.06+0.18+[0 to 0.01])

P(B): 0.18 / (0.06+0.18+[0 to 0.01])

P(???): [0 to 0.01] / (0.06 + 0.18 + [0 to 0.01])

So:

P(A): 0.06 / [0.24 to 0.25]

P(B): 0.18 / [0.24 to 0.25])

P(???): [0 to 0.01] / [0.24 to 0.25]

Producing final probabilities:

P(A): 25% to 24%

P(B): 75% to 72%

P(???): 0% to 4%

“But wait!”, you may say. “This calculation is completely useless! You may be able to get an ‘answer’ out of it, but in the real world, probability exists to support decisions; and your ‘answer’ will throw positive and negative infinities if you try to multiply it to get Expected Value.”

There are two situations where this criticism doesn’t apply. The first is when we are only interested in the ratios between known parts of the distribution: the ratio between the probabilities of A and B remain constant regardless of how ??? behaves. So if you’re contempla...
Bayesianism (Subjective or Objective)
by ChaosHufflepuff — 2018-07-29T08:02:51.153Z

I'm reading a paper called 'Reasonable Doubt and Presumtion of Innocence: The Case of the Bayesian Juror' for a Physics/Policy course I'm taking, and am a bit confused by something in it. Note here that I'm quite new to Bayesianism and do not claim to understand in entirity how it all works.

The claim made is that in pure Bayesianism, all probabilities are subjective (a probability of *you*). As I had understood from initial readings on Bayesianism, it is supposed to be entirely objective (ie you look at the thing you want to determine the probability of, you look at the evidence you have available, and you thusly determine the probability of the thing). As I understand it, this makes Bayesianism objective, at least within the scope of the Bayesian's knowledge.

Is my understanding wrong somewhere? Could some kind and enlightened souls please explain this to me?
On knowing the Future
by Лu Лinveгa — 2018-07-29T01:51:11.167Z

Across peaks of inspired productive days, I get lost inbetween valleys of foggy unfocused wandering.

Some years ago, I begin recording the daily fluctuations of concrete output and work hours, which unsurprisingly maps to the Hand-Eye Dilemma that one would find in the creative rhythms of artists.

I find myself doing design, when a developer, and programming when I should be designing. Atop the creative rhythms, the daily logs reveal a form of stamina that I use to allocate blocks of time to any one task, knowing when the stamina runs out and by when the task must be completed.

Years passed, and each morning, I am told what I should do, that which is most likely to yield the most concrete output per work hour, based on previous patterns of productivity.

It is not optimizing toward longer work hours, but to help me navigate these peaks & valleys by assigning me a task most likely to convert to the optimal amount of concrete work per focus hour.

I sometimes wonder weather it has helped me reach optimal productivity, or has kept me from getting there.
Pretense
by TurnTrout — 2018-07-29T00:35:24.674Z

There's a kind of yearning, to be that person who can do those things - this is self-actualization, yet corrupted. I often feel pulled in this way. I find myself wanting to be a certain person now, to be producing and being and feeling that way now, and I catch myself acting, speaking, signalling as if I were there now. As if I could make people happy by tiling the universe with smiley-faces.

There's a revulsion that comes with this, for me - the sense of wearing a heavy coat, of playing a role, of acting instead of connecting. At times, there is a desire to connect: I begin to speak earnestly, but then comes indecision, a "social acceptance" reflex blunting my emotions and diluting my speech. And then, pain, regret, and shame. Even now, it looms: Can I even post this?

There's a certain lightness I catch and wield, from time to time. The glimmer of a fresh idea, the flow of words straight from the heart through the fingertips, the carefree, liberating simplicity of dropping pretense in a conversation. Why is this so rare?

I feel now the staggering weight of a day's trivialities - the subconscious obeisances paid to circumstance and habit, the pretentious acting out of cached responses, the molding of personality to meet the Past's expectations. There are massive costs - in time, in experience, in moments.

And yet, focusing on "being oneself", one comes to fret whether they're doing it right, or enough, or too much. Self-consciousness takes over, and back on goes the coat.

After CFAR, there was a precious week when I channeled myself during these moments. Pain did not clear its desk for joy, but I paid attention. My life beat to a satisfied rhythm. I felt no urge towards trivial things, towards pretense.

Weeks passed, and I slowly forgot. My experience of the workshop faded into a collection of moments: quietly gazing at the furious red skyline; laughing and singing despite the iceberg inching closer; hearing, and being heard. Ever-so-close bonds loosened, and then crumbled a bit, their essence blown away into the now-yawning distances between us.

I want to take off the coat.