trentmkelly/LessWrong-43k huggingface dataset:
Taking Good Heart Tokens Seriously, So Help Me God
by Jackson Wagner — 2022-04-01T23:29:58.328Z

Summary: We probably can't just hand out a dollar per upvote and expect things to end well. But maybe there are other, less-corrosive ways to encourage Forum contributions:

Maybe Karma could be used as the basis of a voting system for community decisions.
Maybe Karma could be expanded into a pan-rationalist reputation system across multiple websites.
Finally, maybe Karma-allocated grants could help kickstart impact certificate markets.

Assessing the goodness of people's hearts is a complex endeavor fraught with mixed incentives.

How can we reward community involvement without creating bad incentives?

The idea of putting a dollar value on LessWrong and EA karma points has occurred to many people over the years. Clearly, LW / EAF / AF are producing things of value to the world, in a way that (if divided by the total number of upvotes) usually gives a value of at least a few dollars per Karma point in aggregate:

Groups like Lightcone Infrastructure and the Center for Effective Altruism consider it worth their while to spend money and effort directly supporting these sites and encouraging their growth.
The ongoing contributions to Effective Altruism made by hobbyists on these forums are in aggregate probably comparable in scale (in terms of how much they advance the sophistication of EA thought) to the contributions of various respected and well-funded research groups, like Rethink Priorities.
Presumably, most of the people who read this site consider it informative and interesting. Looking at the analytics of my own top-level posts on the EA Forum, I get a ratio of about four upvotes per hour of reader engagement. Considering the high value of people’s time, the attention of readers suggests that posts have some value.

It would be great if we could incentivize people to produce more of this value. But of course we can’t just start handing out money in exchange for Karma, as the Good Heart Project demonstrates. It threatens to create an incentive to mindlessly churn out endless comments purely for dollars, or (assuming the people of LessWrong will always put in the effort to search out and destroy Good-Heart-abusing mutual-upvoting societies wherever they arise) to degrade quality in other ways by encouraging people to produce a higher volume of lower-quality drafts.

So, Good Heart Tokens as currently implemented (direct payments for newly-created karma) are pr...
Interacting with a Boxed AI
by aphyer — 2022-04-01T22:42:30.114Z

A BOXED AI

So, a thought experiment.

We have an AI-in-a-box. By this I mean:

The AI can interact with the world only through a single communication channel.
We control that communication channel.:

We can communicate with the AI at will, it can communicate with us only when we allow it.
We can control what responses the AI is allowed to send to us, doing things like e.g. limitations on the amount of data it is able to send us.

The only way the AI can get out of the box is if we let it out - it cannot e.g. hack its way out by abusing is computer substrate to send radio signals, or some such method. Yes, I'm aware that this is not a thing we currently know how to do. Assume we reach this point anyway.
We know the AI is superhuman in intelligence, but we don't know exactly how much superhuman it is.

We know nothing else about the AI for sure. We think it's Friendly, but maybe it's unFriendly. We don't know.

We cannot safely let the AI out; no matter what it says or does, we can never know for sure that it's not a sufficiently intelligent trick by an unFriendly AI to let us out.

Eliezer has taken some pains to argue that we cannot even talk to the AI: that a sufficiently clever AI can push our buttons to make us let it out, no matter how resolved we may be. And he's provided some compelling arguments that this is the case.

So we should just leave the AI in the box and not talk to it? That way, we can be completely certain that it won't tile the universe in smily faces or some such.

But...well...is this really the optimal solution? If Omega gave you this AI-in-a-box right now, would you stop at "Nope, no way to safely talk to it," without even trying to come up with a way to get some use out of it?

This, then, is a vague stab at how we could get some value out of a probably-Friendly boxed AI in ways that don't risk the end of humanity.

Risks

If you talk to the AI, it may be able to take over your mind and make you let it out. It is (potentially) so much more than human that it sees you as a simple machine, with buttons to push and levers to pull to make you do whatever it wants.

But this relies on your interaction with it having a certain level of complexity.

If the AI plays you at chess, it will win. But what if it plays you at Tic-Tac-Toe? No matter how clever the mind, it is very hard to win a...
Two Forms of Moral Judgment
by aphyer — 2022-04-01T22:13:30.129Z

Minor spoilers for HPMOR, I guess?

SUMMARY

Sometimes, if you do something, people will call it good. Other times, if you do something, people will call it bad.

These words actually cover two different kinds of moral judgment: judgment of actions and judgment of character.

Judgment of actions is evaluating whether these actions are good or bad actions - whether they lead to good or bad consequences. For example, murdering someone is bad because it results in that person being murdered.

Judgment of character is evaluating this action as Bayesian evidence of your character - whether it says good or bad things about the other actions you are likely to take.

Sometimes these two things align: murder is bad on both forms of judgment.

However, confusion can arise when these two forms of judgment lead to different answers. In cases like this, it's valuable to distinguish which kind of judgment you're using, and to understand that these two forms are not necessarily consistent with one another.

THE GUY WITH THE FLAMING SKULL TATTOO

Suppose that Bob gets a large tattoo of a flaming skull on some visible part of his anatomy.

By judgment of actions, it's hard to argue that this act has any particular moral impact.

But by judgment of character, this tattoo serves as a reasonable quantity of Bayesian evidence of bad character. If you see Bob having such a tattoo, you can correctly conclude that Bob is more likely to be violent, more likely to commit crimes, etc.

These two forms of judgment are in tension. Judgment of character suggests that, when you see Bob's tattoo, you should judge him as more likely to be a bad person - that you should be less likely to trust him, less likely to hire him to babysit your children, less willing to meet him in an abandoned parking lot to buy the PS5 he advertised on Craigslist, etc. But judgment of actions declares that Bob has not actually harmed anyone, and there's no reason to punish him.

One resolution to this tension is to distinguish between social and legal punishment. We are often willing to apply social punishment based on judgment of bad character (most people would not disapprove of you for refusing to meet Bob somewhere that isn't public), but not willing to apply legal punishment (most people would disapprove of the police arresting Bob on a charge of 'having a tattoo that we think is Bayesian evi...
Manafold Markets is out of mana 🤭
by Austin Chen — 2022-04-01T22:07:34.081Z

Crossposted from Above The Fold.

The problem

As you may know, we’ve been busy raising our seed round in the last couple of weeks. One piece of feedback we often hear from investors is to be careful with how we issue our M$ (aka

mana

). We conducted an internal audit, which revealed that... Manafold Markets is out of mana!!

We did some brainstorming on how to address this unexpected shortfall. Some ideas:

Charge new users to join Manafold instead of handing out M$ 1,000
Institute

demurrage

, a negative interest rate on unspent mana
Start an OnlyFans and post exclusive photos of our mascot,

Foldy McFoldface

The solution

In the end, we decided to go after one source of nearly limitless mana: tax the rich traders who sit atop our leaderboards!

We were inspired by

Radical Markets

to set up a

Harberger Tax

. This means that all our leaderboard slots are now for sale. Instead of trying to make good predictions in our markets in the hopes of turning a profit, you can just buy that leaderboard position you’ve always wanted! Check it out:

How this works

Every leaderboard slot has an "assessed value": what the current holder thinks their slot is worth.
Slot holders pay a continuous fee of 25% per hour to Manafold.
At any time, you can pay the assessed value of a slot to buy it from the current holder.
The slot is now yours! You can customize the message, or reassess it to a new value.

Try it out now, at

https://manafold.markets/leaderboards

!

Where did Manafold's mana go?

Honestly, we're as puzzled as you are. Leading theories include:

Leaky abstractions in our manabase
One too many floating-point rounding errors
Our newest user Robert');DROP TABLE Balances;--

We'd be happy to pay a bounty to anyone who can help us solve this riddle! Preferably in USD, as we have a lot more of that now than mana.

In other news

It’s been quite a busy day on the internet; who could have predicted all these announcements?

Move over, GPT-3; here come

blog posts generated via prediction market

Our largest funder is now spinning up an

Effective Altruism reality TV show

Don’t miss the

April 2222 edition of the Forecasting newsletter

LessWrong is replacing Karma with

Good Heart Tokens

(Fiction)

Prediction markets tackle corruption in China
The median and mode use less information than the mean does
by Maxwell Peterson — 2022-04-01T21:25:20.916Z

Epistemic status: Unsure. I had this in drafts from a year ago, and am posting it for Goodhart day. (Though it's April 1, all the arguments and statements in this post are things I think are true, with no jokes). I'm interested in arguments against this thesis, and especially interested in thoughts on the question at the end - does the distribution-summarizer corresponding to the L3 or L4 minimizers use more information than the mean (the L2 minimizer)?

The mean, median, and mode are the "big 3" location parameters that most have heard about. But they can have very different properties, and these different properties are related to the fact that the mean uses more information than the median, and the median uses more information than the mode.

Refresher

The mean, median, and mode measure the location of a probability distribution. For the Gaussian distribution, they are all the same, but this isn't the case in general. Here's an example of a Gamma distribution where the three differ:

gamma(shape = 2, rate = 1)

The mean corresponds to the middle of the distribution when weighted by frequency.

The median corresponds to the middle of the distribution, without using the weights. The median is the vertical line that splits a distribution such that 50% of the probability mass is on the left and 50% on the right.

The mode is the highest point on a distribution.

Different amounts of information usage

The median is preserved under a larger set of changes to the data than the mean is. Really, this is often why people use the median: outliers don't knock it around as much as they do the mean. But that ability to resist being knocked around - "robustness" - is the same as the ability to ignore information. The mean's sensitivity is sometimes seen as a liability (and sometimes is a liability), but being sensitive here is the same thing as reacting more to the data. It's good to react to the data: the mean can distinguish between the three different datasets below, but the median can't. The plots below are an example: if all you had was the median, you wouldn't know these three datasets were different; but if all you had was the mean, you would know:

Top: Some data x.
Middle: The data from the top plot, but with the values right of the median shifted 4 to the left.
Bottom: The data from the middle plot, but with the values left of the me

...
If you lose enough Good Heart Tokens, will you lose real-world money?
by Yitz — 2022-04-01T21:11:20.180Z

Please note that the following is satirical, just in case you somehow couldn’t tell.

With the new Good Hearts system replacing Karma, it appears that those who receive a sufficient number of upvotes will gain actual, physical, real-world money. This begs the question: what happens to those who get downvoted?

I have to imagine that the consequences to such a shameful predicament will be dire. After all, per thermodynamic law, there must be a balance of entropy in any closed system. We know that the rationalist movement has been accused of being elitist in the past, aka a closed system. Therefore, for all rationalists who make money, there must be those who lose it. The only reasonable way to lose money as a rationalist is if you are failing to be rational, which means you’re a loser. Being a loser is typified by those who are unpopular in social situations, and downvoting is an easy method to indicate social unpopularity. Therefore, those who are downvoted are losers, and must be punished. Monetarily.
Q.E.D.

An unpopular enough post may even force the writer into debt, at which point the secret MIRI agents scattered where you least expect them (probably still in California though, let’s face it they aren’t moving) will be activated. [At this point I got bored writing this and let GPT-3 complete the rest of this post. Enjoy(?).]
These agents, upon receiving a signal that someone has become too irrational for society, will break into their house in the middle of the night and forcefully remove them from their bed. The agent will then drag the screaming, struggling person to the nearest high place and toss them off.

But before they do so, they’ll make sure to leave behind a note that reads something like “you were downvoted on LessWrong because you are irrationally committed to your current beliefs. Good luck in future lives!”
What's the problem with Oracular AIs?
by FinalFormal2 — 2022-04-01T20:56:26.076Z

I have a superintelligent AI in a box, and I tell it by means of text to explain to me how to create a perfect battery. Its goal is not to maximize the number of perfect batteries or make me understand a perfect battery, its goal is to produce a string that will explain to anyone who reads it how to create a perfect battery. Its only output is the one string, and it will be due in one week. How do I die?
Towards trying to feel consistently energized
by Maxwell Peterson — 2022-04-01T20:46:23.774Z

I go through weeks or months-long periods where I feel exhausted throughout the day and it's been getting worse over the past few years. I thought it might be sleep apnea - my dad has it bad and for him getting tested and hooked up to a Darth Vader mask changed his life - so I did a sleep study at the beginning of March. It was super easy - of course you first have to go into your primary care physician and be like "I feel tired", and she refers you to a sleep doctor, and you go to them and you say "I feel tired", and then they send you home and you come back in two weeks to pick up the device (and then they want you to schedule a follow-up appointment to discuss results! 4 visits overall for one sleep study. My sleep doctor was cool and said she could just summarize the results over MyChart, which she did, so I got out of this with just 3 visits. And if I'd really tried, I could have probably gone to her directly, without talking to my PCP first, although some insurance designs force you to go tell a PCP "I am tired" before you are allowed to visit a specialist, which is a great insurance design if you want to waste everybody's time).

But the actual study was easy. You just tie a strap around your torso and stick some tubes to your nostrils and go to sleep. (The device is the shoddiest-looking thing you've ever seen in your life, and on the paper you sign when you rent it, it says they'll charge you 3 grand if you lose it. I would have guessed this number would be $100.)

The raw study results are a good time - I think there is a special doctor who sits in a tiny basement room and just summarizes the results of sleep studies all day, in his alien sleep-doctor dialect - this dialect of his is why they don't let him out of the basement. From him I learned my clustered central events may represent occult obstruction, which I think means the girl from The Exorcist is sneaking into my room at night and blocking my airway. My RIP index is 100%, which is appropriate, because reading his report made me feel like killing myself. But I asked the sleep doctor I'd talked to to summarize these 5 pages of GPT-2 and she said I had scored a 5.3 on the apnea scale, where numbers from 5-15 indicate apnea, i.e. I basically don't have it. Also I don't snore. So that's out.

My theory for a while has been that I have some extremely snotty circadian algorithm going on, where if...
New Scaling Laws for Large Language Models
by 1a3orn — 2022-04-01T20:41:17.665Z

On March 29th, DeepMind published a paper, "Training Compute-Optimal Large Language Models", that shows that essentially everyone -- OpenAI, DeepMind, Microsoft, etc. -- has been training large language models with a deeply suboptimal use of compute.

Following the new scaling laws that they propose for the optimal use of compute, DeepMind trains a new, 70-billion parameter model that outperforms much larger language models, including the 175-billion parameter GPT-3 and DeepMind's own 270-billion parameter "Gopher".

I'm going to walk through the background of the now-falsified scaling laws from prior to this paper; then I'm going to describe the new laws given by this paper, and why they weren't found earlier; and finally I'll briefly mention some possible implications of this paper.

Independently of the consequences -- this paper is exciting! Machine learning researchers thought they knew laws about how to scale compute optimally, and the laws turned out to be wrong! It's a nice clear instance of science-functioning-in-ways-it-should in ML.

Background

In 2020 OpenAI proposed scaling laws which have since been used (at least implicitly) to guide the training of large models.

These scaling laws attempt to answer several questions. One of these questions is "Given a certain quantity of compute, how large of a model should I train in order to get the best possible performance?"

The answer isn't "as large a model as possible" because, for a fixed quantity of compute, a larger model must be trained on less data. So training a 1-million parameter model on 10 books takes about as many floating point operations (FLOPs) as training a 10-million parameter model on one book.

In the case of very large language models like GPT-3, these alternatives look more like training a 20-billion parameter model on 40% of an archive of the Internet, or training a 200-billion parameter model on 4% of an archive of the Internet, or any of an infinite number of points along the same boundary.

Compute on this scale is not cheap -- so if you're going to be spending 10 million dollars per training run on a model scaled up to be 100x bigger than your toy version of the model, you want principles better than a feeling in your gut to guide how you allocate this compute between "amount of data the model sees" and "how big the model should be."

So if you get 10x more compute, how much bigger do you mak...
5 Tips for Good Hearting
by Thomas Kwa — 2022-04-01T19:47:22.916Z

I've noticed several ways to maximize karma, and since the recent launch of Good Heart tokens, I've become convinced that I should share these with the community, because they will surely lead to better quality content. The tips are listed here and explained in the comments below:

Split good ideas into multiple posts and comments.
Write listicles, especially with titles of the form "X Tips for Y".
Write about current events, rather than topics that have timeless utility.
Write in your style, especially when writing humor.
Write low-effort posts.

Feel free to add your own in the comments!

Note that if you want to upvote this post without giving me net karma, you can downvote this linked comment or this one.
[Link] sona ike lili
by AprilSR — 2022-04-01T18:57:06.373Z

Toki Pona is a language constructed by Sonja Lang, as foretold by linguistic determinism. It is famous for only having about 120 or so words, and yet still being able to express a wide variety of concepts. Today, I am announcing sona ike lili (lit. little bad knowledge), a blog where I will translate famous LessWrong articles into toki pona. Currently, there are two articles: Scope Insensitivity and Taboo Your Words.
Questions about ''formalizing instrumental goals"
by Mark Neyer — 2022-04-01T18:52:11.486Z

Epistemic Status: Autodidact outsider who suspects he has something to add to the conversation about AI risk.
Abstract

This essay raises questions about the methodology, and thus conclusions, reached in the paper “

Formalizing Convergent Instrumental Goals

.” This paper concluded that convergent instrumental goals common to any AGI would likely lead that AGI to consume increasing amounts of resources from any agents around it, and that cooperative strategies would likely give way to competitive ones as the AGI increases in power. The paper made this argument using a toy model of a universe in which agents obtain resources in order to further their capacity to advance their goals.

In response, I argue that simplifications in the model of resource usage by an AI have led to incorrect conclusions. The simplifications, by themselves, do not change the conclusion. Rather, it is the combinatorial interaction of the simplifying assumptions which possibly brings the conclusion into doubt. The simplifying assumptions ‘cover up’ aspects of reality which have combinatorial interactions, and thus the model eliminates these interactions from consideration when evaluating which goals are likely to be convergent instrumental subgoals of an AGI.
The summary of the objection is this:

Absent a totally accurate understanding of reality (and all of the future consequences of its actions), and subject to the random decay and breakdown of any single part of its physiology, an AGI may end up being extremely gentle because it doesn’t want to kill itself accidentally. Taking care of other agents in its environment may end up being the lowest risk strategy for an AGI, for reasons that the original paper simplified away by eliminating the possibility of risks to the AGI.

I understand that I am an amateur here, questioning conclusions in what looks to be an important paper. I hope that that amateur status will not dissuade readers from considering the claim made herein.
This paper is not an argument that AI risk doesn't exist, or isn't existential. On the contrary, it is an argument that a sole focus on the alignment problem ends up obscuring significant differences in risks between different approaches taken to construct an AGI. If the alignment problem is unsolvable (and I suspect that it is), we are ignoring the difference between existing ris...