trentmkelly/LessWrong-43k huggingface dataset:
How Real Moral Mazes (in Bay Area startups)?
by Gordon Seidoh Worley — 2022-04-03T18:08:54.220Z

How real are moral mazes? That is, how much is it the case that middle management is all about internal status competitions rather than about doing real useful things for the business?

In general I don't know. I'm not going to do a survey of businesses to figure this out. But I can say a bit about what I've seen and heard about from friends working in software startups in the SF Bay Area.

Context: I'm a senior staff software engineer at a growing startup. This is a job I roughly describe to people as engineering middle management: I'm not a director or VP, so not literally middle management, but my level is director-equivalent and I have a similar scope of responsibility within the org (multiple teams that are part of a group; I'm responsible for the group rather than individual teams). I've also talked to a lot of people in similar roles past and present through work relationships, friends, and interviewing candidates.

So, the natural question is, how real do I find moral mazes to be?

Short answer: I mostly don't see them. I don't think this is some naivety on my part, either. I've seen and heard of cases where they were starting to grow, and the people responsible were transitioned to other roles. My theory: startups don't have a lot of time for bullshit, even as they are growing fast and have the most opportunity for bullshit to creep in, because bullshit can be the difference between a successful exit with large upside and making nothing (cf. WeWork, Theranos, etc.). So if you're not doing something that directly drives impact for the business, people will notice and not let you get away with it because it's a waste of precious resources needed to make money.

This leads me to a key theory: moral mazes won't exist in places where the actions of individuals have a lot of impact on the success of the business. There's simply too much at stake for everyone to let anyone get away with moral maziness.

There's a bit more of moral maziness at established tech companies that are no longer startups. But even there it's not strongly of the type Zvi talks about, even at companies that suffer from the problem of more engineers than problems to solve. There's just not enough cultural tolerance for bullshit to let moral mazes flourish. I think this is thanks to the orgs being staffed with engineers and former engineers who will walk if the bullshit factor gets too high.

Howev...
[Book Review] Why Greatness Cannot Be Planned: The Myth of the Objective
by Bart Bussmann — 2022-04-03T17:45:23.366Z

Imagine you arrive at work and your boss tells you that instead of attending your daily meetings about benchmarks and milestones you should just do what you find most interesting. What would you do?

Kenneth O. Stanley and Joel Lehman, who are currently both AI researchers at OpenAI, begin their book Why Greatness Cannot Be Planned with this provocative question. Their book makes the argument that when it comes to achieving ambitious goals in society and culture, such as innovating in science, educating our children, or creating beautiful art, objectives are not only useless but actively detrimental.

Suppose that 1000 years ago someone would have come up with the theoretical idea of computation and had set himself the objective to build a computer. Most probably, this person would not have started by researching vacuum tubes, which were however an important stepping stone to the first computer. The invention of the vacuum tube had nothing to do with computers and was instead a product of research in electricity and radio waves. The point is that the stepping stone does not resemble the final product and that by directly optimizing towards an objective it might be harder to reach.

This can easily be seen in a maze that the authors provide as an example. If we would train an AI agent to solve this maze and we would give it the objective to minimize the distance to finish, it would learn to go up, hit the wall, and get stuck there. In order to solve the maze, the agent has to move away from the objective in order to reach the finish. The objective is in this case deceptive and the authors argue that this is true for “just about any problem that’s interesting”.

But what is the alternative to following some objective? Just performing random actions? That doesn't seem terribly efficient either. The book proposes to follow the interesting and the novel instead. The authors developed an algorithm called novelty search, where instead of optimizing an objective, the agent just tries out behaviors that are as novel to it as possible. This algorithm solves the deceptive maze much quicker than objective-based algorithms, even though it is not even trying to succeed!

Why Greatness Cannot Be Planned is a fun and interesting book to read, but is somewhat repetitive and does not always use the most interesting examples. One of the chapters d...
"Don't Get Mad, Get Curious"
by Rubix — 2022-04-03T15:30:44.291Z

Crossposted from https://e-m-morningstar.dreamwidth.org/1066.html

When I was a kid, my mother once found me in the kitchen, swearing at the dishwasher and shoving its filter around. She had me step aside and showed me a better way of finding out what was wrong with the filter: looking for objects stuck in it, moving it from side to side, taking it out and inspecting it and its seating more closely, and so on.
At the time, this looked like magic. The filter wasn’t working, and I was angry -- when I got mad, it felt impossible to do anything other than fight the target of my anger. But my mom was capable of doing otherwise. When faced with the same situation, she calmed down almost immediately and got systematic.
When I said this seemed magical, she told me that she used to fight inoperative appliances too, until she was shown enough times that a systematic approach works better on complicated, broken inanimate objects. From repeated exposure, she learned a mental motion which she called “Don’t get mad, get curious.”
I think there are three broad categories of response to problems (situations where trying what’s worked before isn’t producing good results):

Get mad
Don’t get mad, give up
Don’t get mad, get curious

They’re appropriate to different kinds of problems, and it’s useful to consider in advance which problems call for which reactions. It’s also useful to learn how to switch modes on purpose. This post covers which contexts call for which reactions; how to switch modes is an open question, and approaches tend to be highly individualised.
Getting mad is useful when:

You’re being mistreated,
Both submission and strategic action have failed repeatedly,
Future cooperation is off the table or isn’t worth it.

Getting mad is best used as stop energy: it’s a way of getting someone to stop doing a thing you dislike, to go away and leave you alone, or to give up their claim on some resource. It’s a bad way to convince someone about matters of fact, it burns goodwill (if any exists), and it makes you less capable of strategic thought, which may put you at risk.
Giving up is useful when getting mad wouldn’t serve your values and curiosity has produced a lot of dead ends. It’s an adaptive response if you’re sad and tired, and don’t expect more negotiation to help your position now, but want to leave the door open for future discussion and potential compromise.
Giving up helps you ...
The doomsday argument is normal
by avturchin — 2022-04-03T15:17:41.066Z

TL;DR: crazy predictions based on anthropic reasoning seem crazy only because they contradict our exaggerated expectations about too good future.

Let’s look at the two statements:

I am in the middle of the set of all humans ranged by birth rank and therefore large human civilization will exist for only a few more millennia at least.
I am in the very beginning of human history and humanity will exist for billions of years and will colonize the whole galaxy.

The first statement is the Doomsday argument in a nutshell and it is generally regarded as wrong. It could be false for two reasons: either the conclusion is false, or its logic is false.

Often someone argues that DA logic must be false because the conclusion is false, and therefore optimistic statement 2 is true. That is, we will survive for billions of years; therefore, we will not die in the next millennia, and thus logic of DA is wrong.

However, the optimistic statement 2 is only true for a person who is deep into transhumanism, nanotech etc but ignores x-risks. For many people, the doomy statement 1 is more probable, especially for those, who are deep into climate change, nuclear war risks and etc.

For a techno-optimist, the conclusion of DA is wrong not because it is inherently wrong, but because it contradicts our best hopes for the great future – so hating DA is wishful thinking.

Everything adds up to normality. As DA is using mediocrity thinking, it is “normal” by definition, in a tautological sense. I am typical, therefore, I am in the middle, therefore, the end is as far as the beginning. DA doesn’t say that the end is very near.

But the end becomes surprisingly near if we use the birth rank for the calculation, but the real clock time for the timing of the end. As I said in “Each reference class has its own end” we should define “the end” in the same terms as we define the reference class. Therefore, being in the middle of birth rank is neither surprising nor especially bad. It only says that there will be tens of billions of births in the future. It even doesn’t say that there will be a catastrophe in the end.

The DA prediction about being in the middle of the birth ranks becomes bad and surprising only when we compare it with the expected exponential growth of the population ( or very high plato). In that case, all these billions of births will happen in the next millenni...
[Invisible Networks] Goblin Marketplace
by Kaj_Sotala — 2022-04-03T11:40:04.393Z

Written for crtlcreep’s Invisible Networks 2022 challenge: “invent a social network each day”, today’s prompt: “Goblin Marketplace”.

“The Goblin Marketplace is that way. Assuming that you have ears on your head, you can’t miss it.”

The expression is not metaphorical: even if you were deaf, you would hear the sounds of the Marketplace. That is, assuming that nobody had cut your ears off. In that case, your ears would continue to hear it, but they wouldn’t communicate the sound to your brain.

Too bad you’re not a goblin.

The screaming tends to carry the loudest. Obviously, the Goblin Marketplace sells goblins. To help prove that the goods are authentic, there are campfires scattered around the marketplace, cut-off goblin heads roasting on top of them. They’re the ones who are screaming, for hours and hours at a time.

Next to the campfire, you’ll find a market stall. It’s staffed by the same goblin’s cut-off hands, or maybe just one of them, depending on whether the other one has sold itself yet. Most goblins are left-handed, so the right hand will go first, leaving the most important one to carry out the business.

If you are proficient in goblin tapping language, you can communicate with the hands directly, the two of you spelling out words by tapping patterns on each other’s palms. If not, you can bring your own translator, or employ one of the many hanging around the marketplace.

What’s usually for sale are the goblin’s feet, hands, and possibly the head, if it’s a place where heads are sold and it’s old enough that it wouldn’t regenerate a new body anymore. Or if you are willing to pay a lot of money. Otherwise, another goblin would pick up the head at the end of the day and go stick it to a spawning pool, letting it grow new body parts for sale.

The prices will depend on a number of factors, such as the goblin’s age, the part that you want to buy, and who the other owners are. Goblins will eventually die of old age, and any body parts will have the same age as the head they’ve been grown from. So a very old goblin’s parts might go for cheaper, since they won’t last for very long. On the other hand, an old goblin might have had the time to generate and sell quite a few body parts. They would then all be connected to each other, letting you send messages to anyone else who owned such parts.

Tap a message on a goblin’s hand, and your friend who owns another hand fro...
Avoiding Moral Fads?
by Davis_Kingsley — 2022-04-03T09:15:59.216Z

(content warning: murder, medical abuse)

A few years ago, my old middle school and another school in its district were renamed; they had originally been named after once-esteemed educational figures who were now in disrepute thanks in large part to their links to eugenics. I believe at the time I may have thought that it was odd that the schools had not been renamed sooner -- eugenics had already been discredited for decades at that point.

But at the time that these men lived, eugenics was not discredited. In fact, it was extremely popular. While eventually things changed, much of society was taken in first -- with terrible consequences. Unjust laws were passed based on eugenic principles in many nations, perhaps most notably in Nazi Germany. Hundreds of thousands of people were killed under government eugenics programs while hundreds of thousands more were involuntarily sterilized. Nowadays these acts are widely viewed as murder and abuse, but it took society some time to reach that conclusion and in the meantime things went pretty badly.

Worse still, eugenics is not the only such disgrace. You may well have heard of the lobotomy, an invasive medical procedure that is now widely viewed as destructive and abusive. However, when the lobotomy was first developed, its creator was not condemned as a medical abuser and removed from his position -- in fact, he received the Nobel Prize for his creation! Tens of thousands of people were lobotomized before lobotomies became viewed as abusive and were banned or greatly restricted in many jurisdictions.

Similarly, there was an extended period in the 1980s and 1990s where many scientists and media figures believed that techniques known as "recovered memory therapy" were helping people recover suppressed memories of abuse they had actually suffered as children -- now the general view is that this therapy was creating false memories, not bringing hidden true ones to the surface. Unfortunately, before recovered memory therapy was discredited its "revelations" led to many people being falsely accused of child abuse and in some cases even Satanic activity, and some had their reputations destroyed or were unjustly imprisoned on the basis of these "recovered" memories.

All three of these cases -- eugenics, lobotomies, and recovered memory therapy -- are instances where bad moral and medical thinking led to grave abuses. Worse still...
AI Governance across Slow/Fast Takeoff and Easy/Hard Alignment spectra
by Davidmanheim — 2022-04-03T07:45:57.592Z

It has been suggested that in a rapid enough takeoff scenario, governance would not be useful, because the transition to superintelligence would be too rapid for human actors - whether governments, corporations, or individuals - to respond to. This seems to imply that we only care about takeoff speed. And if that is the only relevant factor, the case for governance only applies if you believe slow takeoff is likely. Of course, it also matters how long we have until takeoff - but even so, I think this leaves a fair amount on the table in terms of what governance could do, and I want to try to make the case that even in that world, governance (still defined broadly1) is important - though in different ways.

The Easy/Hard Spectrum

To make the argument, I will lay out three possibilities about AI alignment which are orthogonal to takeoff speed and timing; alignment-by-default, prosaic alignment, and provable alignment. These are actually somewhat of a spectrum, with the three scenarios spaced along it. In any case, for each possibility, governance needs to accomplish very different things in order to be successful, according to the above definition - and the relationship with takeoff speeds seems important, but not fully determinative.

The first possibility, alignment-by-default, is that if we train systems via reinforcement learning or similar, then even without particular effort to solve alignment, all systems which are successful end up learning policies and goals close enough to human values that they are beneficial and influenceable. In the slower takeoff case, initially, governance looks a lot like human governance, making sure that actors, both human and AI, can cooperate and follow mutually understood and agreed upon rules. Later, and in the faster takeoff case, our efforts towards governance become irrelevant as the AI systems replace human structures, or improve them.

The second possibility, prosaic alignment, is that alignment of artificial intelligence systems is somewhat difficult, but achievable via approaches which can be developed. So some systems will be aligned, but without oversight, unaligned systems are possible or likely. In this case, the key task of governance is to ensure that all early HLMI/PASTA/AGI systems undergo robust alignment procedures. Prior to the emergence of such systems, many tasks will be useful for ensuring this outcome...
Practical use of the Beta distribution for data analysis
by Maxwell Peterson — 2022-04-03T07:34:26.483Z

Many data scientists and data analysts working today use the Gaussian (Normal) distribution when computing probabilities from yes/no count data. The appropriate distribution for binary count data is the Beta distribution, and the Gaussian is only an approximation. This approximation works well for probabilities close to 0.5, and on medium-to-large data, but breaks down on small data and data with probabilities close to 0 or 1. Avoiding the breakdown of the approximation is important when computing uncertainty intervals on small data. The Beta distribution is extremely easy to use in languages like Python and R, and should be preferred.

Examples

An example where the Gaussian approximation is fine

You work for a fancy restaurant. Fancy restaurants tend to have small menus. Your restaurant is extremely fancy, and thus has a 2-item menu: patrons can choose steak, or they can choose chicken. You want to compute the probability that a patron next week will order chicken, along with the uncertainty intervals around this probability. In the past week, 40 people got chicken, and 60 got steak. Here are the Gaussian and Beta distributions for n=100 and p=0.40:

Similar to how the Gaussian takes two parameters - the mean and standard deviation - the Beta distribution takes two parameters too. The first parameter is how many "yes"'s there were (in this case, chicken orders), the second, how many "no"'s there were. Here, that gives Beta(40, 60).

The Gaussian approximation works very well here - it's nearly identical to the exact Beta solution. The 95% interval for the Beta is (0.307, 0.497), and the 95% interval for the Gaussian is (0.304, 0.496). For both, the mean is 0.40, like we'd expect. The uncertainty intervals tells us that from the experience of just 100 patrons, there's a 95% probability that between 30% and 50% of customers will order chicken next week.

An example where the approximation breaks

You're doing a test week where, instead of chicken, the restaurant serves guinea pig. So people can choose guinea pig or steak. Guinea pigs are seen as more like pets in the US, and also (from personal experience) just don't taste terribly good, so at the end of the week, only one adventurous soul has ordered it. 1 guinea pig, 99 steak. Check out the distributions:

I've marked 0 probability with a heavy black vertical line. The Gaussian approximation extends ...
What are some ways in which we can die with more dignity?
by Chris_Leong — 2022-04-03T05:32:58.957Z

Eliezer's recent kidding not kidding death with dignity post suggests that our chances of survival are so low that we should just focus on going out with some semblance of dignity (ie. "at least we made an attempt that wasn't truly pathetic).

Or at least that's what it seems to be claiming initially. If you read it to the end it becomes clear that Eliezer is proposing a frame for thinking about how to act in scenarios with a low probability of success[1]. In particular, he seems to be criticizing the tendency of people to say "well we need to assume X because otherwise we would be doomed anyway" as the tendency is to power on multiple assumptions and end up focusing on and over specific-case. In contrast, Eliezer suggests that it is better to position yourself sucks that you would be able to leverage positive moral violations in general as then you aren't just betting on one particular scenario.

I often find that reframing a problem is quite conducive to producing solutions, so I'm asking how we could actually die with more dignity[2].

^

I find it fascinating how post-rationalist this is.

^

People could just post their thoughts on the original post, but this question serves as a nudge to actually attempt to generate solutions rather than to just grimly reflecting on it.
General Thoughts on Less Wrong
by Chris_Leong — 2022-04-03T04:09:35.771Z

General disclaimer: Obviously there's always going to be more that could be done and it's better to do a few things well, rather than many things poorly, but I still think it is worth being conscious of areas that could be improved[1].

I'm still strongly in favor of adding sub-communities. I know that there were worries before that there wasn't sufficient volume, but I suspect that creating new spaces for discussion will create their own volume. Further, I suspect that carefully curating the subcommunities that people are allowed to create would prevent some of the risks of this causing division in the community. I think that the strongest argument here comes from looking at Reddit. Reddit simply wouldn't be Reddit with its wondrous and diverse range of communities if it had tags instead of sub-communities.
The resources section of the community page still seems underdeveloped. I don't this is a huge priority but this could be a good thing for an intern to work on.
Most Wiki pages still seem rather bare. I would love to see Less Wrong develop an internship program where interns spend a significant amount of their time focusing on improving pages on the Wiki. I could imagine a world where the Wiki pages ends up being one of the first places you head when you want to know about a topic.
Short-form: Most people still seem to be posting their thoughts to Facebook or Twitter. It feels awkwardly integrated to me, but some short-form posts are seeing reasonable levels of engagement. I think for this feature to really reach its full potential there has to be a separate item on the sidebar just for short-form[2] and that it should display these posts in a feed format very similar to Facebook. This feed should be customised based on who you follow and the posts that you've liked/indicated that you aren't interested in.
More generally, I have this vague sense that this community isn't reaching its full potential. I guess a large part of this is that everyone (including me!) has their own distinct interests and perspectives and this means that we don't coordinate very well on larger-scale (or even mid-scale) projects. Effective Altruism is a very similar community, but they seem to be much better at this. I don't expect this to be resolved quickly. In the same way, it took far too long and many failed attempts for LW to be revived. That said, I have no idea what L

...