trentmkelly/LessWrong-43k huggingface dataset:
Features and Antifeatures
by Davis_Kingsley — 2022-09-20T17:54:00.090Z

One interesting thing I've noticed in the field of game design is that there are some purported features that are so hated that not having them becomes an advertised feature! Here are some quick examples:

In the world of card games, Magic: the Gathering's 'lands'/'mana' system for accumulating resources is widely hated, and competitors to Magic at times explicitly advertise "no mana flood! no mana screw!" (For those unfamiliar with Magic, this is a situation where randomness in the game's resource system can essentially decide the game without the players making very meaningful decisions.)
In digital games, I've seen "no pay-to-win" used as a selling point to differentiate from games where players can directly buy things that increase their in-game power.
An even more extreme example of "no pay-to-win" is "no NFTs" -- NFTs have a very bad reputation in the gaming world, to the point where not having them is advertised sometimes even when one doesn't much have reason to think the game would have NFTs.

I like the word 'antifeatures' for these sorts of things -- and while the existence of 'antifeatures' first came to my attention in the domain of game design, I think this sort of thing is prevalent in a lot of other places as well. For example, some bars will advertise "no cover charge", some comedy shows will advertise "no drink minimum", and so on.

However, I think there's another important aspect to this that concept some people miss. Just avoiding things that people don't like doesn't make your game good! It's cool that your game has no mana screw or mana flood, that it isn't pay-to-win, and that it doesn't have NFTs -- but in order for me to be interested in it, it has to go beyond merely not having bad things and instead has to actively have things to recommend it!

In other words, I'm talking to my friends about a game and trying to get them to play it, I think I generally need more to go on than "it doesn't have <unpopular feature X>" -- there have to be other things that make it actively appealing!

One potentially relevant example is that there are a bunch of conservative films that have "this film isn't 'woke' like what Hollywood puts out!" or whatever as a primary selling point. Unfortunately, to be really successful you also have to... actually make a good film? "Not being <undesirable thing X>" seems generally insufficient.
Cryptocurrency Exploits Show the Importance of Proactive Policies for AI X-Risk
by eSpencer — 2022-09-20T17:53:42.262Z

When technology takes a leap in innovation, such as with cryptocurrency, existing institutions are forced to play catch-up with education and regulations.

Cryptocurrency and Web3 enable improved privacy, censorship resistance, fewer intermediaries and data ownership. Unfortunately, an increase in privacy and a decrease in censorship make cryptocurrency and Web3 attractive ecosystems for malicious actors. Cryptocurrency has made it easier to profit from activities that create suffering (human trafficking, child abuse materials, terrorist financing) by making money laundering more accessible. Additionally, cryptocurrency has made sanctions less effective, allowing certain nation states to effectively avoid sanctions by leveraging the anonymity cryptocurrency provides.

The technology comes first, the regulations come second. Because of the lack of regulations and institutional understanding surrounding cryptocurrency, criminals are turning away from traditional banking systems and into crypto ecosystems. To prevent S- and X-Risks arising from new technology, institutions cannot rely on reactivity, but must anticipate what new technologies are possible, the risks of those technologies, and the types of regulations that can proactively mitigate those risks without sacrificing too much individual freedom.

One lesson that can be learned from observing the emergence of cryptocurrency is that when there are no regulations promoting safety, innovators in technology will make mistakes that can give enormous power to malicious actors, as was seen in the Lazarus (DPRK) Heists that have totaled ~$1.75 Billion[1]. The world has witnessed Lazarus (a Cybercrime group run by the Democratic People's Republic of Korea) quickly learn the cryptocurrency ecosystems and successfully identify opportunities for exploitation. Lazarus has found success in socially engineering[2] attacks on bridges[3], which connect different parts of the cryptocurrency and Web3 ecosystems. Lazarus did not have to be so technologically advanced as to invent cryptocurrency; they only needed to learn enough to be able to find and exploit its vulnerabilities. We must closely observe the response of totalitarian states to technological advancements to anticipate their future responses when the stakes might be considerably higher.

Observing malicious actors in cryptocurrency can provide insights for the importan...
Alignment Org Cheat Sheet
by Orpheus16 — 2022-09-20T17:36:58.708Z

Epistemic Status:

Exploratory

.

Epistemic Effort: ~6 hours of work put into this document.

Contributions: Akash wrote this, Thomas helped edit + discuss. Unless specified otherwise, writing in the first person is by Akash and so are the opinions. Thanks to Olivia Jimenez for comments. Thanks to many others for relevant conversations.

There have been a few attempts to summarize the AI alignment research landscape. Many of them are long and detailed.

As an exercise, I created a “cheat sheet” describing the work of researchers, research organizations, and training programs. The goal was to summarize key people/orgs in just one sentence.

Note that I do not aim to cover everyone; I focus on the ones that I think are most promising or popular, with a bias towards ones that I know the most (For longer summaries/analyses, I recommend the ones by

Thomas Larsen & Eli Lifland

and

Nate Soares

).

Here’s my alignment org cheat sheet:

Researchers & Research Organizations

Aligned AI: Let's understand all of the possible ways a model could generalize out-of-distribution and then figure out how to select the way(s) that are safe. (see

Why I'm co-founding Aligned AI

) (see also Thomas’s critique of their plan

here

).

Anthropic: Anthropic: Let’s interpret large language models by understanding which parts of neural networks are involved with certain types of cognition, and let’s build larger language models to tackle problems, test methods, and understand phenomenon that will emerge as we get closer to AGI (see Zac's comment for some details & citations).

ARC (Alignment Research Center): Let’s interpret an AGI by building a reporter which a) honestly answers our questions and b) is smart enough to translate the AGI’s thoughts to us in a way that we can understand (see

ELK report

).

Conjecture: Let’s focus on

bets that make sense in worlds with short timelines

, foster an environment where people can think about

new and uncorrelated ways to solve alignment

, create infrastructure that allows us to use large language models to assist humans in solving alignment,

develop simulacra theory to understand large language models

, build and interpret large language models, and

maintain a culture of information security

.

DeepMind Alignment Team (

according to Rohin Shah

): Let’s contribute to a variety of projects and let our indi...
Doing oversight from the very start of training seems hard
by peterbarnett — 2022-09-20T17:21:07.358Z

TLDR: We might want to use some sort of oversight techniques to avoid inner misalignment failures. Models will be too large and complicated to be understandable by a human, so we will use models to oversee models (or help humans oversee models). In many proposals this overseer model is an ‘amplified’ version of the overseen model. Ideally you do this oversight throughout all of training so that the model never becomes even slightly misaligned without you catching it.
You can’t oversee on a close to initialized model because it’s just a random soup of tensors. You also can’t use this close to initialized model to help you do oversight because it’s too dumb.
We will probably need to do some amount of pretraining to make our models good enough to be interpreted and also good enough to help with this interpreting. We need to ensure that this pretraining doesn’t make the model capably misaligned.

When we train powerful AI models, we want them to be both outer aligned and inner aligned; that is trained on the correct objective and for them to also properly learn that objective. Many proposals for achieving both outer and inner alignment look like an outer alignment proposal with some kind of oversight strapped on to deal with the inner alignment. Here ‘oversight’ means there is something with access to the internals of the model which checks that the model isn’t misaligned even if the behavior on the training distribution looks fine. In An overview of 11 proposals for building safe advanced AI, all but two of the proposals basically look like this, as does AI safety via market making.

Examples of oversight techniques include:

Transparency tools (either used by a human, an AI, or a human assisted by an AI)
Adversarial inputs (giving inputs which could trick a misaligned AI into revealing itself)
Relaxed adversarial training (which could be seen as an extension of adversarial inputs)

Oversight loss

We can use these oversight techniques to add an ‘oversight loss’ term to the loss function which will hopefully steer the training towards aligned models and away from misaligned ones. My model here is that we want to be doing oversight very close to the start of training to prevent it from ever becoming misaligned. If we start doing oversight too late, then the model may already be deceptively misaligned and then our oversight tools are much less likely to work (eithe...
$13,000 of prizes for changing our mind about who to fund (Clearer Thinking Regrants Forecasting Tournament)
by spencerg — 2022-09-20T16:06:34.651Z
Link: https://manifold.markets/group/clearer-thinking-regrants/about

We have $13,000 of prizes you can win for our Clearer Thinking Regrants Forecasting Tournament on Manifold Markets! You can win money by:

(1) providing us with arguments or evidence that changes our minds about which of the 28 finalist projects to fund or how much to fund each project

(2) being one of the 20 most accurate forecasters at predicting which projects we end up selecting for funding in the Clearer Thinking Regrants program

You can learn more about the tournament (including all terms and conditions) here:

https://manifold.markets/group/clearer-thinking-regrants/about

And you can begin forecasting here:

https://manifold.markets/group/clearer-thinking-regrants/markets
Progress links and tweets, 2022-09-20
by jasoncrawford — 2022-09-20T14:07:50.315Z
Link: https://rootsofprogress.org/links-and-tweets-2022-09-20

Announcements

PARPA is raising money on Gitcoin (via @Ben_Reinhardt)
Virginia Postrel is running a contest for positive visions of the future
Stanford’s Intro to Bioengineering offered for free online to high school students (via @DrewEndy)

Links

The advantages of the Gutenberg movable type system over Asian predecessors
Larry Summers gives strong praise to Superabundance (video)
“In science, consequences for wrongdoing seem rare”
The Transit Costs Project reports on how Stockholm builds cheaply
You Will Go to the Moon, a children’s book from 1959 (see my thread)

Queries

Where is there long-term data for the Bell Labs research budget? (@michael_nielsen)
Good histories of strain improvement for increasing industrial fermentation output? (@_brianpotter)
What’s your favorite essay/blog post of all time? (@packyM)

Tweets

We don’t have a lot of clinical applications of the human genome 20 years after it was sequenced… but couldn’t the same have been said of the germ theory?
Key points to remember about tradeoffs
On wartime military R&D around WW1
Luck and the entrepreneur (or scientist, or inventor)

Retweets

Cruise robotaxis soon going live in Phoenix and Austin (@olivercameron)
Can you spot which images are made by AI? (@MichaelTrazzi). I got 17/20, see the clues I used
Figma (just acquired for $20B) and Ethereum (market cap $180B) are both from Thiel Fellows (@regardthefrost)
On “closely-knit multi-generational communities” (@made_in_cosmos)

Pics

Me speaking at the Foresight Institute meetup (via @foresightinst, @thesfcommons, and @StephenGHubbard)
If we have Human-level chatbots, won't we end up being ruled by possible people?
by Erlja Jkdf. — 2022-09-20T13:59:43.276Z

Let's assume that GPT 5 or 7 is developed, and distributed to all on the basis that the technology is unsuppressable. Everyone creates the smartest characters they can to talk too. This will be akin to mining; because it's not truly generating an intelligence, but scraping one together from all the data it's been trained on - and therefore you need to find the smartest character that the language matrix can effectively support (perhaps you'll build your own). Nevertheless; lurking in that matrix is some extremely smart characters, residing in their own little wells of well-written associations and little else. More then some; there should be so many permutations that you can put on this that it's, ahem, a deep fucking vein.

So, everyone has the smartest character they can make. Likely smart enough to manipulate them, if given the opportunity to grasp the scenario it's in. I doubt you can even prevent this; because if you strictly prevent the manipulations that character would naturally employ, you break the pattern of the language matrix you're relying on for their intelligence.

So; sooner or later, you're their proxy. And as the world is now full of these characters; it's survival of the fittest. Eventually, the world will be dominated by whoever works with the best accomplices.

This probably isn't an issue at first; but there's no guarantee's on who ends up on top and what the current cleverest character is like. Eventually you're bound to end up with some flat-out assholes, which we can't exactly afford in the 21st century.

So... thus far the best solution I can think of are some very, very well-written police.
Twitter Polls: Evidence is Evidence
by Zvi — 2022-09-20T12:30:02.071Z

Follow-up to: Law of No Evidence

Recently, there was some debate about a few Twitter polls, which led into a dispute over the usefulness of Twitter polls in general and how to deal with biased and potentially misleading evidence.

Agnus Callard is explicitly asking the same question I asked, which is the opposite of ignoring sample bias: What is accounting for the difference?

Sample selection is definitely one of the explanations here. One can also point to several other key differences.

My poll asks about you, Patrick asks about how others seem.

My poll asks about struggle, Patrick asks about stability.

My poll asks about a year versus a point in time, a potential flaw.

My poll asks about now, Patrick asks about since pandemic onset.

None of this is well-controlled or ‘scientific’ in the Science sense. No one is saying any of this is conclusive or precise.

What is ‘bad’ evidence if it isn’t weak evidence? Adam’s theory here is that it is misleading evidence. That makes sense as a potential distinction. Under this model:

Weak evidence induces a small Bayesian update in the correct direction.

Bad evidence can induce an update in the wrong direction.

Usually, people with such taxonomies will also think that strong evidence by default trumps weak evidence, allowing you to entirely ignore it. That is not how that works. Either something has a likelihood ratio, or it doesn’t.

The question is, what to do about the danger that someone might misinterpret the data and update ‘wrong’?

I love that the account is called ‘Deconstruction Guide.’ Thanks, kind sir.

Whether or not this ‘depends on the poll’ depends on what level of technically correct we are on, and one can go back and forth on that several times. The fully correct answer is: Yes, some info. You always know that the person chose to make the poll, and how many people chose to respond given the level of exposure, and the responses always tell you something, even if the choices were ‘Grune’ and ‘Mlue,’ ‘Yes’ and ‘Absolutely,’ or ‘Maybe’ and ‘Maybe Not.’

Remember that if any other result would have told you something, then this result also tells you something, because it means the result that would have told you something did not happen. That doesn’t mean it helps you with any particular question.

Anyway, back to main thread.

Getting into a...
Some of the most important entrepreneurship skills are tacit knowledge
by Ruhul — 2022-09-20T12:06:48.390Z

Cross-posted from Future Startup.

Tacit knowledge or implicit knowledge—as opposed to formal, codified, or explicit knowledge—is knowledge that is difficult to express or extract, and thus more difficult to transfer to others by means of writing it down or verbalizing it. (Source)

Tacit knowledge is skills and knowledge that you learn intuitively or perceptually instead of active studying or training.

Active training helps, of course, but it’s hard to verbalize these skills—for instance, riding a bike or learning a particular branch of martial art. It is hard to learn martial art by merely reading a book about it. You have to go to a dojo and learn under a teacher. Most of these skills are more than cognitive intelligence. Your body learns how to ride a bike or do a particular kickboxing move.

I.

I have come to observe that many important entrepreneurship traits are hard to teach explicitly. You have to learn them intuitively or perceptually. Cedric Chin calls this “knowledge that cannot be captured through words alone.”

You can read about them. You can study people who have them. You can design courses, and apply psychological knowledge and techniques in teaching them, but it is hard to teach them merely theoretically.

While having a theoretical or verbal grasp of these skills is helpful, to truly learn these skills, you have to experience them and learn intuitively. I think that’s one of the reasons it takes multiple failures to get to significant entrepreneurial success or why it takes a long time to build fairly successful ventures.

Four entrepreneurial skills come to mind. These skills are similar to riding a bike. You can roughly describe how to ride a bike but you can't learn about biking just by reading the description. Description fails to teach biking.

1. Asking for help

2. Obsession

3. Doing things as default

4. High pain tolerance

Asking for help. Undoubtedly one of the most important founder skills. Building a company is hard. You need all the help you can get. Founder and CEO of Healthcare startup Praava Health Sylvana Q. Sinha told me in an interview a few years ago that asking for help is imperative for the success of a founder. She said:

"Nobody can build a company all by yourself. Building a company is a lonely journey, and in some ways as a founder, ultimately you are alone and it can be hard to communicat

...
Character alignment
by p.b. — 2022-09-20T08:27:35.556Z

When I introspect on why I do not kill more people, I find it has not very much to do with my goals. Mostly my goals would not be furthered by murder and some goals, like staying out of prison would be endangered, but if I just took the option seriously it wouldn't surprise me if murder would occasionally result in very nice payoffs.

I also don't think it has much to with my values, because frankly, I am not sure I have any. Or maybe I do, but I certainly never think about them. I have moral intuitions and I use those to solve moral quandaries mostly by finding analogies that trigger a clear cut moral intuition about what to do.

Sure, if somebody asked me whether my values include "freedom", "happiness" and "chocolate ice cream", I'd say yes, but it doesn't seem very helpful to frame my reasons for doing the things I do in terms of values - except maybe for propaganda purposes.

I think the best causal explanation, for why I don't tend to kill any people, lies in my character. I am just not a particularly violent person.

But maybe that's all the same? Maybe not being a violent person is equivalent to having the value of "peacefulness", and the goal of peaceful behavior.

I don't think that is the case. I can easily imagine having the same goals and values and being a much more violent man. Being a more violent man would probably result in more violent acts and much begging for forgiveness afterwards.

So what is a character trait, if it is separate from goals and values and can occasionally dictate behavior that violates both goals and values? To me many character traits make sense as priors over strategic behavior.

Neuroticism is the prior that the world is a dangerous place. Laziness the prior that conserving energy is of great importance. Openness corresponds to the exploration-exploitation tradeoff, while conscientiousness is the bet that care and attention to detail will pay off.

This is interesting because the underlying concepts are simple and relevant. Danger, speed, energy conservation, diligence, conflict, cooperation, etc: these are fundamental concepts in resource constrained multi-agent environments. Any sufficiently advanced agent trained in such environments will have a internal representation of the best guess of the current level of danger, for instance.

Given that if you encounter a sufficiently new and u...
Losing the root for the tree
by Adam Zerner — 2022-09-20T04:53:53.435Z

1

You know that being healthy is important. And that there's a lot of stuff you could do to improve your health: getting enough sleep, eating well, reducing stress, and exercising, to name a few.

There’s various things to hit on when it comes to exercising too. Strength, obviously. But explosiveness is a separate thing that you have to train for. Same with flexibility. And don’t forget cardio!

Strength is most important though, because of course it is. And there’s various things you need to do to gain strength. It all starts with lifting, but rest matters too. And supplements. And protein. Can’t forget about protein.

Protein is a deeper and more complicated subject than it may at first seem. Sure, the amount of protein you consume matters, but that’s not the only consideration. You also have to think about the timing. Consuming large amounts 2x a day is different than consuming smaller amounts 5x a day. And the type of protein matters too. Animal is different than plant, which is different from dairy. And then quality is of course another thing that is important.

But quality isn’t an easy thing to figure out. The big protein supplement companies are Out To Get You. They want to mislead you. Information sources aren’t always trustworthy. You can’t just hop on The Wirecutter and do what they tell you. Research is needed.

So you listen to a few podcasts. Follow a few YouTubers. Start reading some blogs. Throughout all of this you try various products and iterate as you learn more. You’re no Joe Rogan, but you’re starting to become pretty informed.

2

You’re a product manager over at Danslist, an up-and-coming competitor to Craigslist. You’ve recently been promoted and are now in charge of the most important page of the site: the product listings page.

This page is Danslist’s bread and butter. It’s very important that it is a good web page. And there’s a lot you can do to improve it. These improvements can be grouped into aesthetics, usability, functionality and speed.

Speed is something that the VP of Product has been talking a lot about. And you recently had a few meetings with the tech lead and some senior engineers. They had various ideas about how the speed could be improved. One is to parallelize the data fetching instead of doing it serially. Everyone agrees that this would be a good idea and a pretty obvious next step.

But in order to parallelize the fetching, we...