trentmkelly/LessWrong-43k huggingface dataset:
Research agenda - Building a multi-modal chess-language model
by p.b. — 2022-04-07T12:25:18.502Z

This is one of the posts that detail my research agenda, which tries to marry Deep Learning and chess to gain insight into current AI technologies.

How "the world", "what the model knows about the world" and "what the models says about the world" hang together seems to me to be a core question for prosaic alignment. In chess the state of the world can be automatically analysed by powerful engines.

This extends to the detection of high level concepts, like zugzwang, initiative, king safety, development, corresponding squares, etc. Therefore a multi-modal chess language models truthfulness and symbol grounding capabilities could be automatically quantified on several levels of complexity.

Of course it would also just be awesome.

There is current research into how to create chess comments. The probably best of such tries, Chis Butner's amazing project "Chess Coach", gives a nice overview.

Generally speaking, the quality of these generated comments is low. They are frequently nonsensical or only loosely connected to the position. To a large part, this must be due to the small dataset sizes, which range from 300,000 to 1 million position/comment pairs. But another problem is that the existing datasets contain mostly very low quality comments by weak players.

There are several avenues to improve these models:

Pretraining is a big one:

I plan on using an encoder-decoder architecture that encodes chess positions and decodes into natural language.

The encoder will be pretrained on chess move prediction and possibly several other pretraining tasks like predicting the outcome, predicting an engine score, predicting future piece occupation of squares, predicting the length of the game, predicting mate, etc.

The decoder will be pretrained (or finetuned) on natural language as chess related as I can make the dataset. The idea is to plug an encoder that knows chess very well, into a decoder that is already proficient at making grammatical and non-selfcontradictory statements.

However, training this pretrained architecture on only several hundred thousand position-comment pairs is probably not going to be enough. So a key part of my research roadmap is to create a new and much larger dataset.

Partly, this will be synthetic. Connecting moves and squares and pieces and checks and mates and stalemates with the correct phrase and vocabulary is not ...
Predicting a global catastrophe: the Ukrainian model
by RomanS — 2022-04-07T12:06:32.804Z

The large-scale Russian invasion into Ukraine provides an interesting model of how governments, pundits, prediction markets, and the common folks would predict an imminent global catastrophe.

Prediction markets

The Metaculus community successfully predicted that Russia will invade Ukraine:

>50% at 2022-01-18 (1 month before the invasion)
>75% at 2022-02-19 (5 days before)

If I remember correctly, GJP superforcasters were similarly successful, although they were a bit slower.

Pundits

I have an impression that the vast majority of pundits failed to predict it, including Matthew Yglesias and Scott Alexander.

A somewhat funny example is an acquaintance of mine, whom I'll not name for privacy reasons. The guy was very motivated to make the right prediction, he has an excellent track record on Metaculus and a deep understanding of the Russian politics, and he is living in Russia. Nevertheless, he was confidently wrong until the day before the invasion.

On the other hand, there is Oleksiy Arestovych who predicted the war in 2019, including detailed and surprisingly correct predictions of the Russian military movements. Before that, he successfully predicted the Russian invasion of 2014.

Governments

According to Arestovych (who is a military adviser to the president of Ukraine), the Ukrainian leadership knew that Russia will invade, and knew the date of the invasion. But the leadership decided to keep it from the general public for military reasons (to prevent a mass panic that would paralyze preparations for the war) .

The NATO and the US leadership have issued many public warnings about the imminent war.

The general public

The warnings were mostly dismissed or even ridiculed by common people, in both Russia and Ukraine.

According to a poll conducted 3 days before the invasion, only 35% of Ukrainians believed that Russia will invade.

On the other hand, some top Ukrainian politicians and billionaires (and their families) left Ukraine well in advance, some - already in January, more than 1.5 months before the invasion. I think that the most likely explanation for that is a leaked insider info.

Some lessons

We could try to extrapolate the data to the case of an imminent global catastrophe. For example, an imminent large asteroid impact or a nuclear war.

If prediction markets say that the global catastr

...
Truthfulness, standards and credibility
by Joe Collman — 2022-04-07T10:31:53.103Z

-1: Meta Prelude

While truthfulness is a topic I’ve been thinking about for some time, I’ve not discussed much of what follows with others. Therefore, at the very least I expect to be missing important considerations on some issues (where I’m not simply wrong).

I’m hoping this should make any fundamental errors in my thought process more transparent, and amenable to correction. The downside may be reduced clarity, more illusion-of-transparency…. Comments welcome on this approach.

I don’t think what follows is novel. I’m largely pointing at problems based on known issues.
Sadly, I don’t have a clear vision of an approach that would solve these problems.

0: Introduction

…our purpose is not to give the last word, but rather the first word, opening up the conversation… (Truthful AI)

I’d first like to say that I believe some amount of research on truthfulness to be worthwhile, and to thank those who’ve made significant efforts towards greater understanding (including, but not limited to, the authors of

Truthful AI

(henceforth TruAI)).

No doubt there’s some value in understanding more, but my guess is that it won’t be a particularly fruitful angle of attack. In all honesty, it seems an inefficient use of research talent to me - but perhaps I’m missing something.

Either way, I hope the following perspective will suggest some useful directions for conversation in this area.

[Note: section numbers refer to this document unless “TruAI…” is specified]
[I'll be assuming familiarity with TruAI throughout, though reading the full paper probably isn't necessary so long as you've seen the executive summary in the post]

My current belief is that near-term implementation of the kind of truthfulness standards talked about in TruAI would be net negative, for reasons I’ll go on to explain. To me it seems as if we’d be implementing a poor approximation to a confused objective.

A high-level summary of my current view:

Narrow truthfulness looks approachable, but will be insufficient to prevent manipulation.
Broad truthfulness may be sufficient, but is at least as hard as intent alignment.
Truthfulness amplification won’t bridge the gap robustly.
Achieving increased trust in narrow truthfulness may lead to harm through misplaced trust in broad truthfulness.
Achieving narrow truthfulness may simply move the harm outside its scope.

For much of what follows the l...
How to train your transformer 
by p.b. — 2022-04-07T09:34:52.465Z

In this blogpost I describe a transformer project, how I proceeded, the results, the pitfalls and more. At the end of the post I will repeat all salient points in a TL;DR, so if you are short of time, just skip ahead.

I originally published this on a company blog. The blog has been down for the last couple of months and seems unlikely to be up again any time soon. I republish it here, to give context and additional detail for my research agenda into transformer capabilities.

The transformer is currently the hottest Deep Learning model. Introduced in 2017, it was initially conceived to give Google translate a huge quality boost, but quickly turned out to be useful for many more application besides machine translation.

The two main workhorses of Deep Learning used to be convolutional neural networks for image processing and any data that has a vaguely similar format and LSTMs for sequence processing or any data that can, sensibly or not, be interpreted as a sequence.

The story of the last four years has been one of transformers or transformer-inspired architectures (look for the key words “attention based”), slowly out competing ConvNets and LSTMs in one domain after another. I already mentioned machine translation, but any kind of language modelling achieved a huge boost in quality by switching to the transformer architecture.

NLP, “natural language processing”, finally had its “image net moment”, when the transformer-based BERT-model turned out to just be better at everything NLP than anything else. Automatic text generation of any form and kind (satire, poetry, python-code, dungeons and dragons, …) was made possible by GPT-2 and recently GPT-3.

In one of the most exciting scientific breakthroughs of recent history, Google subsidiary Deepmind solved the protein folding problem (for certain definitions of “solved”). One key difference between the protein folding programs Alphafold2 (solved protein folding) and Alphafold1 (was pretty good) … you guessed it: a transformer-inspired building block.

Other advances achieved with transformers include CLIP, an open-ended image recognizer and DALL-E, a model that can create images from a short textual description.

Interestingly, the transformer model is not naturally a sequence model or an image model. Instead, the attention mechanism that lies at its core is an operation over s...
Finding Useful Things
by Johannes C. Mayer — 2022-04-07T05:57:47.058Z

There is a very important problem that seems to have no real solution. How do you find useful things that you don't yet know about? This is sort of the problem of unknown unknowns, but not quite. That is only the first part. Once you have learned of the existence of a potentially useful thing, it is often not easy to evaluate if it is actually useful. Especially once you take into account all the time that you would need to invest into the thing before it can become useful.

The best way to evaluate if a thing is useful is to just learn the thing. I don't mean learn about the thing, which is of course also necessary, but to learn the actual thing.

For example, let's assume you want to improve how well you can visually imagine things. If you want to evaluate how much learning to draw can help you with mental visualization, then the best evidence you can gather is to learn drawing, and see for yourself. Ideally, make predictions beforehand of how you would expect this to change your ability to visualize.

However, even if you only want to evaluate the intermediate drawing skill level, this will take hundreds of hours.

When investing so much time you need to watch for the Sunk-Cost Fallacy and Rationalisations. After investing hundreds of hours, it might be hard to admit that they were all wasted. But of course, the problem is that ideally, we want to make a time investment only if it would be worth it.

Let's briefly step back and consider the first part of the problem again. How can you learn about important things? Even if someone thinks of googling "What should I do with my life?", they won't get any of the things that I right now think were important for me to find. If you google this phrase, there is nothing about AI alignment, nothing about 80,000 hours, and nothing about effective altruism. I am not saying that what you get is all bad. I actually think I got at least one good resource talking about what makes work satisfying. And I am not saying that everybody would necessarily end up finding those resources that I named to be valuable. I just want to make the point that finding the things that I would now consider high-quality resources is probably more difficult than one quick google search. And before I knew about them I did not even try a google search.

I learned about AI alignment by watching random YouTube videos, until I came across one with Sam Harris on...
Book Review: A PhD is Not Enough
by ljh2 — 2022-04-07T05:15:51.095Z

Preface

I want a PhD before I die.

Why? Same reason(s) other people might. Something something passion.

Cynical reasons aside-- nothing else useful to do with the major; it's a bureaucratic checkmark; I want Tenure and coast the rest of my life-- people want to have impact in their work and contribute to the world, and a PhD is what many view as an avenue to do that.

And thus, Feibelman knocks these starry-eyed graduates down a peg-- It's Not Enough. "Enough" to do what? To Do Great Things, Peter J. Feibelman answers.

Well, I paraphrased. Here's his direct answer (emphasis mine)--

"This book is meant for those who will not be lucky
enough to find a mentor early, for those who naively
suppose that getting through graduate school, doing a
postdoc, etc., are enough to guarantee a scientific ca-
reer. I want you to see what stands between you and a
career, to help you prepare for the inevitable obstacles
before they overwhelm you." -- Preface p. xiii

Or, as I interpret it, you won't change the world with just a PhD. Hence, our eponymous title.

My immediate questions as I read this were the following-- who are you, and why should I keep reading this book?

Peter J. Feibelman is a rather renowned (retired?) physicist in the materials science world. Here is his Google scholar profile, and you can immediately tell he's made some significant inroads to... honestly I'm not entirely sure. Graphene research? I'm not a materials science expert. Furthermore, he is rather well honored too by the American Institute of Physics as well.

The takeaway here is that Feibelman is a very accomplished, intelligent researcher who has spent decades uplifting the materials science world from his corner at Sandia National Laboratories.

But he's no von Neumann. No Terrence Tao, or Gauss. He is not a generational genius who very likely would have succeeded regardless of academia and makes discoveries while placing gold crowns in bathtubs.

But, just because you haven't heard of him, does that mean he didn't do some Great Things (TM)? Do you think he was some no-name researcher who didn't advance the state of the world and push our understanding of X thing?

If you do, then this review will be pointless for you, because the biggest appeal of the book is its approachability and reality.

We stand on the shoulders of giants, but you don't have to be an actual gi...
Setting the Brains Difficulty-Anchor
by Johannes C. Mayer — 2022-04-07T05:04:54.411Z

Epistemic Status: Based on my first-hand experience. It is possible that I overestimate the strength of the effect, but I feel fairly confident that it exists.

I often observe, that when I do something that I find very difficult for some time, I get used to the difficulty. Importantly, "getting used to" seems to be a global attribute of my mind. When I switch to another easier activity, that activity seems easier compared to how easy it seemed, before doing the difficult activity. Normally easier also corresponds to the activity being more engaging. Probably because any progress you make seems to require less effort compared to the difficult task. This in turn means that you will make faster progress when working on the easier task than you would have otherwise.

I think of this as setting a difficulty anchor. You do something difficult, and if you do it for long enough then your brain is using the difficulty of that task to evaluate the difficulty of other tasks. The same effect seems to also apply to easy tasks. Playing computer games for very long will make other things seem harder because it will be harder to make progress/dopamine. After all most games are optimized for the player experiencing lots of progress/dopamine.

The difficulty I am talking about is the difficulty in terms of how hard you need to think. It is difficult, to never be late, or to shoot someone in the head in Counter-Strike (if the other player is competent). But that seems different. Never being late is mainly difficult because of things outside of your control. And once you are good at Counter-Strike, what your brain does might still be complex, but in some sense, it is not difficult. If you are competent, you already embedded the complex algorithms that are necessary to play the game in your neural network, and you can simply execute them. Only tweaking them slightly, depending on the specific situation you are in.

I am talking about the sense of difficulty, in which learning mathematics is difficult. Most of the time when you learn math (somewhat depending on your style), you learn new concepts, and how these concepts relate to other concepts. And this seems to strain the brain a lot.

Probably learning new things is one of the most difficult things for the brain. In my experience, learning how to draw, or learning a language, seem difficult in the same sense. However, at least for lear...
How I Got So Much GHT
by Gordon Seidoh Worley — 2022-04-07T03:59:36.538Z

With Good Heart Week nearing its end and my schedule unlikely to let me write another post, I have one last one to share about how I managed to earn as much GHT as I did.

As of this writing I'm in the top 5. I've been as high as #3 but currently I'm sitting at #4. How did I do it?

Well, to start off, when we got the initial announcement on April 1st I tried to think of something that would be both a fun use of Good Heart Tokens and would earn me a lot of them. The best idea I came up with after a few minutes of thinking was a donation lottery.

That was all in good fun for April Fool's Day, but then when GHT got extended to a week long I had to come up with some plan to produce real posts if I wanted to earn GHT. Since I had committed the lottery to pay out at least $1000 regardless of what happened, I now had an incentive: earn as much GHT as possible to offset the amount I was personally going to have to donate!

I wrote a shortform post asking folks for ideas on things they might like to see me write. I didn't get any bites.

Later in the day, I thought to myself "what the heck, this is a week for writing things I wouldn't otherwise feel were worth the effort to write, but given the changed incentives we've cross the threshold where, at least for a week, I'm willing to spend time writing things that are less likely to be novel or enduring in exchange for GHT".

That said, I didn't want to spend a lot of time writing since I still have to do my day job. "Oh, hey!" went my brain, "what if I wrote about my day job!".

So I ended up doing just that. Most of the posts I wrote this week were on the subject, either directly or obliquely:

Becoming a Staff Engineer
How Real Moral Mazes (in Bay Area startups)?
Software Engineering: Getting Hired and Promoted
You get one story detail

The last one isn't explicitly about software engineering, but it does reflect a lot of what I've learned about communicating effectively in a large organization.

I tried to think of other things to write about. I came up with two other ideas of things that I've talked about at various times but didn't recall having written down:

My Superpower: OODA Loops
My Recollection of How This All Got Started

I also posted one question to try to get some additional conversations going, but so far it hasn't gotten much traction:

What are rationalists worst at?

In all cases I followed a few rules with these posts:

Write abo

...
What Should We Optimize - A Conversation
by Johannes C. Mayer — 2022-04-07T03:47:42.439Z

The following is a lightly edited chat conversation from 2022-04-07. My interlocutor requested to be anonymized as Jason. This conversation sprang out of Jason reading about my transhuman dream.

(The conversation starts out a bit slow, but overall covers a lot of ground.)

Jason:
Maxing positive experiences, and minimizing negative experiences seems misguided. I am convinced that positive experience is purely defined by its contrast to negative experience. Without negative experience, the positive experience is just what you do, it's normal.

I think a better goal would be to minimize suffering. After all, people even like negative experiences (watch horror movies), but nobody likes suffering.

Maybe that’s what you meant all along, and I just interpreted the statement too literally.

Johannes:
You don't think about bad things if you have a positive experience, like laughing hard, or having an orgasm. Also after your logic, minimizing suffering seems wrong. It seems more like you should be in favor of everybody having at least one intensely negative experience. And then only after that, you would minimize suffering.

Jason:
Positive/negative experience is more like positive/negative pressure, not charge. Let me propose a model:

We rate experiences not by an absolute measure, but by an implicitly defined (relative) ordering among them. The things in the bottom half are what we call negative experiences, and the upper half are the positive ones (roughly speaking).

This ordering seems to be defined subconsciously. So you may not consciously think "Man, cumming is so much better than driving to work", but that thought may be happening subconsciously: The computation that tells you that this is a positive experience. A table lookup if you will. Sleep or other processing of experience shapes the ordering/table.

So like with pressure, if you raise the pressure ("remove" the "negative" experience), you're just left with a smaller range, which still has a bottom and top half.

And we know this to be true experimentally (hedonic treadmill).

So what about suffering? I think in this model, suffering is a subset of the negative experience. Just defining it as the worst negative experience doesn't seem complete. Suffering seems like an orthogonal metric (though less correlated with "positive" experience).

I don't understand your point about "after your logic minimizing suffering seems wron...