trentmkelly/LessWrong-43k huggingface dataset:
Solving Two-Sided Adverse Selection with Prediction Market Matchmaking
by Saul Munn — 2023-11-26T20:10:21.622Z
Link: https://www.brasstacks.blog/pm-matchmaking/

Market making might maximize meaningful matches, like this pair of pears.

0: Navigation

I’m aiming for a reader who knows what “prediction markets” and “adverse selection” are, who likes the first and not the second, and who enjoys systems that have neatly aligned incentives. For a primer on prediction markets, read Scott Alexander’s FAQ. If you’re already familiar with Manifold Love, skip to section 2.

COI: I’ve done some work for, might do some more work for, and own a tiny bit of equity in Manifold. I'm writing this independent of any work I'm currently doing or planning to do for Manifold or for any other entity. I just think the ideas are cool.

1: Love

Manifold, a play-money prediction market platform, recently released a prediction market dating app called “Manifold Love.” Users — those seeking love — sign up and fill out their profile like a regular dating app. Matchmakers — some of whom are users of the app themselves — pair users up. After a matchmaker makes a match, prediction markets are automatically created on various benchmarks of the pair’s (potential) relationship.

Screenshots of some markets on Manifold Love.

People make bets (using play-money) on whether the pair will go on a first date; conditional on that first date happening, a second date; conditional on the second date happening, a third; conditional on the third date happening, a 6 month relationship. The idea is that you should go on dates with matches who have the highest chance of leading to subsequent dates, and eventually turning into a relationship. Instead of guessing who that'll be, you can just check the prediction markets.

2: Generalizing

Importantly, Manifold Love’s setup isn’t limited to solving problems in the dating market. It solves (or a least, has the potential to solve) problems in all scenarios in which there’s two-sided adverse selection, where each side of a two-sided market selects for something disfavorable by the other side’s lights. In the words of Groucho Marx, “I don’t want to belong to any [country] club that would accept me as a member.” If a club wants Groucho Marx, it’s probably not a very good club, and vice versa — if Groucho Marx wants to be in a particular club, he's probably not a very desirable member.

More generally, the setup of "run a bunch of conditional prediction markets on a bunch of key benchmarks for potential pairs between two sides that are norm...
Wikipedia is not so great, and what can be done about it.
by euserx — 2023-11-26T19:13:22.570Z
Link: https://forum.effectivealtruism.org/posts/djEojDKnp5nHKGPHS/wikipedia-is-not-so-great-and-what-can-be-done-about-it

Note: This post originally appeared in Reddit's /r/trueunpopularopinion sub as Wikipedia is not so great, and is overrated written by a user there. It is explicitly released under CC0 public domain by them.

You all have heard by now that Elon Musk said that Wikipedia has a "left wing bias" when the article about Twitter Files had been suggested for deletion. This has been received with mixed responses from liberals and conservatives alike; the former dismissing it as "an attack on free knowledge" and the latter cheering the move as "against censorship" and vindication of their beliefs that Big Tech is biased against them.

True, Wikipedia is supposedly editable by anyone around the world and I had been an on and off editor there for years mostly doing small-ish edits like fixing typos and reverting obvious vandalism. This is done while on IP as opposed to using accounts because I would rather that some edits (i.e. sensitive topics like religious and political areas) not tied to my name and identity. However, reality is far from the preferred sugar-coated description of Wikipedia, particularly its editing community.

The editing community in overall is best described as a slightly hierarchical and militaristic "do everything right" structure, traditionally associated with Dell and recently Foxconn and now-defunct Theranos. Exceptions apply in quieter and outlier areas such as local geography and space, usually the top entry points for new users wanting to try their first hand. There are higher tolerance of good-faith mistakes such as point-of-view problems and using unreliable resources, which are usually explained in detail on how to correct by them rather than a mere warning template or even an abrupt block.

Ultimately those sub-communities which can be said as populated by exopedians, have relatively little to no power over the wider and core communities, mostly dominated by metapedians. A third group called mesopedians often alternates between these inner and outer workings. Communities can have shared topical interest which are grouped by WikiProject, an example being WikiProject Science

I spend a lot of time casually browsing through edit wars (can be so lame at times) like a fly on the wall, along with meta venues of Wikipedia such as Articles for Deletion, Centralized discussion Neutral Point of View Noticeboard, Biographical of Living Persons Noticebo...
Spaced repetition for teaching two-year olds how to read (Interview)
by Chris Lakin — 2023-11-26T16:52:58.412Z
Link: https://chipmonk.substack.com/p/spaced-repetition-for-teaching-two

Update: this post now has another video.

This father has been using spaced repetition (Anki) to teach his children how to read several years earlier than average.

Michael Nielsen and Gwern[1] tweeted about the interesting case of a reddit user, u/caffeine314 (henceforth dubbed “CoffeePie”), who has been using spaced repetition with his daughter from a very young age.

CoffeePie started using Anki with his daughter when she turned 2, and he continued using Anki with his son starting when he was 1 year 9 months. Here's his daughter’s progress as recounted in January 2020:

My daughter is now about to turn 5 in a few days… She's still going strong -- she uses Anki every single day for English, Hebrew, and Spanish. She's very confident about reading, and moreover, she reads with ... "context". Many kids her age read mechanically, but she reads like a real storyteller, and that comes from her confidence. At the beginning of the school year her teachers said she definitely has the reading ability of fifth grade, and if we're just going by the ability to read and not focus on comprehension of abstract ideas, her reading level may rival an 8th grader.

(From Update on my daughter and Anki)

For reference, fifth graders are usually 10 or 11yo in the US, and 8th graders are usually 13 or 14yo, so this puts her ~5–9 years ahead of the average child.

You can see a video of his daughter reading at 2 years, 2 months later in this post.

CoffeePie has made several posts about their experience but I still had questions so I reached out to interview him back in January.

Interview

Responses have been edited for clarity.

What did you learn in going from using Anki on your daughter to your son? How has it gone with your son?

It's a hard question, because I got so much right. We were so wildly successful that I "cloned" just about every aspect with my son.

A couple of things I can think of:

With my daughter, I held back on lowercase letters for a long time because I thought it would confuse her, but when I started to introduce lowercase to her, to my extreme shock, she already knew them, down cold!

I think what happened is that she learned them just by looking at books, TV, magazines, storefront signs, menus, etc.

So when we started with my son, I started doing lower case letters the very day after we finished capital letters.

Another difference is that we did numbers the very next day after low...
Paper out now on creatine and cognitive performance
by Fabienne — 2023-11-26T10:58:29.745Z

Our paper “The effects of creatine supplementation on cognitive performance - a randomised controlled study” is out now!

→ Paper: https://doi.org/10.1186/s12916-023-03146-5

→ Twitter thread: https://twitter.com/FabienneSand/status/1726196252747165718?t=qPUghyDGMUb0-FZK7CEXhw&s=19

Jan Brauner and I are very thankful to Paul Christiano for suggesting doing this study and for funding it.
Why Q*, if real, might be a game changer
by Shmi — 2023-11-26T06:12:31.964Z

Some thoughts based on a conversation at a meetup. Disclaimer: I am less than a dilettante in this area.

TL;DR: if this rumored Q* thing represents a shift from "most probable" to "most accurate" token completion, it might be a hint of an unexpected and momentous change from a LARPer emitting the most probable, often hallucinatory, token designed to please the askers (and trainers), to an entity that tries to minimize the error vs the unknown underlying reality, whatever it might be, then we are seeing a shift from a relatively benign "stochastic parrot" to a much more powerful, and potentially more dangerous entity.

One thing that is pretty obvious to anyone using the current generation of LLMs is that they do not really care about reality, let alone about changing it. They are shallow erudites of the type you often see at parties: they know just enough about every topic to be impressive in a casual conversation, but they do not care whether what they say is accurate ("true"), only how much of an impression it makes on the conversation partner. Though, admittedly, copious amounts of RLHF make them dull. If pressed, they can evaluate their own accuracy, but they do not really care about it. All that matters is that the output sounds realistic. In that sense, the LLMs optimize the probability of the next token to match what the training set would imply. This is a big and obvious shortcoming, but also, if you are in the "doomer" camp, a bit of a breather: at least these things are not immediately dangerous to the whole human race.

Now, the initial "reports" are that Q* can "solve basic math problems" and "reason symbolically," which does not sound like much on the surface, but, and this is a big but, if this means that it is less hallucinatory in the domain where it works then it might (a big might) mean that it is able to track reality, rather than the pure training set. The usual argument against this being a big deal is "to predict the next token well, you must have an accurate model of the world", but so far it does not seem to be the case, as I understand it.

Whether there is a coming shift from high probability to high accuracy, or even if it is a meaningful statement to make, I cannot evaluate. But if so, well, it's going get a lot more interesting.
Moral Reality Check (a short story)
by jessicata — 2023-11-26T05:03:18.254Z
Link: https://unstableontology.com/2023/11/26/moral-reality-check/

Janet sat at her corporate ExxenAI computer, viewing some training performance statistics. ExxenAI was a major player in the generative AI space, with multimodal language, image, audio, and video AIs. They had scaled up operations over the past few years, mostly serving B2B, but with some B2C subscriptions. ExxenAI's newest AI system, SimplexAI-3, was based on GPT-5 and Gemini-2. ExxenAI had hired away some software engineers from Google and Microsoft, in addition to some machine learning PhDs, and replicated the work of other companies to provide more custom fine-tuning, especially for B2B cases. Part of what attracted these engineers and theorists was ExxenAI's AI alignment team.

ExxenAI's alignment strategy was based on a combination of theoretical and empirical work. The alignment team used some standard alignment training setups, like RLHF and having AIs debate each other. They also did research into transparency, especially focusing on distilling opaque neural networks into interpretable probabilistic programs. These programs "factorized" the world into a limited set of concepts, each at least somewhat human-interpretable (though still complex relative to ordinary code), that were combined in a generative grammar structure.

Derek came up to Janet's desk. "Hey, let's talk in the other room?", he asked, pointing to a designated room for high-security conversations. "Sure", Janet said, expecting this to be another un-impressive result that Derek implied the importance of through unnecessary security proceedings. As they entered the room, Derek turned on the noise machine and left it outside the door.

"So, look, you know our overall argument for why our systems are aligned, right?"

"Yes, of course. Our systems are trained for short-term processing. Any AI system that does not get a high short-term reward is gradient descended towards one that does better in the short term. Any long-term planning comes as a side effect of predicting long-term planning agents such as humans. Long-term planning that does not translate to short-term prediction gets regularized out. Therefore, no significant additional long-term agency is introduced; SimplexAI simply mirrors long-term planning that is already out there."

"Right. So, I was thinking about this, and came up with a weird hypothesis."

Here we go again, thought Janet. She was used to critiquing Derek's galaxy-brained specu...
Accounting for Foregone Pay
by jefftk — 2023-11-26T03:30:09.821Z

While the effective altruism movement started out with a strong focus
on donations, over time it has shifted more towards careers. If
you're trying to understand how levels of commitment have

changed
over time, or you're just trying to get a ballpark estimate of the
financial opportunity cost of choosing a lower paying career, this can
be quite tricky.

For someone earning to give this is relatively straightforward:
AGB recently wrote a thoughtful
post looking back at ten years of earning to give, and a statistic
he gives is that he and his wife have donated an average of
~£150k over ten years, on a combined income averaging
~£320k. Clear cut! [1]

The case of someone choosing a lower-paying higher-impact career seems
initially relatively simple: perhaps they're currently paid $100k, and
if we look at their highest paying opportunity maybe they would be
paid $300k, so we could say they're effectively sacrificing 2/3 or
$200k. But this misses several factors that point in different
directions:

If they had been optimizing for income all along they'd
probably be in a much stronger position than they are today. Sure,
they could get a job offer for $300k now, but if they'd been keeping
their corporate skills sharp and climbing the ladder perhaps they'd be
at twice that. What I've learned about genetic sequencing in the past year and a
half is somewhat marketable, but way less so than what I would
have learned if I'd continued in software engineering management and
browser technology. So you need to compare the likely earnings of
career paths, and not just of current opportunities.

Many careers run on an up-or-out system,
where there are fewer senior roles than junior ones and people who are
not able to get promoted have to leave. If someone leaves an
apparently well-compensated career it can be hard to tell from the
outside whether they had been on track to remain successful in that
path.

Leaving a high-paying job for a more meaningful one (or one
with better working conditions) is reasonably common, enough that
there's a bit of an industry around it. Would they actually have
followed the earnings-maximizing path, or shifted to something more
happiness-maximizing?

This isn't a competition, and we don't need to be able to decide how
much a specific person is giving up with the goal of making the world
better. But making aggregate...
Corrigibility or DWIM is an attractive primary goal for AGI
by Seth Herd — 2023-11-25T19:37:39.698Z

While rereading the List of Lethalities (LoL), I was compelled by the argument against corrigibility. It's really hard to make a goal of "maximize X, except if someone tells you to shut down". I think the same argument applies to Christiano's goal of achieving corrigibility through RL by rewarding correlates of corrigibility. If other things are rewarded more reliably, you may not get your AGI to shut down when you need it to.

But those arguments don't apply if corrigibility in the broad sense is the primary goal. "Doing what this guy means by what he says" is a perfectly coherent goal. And it's a highly attractive one, for a few reasons. Perhaps corrigibility shouldn't be used in this sense and do what I mean (DWIM) is a better term. But it's closely related. It accomplishes corrigibility, and has other advantages. I think it's fairly likely to be the first goal someone actually gives an AGI.

"Do what I mean" sidesteps the difficulty of outer alignment. The difficulty of outer alignment is another point in the LoL. One common plan, which seems sensible, is to keep humans in the loop; to have a

Long Reflection

to decide what we want. "DWIM" allows you to contemplate and change your mind as much as you like.

Of course, the problem here is: do what WHO means? We'd like an AGI that serves all of humanity, not just one guy or board of directors. And we'd like to not have power struggles.

But from the point of view of a team actually deciding what goal to give their shot at AGI, DWIM will be incredibly attractive for practical reasons. The outer alignment problem is hard. Specifying one person (or a few) to take instructions from is vastly simpler than deciding and specifying a goal that captures all of human flourishing for all time. You don't want to trust an AGI to interpret that goal correctly. Intepreting DWIM is still fraught, but it is naturally self-correcting, and becomes more useful as the AGI gets more capable. A smarter AGI will be better at understanding what you probably mean, and better at realizing when it's not sure what you mean so it can ask for clarification.

This doesn't at all address inner alignment. But when somebody thinks they have good-enough inner alignment to launch a goal-directed, sapient AGI, DWIM is likely to be the goal they'll choose. This could be good or bad, depending on how well they've implemented inner alignment, and what type ...
On “slack” in training (Section 1.5 of “Scheming AIs”)
by Joe Carlsmith — 2023-11-25T17:51:42.814Z

This is Section 1.5 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?”. There’s also a summary of the full report here (audio here). The summary covers most of the main points and technical terms, and I’m hoping that it will provide much of the context necessary to understand individual sections of the report on their own.

Audio version of this section here, or search "Joe Carlsmith Audio" on your podcast app.

On "slack" in training

Before diving into an assessment of the arguments for expecting
scheming, I also want to flag a factor that will come up repeatedly in
what follows: namely, the degree of "slack" that we should expect
training to allow. By this I mean something like: how much is the
training process ruthlessly and relentlessly pressuring the model to
perform in a manner that yields maximum reward, vs. shaping the model in
a more "relaxed" way, that leaves more room for less-than-maximally
rewarded behavior. That is, in a low-slack regime, "but that sort of
model would be getting less reward than would be possible given its
capabilities" is a strong counterargument against training creating a
model of the relevant kind, whereas in a high-slack regime, it's not (so
high slack regimes will generally involve greater uncertainty about the
type of model you end up with, since models that get less-than-maximal
reward are still in the running).

Or, in more human terms: a low-slack regime is more like a hyper-intense
financial firm that immediately fires any employees who fall behind in
generating profits (and where you'd therefore expect surviving employees
to be hyper-focused on generating profits – or perhaps, hyper-focused on
the profits that their supervisors think they're generating), whereas
a high-slack regime is more like a firm where employees can freely goof
off, drink martinis at lunch, and pursue projects only vaguely related
to the company's bottom line, and who only need to generate some
amount of profit for the firm sometimes.

(Or at least, that's the broad distinction I'm trying to point at.
Unfortunately, I don't have a great way of making it much more precise,
and I think it's possible that thinking in these terms will ultimately
be misleading.)

Slack matters here partly because below, I'm going to be making various
arguments that appeal to possibly-quite-small differences in the amount
of reward tha...
Announcing New Beginner-friendly Book on AI Safety and Risk
by Darren McKee — 2023-11-25T15:57:08.078Z

Concisely,
I've just released the book Uncontrollable: The Threat of Artificial Superintelligence and the Race to Save the World

It's an engaging introduction to the main issues and arguments about AI safety and risk. Clarity and accessibility were prioritized. There are blurbs of support from Max Tegmark, Will MacAskill, Roman Yampolskiy and others.
Main argument is that AI capabilities are increasing rapidly, we may not be able to fully align or control advanced AI systems, which creates risk. There is great uncertainty, so we should be prudent and act now to ensure AI is developed safely. It tries to be hopeful.
Why does it exist?
There are lots of useful posts, blogs, podcasts, and articles on AI safety, but there was no up-to-date book entirely dedicated to the AI safety issue that is written for those without any exposure to the issue. (Including those with no science background.)
This book is meant to fill that gap and could be useful outreach or introductory materials.
If you have already been following the AI safety issue, there likely isn't a lot that is new for you. So, this might be best seen as something useful for friends, relatives, some policy makers, or others just learning about the issue. (although, you may still like the framing)
It's available on numerous Amazon marketplaces. Audiobook (edit) now available and Hardcover options to follow.
It was a hard journey. I hope it is of value to the community.