trentmkelly/LessWrong-43k huggingface dataset:
Ruining an expected-log-money maximizer
by philh — 2023-08-20T21:20:02.817Z

Suppose you have a game where you can bet any amount of money. You have a 60% chance of doubling your stake and a 40% chance of losing it.

Consider agents Linda and Logan, and assume they both have £11. Linda has a utility function that's linear in money (and has no other terms), ULinda(m)=m. She'll bet all her money on this game. If she wins, she'll bet it again. And again, until eventually she loses and has no more money.

Logan has a utility function that's logarithmic in money, ULogan(m)=ln(m). He'll bet 20% of his bankroll every time, and his wealth will grow exponentially.

Some people take this as a reason to be Logan, not Linda. Why have a utility function that causes you to make bets that leave you eventually destitute, instead of a utility function that causes you to make bets that leave you rich?

In defense of Linda

I make three replies to this. Firstly, the utility function is not up for grabs! You should be very suspicious any time someone suggests changing how much you value something.

"Because if Linda had Logan's utility function, she'd be richer. She'd be doing better according to her current utility function." My second reply is that this is confused. Before the game begins, pick a time t. Ask Linda which distribution over wealth-at-time-t she'd prefer: the one she gets from playing her strategy, or Logan's strategy? She'll answer, hers: it has an expected wealth of £1.2t. Logan's only has an expected wealth of £1.04t.

And, at some future time, after she's gone bankrupt, ask Linda if she thinks any of her past decisions were mistakes, given what she knew at the time. She'll say no: she took the bet that maximized her expected wealth at every step, and one of them went against her, but that's life. Just think of how much money she'd have right now if it hadn't! (And nor had the next one, or the one after….) It was worth the risk.

You might ask "but what happens after the game finishes? With probability 1, Linda has no money, and Logan has infinite". But there is no after! Logan's never going to stop. You could consider various limits as t→∞, but limits aren't always well-behaved2. And if you impose some stopping behavior on the game - a fixed or probabilistic round limit - then you'll find that Linda's strategy just uncontroversially gives her better payoffs (according to Linda) after the game than Logan's, when her probability of being bankrupt i...
Steven Wolfram on AI Alignment
by Bill Benzon — 2023-08-20T19:49:28.953Z

Joe Walker has a general conversation with Wolfram about his work and things and stuff, but there are some remarks about AI alignment at the very end:

WALKER: Okay, interesting. So moving finally to AI, many people worry about unaligned artificial general intelligence, and I think it's a risk we should take seriously. But computational irreducibility must imply that a mathematical definition of alignment is impossible, right?

WOLFRAM: Yes. There isn't a mathematical definition of what we want AIs to be like. The minimal thing we might say about AIs, about their alignment, is: let's have them be like people are. And then people immediately say, "No, we don't want them to be like people. People have all kinds of problems. We want them to be like people aspire to be.

And at that point, you've fallen off the cliff. Because, what do people aspire to be? Well, different people aspire to be different and different cultures aspire in different ways. And I think the concept that there will be a perfect mathematical aspiration is just completely wrongheaded. It's just the wrong type of answer.

The question of how we should be is a question that is a reflection back on us. There is no "this is the way we should be" imposed by mathematics.

Humans have ethical beliefs that are a reflection of humanity. One of the things I realised recently is one of the things that's confusing about ethics is if you're used to doing science, you say, "Well, I'm going to separate a piece of the system," and I'm going to say, "I'm going to study this particular subsystem. I'm going to figure out exactly what happens in the subsystem. Everything else is irrelevant."

But in ethics, you can never do that. So you imagine you're doing one of these trolley problem things. You got to decide whether you're going to kill the three giraffes or the eighteen llamas. And which one is it going to be?

Well, then you realise to really answer that question to the best ability of humanity, you're looking at the tentacles of the religious beliefs of the tribe in Africa that deals with giraffes, and this kind of thing that was the consequence of the llama for its wool that went in this supply chain, and all this kind of thing.

In other words, one of the problems with ethics is it doesn't have the separability that we've been used to in science. In other words, it necessarily pulls in everything, and we don't get to s

...
What value does personal prediction tracking have? 
by fx — 2023-08-20T18:43:58.059Z

I've seen a lot of people on LW use tools like Predictionbook or Fatebook for tracking personal predictions in their daily life. However, I wasn't able to find any information on specific reasons to do that. Fatebook lists a few vague points on its about page, but they aren't really elaborated on.

Is personal prediction tracking just a more interesting version of a calibration test? So you just track your predictions about your life and then look at the calibration graph to check if you're over- or underconfident? Or are there any other benefits?

The aforementioned Fatebook page lists "make better decisions" and "communicate more clearly". I can see how phrasing your beliefs as probabilities instead of "yes/no" questions could help with that - but how does prediction tracking in particular help with it?
Jan Kulveit's Corrigibility Thoughts Distilled
by brook — 2023-08-20T17:52:36.326Z

Many thanks to Jan for commenting on a draft of this post.

There were a lot of great comments on "Let's see you write that corrigibility tag". This is my attempt at expanding Jan Kulveit's comment[1], because I thought it was useful, and should be read more widely. This post may not accurately represent Jan's thoughts. This post took approximately 3 hours to write.

Introduction

'Corrigible' is a word that is used in a lot of different ways in AI alignment. Here, we're using Eliezer Yudkowsky's framing from mad investor chaos and the woman of asmodeus (a story about a lot of things, but also about alignment). In the story, there is a world with plans to create a limited AI if needed in an emergency, with as many safeguards as they have come up with at the time.

The initial craft doesn't have to be literally perfect to work perfectly in the end, it just has to be good enough that its reflection and self-correction ends up in exactly the right final place.

Eliezer asked if LessWrong could write a list of safeguards for a limited AI like this without his input. Eliezer's answer is available here; this post is an explanation of Jan's answer. Jan notes that his answer was written in Eliezer's frame, not the frame he would normally think in.

We'll look at examples of five types of tool; those for reducing instrumental convergence and reducing unbounded rationality, 'traps', oversight methods, and interpretability methods.

Five Types of Tool

Against Instrumental Convergence

"Instrumental convergence is the theorized tendency for most sufficiently intelligent agents to pursue potentially unbounded instrumental goals, such as self-preservation and resource acquisition".

Instrumental convergence precludes corrigibility. For instance, many theoretical agents will not allow their goal function to be changed, because this will reduce their ability to achieve their current goals. Additionally, pursuit of instrumental goals can break some of the other corrigibility methods. Preventing instrumental convergence is therefore key to achieving corrigibility. Jan outlines three tools that could be used to help prevent instrumental convergence: preventing resource acquisition, uncertainty about goals, and making the AI want to not exist in the future.

Resource acquisition can be disincentivized by using a mutual information measure between the AI and distant parts of the en...
Memetic Judo #3: The Intelligence of Stochastic Parrots v.2
by Max TK — 2023-08-20T15:18:24.477Z

There is the persistent meme that AIs such as large language models (ChatGPT etc.) do, in a fundamental sense, lack the ability to develop human-like intelligence.

Central to it is the idea that LLMs are merely probability-predictors for the-next-word based on a pattern-matching algorithm, and that they therefore cannot possibly develop the qualitative generalization power and flexibility characteristical of a human mind.
In that context, they are often dismissed as "stochastic parrots", suggesting they just replicate without any true understanding.

Example Argument

Large Language Models are just stochastic parrots - they simply replicate patterns found in the text they are trained on and therefore can't be or become generally intelligent like a human.

Another Popular Variant

The AIs don't produce output that is truly novel or original, they just replicate patterns and (somehow) combine them or "mash them together".

I will explain later why I think that both are essentially equivalent.

Just Parrots

The problem with this argument as stated is not in the premise (that LLMs are, essentially, probabilistic pattern replicators - this is essentially correct), it is that the conclusion does not directly follow from the premise (a non sequitur).

When I meet proponents of it, usually they do not have a convincing explanation for why the parrots cannot be generally intelligent.

While I believe that the characterization of large language models as "stochastic parrots" is not strictly incorrect, it is certainly misleading. The right approach is to convince the sceptic to not underestimate their potential.

Don't underestimate him.

The Functional Brain Argument

There are strong reasons to assume that the non-existence of generally intelligent mathematical algorithms would violate the concept of brain physicalism - the latter seeming to be the default-position among neuroscientists.

Humans are general intelligences.

The human brain is a physical system.

The behavior of arbitrary physical systems can be approximated by (arbitrarily complicated) mathematical functions or algorithms (in other words: they can be simulated).

If all three premises are true, it follows that there must exist functions or algorithms that are (or behave like) general intelligences.

It has been proven that neural nets (as long as they are large enough) can approximate arbitrary functions.

Theref...
“Dirty concepts” in AI alignment discourses, and some guesses for how to deal with them
by Nora_Ammann — 2023-08-20T09:13:34.225Z

Meta: This is a short summary & discussion post of a talk on the same topic by Javier Gomez-Lavin, which he gave as part of the PIBBSS speaker series. The speaker series features researchers from both AI Alignment and adjacent fields studying intelligent behavior in some shape or form. The goal is to create a space where we can explore the connections between the work of these scholars and questions in AI Alignment.

This post doesn't provide a comprehensive summary of the ideas discussed in the talk, but instead focuses on exploring some possible connections to AI Alignment. For a longer version of Gomez-Levin’s ideas, you can check out a talk

here

.

"Dirty concepts" in the Cognitive Sciences

Gomez-Lavin argues that cognitive scientists engage in a form of “philosophical laundering,” wherein they associate, often implicitly, philosophically loaded concepts (such as volition, agency, etc.) into their concept of “working memory.”

He refers to such philosophically laundered concepts as “dirty concepts” insofar as they conceal potentially problematic assumptions being made. For instance, if we implicitly assume that working memory requires, for example, volition, we have now stretched our conception of working memory to include all of cognition. But, if we do this, then the concept of working memory loses much of its explanatory power as one mechanism among others underlying cognition as a whole.

Often, he claims, cognitive science papers will employ such dirty concepts in the abstract and introduction but will identify a much more specific phenomena being measured in the methods and results section.

What to do about it? Gomez-Lavin’s suggestion in the case of CogSci

The pessimistic response (and some have suggested this) would be to quit using any of these dirty concept (e.g. agency) all together. However, it appears that this would amount to throwing the baby out with the bathwater.

To help remedy the problem of dirty concepts in working memory literature, Gomez-Lavin proposes creating an ontology of the various operational definitions of working memory employed in cognitive science by mining a wide range of research articles. The idea is that, instead of insisting that working memory be operationally defined in a single way, we ought to embrace the multiplicity of meanings associated with the term by keeping track of them more explici...
Call for Papers on Global AI Governance from the UN
by Chris_Leong — 2023-08-20T08:56:58.745Z
Link: https://www.linkedin.com/posts/un-tech-envoy_%3F%3F%3F%3F-%3F%3F%3F-%3F%3F%3F%3F%3F%3F-%3F%3F-%3F%3F-activity-7097568680066068481-egyT?utm_source=share&utm_medium=member_ios

Copied from their LinkedIn page:

📢 𝐂𝐚𝐥𝐥 𝐟𝐨𝐫 𝐏𝐚𝐩𝐞𝐫𝐬 𝐨𝐧 𝐆𝐥𝐨𝐛𝐚𝐥 𝐀𝐈 𝐆𝐨𝐯𝐞𝐫𝐧𝐚𝐧𝐜𝐞🌍

Exciting news! As we gear up for the High-level Advisory Body on AI, we're inviting thought leaders, researchers, and enthusiasts to contribute short papers (~2000 words) on pivotal themes:

1️⃣ Key Issues on Global AI Governance: Dive into studies and recommendations that the High-Level Advisory Body on AI should prioritize, especially those needing global governance attention.

2️⃣ Current Efforts in Global AI Governance: Share analyses on bilateral, multilateral, and inter-regional initiatives. We're keen on understanding varying philosophical approaches, critiques, and suggestions.

3️⃣ Models in Global AI Governance: Whether you're analyzing existing models or proposing fresh perspectives, we're all ears. Surveys and analyses of other proposals are also encouraged.

We champion diversity! 🌈 We're eager to hear from a myriad of groups, regions, and methodologies. Your insights will serve as foundational material (with due credit) for the High-Level Advisory Body on AI.

📅 Deadline: 30 September

📧 Submission: Send your paper (hyperlink or PDF) to techenvoy@un.org. Ensure your title page has the author(s) details, affiliation, and contact info. If it's an Executive Summary of a more extensive piece, kindly attach the main paper's link.
How do I read things on the internet
by Vlad Sitalo — 2023-08-20T05:43:47.494Z
Link: https://vlad.roam.garden/How-do-I-read-things-on-the-internet

This is a linkpost - I recommend reading it at the original URL for a better reading UX

I have a somewhat elaborate process for reading things that I find on the web. I've been inspired to share it because after many a long iteration it finally feels adequate!

Reading things on the web seems like it should be easy, and yet - I've been failing at it for years! 🙀

In this article I explore my current workflow and challenges that made it into what it is today.

Workflow

When I see something new I want to read - save it to Readwise Reader (it also becomes an SRS item in Roam)
When in "podcast mode" (doing house chores, biking, etc) - queue up a TTS version of an article.
Each day after lunch - read one or more pieces suggested by Spaced Repetition in Roam - reschedule others
Highlights and notes are synced into Roam and selectively become SRS items

The goal of the workflow is to: Enable you to reliably read things you want to read (and retain learnings from it) while minimizing effort and attention spent.

Discovery

This is part of the pipeline that received relatively less optimization attention, mostly by virtue of me suffering from abundance of content rather than scarcity. I include it primarily for completeness’ sake.

Some ways in which discovery is happening for me are as follows:

Follow-up from previous things I read

people often link related content
Ampie extension is helpful to discover the broader conversation about a given piece - which includes links to related content

Recommendations from friends - sent directly or via social media
Aggregator newsletters (Indie Hackers, https://rationalnewsletter.com/)

I like aggregator newsletters because they introduce an additional layer of curation over raw subscription streams like RSS.

I'm also subscribed to a few "normal" newsletters

I'd love to see a Goodreads-style platform created for discovering and tracking articles.

Ampie gives me some of the same benefits, but less systematically
Curius.app is a more social version of Hypothes.is and also covers some of the similar ground

Reading Inbox

When I encounter something that I think would be worth my attention - I save it to Readwise Reader

This also serves as a trigger to create an SRS item in Roam

The first problem of reading things on the internet is that there are too many things out there one is tempted to read.

Even if you have a good curation process there is always too much ...
AI Forecasting: Two Years In
by jsteinhardt — 2023-08-19T23:40:04.302Z
Link: https://bounded-regret.ghost.io/scoring-ml-forecasts-for-2023/

Two years ago, I commissioned forecasts for state-of-the-art performance on several popular ML benchmarks. Forecasters were asked to predict state-of-the-art performance on June 30th of 2022, 2023, 2024, and 2025. While there were four benchmarks total, the two most notable were MATH (a dataset of free-response math contest problems) and MMLU (a dataset of multiple-choice exams from the high school to post-graduate level).

One year ago, I evaluated the first set of forecasts. Forecasters did poorly and underestimated progress, with the true performance lying in the far right tail of their predicted distributions. Anecdotally, experts I talked to (including myself) also underestimated progress. As a result of this, I decided to join the fray and registered my own forecasts for MATH and MMLU last July.

June 30, 2023 has now passed, so we can resolve the forecasts and evaluate my own performance as well as that of other forecasters, including both AI experts and generalist "superforecasters". I'll evaluate the original forecasters that I commissioned through Hypermind, the crowd forecasting platform Metaculus, and participants in the XPT forecasting competition organized by Karger et al. (2023), which was stratified into AI experts and superforecasters.

Overall, here is how I would summarize the results:

Metaculus and I did the best and were both well-calibrated, with the Metaculus crowd forecast doing slightly better than me.

The AI experts from Karger et al. did the next best. They had similar medians to me but were (probably) overconfident in the tails.

The superforecasters from Karger et al. did the next best. They (probably) systematically underpredicted progress.

The forecasters from Hypermind did the worst. They underpredicted progress significantly on MMLU.

Interestingly, this is a reverse of my impressions from last year, where even though forecasters underpredicted progress, I thought of experts as underpredicting progress even more. In this case, it seems the experts did pretty well and better than generalist forecasters.

What accounts for the difference? Some may be selection effects (experts who try to register forecasts are more likely to be correct). But I'd guess some is also effort: the expert "forecasts" I had in mind last year were from informal hallway conversations, while this year they were formal quantitative predictions with some (small) m...
Four management/leadership book summaries
by Nikola Jurkovic — 2023-08-19T23:38:53.795Z

Preface

A while ago I read four books on leadership and management and wrote summaries of them. The summaries are slightly biased in that they are mostly what I found to be useful and how I interpreted the books. I don’t necessarily endorse the points made by these books. If you are interested in management I would suggest you read these, with Managing to Change the World as my strongest recommendation.

These books mostly talk about how to manage a team of people that are being paid to do some work. If you are managing a team of volunteers, the dynamics change significantly and you, as a team leader, have much less authority over how people behave. It is important to set clear expectations with people in such a context (how many hours a week they’re willing to work is the most important variable).

Managing to Change the World

Non-profits should be more, not less, ruthless than for-profits. For-profits focus on good organization and management because they want to make more money and being organized and well-managed is useful for achieving that. Non-profits are saving lives, and so they should be much more ruthless when it comes to achieving their goals, because so much more is at stake.
The point of managing is to impact the world. A well-managed team is more effective as it has someone on the team who keeps the big picture in mind, oils the machine, and makes sure everyone is working towards the goal.
The main duty of managers is to make sure that tasks get done. However, it is not in the manager's capacity to do all the tasks, so other people must do them instead.
When delegating a task, make sure that expectations are clear. The person who was assigned the task should be able to repeat back to you what you want them to do, including the task itself, what success looks like, why it's important, and when the deadline is.
After a task is delegated, make sure that it's progressing as intended. This might take the form of a one-on-one check-in call or an email. Do not blindly assume that tasks get done when people commit to them. People often run into obstacles and checking in can help overcome those.
After a task is completed (or failed), keep people accountable. Make sure employees know you notice when a task has gone well or badly.
Provide quick and private negative feedback. If someone is doing something wrong, tell them as soon as you can. Say i

...