trentmkelly/LessWrong-43k huggingface dataset:
UDT shows that decision theory is more puzzling than ever
by Wei Dai — 2023-09-13T12:26:09.739Z

I feel like MIRI perhaps mispositioned FDT (their variant of UDT) as a clear advancement in decision theory, whereas maybe they could have attracted more attention/interest from academic philosophy if the framing was instead that the UDT line of thinking shows that decision theory is just more deeply puzzling than anyone had previously realized. Instead of one major open problem (Newcomb's, or EDT vs CDT) now we have a whole bunch more. I'm really not sure at this point whether UDT is even on the right track, but it does seem clear that there are some thorny issues in decision theory that not many people were previously thinking about:

Indexical values are not reflectively consistent. UDT "solves" this problem by implicitly assuming (via the type signature of its utility function) that the agent doesn't have indexical values. But humans seemingly do have indexical values, so what to do about that?

The commitment races problem extends into logical time, and it's not clear how to make the most obvious idea of logical updatelessness work.

UDT says that what we normally think of as different approaches to anthropic reasoning are really different preferences, which seems to sidestep the problem. But is that actually right, and if so where are these preferences supposed to come from?

2TDT-1CDT - If there's a population of mostly TDT/UDT agents and few CDT agents (and nobody knows who the CDT agents are) and they're randomly paired up to play one-shot PD, then the CDT agents do better. What does this imply?

Game theory under the UDT line of thinking is generally more confusing than anything CDT agents have to deal with.

UDT assumes that the agent has access to its own source code and inputs as symbol strings, so it can potentially reason about logical correlations between its own decisions and other agents' as well defined mathematical problems. But humans don't have this, so how are humans supposed to reason about such correlations?

Logical conditionals vs counterfactuals, how should these be defined and do the definitions actually lead to reasonable decisions when plugged into logical decision theory?

These are just the major problems that I was trying to solve (or hoping for others to solve) before I mostly stopped working on decision theory and switched my attention to metaphilosophy. (It's been a while so I'm not certain the list is complete.) As far as I kn...
Alignment & Capabilities: what's the difference?
by johnhalstead — 2023-09-13T11:48:33.486Z

In the AI safety literature, AI alignment is often presented as conceptually distinct from capabilities. However, (1) the distinction seems somewhat fuzzy and (2) many techniques that are supposed to improve alignment also improve capabilities.

(1) The distinction is fuzzy because one common way of defining alignment is getting an AI system to do what the programmer or user intends. However, programmers intend for systems to be capable. eg we want chess systems to win at chess. So, a system that wins more is more intent aligned, and is also more capable.

(2) eg This Irving et al (2018) paper by a team at Open AI proposes debate as a way to improve safety and alignment, where alignment is defined as aligning with human goals. However, the debate also improved the accuracy of image classification in the paper, and therefore also improved capabilities.

Similarly, Reinforcement learning with human feedback was initially presented as an alignment strategy, but my loose impression is that it also made significant capabilities improvements. There are many other examples in the literature of alignment strategies also improving capabilities.

**

This makes me wonder whether alignment is actually more neglected that capabilities work. AI companies want to make aligned systems because they are more useful.

How do people see the difference between alignment and capabilities?

I also posted this on the EA Forum, but I'm curious to see what people here think.
Duty to rescue / Non-assistance à personne en danger
by Thomas Sepulchre — 2023-09-13T09:49:35.964Z

If you are waiting in a bank lobby, and an 83-year-old man collapses, do you have a moral duty to try to rescue him, call an ambulance, or provide any kind of assistance? Or, more importantly, do you have any legal duty to do so, i.e. can you be held liable for failing to come to rescue someone when you could have? Apparently, if you live in Germany, the answer is yes. In most of the US, the answer is no.

Duty to rescue

The Non-assistance à personne en danger is (in French) the offense committed when someone doesn't help another person who is in danger. It is an offense in France, Belgium, Quebec (but not the whole of Canada), and, as far as I can tell, most French-speaking countries. In the US, as well as other English-speaking countries, it is known as Duty to rescue, but, besides the non-literal translation, one key difference between the two languages is that it is not an offense at all.

It is actually an offense in only a small number of countries. The wikipedia page provides this very informative map, with duty to rescue in blue, good Samaritan law in cyan, and red, orange and grey being varying levels of no duty to rescue / no data. The good Samaritan law is an interesting concept, where a bystander still has no duty to rescue, but the law provides some protection to anyone who tries to rescue them, if they cause harm by doing so.

The lack of duty to rescue in most countries goes strongly against my intuitions (those intuitions being probably influenced by the law), because it introduces doing nothing as a special and privileged kind of choice. In Europe (at least according to this map), in a situation where someone is in danger, you, as a bystander, should take the action which helps the most (and this may or may not be anything, depending on the situation). In most of the US, apparently, one should either take the action which helps the most, or do nothing. Doing nothing is some kind of get out of jail free card. Actually no, it is a literal get out of jail free card. This introduces some asymmetry between action and inaction, where one is liable for bad actions, but one cannot be liable for inaction. It also incentivises inaction.

On a lighter note, this lack of duty to rescue is the plot of Little Lost Robot, one of the short stories in Isaac Asimov's I, Robot. Without spoiling too much, as we all know, the first law states: "A robot ...
The Flow-Through Fallacy
by Chris_Leong — 2023-09-13T04:28:28.390Z

There is something of an informal reasoning fallacy where you promote something that seems like something that we should "surely do", something that seems obviously important, but where we haven't thought through all of the steps involved and so we aren't actually justified in assuming that the impact flow through.

Here are some examples:

The Dean of a computer science department thinks that "surely, it's important to produce not only technically proficient graduates but those who use their skills for good". So they mandate that all students must do a "technology and ethics class". The only problem is that none of their professors are an expert in this nor even interested in it, so it ends up being a poorly taught and run subject such that students put in the absolute minimum effort and forget everything a week after the exam.
The prime minister of a country wants to reduce crime. He notices that the police department is severely underfunded so he significantly increases funding. Unfortunately, they're so corrupt and nepotistic that the department is unable to spend the funds effectively.
A government wants to increase recycling, so they create a "national recycling day" reasoning “surely this will increase recycling. Unfortunately, most people end up ignoring it and, out of those who actually are enthusiastic, most make an extra effort to recycle for a few days or even a week, then basically forget about it for the rest of the year. So it ends up having some effect, but it’s basically negligible.

In each of these cases, the decision maker may not have chosen the same option if they'd taken the time to think it through and ask themselves if the benefits were likely to actually accrue.

Domain experience can be helpful to know that these kinds of issues are likely to crop up, but so is the habit of Murphyjitsu'ing your plans.

To be clear, I'm not intending to refer to the following ways in which things could go wrong:

Side-effects: The government introduces snakes to reduce the rodent population, but this has the side-effect of causing more people to gain snake bites.
Reactions: Amy donates $10 million to the Democrats. Bob hears about this and decides to donate $20 million to the Republicans in response
Value confusion: James spends years acquiring his Pokemon card collection as a kid. He builts an amazing Pokemon card collection, but regrets all the time and money he

...
Book review: The Importance of What We Care About (Harry G. Frankfurt)
by David Gross — 2023-09-13T04:17:16.823Z

This is more of a summary or paraphrase than a review, really. The Importance of What We Care About is 13 essays by philosopher Harry G. Frankfurt. These are my one-paragraph summaries of each:

If you do something because you willed yourself to do it, you are responsible for having done it, even if you could not in fact have done otherwise. The fact that your action was done under coercive threat or that it was predetermined does not absolve you of responsibility for it if the actual reason you took the action is because you intended to. For you to be blamable/creditable for some action does not require that you could potentially have done something else, contrary to centuries of previous theorizing about ethics.
A perhaps defining property of “persons” is that we can form “second-order volitions”: desires about our desires. Because a person may wish that her will were directed toward wishes other than her current wishes, it is for persons that questions of “free will” arise: do I have the ability to choose my own wishes, or do I merely have the ability to work toward the fulfillment of whatever unchosen wish currently predominates?
Threats and offers are two ways that one person may try to influence someone to exercise his will in a particular way. For either of these to rise to the level of “coercion,” or at any rate, “compulsion” — in a sense that absolves the person coerced of moral responsibility for the resulting action — it must go beyond merely influencing his will and become more like imposing one’s own will upon him.
Doing something because it is the lesser of available evils, and therefore something one would rather not have to do, is still to do it freely and responsibly. Though being responsible for doing the lesser of available evils doesn’t necessarily make you blameworthy for the evil. If indeed it was the lesser of available evils, that is enough to deflect the blame.
Things we do may be active (actions) or passive (mere happenings). “Passions” (e.g. anger) also may be active or passive in this way: some passions we own as part of our persons (they are “internal”), others take hold of us without our consent (“external”). A person may externalize a passion by rejecting it or refusing to cooperate with it or even by submitting unwillingly to it, or can internalize one by agreeing with it, going along with it, incorporating it into one’s self image.

...
Padding the Corner
by jefftk — 2023-09-13T01:30:04.009Z

There's a shelf in my dad's kitchen, about four feet off the ground.
I remember when I stopped being short enough that I could walk under
it: ouch! I didn't get a concussion or anything, but it was pretty
unpleasant. My sisters and cousins also remember bonking their heads
on it. I noticed my oldest was getting just tall enough, and started
to tell her this story warning her about it.

While I was telling the story I realized how silly this was, and
stopped what I was doing to put some padding on the corner.

A washcloth folded over and screwed to the shelf doesn't look
wonderful, but it's now a couple years later, three of the cousins are
tall enough to intersect with it, and no one has gotten hurt. Maybe
someday we can do something more elegant, but in the meantime fixing
sharp corners beats warning people.

Comment via: facebook, mastodon
Should an undergrad avoid a capabilities project?
by Double — 2023-09-12T23:16:39.817Z

Although I'm writing this about my personal situation and project, I expect others to be in this situation as well.

I'm thinking about joining ProjectX. It sounds like a perfect opportunity to work with AI, gain experience, and potentially make something I'm proud of. Unfortunately, it is about improving AI efficiency, which I read as "capabilities". When capabilities increase, doomsday gets nearer and more likely.

More information about the project:

"ProjectX: ML Research Competition

ProjectX is the world’s largest undergraduate machine learning research competition with competing teams from top universities across the world. 3 winning teams will each be awarded a cash prize of CAD 20,000, and all participants will be invited to attend the annual University of Toronto AI Conference in mid January 2024. Last year's competition was sponsored by Google, Intel and Nvidia, and featured keynote speaker James Van Loon.

This year's focus is Efficient AI: investigating innovative methodologies and ML approaches to create AI models that significantly reduce power consumption without compromising performance.

We are fielding a team of 3-6 students to represent Northwestern this year. The team will work on their chosen problem from October to December 2023, culminating in a paper describing their findings.

More info about this year's competition

Sample paper from last year's winning team"
[Linkpost] Contra four-wheeled suitcases, sort of
by Gunnar_Zarncke — 2023-09-12T20:36:02.412Z
Link: https://dynomight.substack.com/p/luaggage

Are simple, rugged, reliable systems better or are fancy, sophisticated, complex ones better?

The article goes in with a preconception, analyzes suitcases, cars, hiking gear, food, cups, space pens, people, startups, soccer, headphones, weight, rockets, rockets, ships, and banks, and comes to the conclusion:

I hoped that this exercise would show that simple, rugged, reliable systems are systematically better than fancy fragile ones. But it didn’t. Sometimes rugged wins and sometimes it doesn’t.

It says more about trends and patterns he found. Go read the short article to find out more.

I liked that it notices confusion and babbles lots of interesting categories.
Seeking Feedback on My Mechanistic Interpretability Research Agenda
by RGRGRG — 2023-09-12T18:45:08.902Z

Why this post

I’ve been doing MI research full-time for about three months and since my current grant is ending soon and I recently received a Lightspeed rejection, now seems like a good time to take some time away from object-level work to reflect on direction and next steps. I spent a few hours writing a draft, realized I thought it was too complicated (for the sake of sounding complicated?) and rewrote what I really want to do and why in ~750 words. I intend to incorporate feedback in an OpenPhil early career application (which I will submit in a few days). I am also applying to AI research position(s) and if I receive a position, this will help to inform what to focus on.

Agenda

In my mind, the goal of mechanistic interpretability research is to sufficiently understand what state-of-the-art networks are doing, so we can examine whether any dangerous behaviors are occurring. I expect automated techniques to be an important step towards understanding such models (Examples: 1, 2, 3). However, I feel we are missing an important foundation since we’ve yet to fully understand what small language models are doing, and thus, don’t know the set of things that automated techniques should be detecting. (Should these techniques be searching for reasoning?, specific behaviors?, memorized information?, superposition?). I believe that we first need to conduct a meticulous, low-level exploration of shallow language models to guide our analysis of larger models.

With the release of the TinyStories models, we now have small language models that produce reasonable text completions (of children’s stories). TinyStories-1Layer-21M is a one-layer model in this family that will serve as the focus of my research. I will conduct low-level mechanistic interpretability on this model using Anthropic’s Circuits Framework. I believe that analyzing this network using circuits will be feasible given its relatively smaller size and the limited number of pathways through the network.

I strongly believe that there will be rules and patterns that exist in this 1-layer model that will serve as a guideline of things we can search for in larger models. (Of course, there are almost certainly rules and patterns in larger models trained on a more diverse data set that do not exist in this model, but this should serve as a good starting p...
Automatically finding feature vectors in the OV circuits of Transformers without using probing
by Jacob Dunefsky — 2023-09-12T17:38:49.207Z

Let's say that you're working to understand how a given large language model computes its output on a certain task. You might be curious about the following questions:

What are some linear feature vectors that are responsible for the model's performance on a given task?
Given a set of different tasks, to what extent are the same features responsible for the model's outputs on these different tasks?
How do nonlinearities such as LayerNorm transform the features that are used for a given task?
Given a pair of features, if the value of one feature for an input is high, then does this imply that the value of another feature for that input is likely to be high?
How can I find a steering vector to add to the activations of a model that increase the model's output on one task while decreasing its output on another?

Recently, I've been working on developing a general method that yields partial answers to these questions. Notably, unlike probing methods for finding feature vectors, this method can find approximate answers to these questions with zero forward passes on data, yielding improved answers with very few additional datapoints[1]. This post introduces this method for finding feature vectors, dubbed "observable propagation"; puts forth theoretical results regarding the analysis of feature vectors yielded by observable propagation; and presents the results of some preliminary experiments testing this method.

Note that this is all very much a work in progress! There are many more experiments that I am currently running, in order to obtain a better understanding of the utility and limits of observable propagation. Additionally, I still have to clean up my code before making it public (although the methods described here shouldn't be too hard to re-implement). But rather than wait for everything to be finished before sharing the results that I do have, I figured that it would be best to present the work that I've completed thus far to the community.

The contributions of this work are as follows:

This work introduces the concept of "observables" that generalizes the pattern of considering a model's performance on a task by looking at logit differences. A number of new techniques for analyzing Transformers opens up once one starts treating observables as concrete objects amenable to study.
This work presents a method called "observable propagation" that allows one to find fea

...
Startup Roundup #1: Happy Demo Day
by Zvi — 2023-09-12T13:20:03.883Z

There were a bunch of discussions recently related to issues surrounding Y-Combinator, related as usual to their annual demo day. It seemed worth splitting them off into a post.

Bidders at Auction Mostly Think Prices Are Too High

YC is in session, so all the usual talk is back.

Paul Graham: As happens every single YC batch, investors complain that the valuations are too high, and the startups raise anyway.

Kush: Is there any specific reason why you think this happens?

Paul Graham: Some investors just cannot grasp the implications of the fact that all the returns are concentrated in the big wins. The top tier investors get it though; you’ll rarely lose one of them over price.

Amal Dorai: Isn’t the dynamic range of YC valuations narrower than the companies themselves? We see pretty much a 2x range that they’re raising in, but >10x variance in current traction. The high flyers are fine but those riskier companies still finding their footing are behind the 8 ball.

Paul Graham: The variation in valuations is much smaller than the variation in quality. That’s what determines risk, not their current traction. If current traction were a perfect or even good predictor, investing would be trivially easy.

David Tran: “You’ve found the market price when buyers complain but still pay”

Dan Grey: What I’ve seen investors saying is that they would rather wait a year and then invest in the next round – which happens at a similar price but with much more proof. If the initial round is not offering the right level of upside for the risk, why should investors jump in?

Paul Graham: That was an amusingly novel variant. I only saw one investor say that. The reason it’s a dumb strategy is that choosing companies whose valuation hasn’t increased a year later selects for bad ones.

This model tells me that the lower valuation, less exciting companies in YC are likely overvalued, and the higher valuation, more exciting companies in YC are highly undervalued.

I see two distinct dynamics here, the one Graham and Dorai raise, and Tran’s.

Tran’s is the core reason investors complain prices are too high. Of course most investors think the YC prices are too high. There are tons of investors chasing not that many YC companies with not that much space in their rounds. YC companies have many advantages over non-YC companies, including all three of:

Selection effects (I don’t th

...