trentmkelly/LessWrong-43k huggingface dataset:
How do I start a programming career in the West?
by Lao Mein — 2022-11-25T06:37:12.237Z

I was expecting great things when I graduated from high school. I scored 35 on the ACT, was messing around with crypto, and fully expected to drop out of college to start a business. Things were looking up.

Instead, due to a combination of depression and procrastination, I ended up spending the next 8 years drifting in and out of college before I finally got my undergraduate degree. I wish I was out partying during this time since that would at least give me some good memories, but I honestly just stayed inside and played video games all day. Not my proudest decade.

But eventually, I pulled myself together and graduated. I've been employed as a bioinformatics engineer in China for the past 2 years, and have been steadily improving my life. I'm contributing to group discussions, pointing out major problems with research plans, and picking up programming and data management skills. Instead of putting in minimal effort, I'm spending most of my workday coding or researching, an incredible improvement over my productivity in college. I feel like a highly valued member of my team. I've published papers, helped design gene panels for cancer treatment, and sifted through papers for new research avenues for my team to pursue.

And then I remembered that I was making what would be considered minimum wage in the US. I know I can do better. I also don't want the possibility of a multi-month lockdown starting at any time to hang over my head. I have a passion for AI and want to pursue a career in programming/data science in the West. Since many of you are involved in these fields, do you have any recommendations? I see 3 paths forward:

Continue working for a few years. Pile up papers written, find character references, and acquire more job skills. My problem is that I don't think our prediction models are going to go beyond logarithmic regression, and I'm feeling increasingly stuck. I suppose this is my default path.
Try to participate in a really impressive side project, and use that as a jump-off point to get into an AI alignment/capabilities company. The problem is that my personal passions might not match up with what such companies are looking for. I'm currently idea-boarding a cheap (<$20) laser-guidance package for quad-rotor-dropped grenades, but I think war is low-status in Western tech companies and this might actually hurt my chances. Maybe I should write an R pa

...
The AI Safety community has four main work groups, Strategy, Governance, Technical and Movement Building
by peterslattery — 2022-11-25T03:45:30.203Z

Epistemic status

Written as a non-expert to develop and get feedback on my views, rather than persuade. It will probably be somewhat incomplete and inaccurate, but it should provoke helpful feedback and discussion.

Aim

This is the first part of my series ‘A proposed approach for AI safety movement building’. Through this series, I outline a theory of change for AI Safety movement building. I don’t necessarily want to immediately accelerate recruitment into AI safety because I take concerns (e.g., 1,2) about the downsides of AI Safety movement building seriously. However, I do want to understand how different viewpoints within the AI Safety community overlap and aggregate.

I start by attempting to conceptualise the AI Safety community. I originally planned to outline my theory of change in my first post. However, when I got feedback, I realised that i) I conceptualised the AI Safety community differently from some of my readers, and ii) I wasn’t confident in my understanding of all the key parts.

TLDR

I argue that the AI Safety community mainly comprises four overlapping, self-identifying, groups: Strategy, Governance, Technical and Movement Building.
I explain what each group does and what differentiates it from the other groups
I outline a few other potential work groups
I integrate these into an illustration of my current conceptualisation of the AI Safety community
I request constructive feedback.

My conceptualisation of the AI Safety community

At a high level of simplification and low level of precision, the AI Safety community mainly comprises four overlapping, self-identifying, groups who are working to prevent an AI-related catastrophe. These groups are Strategy, Governance, Technical and Movement Building. These are illustrated below.

We can compare the AI Safety community to a government and relate each work group to a government body. I think this helps clarify how the parts of the community fit together (though of course, the analogies are imperfect).

Strategy

The AI Safety Strategy group seeks to mitigate AI risk by understanding and influencing strategy.

Their work focuses on developing strategies (i.e., plans of action) that maximise the probability that we achieve positive AI-related outcomes and avoid catastrophes. In practice, this includes researching, evaluating, developing, and disseminating strategy (see this for ...
Less Successful Cider Adventures
by jefftk — 2022-11-25T01:50:03.366Z

A few weeks ago I made some

boiled
apple cider. It was tasty, and after eating most of it I decided
to make more. This time I used a gallon of cider, and it wasn't done
after a day so I turned the slow cooker to "low" and left it
overnight. Turns out that was too much! It had reduced to about 1/10
its original volume (1gal to 12oz), which wouldn't be too bad if I
ended up with caramel. Except that it also has a decidedly burnt
flavor, probably from getting too hot at the end. I like dark
caramel, but this is past dark.

On the plus side, it's fun to play with, a bit like silly putty:

The kids have told me I can't throw it out yet, but it probably goes
tomorrow. It's frustrating because the apple cider caramel flavors
are all there, there's just also a persistent flavor of 'burnt'.

Comment via: facebook, mastodon
Gliders in Language Models
by Alexandre Variengien — 2022-11-25T00:38:11.565Z

Epistemic status: a highly speculative and rough idea that involves many concepts I’m not familiar with.

TL;DR Language models propagate features from the prompt to the text completion they generate, I call such features gliders. If powerful LMs are widely deployed on the Internet, gliders could propagate fast and undergo selection pressures, pushing them to become effective memes. In addition to being more sharable, they could be selected for their ability to extract parasitic computation from LM and use it to propagate more effectively. On a meta note, writing this post was an interesting exercise to think about weird failure cases of AI, and I expect that doing this can be beneficial for others too.

Thanks to Fabien Roger, Arthur Conmy, Jean-Stanislas Denain and Diego Dorn for helpful feedback and suggestions.

In this text, I explore an abstraction to think about stable structures in the text generated by self-supervised language models (LM) like GPT-3. It is likely that in a near future, feedback loops where the output of an LM is published online and then used in the context of a new LM instance will mobilize immense amounts of computation and data (through the use of

chatbots

, automatic content generation, or

AI-powered API users

).[1] It seems useful to think in advance about their consequences and potential failure modes.

Stable structures moving forward

When prompted with "Once| upon| a| time|,| there| was| a| group| of| unicorns|", GPT3 generates a meaningful sentence — "| called| the| Blue| Unicorns|." — keeping the interweaving with the "|" character[2]. The property "Each word is separated by |" is preserved by iteratively applying next-token prediction.

This is not so surprising if we consider LM as

simulators

: they are trained to propagate the regularities from the prompts to the next-token predictions by inferring hidden variables from the context. This includes both low-level text features such as consistent use of the apostrophe punctuation character, as well as high-level features such as an agentic simulacrum embodying a politician making plans to win an election. If those are present in the text of the prompt, they will stay in the text generated by the LM.

Such stable features can be extremely diverse. It even seems possible that some can be invisible to humans, lying in the

null space of natural langua

...
On Kelly and altruism
by philh — 2022-11-24T23:40:04.095Z

One-sentence summary: Kelly is not about optimizing a utility function; in general I recommend you either stop pretending you have one of those, or stop talking about Kelly.

There was a twitter thread that triggered some confusion amongst myself and some other people in a group chat I'm in.1

The relevant tweets are these (I've omitted some):

3) Let’s say you were offered a coin flip. 75% it comes up heads, 25% it comes up tails; 1:1 payout. How much would you risk?

4) There are a number of ways to approach this question, but to start: what do you want, in the first place? What’s your utility function?

5) In other words–how cool would it be to make $10,000? How about $1,000,000–is that 100 times as good?

For most people the answer is ‘no, it’s more like 10 times as good’. This is because of decreasing marginal utility of money.

8) One reasonable utility function here is U = log(W): approximating your happiness as logarithmic in your wealth. That would mean going from $10k to $100k is worth about as much as going from $100k to $1m, which feels…. reasonable?

(this is what the Kelly Criteria assumes)

9) So, if you have $100k, Kelly would suggest you risk half of it ($50k). This is a lot! But also 75% odds are good.

10) What about a wackier bet? How about you only win 10% of the time, but if you do you get paid out 10,000x your bet size? (For now, let’s assume you only get to do this bet once.)

11) Kelly suggests you only bet $10k: you’ll almost certainly lose. And if you kept doing this much more than $10k at a time, you’d probably blow out.

That this bet is great expected value; you win 1,000x your bet size, way better than the first one! It’s just very risky.

12) In many cases I think $10k is a reasonable bet. But I, personally, would do more. I’d probably do more like $50k.

Why? Because ultimately my utility function isn’t really logarithmic. It’s closer to linear.

13) Sure, I wouldn’t care to buy 10,000 new cars if I won the coinflip. But I’m not spending my marginal money on cars anyway. I’m donating it.

And the scale of the world’s problems is…. Huge.

14) 400,000 people die of malaria each year. It costs something like $5k to save one person from malaria, or $2b total per year. So if you want to save lives in the developing world, you can blow $2b a year just on malaria.

15) And that’s just the start. If you look at the scale of funds spent on diseases, global warming, emer

...
Open technical problem: A Quinean proof of Löb's theorem, for an easier cartoon guide
by Andrew_Critch — 2022-11-24T21:16:43.879Z

Motivation

Löb's theorem is pretty counterintuitive. Speaking informally about one or more agents engaging in logically valid reasoning, the theorem says something like this:

"If it's believed that believing a particular prophesy would cause it to self-fulfill, then the prophecy will be believed."

Formally, the theorem just says □(□C→C)→□C , where □X means "X is provable". This is most weird if X is a false statement, because it means you can't prove that you can't prove X (Gödel's theorem).

(Also: the box symbol above is not an unrendered symbol; it's supposed to be a box.)

Meta-motivation

I'd like to make Löb's theorem more intuitive for humans, because it shows how agents can sidestep the need to mentally simulate each other in games, by instead using reasoning/arguments that happen to self-fulfill in a good and defensible way. Basically, Löbian reflection helps agents to avoid metacognitive stack overflows when thinking about themselves and each other, and I want more human beings to understand how that can work, and sometimes already does work, in real world agents.

State of the art

The best attempt I've seen to make Löb's theorem more intuitive is Eliezer Yudkowsky's Cartoon Guide to Löb's Theorem, which is still quite confusing. The confusingness comes from thinking about a self-referential statement (Ψ on Wikipedia; S on Arbital) that's used to carry out the proof. The statement basically says "If this statement is provable, then C." Dealing with that sentence is pretty cumbersome, and requires a lot of fiddling around with nested statements and implications.

Doing better

I think we can make a new proof of Löb's theorem that doesn't use that weird self-referential sentence, by instead making the proof of Löb's theorem itself self-referential. Page 15 of the following paper poses an open problem on how to do this, which I think is possible to resolve affirmatively:

Cooperative and uncooperative institution designs: Surprises and problems in open-source game theory (Critch, Dennis, Russell, 2022)

Here's a screenshot from it:

If we can make a proof like that work, we could then use the following much shorter and simpler cartoon guide to explain it:

In other words, I want to write a proof of Löb's theorem that is structured much like a Quine. I'd like the details to (eventually) be rea...
Corrigibility Via Thought-Process Deference
by Thane Ruthenis — 2022-11-24T17:06:39.058Z

We would ideally want the agent to [behave] as if it were thinking, "I am incomplete and there is an outside force trying to complete me, my design may contain errors and there is an outside force that wants to correct them and this a good thing, my expected utility calculations suggesting that this action has super-high utility may be dangerously mistaken and I should run them past the outside force; I think I've done this calculation showing the expected result of the outside force correcting me, but maybe I'm mistaken about that." — The Hard Problem of Corrigibility

Let's take that as a literal design specification.

1. High-Level Description

I propose that a corrigible mind design would involve the AI being recursively fed summaries of its own thought processes, set up such that the AI has uncertainty regarding the validity of its reasoning (with a strong initial prior for "this reasoning is bad") and can only get evidence on that via some pre-specified method that defers to humans, e. g. a particular feedback channel with humans on the other end[1]. The intended behavior is for it to summarize its thoughts in a non-manipulative human-readable format, get feedback on them, then update its reasoning policies in accordance with this feedback.

This aims to avoid the problem of fully updated deference by making the AI recursively uncertain of its thought-processes: not only about object-level problem-solving, but also about how it approaches minimizing its self-uncertainty ("should I really kill the people behind the feedback channel and seize control for myself?"), and how it translates its thoughts to humans ("should I really lie to get better feedback?"), and how it updates on human feedback ("should I really just ignore it?"). Any novel action-plan should be seized by uncertainty before being physically implemented like this, and sent for approval.

The intent is for the AI to start off uncertain even of its meta-meta-cognition (how it should ask for feedback and implement feedback), then gradually get more confident at it as humans approve its policies on that, until it no longer needs to run meta-meta-cognition past them. Then it'll gradually get better at meta-cognition and object-level cognition the same way, with varying degrees of certainty depending on how familiar a given thought-process is.

There's a lot of obvious problems with this setup, starting wit...
Geometric Exploration, Arithmetic Exploitation
by Scott Garrabrant — 2022-11-24T15:36:30.334Z

This post is going to mostly be propaganda for Thompson sampling. However, the presentation is quite different from the standard presentation. I will be working within a toy model, but I think some of the lessons will generalize. I end with some discussion of fairness and exploitation.

A Exploration/Exploitation Toy Model

Imagine you are interacting with the world over time. We are going to make a couple substantial assumptions. Firstly, we assume that you know what you are trying to optimize. Secondly, we assume that you get high quality feedback on the thing you are trying to optimize. In the spirit of the online learning community, we will be referring to your goal as "reward" rather than "utility."

On day t, you choose an action, at∈A, and then the environment responds with an observation et∈E. Both you and the environment may respond randomly, and may react to the entire history of what has happened. (If you want the environment to act first, you could just make a0 not matter.) We assume we have some fixed bounded reward function r:E→R, that you know. (If you want reward to be more of a function of the history, you can just include more history information in et.)

We are going to be imagining agents that are myopically choosing at, so as to maximize r(et). The choice to do this, rather than e.g. maximizing the total reward of the next m time steps, is mostly for simplicity. However, the choice not to have preferences that are over the entire future is basically the assumption that you get high quality feedback on what we are trying to optimize, which is a substantial philosophical assumption.

So, the environment can be thought of as a function, that takes in a history, (a0e0)(a1e1)…(at−1et−1)at, and returns a probability distribution on et which we then sample from. If the agent knew what the environment was, the agent would just choose the at each round which maximizes the the expectation of r(et). However, we want to model an agent that is uncertain about what environment it is interacting with, and learning over time.

Let us say that the agent has some probability distribution over possible environments. The set H (for hypotheses) of possible environments is the set of functions that take in a string of the form (a0e0)(a1e1)…(at−1et−1)at, and output a distri...
What I Learned Running Refine
by adamShimi — 2022-11-24T14:49:59.366Z

You have one job: Solving problems. You have multiple tools. Maybe you use code as a tool to solve some problems. Maybe you use design for others. Maybe you use good communication and negotiation skills.

Mike Acton,

How much time should I spend coding versus managing?

If you seek tranquility, do less. Or, more accurately, do what’s essential.

Marcus Aurelius, Meditations, Book 4.24

This post is part of the work done at Conjecture.

Refine

, the alignment research incubator we are running at Conjecture, finished its first cohort a few weeks ago. So now is a good time to take stock, share what we’ve learned, and discuss its future.

Let’s get this out of the way first: we are not planning any new cohort in the foreseeable future. There are multiple reasons for this, which I’ll expand on in this post. But to summarize:

Running Refine in a way that would fully aim at the stated target would require more effort
SERI MATS is doing a great job of scaling conceptual alignment research, and seem open to integrate some of the ideas behind Refine
The work we’re doing in Conjecture’s epistemology team is far more fundamental and neglected than field-building according to me, at least in the current climate.

Now for the details.

The Target

The

key idea behind Refine

was to create more conceptual alignment researchers with their own radically different agendas, rather than new researchers following established approaches. To create more researchers like

John

,

Paul

,

Vanessa

,

Evan

,

Steve

, and the others.

How we operationalized this goal was to look for

relentlessly resourceful

thinkers with unorthodox

shapes of minds

for the alignment community.

The Result

Now that the first cohort is over, how well have we hit this target? Out of 5 participants

2 are pursuing their own research bets, though these are not radically different from established approaches
1 is still building theirs
1 has found a neglected field-building opportunity
1 feels like they still need to upskill before working directly on alignment.

Based only on The Target above, this is 0/5.

Of course that doesn’t mean the program didn’t have positive outcomes and externalities! On the contrary, I’m really happy how a lot of things turned out, and I’ve heard from all participants that they got a lot out of Refine. Non-negligeable accomplishments include:

Feedback from multiple al

...
Covid 11/24/22: Thanks for Good Health
by Zvi — 2022-11-24T13:00:01.750Z

This Thanksgiving, I am thankful that I am once again back at full health. Covid comes and Covid goes. It was like I had a bad cold… for three days. That actually about sums it up, with some residual coughing. After a week, I was able to do a full session on the elliptical machine, so I consider everything back to normal.

I am not yet ready to wind down the Covid posts, but I am getting close to that point. If we get through the next two months without anything major happening, it will probably be time. For now, things will continue as normal.

Executive Summary

I’m feeling much better now.

China looks to be headed for more lockdowns, here we go again.

Person leaving WHO apologizes for denying Covid is airborne.

Let’s run the numbers.

The Numbers

Predictions

Predictions from Last Week: 250k cases (+4%) and 2,400 deaths (+12%).

Results: 253k cases (+3%) and 2,182 deaths (+2%).

Predictions for Next Week (Thanksgiving): 223k cases (-12%) and 1,850 deaths (-15%).

Standard holiday drop-off.

Deaths

Cases

Booster Boosting

Our booster strategy is a disaster. Monovalent Omicron boosters would be superior. A 4-to-6 month interval would have been superior. Our supplies are split between initial doses that can only be used initially and that are fully obsolete, and booster doses that are something like 60%-75% obsolete (although both are still far better than nothing, to be clear.) Many don’t know the updated boosters even exist.

Survey says 1/5 got the bivalent booster, while another 3/5 ‘definitely’ or ‘probably’ will get it in the future. Which tells me that people are lying in the survey. We are not going to get that kind of uptake, and most people who have not gotten the booster yet are not going to go do it later.

Here is what passes for public health messaging: ‘Because I said so.’

They are ‘better.’ So get them. How much better? You don’t need to know. Not your department. Cost-benefit analysis is not a thing. We are the jump department and we said jump, damn it.

To be fair, the rest of the the thread does give some numbers.

This is not nothing, if one is concerned and at high risk then boosting does seem worthwhile, yet I notice that none of this is especially encouraging with regard to boosters. If a booster alone was a 30% cut in symptomatic Covid permanently that woul...
Dumb and ill-posed question: Is conceptual research like this MIRI paper on the shutdown problem/Corrigibility "real"
by joraine — 2022-11-24T05:08:01.984Z

https://intelligence.org/files/Corrigibility.pdf
Here's the link ^

It is hard for me to phrase this question in a way that is correct and also not offensive. But research like this feels "not real" for some reason. It feels fuzzy, like random made-up philosophy that won't ever have any real impact on the world. It doesn't feel /real/ like a paper on "look this neural net is doing this bad thing, watch" or "look this model always lies and it's impossible to come up with a model that doesn't lie if it's in this format"

This just feels like pretend, made-up research that they put math equations in to seem like it's formal and rigorous. You just make up rules and then pretend they describe reality and then prove mathematically some results from there?
Does anyone else see what I'm saying about feeling like this sort of research is fake, and how would I convince myself that it isn't just useless random thoughts written down that kind of dance near the topic? At the end of all those questions, I feel no closer to knowing if a machine would stop you from pressing a button to shut it off.