trentmkelly/LessWrong-43k huggingface dataset:
Observation
by LoganStrohl — 2022-02-19T18:47:08.917Z

Imagine that you meet someone you’re attracted to at a party. At one point, they smile at you, and you notice. You’re pretty sure they like you, but you really want to know whether they like you like you.

You don’t act on this in any particular way, but you do spend the whole next week thinking about it. You think about other people who have been into you, and about people who have not, and the differences between them. You muse about what sort of taste in romantic partners you imagine the person might have. By the end of the week, you’re weighing your virtues and vices, trying to decide whether you’re even worthy of love.

(If this seems alien to you, I hope it is at least true to your experiences of some humans.)

In the moment when you noticed you were attracted to the person, you made an observation. In the moment when you noticed their smile, you made another. In the moment when you noticed your curiosity, you made another.

But as soon as you vanished into your own musings, you were no longer making observations. You were no longer collecting data. Instead, you were interpolating, extrapolating, filling in the gaps with stories and guesses, processing and reprocessing. Everything that followed, in the week after the party, took place inside your map—analysis, interpretation, reasoning, reflection.

In Arthur Conan Doyle’s “A Scandal in Bohemia,” Sherlock Holmes lectures Watson on the difference between seeing and observing:

“You see, but you do not observe. The distinction is clear. For example, you have frequently seen the steps which lead up from the hall to this room.”

“Frequently.”

“How often?”

“Well, some hundreds of times.”

“Then how many are there?”

“How many? I don't know.”

“Quite so! You have not observed. And yet you have seen. That is just my point. Now, I know that there are seventeen steps, because I have both seen and observed.”

I don’t know how many steps there are on the staircase up to my own living room, either. Setting aside the question of prioritization, and whether I should be turning my attention there—what is it, exactly, that Watson and I are doing with the steps?

My guess is that we’ve taken some initial impressions—a few moments of impact from the external world—and used those points to draw a constellation. Every time we walk up the steps, we do almost all of our processing on the constellation, rather than on the points of light in the sky.&...
Alignment researchers, how useful is extra compute for you?
by Lauro Langosco — 2022-02-19T15:35:31.751Z

TLDR

If you work in AI alignment / safety research, please fill out this form on how useful access to extra compute would be for your research. This should take under 10 minutes, and you don't need to read the rest of this post beforehand—in fact it would be great if you could fill out the form right now.

Introduction

I want to get an idea of how much demand there is for a university-independent organization that manages a compute cluster for academic AI alignment groups and independent researchers. Currently I don’t know anybody who is willing to run such an organization, but if demand is large one could either actively look for people to run such a project or find an existing organization that is willing to take it on.

Main idea

Non-industry AI safety research organizations have a hard time procuring compute. Groups spend many researcher-hours on managing servers on a relatively small scale. Common obstacles are 1) having to deal with university bureaucracy (e.g. regarding hiring, engineer wages, and procurement) and 2) missing out on economies of scale.

Proposal: A university-independent organization that provides access to compute for academic AI alignment research groups as well as independent researchers. Such an organization could pay high wages for its engineers (compared to academic labs) and benefit from economies of scale.

Potential benefits

Time saved: currently, researchers spend time applying for compute grants, setting up and maintaining servers, and setting up software environments after switching between systems. Easy access to large amounts of compute would avoid most of these time costs.

Expanded capability to do research: a centralized organization could afford to manage larger compute clusters than those usually used by individual labs. The difference is even larger for independent researchers, who might not have access to large-memory GPUs at all.

Potential problems

Gatekeeping: just like with other resources such as funding, deciding who gets access can be hard and risks becoming a political problem. OTOH, subject-specific grants are common / accepted within academia. Still, management of access would have to be done carefully.

Demand: some groups have access to large university clusters and may not need this service. I’m currently uncertain about how large the demand for this is.

Leadership: even if it were clear that this is a good

...
Mechanism design / queueing theory for government to sell visas
by RyanCarey — 2022-02-19T12:33:23.038Z

This question is just for my interest. Suppose you, the US government, have a finite supply of 65k H1B visas that you're prepared to offer per year, and you want them to go to the most economically valuable applicants. You could set a fixed fee, but then you'll run out of visas by the end of the year. You could have a blind auction every few days, but then companies have to wait until an auction happens to be certain of the outcome. It seems like instead, it should be possible to set a dynamic fee, that always offers a company a price to purchase any number of H1Bs (less than the total amount remaining), but that ensures that a number near to the total are offered per year?

To summarise, and abstract away the details of the scenario, I'm looking for a mechanism that works for the case where there is a finite supply of n items that has the following properties:

you always offer a price for any number m items that is less than the total remaining
when a customer makes a purchase, they definitely receive the items
about as much economic value is created, as possible, from the sale
you earn about as much revenue as possible

Presumably there's some intersection of queuing their and mechanism design that is designed to answer such a question?
Thoughts on Dangerous Learned Optimization
by peterbarnett — 2022-02-19T10:46:24.445Z

Epistemic status: confused.

I think there are two importantly different types of learned optimization which could be learned by an AI system, one of which is dangerous and one which isn’t. I am quite confused about this; unsure whether or not this distinction is real or non-trivial.

The two types of optimization that I see are:

Optimizing for a goal
Performing some kind of optimization/search process as part of accomplishing a task

For the first type (goal optimization), the system is ‘trying’ to do something, in the same way that a human might ‘try’ to accomplish a goal. The agent takes actions that affect the external environment, while in pursuit of its goal. I think this can happen most easily in an RL context, where we are training the system explicitly to optimize. I think this is dangerous and has the potential for misalignment. Examples of this would include, getting to a location as fast as possible, maximizing the number in a bank account, or maximizing presses of a reward button.

For the second type (search process), the system isn’t ‘trying’ to do anything, and part of its algorithm includes a search algorithm; stochastic gradient descent may just find a search algorithm which is useful for accomplishing a task. Examples of this include finding the largest number in a list, using gradient descent to find the position of the minimum of a function, or a language model performing some kind of search over its inputs to help it answer a question. I don’t see strong arguments about why this type of optimization/search would be dangerous. I think this kind of search process would be more like ‘find the word/vector which best satisfies this criterion’, and less like ‘change the state of the world to optimize for this criterion’. An example of this style of optimization in GPT is talked about

here

.

I’m mainly thinking about the development of search processes in the context of supervised learning and language models (systems which don’t explicitly take actions). These systems are often called Oracle AIs, and I think they will be safe if they are trained offline (not online as in

Predict-O-Matic

). This also seems similar to Eric Drexlers’s

Comprehensive AI Services

.

This distinction seems similar, but not identical to “

Search-in-Territory vs Search-in-Map

”, and also “

Selection vs Control

” and “

Mesa-Search vs Mesa-Control

”. Goal optimizat...
SPARC Applications Are Now Live!
by CitizenTen — 2022-02-19T02:15:49.322Z

The Summer Program for Applied Rationality and Cognition (SPARC) is a camp intended for really bright high school students to get together and become more “agenty.” It was originally spun up by people around CFAR to get math students interested in AI alignment and EA, though the focus has now expanded to include more than that. I attended SPARC last year, and it was sufficiently incredible that I want to bring the news to any high schoolers here that read Lesswrong.

Please apply – You won’t regret it!

As somebody who was and still is a frequent reader of LessWrong, it might not be at first apparent that getting to interact with others your age and talking about many of the same ideas would be so pivotal. But your monkey brain will disagree. In theory, you know that agenty people exist, who work on interesting things, and that you could be one of them, but until you actually meet them, it doesn’t become real. Furthermore, finding your “tribe” of young individuals like yourself is not something to be overlooked, especially if you don’t live in a major rationalist hub. Finding people you click with is genuinely amazing. Anyway, I don’t want to go on for too long, but even if this sounds even marginally appealing, please do actually sit down and writing yourself a reminder to apply. Be the agent you want to see in the world :)

How do I apply?

https://www.sparc-camp.com/apply

Note: The program is free for all participants, with financial assistance for travel available for those who need it.

Applications are due March 6th.

More firsthand accounts of SPARC can be found here:

https://www.sparc-camp.com/

https://www.quora.com/What-benefits-do-students-get-from-attending-the-SPARC-summer-program

https://www.quora.com/What-is-it-like-to-attend-SPARC-1
HCH and Adversarial Questions
by David Udell — 2022-02-19T00:52:29.949Z

This is a paper I wrote as part of a PhD program in philosophy, in trying to learn more about and pivot towards alignment research. In it, I mostly aimed to build up and distill my knowledge of IDA.

Special thanks to Daniel Kokotajlo for his mentorship on this, and to Michael Brownstein, Eric Schwitzgebel, Evan Hubinger, Mark Xu, William Saunders, and Aaron Gertler for helpful feedback!

Introduction

Iterated Amplification and Distillation (IDA) (Christiano, Shlegeris and Amodei 2018) is a research program in technical AI alignment theory (Bostrom 2003, 2014; Yudkowsky 2008; Russell 2019; Ngo 2020). It’s a proposal about how to build a machine learning algorithm that pursues human goals, so that we can safely count on very powerful, near-future AI systems pursuing the things we want them to once they are more capable than us.

IDA does this by building an epistemically idealized model of a particular human researcher. In doing so, it has to answer the question of what “epistemic idealization” means, exactly. IDA’s answer is one epistemically idealized version of someone is an arbitrarily large number of copies of them thinking in a particular research mindset, each copy thinking for a relatively short amount of time, all working together to totally explore a given question. Given questions are divided into all significant sub-questions by a hierarchy of researcher models, all relevant basic sub-questions are answered by researcher models, and then answers are composed into higher-level answers. In the end, a research model is able to see what all his relevant lines of thought regarding any question would be, were he to devote the time to thinking them through. Epistemic idealization means being able to see at a glance every relevant line of reasoning that you would think through, if you only had the time to.

One worry for IDA is that there might exist sub-questions researcher models in the hierarchy might encounter that would cause them to (radically) reconsider their goals. If not all models in the hierarchy share the same goal-set, then it’s no longer guaranteed that the hierarchy’s outputs are solely the product of goal-aligned reasoning. We now have to worry about portions of the hierarchy attempting to manipulate outcomes by selectively modifying their answers. (Take, for example, a question containin...
Encouragement to Instill Confidence?
by Zvi — 2022-02-18T22:30:03.722Z

I found this interesting and decided to overthink it.

Before proceeding, pause to think about whether this seems right to you and why.

The thesis here is something like:

There is a right amount of confident to be.
Encouragement proportionally raises confidence.
(Implicit) Encouragement does not do other important things.
→People often need encouragement.
People tend to be less confident than they seem.
Your estimate of their confidence will be based on how confident they seem.
→You will tend to overestimate their confidence level.
→You will tend to underestimate how much they need their confidence raised.
Your estimate of how much encouragement raises confidence is unbiased.
→You should give people more confidence than you expect them to need.

Also implicit is that ‘more’ is about the right level of adjustment here.

Once again: Does this seem right? Which steps seem right to you?

1: There is a right amount of confident to be → Mostly True.

Confidence is strange. It is often a partially-self-fulfilling prophecy. It has various different effects. When founding a startup it can only be accurate at 0% but you will be better off with it higher, when looking to date similar dynamics often apply. Still, we can have one-dimensional ethics, and declare that of all the options there is almost always a best one. And under uncertainty, there is almost always a best one to push someone else towards.

2: Encouragement proportionally raises confidence → True in expectation.

In general, encouraging someone will tend to raise their confidence. Some methods of encouragement do this more than others. Saying ‘that will work for sure’ will tend to focus its effects more on increasing confidence and in turn making action and hopefully success more likely. Saying ‘it’s good to attempt hard things, I’ll be proud of you for trying, good luck’ or ‘you will almost certainly fail or get turned down but the odds are so good you need to do it anyway’ might have different effects.

3: (Implicit) Encouragement does not do other important things → False

Encouragement provides information, changes probability of various actions, changes social context and who can get or expect credit and blame from who, will effect credibility and so on. Encouragement is in part a claim that has a truth value, sharing knowledge and your guess on their chances of success and the wisdom and expected value of their actions. Often en...
Misc thematic links
by HoldenKarnofsky — 2022-02-18T22:00:23.466Z

These are mostly links that contain some sort of interesting update or different perspective on stuff I've covered in past pieces.

Misc

I recently wrote a book non-review explaining why I haven’t read The Dawn of Everything. Something I didn’t know when I wrote this was that 8 days earlier, Slavoj Zizek had written a lengthy review of the recent Matrix movie that only revealed at the very end that he hadn’t seen it. A new trend in criticism?

18 charts that explain the American economy - I thought this was an unusually good instance of this “N charts that explain X” genre. Especially if you like “busted charts” where, instead of nice smooth curves showing that X or Y is on increasing/decreasing, you see something really weird looking and realize you’re looking at a weird historical event, like these:

I’m really into busted charts because I’m into focusing our attention on events so important you don’t have to squint to see them.

A study implying that the Omicron boosters I advocated for wouldn’t have helped even if we had rolled them out in time. Hey, I still think we should have tried it back when we didn’t know (and we still don’t know for sure), but I like linking to things showing a previous take of mine turned out wrong.

I’m excited about the idea of professional forecasters estimating probabilities of future events (more here and here), but I have no evidence to contradict this tweet from someone who’s been working in this industry for years:

That’s why despite years of forecasting and 1000+ questions answered it is surprisingly hard to find an example of a forecast which resulted in a change of course and a meaningful benefit to a consumer

— Michael Story ⚓ (@MWStory) January 17, 2022

A more technical analysis (which I have skimmed but not digested) of the same point made in This Can’t Go On: that our current rate of economic growth doesn’t seem like it can continue for more than another 10,000 years or so. This paper is looking at more fundamental limits than my hand-wavy “how much can you cram into an atom?” type reasoning.

AI

True that:

I'm old enough to remember when protein folding, text-based image generation, StarCraft play, 3+ player poker, and Winograd schemas were considered very difficult challenges for AI. I'm 3 years old.

— Miles Brundage (@Miles_Brundage) February 7, 2022

Here’s a fun piece in the “nonfiction science...
Abundance and scarcity; working forwards and working backwards
by Quinn — 2022-02-18T19:05:38.974Z

Epistemic status: I was bet $100 that I would not write this. Processing drastic opportunities, this is an extremely decision-relevant post and if my friends are right millions of dollars are lying in it.

Managing expectations: I am not writing a guide to hacking the mindset, I'm journaling my progress through reasoning about what hacking the mindset would look like.

Citational status: I am not naming names in this post, but if you recognize yourself in it I'm sending you much thanks and love

Content warning: cash, rich people problems.

I'm not great at "shut up and multiply"-ing my life. I have a scarcity mindset, probably from the years I was so poor that the (literal) man on the street (literally) stopped me to tell me I looked too skinny, that he could tell I wasn't eating enough. In 2019 when I was managing teaching assistants for a code bootcamp for 17.5/hr, I felt rich, in such a way that my bones don't feel too much richer now that I've 6x'd that. I don't think philanthropic goals are really in my bones. Sure I'll donate at least 20k this year, just because it feels like not doing that would be stupid, but when someone tells me about an opportunity to 2x or 10x from where I'm at now, my bones don't immediately say "ah yes, for those people I'd help with my donations", my response - my intuitive, gut response - is still "what? what for? I'm already rich". Yes, there are other reasons for savings: runway for self-funded alignment research, funding other alignment researchers, buying a house, etc. System 2, post-hoc, I can say a variety of reasons why more money is good, but I can't get system 1 to believe it. My system 1 just associates abundance with trivially/mindlessly getting takeout every meal because I don't like to invest time into food, and returns "we won". (I still don't get delivery, because I think that's too decadent, but that's another story).

But takeout every meal is not the story of abundance mindset, and in this post I will try to explain the difference.

I have a job that pays me in the 96th percentile of one of the richest countries, and I have massive intellectual satisfaction at work. Having been cash poor, and having been in stupid jobs that I hated, I've become rather conservative. My brain thinks that from "just like a capitalism" perspective (where you might nominally be creating nonzero prosocial value, but your emphasis is on just ...
Non-Fungible Costs
by hawkebia — 2022-02-18T17:15:12.897Z

You and your partner live in a cozy one-bedroom in the city. You enjoy being in the city bustle. It's got a quiet, sunny, plant-filled patio; and is close to all your favourite bars and restaurants.

But you've just had a baby. Work is a 40 minute commute. You can afford the current rent, but it feels expensive for the size. And you're just not visiting those bars and restaurants as often. Maybe you should move to that large apartment outside the city, and closer to both your offices?

So like good rationalists, you reason through the benefits, costs, degrees of certainty etc. and decide to move.

A few months in, your partner says they may be having regrets. You aren't. You feel great about the decision. But you're a loving and supportive partner. You both review the decision making process again with an open mind; and your partner arrives at the same decision. Yes, moving made sense.

You end up having to repeat this a few more times, till it all blows up in a big fight. How can they be so irrational?! How can one regret a decision, yet chose it over and over again? Either they have no clue what they want, or just aren't being honest while evaluating choices.

Or, there is another explanation - that unlike money, the costs and benefits are not fungible. They represent mostly independant sets of desires, which can't be substituted for one another. Sure your partner values the convenience, space, affordability etc. more than the bustle and social life of the city. But the cost of losing those things didn't suddenly go to zero once the decision was made.

Outside of pure monetary decision making, most choices pit at least some independant desires against one another. There is a case to be made that the negative reaction to a non-fungible cost, would be the same as if there were no offsetting benefit at all! And that should be considered completely expected and rational.

This applies to everything from life decisions, to public policy, altruism and company values. eg.

Automation / Moving to Renwables will lead to some people losing their jobs, but a net increase in well-paying jobs. But people, their livelihoods, their personal dignity aren't fungible; or at least they shouldn't be!

Facebook / Social Media is a net benefit to society. Leadership at these companies strongly believe this, as do I. It removes gatekeepers and democratizes speech. The increased access to informat...