trentmkelly/LessWrong-43k huggingface dataset:
Saying no to the Appleman
by Johannes C. Mayer — 2022-04-29T10:39:48.693Z

Today I was overhearing an interesting conversation between my father and a door-to-door salesperson. It was interesting to witness the inability of my father to say no, and the tactics that the salesperson used. Nobody knew that I was listening.

My father tried to tell that salesperson no perhaps 5 times. My father was always trying to fish for arguments to justify the no. The salesperson seemed to be listening carefully, making sounds that indicate acknowledgment and understanding. But then when he began to speak again, he ignored the argument and tried to give reasons why buying apples is good. Or he would seem to offer something special to my father by reducing the minimum order quantity.

As an argument, my father said something along the lines of there being many people in the household that all need to be financially supported, and that therefore buying these overpriced apples is not a good choice. Though he did not mention the apples being overpriced. And the salesperson without a moment's hesitation interjected that if there are so many people, then it would be better to buy 40kg instead of 20. And he managed to say it in a joking tone, making it not seem offensive.

(So a joking tone of voice can be used to put options on the table that would be socially inappropriate otherwise, without taking a reputation hit. Interesting.)

And for offering something special he used a cute little psychological trick. Ideally, you would like to sell as much as possible. Having a minimum buy quantity is good to check the reaction of the potential buyer. If it is only a bit above what he would like to spend you might be able to talk him into buying more than he otherwise would. If he does not seem like he is going to take the offer you can just lower the minimum buy quantity. And then it seems like the seller is doing you a favor. But of course, selling anything is better than nothing. And the seller did never offer to lower the price.

He only talked about that there is some other place my father probably never heard of, where the apples cost 5$, and that therefore his offer of 4$ per 1kg of apples must be good.

The seller started with 30kg being the minimum order quantity the first time he showed up. Each time after that my father said no I think. And each time the seller lowered the minimum order quantity. From 30 to 24 to 20. And this time he lowered it again to 15, and t...
Prize for Alignment Research Tasks
by stuhlmueller — 2022-04-29T08:57:04.290Z

Can AI systems substantially help with alignment research before transformative AI? People disagree.

Ought is collecting a dataset of alignment research tasks so that we can:

Make progress on the disagreement
Guide AI research towards helping with alignment

We’re offering a prize of $200-$2000 for each contribution to this dataset.

The debate: Can AI substantially help with alignment research?

Wei Dai

asked the question

in 2019:

[This] comparison table makes Research Assistant seem a particularly attractive scenario to aim for, as a stepping stone to a more definitive [AI Safety] success story. Is this conclusion actually justified?

Jan Leike

thinks so

:

My currently favored approach to solving the alignment problem: automating alignment research using sufficiently aligned AI systems. It doesn’t require humans to solve all alignment problems themselves, and can ultimately help bootstrap better alignment solutions.

Paul Christiano

agrees

:

Building weak AI systems that help improve alignment seems extremely important to me and is a significant part of my optimism about AI alignment. [...] Overall I think that "make sure we are able to get good alignment research out of early AI systems" is comparably important to "do alignment ourselves." Realistically I think the best case for "do alignment ourselves" is that if "do alignment" is the most important task to automate, then just working a ton on alignment is a great way to automate it. But that still means you should be investing quite a significant fraction of your time in automating alignment.

Eliezer

doesn't

:

"AI systems that do better alignment research" are dangerous in virtue of the lethally powerful work they are doing, not because of some particular narrow way of doing that work. If you can do it by gradient descent then that means gradient descent got to the point of doing lethally dangerous work. Asking for safely weak systems that do world-savingly strong tasks is almost everywhere a case of asking for nonwet water, and asking for AI that does alignment research is an extreme case in point.

Everyone would likely agree that AI can help a little, e.g. using next word prediction to write papers slightly faster. The debate is about whether AI can help enough with alignment specifically that it substantially changes the picture. If AI alignment is 70% easy stuff we can automate and 3...
Increasing Demandingness in EA
by jefftk — 2022-04-29T01:20:01.507Z

In thinking about what it means to lead a good life, people often
struggle with the question of how much is enough: how much does our
morality demand of us? People have given a wide range of answers to
this question, but effective altruism has historically
used "giving 10%".
Yes, it's better if you donate a larger fraction, switch to a job
where you can earn more, or

put your
career to use directly, but if you're giving 10% to effective
charity you're doing your share, you've met the bar to consider
yourself an EA, and we're happy to have you on board.

I say "historically", because it feels like this is changing; I
think EAs would generally still agree with my paragraph above, but
while in 2014 it would have been uncontroversial now I think some
would disagree and others would have to think for a while.

EA started out as a funding-constrained movement. Whether you looked
at global poverty, existential risk, animal advocacy, or movement
building, many excellent people were working as volunteers or well
below what they could earn because there just wasn't the money to
offer competitive pay. Every year GiveWell's total room
for more funding was a multiple of their money moved. In this
environment, the importance of donations was clear.

EA has been pretty successful in raising money, however, and the
primary constraint has shifted from money to people. In 2015, 80k made
a strong case for focusing on what people can do directly, not
mediated by donations, and this case is even
stronger today. Personally, I've found this pretty convincing, though in 2017 I
decided to return to
earning to give because it still seemed like the best fit for me.

What this means, however, is that we are now trying to build a
different sort of movement than we were ten years ago. While people
who've dedicated their careers toward the most critical things
have made up the core of the movement all along, the ratio of impact
has changed.

Imagine you have a group of people donating 10% to the typical mix of
EA causes. You are given the option to convince one of them to start
working on one of 80k's priority areas,
but in doing so N others will get discouraged and stop donating. This
is a bit of a false dilemma, since ideally these would not be in
conflict, but let's stick with this for a bit because I think it is
illustrative. In 2012 I would have put a pretty...
What is a training "step" vs. "episode" in machine learning?
by Evan R. Murphy — 2022-04-28T21:53:24.785Z

My impression is that steps and episodes are both time periods in a training process, and that these terms are somewhat common in RL. An episode is larger than a step and usually contains many steps.

Is this correct?

Some related questions:

Are rewards issued at the end of steps, at the end of episodes, or it could be either/both just depending on the particular training process being used?
Is a step the smallest unit of time in which any observable activity happens? Or could there be a lot of things going on within a single "step"?
Do these units of time persist in deployment, or are they only relevant during training?

I'd appreciate answers to any of these questions if you know them. (It's ok if you don't know all the answers or don't have time to include them all in a single answer.)

What's the context?

I have seen these terms come up in discussions about myopia in AI.

For example, in Evan Hubinger's post about AI safety via market making, he invokes the terms "per-step" as well as "per-episode" myopia:

Before I talk about the importance of per-step myopia, it's worth noting that debate is fully compatible with per-episode myopia—in fact, it basically requires it. If a debater is not per-episode myopic, then it will try to maximize its reward across all debates, not just the single debate—the single episode—it's currently in. Such per-episode non-myopic agents can then become deceptively aligned, as they might choose to act deceptively during training in order to defect during deployment. Per-episode myopia, however, rules this out. Unfortunately, in my opinion, per-episode myopia seems like a very difficult condition to enforce—once your agents are running multi-step optimization algorithms, how do you tell whether that optimization passes through the episode boundary or not? Enforcing per-step myopia, on the other hand, just requires detecting the existence of multi-step optimization, rather than its extent, which seems considerably easier. Thus, since AI safety via market making is fully compatible with per-step myopia verification, it could be significantly easier to prevent the development of deceptive alignment.

Understanding these terms more clearly will help me understand proposals about myopia, which is relevant to some research I'm doing right now. These also seem like basic or generally important concepts in ML which I'd like to understand better.
Facts Matter
by mrdlm — 2022-04-28T21:19:38.599Z

There comes a time in the life of a student of philosophy when they start thinking facts aren’t important. What starts off as a passion to stamp out falsehoods and replace them with truths, morphs into an acute uninterest, and eventually disdain towards facts. I have suffered from this mental disorder for a long time, but have recently gotten cured. So here I am, standing on the other side, healthier and wiser, and preparing to persuade you.

Philosophy is a study of certain kinds of general ideas that are abstracted from particular details. So we talk about morality as a general principle, instead of focusing on real-life, on-the-ground policies. Making correct assessments about policies requires one to know many facts about what’s going on, and acquiring those facts comes with messy procedures, dirty hacks, approximations, and uncertainties. You can’t write a syllogism and marvel at the knowledge that if the argument is sound, and if the premises are true, then the conclusion must be true. Real-life isn’t like that. It’s not clear what’s going on out in the world because everything is too complex. Too many different things happening all at once, and no good ways of cleanly parsing them out. To the degree that anything can be understood, the job is left to specialists in each field. Philosophers, on the other hand, stand quite far back, look at the final results, discard all the unnecessary details, and abstract ideas from them to talk about.

Instead of being curious about acquiring more facts, and consequently enriching their picture of a situation, the compulsion is to wash away any messiness so that only pristine abstractions remain. And upon those beautiful structures, they apply the tools of logic and reasoning smoothly. Sometimes principles are plucked out from the facts and clothed in fancy made-up fictional settings called Thought Experiments. The working of the fine machinery of philosophy can only be witnessed properly when the problems are diluted and transposed in this manner, to the point where all that is ever achieved are some intellectual stimulations for the participants. The problems will remain intact after all the discussions are finished; the real solutions and their implementations are left to the men and women without philosophy degrees, who’re on the ground putting their hands to use.

But even if you’re not interested in solving real pro...
Is alignment possible?
by Shay — 2022-04-28T21:18:25.891Z

Is AGI alignment even possible in the long term? Will AGI simply outsmart our best defenses? It would be, after all, superhuman (and by an enormous margin). Isn’t it likely that an AGI will recognize what actions humans took to control it and simply undo those controls? Or just create a novel move, like AlphaGo did, and completely sidestep them. An AGI could also just wait until conditions are favorable to take charge. What is time to an immortal intelligence? Especially time as short as a few human lifespans. Unless mis-alignment is physically impossible, it seems as if all attempts will ultimately be futile. I hope I’m wrong.
Two Prosocial Rejection Norms
by Emrik — 2022-04-28T20:53:15.850Z

When I tell people that I'm shy, the universal response I get is "wow, you don't seem shy at all!"

Problem statement

Pop culture psychology has a single concept for "debilitatingly reluctant to approach people", and it's called "shyness" or "social anxiety" and is explained by "fear of rejection". I am debilitatingly reluctant to approach people, but I am very low on fear of rejection, what gives?

I propose that my 'shyness' can be explained by very high levels of what I'll call meta-fear of rejection, and fear of fake willingness.

Symptoms of being kind.

Friends I've talked to about this all identify as having these to different degrees, and what works out to be optimal social norms for each person depends on this 'rejection profile'.

But before we go into it, let's be clear about what the goal is: We want to maximise opportunities for mutually beneficial interactions. Popular advice is often all about benefiting the receiver of said advice, so people are encouraged to "just walk up to them!" to overcome fear of rejection with little consideration for how this might impact others. And this is particularly useless as advice for people with high levels of 3 and 4.

Another obstacle to forming optimal social norms is the fuzziness of the language we use to express invitations, requests, and offers.

O how I lament that my invitations are interpreted as requests sometimes.

Sometimes people pretend to make invitations when they're really making requests, and this lets them benefit from the other person's willingness while claiming to not be indebted to them.

It was just an invitation, I don't owe you anything!

Conversely, sometimes people pretend to have heard a request when the other person actually intended to invite them, and that way they can claim to be up a favour.

But I only did that as a favour to you!

These are all forms of social manipulation enabled by a fuzzy language. And the deception wouldn't work without a high frequency of honest mistakes in this area.

So when I walk up to someone and invite them "hey, wanna be friends?" I have to worry about 1) fear of rejecting, and 2) them mistakenly or intentionally misunderstanding my invitation as a request. I just want to be able to communicate my invitation as exactly what it is, and for them to turn me down without hesitation and pain on their part.

Two marginally better social norms

Rejection aid

When you approach someon...
Dath Ilan vs. Sid Meier's Alpha Centauri: Pareto Improvements
by David Udell — 2022-04-28T19:26:26.664Z

Epistemic status: Rambly; probably unimportant; just getting an idea that's stuck with me out there. Small dath-ilan-verse spoilers throughout, as well as spoilers for Sid Meier's Alpha Centauri (1999), in case you're meaning to get around to that.

The idea that's most struck me reading Yudkowsky et al.'s dath ilani fiction is the idea that dath ilan is puzzled by war. Theirs isn't a moral puzzlement; they're puzzled at how ostensibly intelligent actors could fail to notice that everyone can do strictly better if they could just avoid fighting and instead sign enforceable treaties to divide the resources that would have been spent or destroyed fighting.

This … isn't usually something that figures into our vision for the science-fiction future. Take Sid Meier's Alpha Centauri (SMAC), a game whose universe absolutely fires my imagination. It's a 4X title, meaning that it's mostly about waging and winning ideological space war on the hardscrabble space frontier. As the human factions on Planet's surface acquire ever more transformative technology ever faster, utterly transforming that world in just a couple hundred years as their Singularity dawns … they put all that technology to use blowing each other to shreds. And this is totally par for the course for hard sci-fi and the human future of the vision generally. War is a central piece of the human condition; always has been. We don't really picture that changing as we get modestly superintelligent AI. Millenarian ideologies that preach the end of war … so preach because of how things will be once they have completely won the final war, not because of game theory that could reach across rival ideologies. The idea that intelligent actors who fundamentally disagree with one another's moral outlooks will predictably stop fighting at a certain intelligence threshold not all that far above IQ 100, because fighting isn't Pareto optimal … totally blindsides my stereotype of the science-fiction future. The future can be totally free of war, not because any single team has taken over the world, but because we got a little smarter about bargaining.

Let's jump back to SMAC:

This is the game text that appears as the human factions on Planet approach their singularity. Because the first faction to kick off their singularity will have an outsized influence on the utility function inherited by their superintelligence, late-game w...
A Parable Of Explainability
by George3d6 — 2022-04-28T16:46:24.280Z
Link: https://www.epistem.ink/p/a-parable-of-explainability

Ok, sure, machine learning is great for making predictions; But you can’t use it to replace scientific theory. Not only will it fail to reach generalizable conclusions, but the result is going to lack elegance and explainability. We won’t be able to understand it or build upon it!

What makes an algorithm or theory or agent explainable?

It's certainly not the ability o "look inside"; We're rather happy assuming that block boxes, such as brains, are capable of explaining their conclusions and theories. We scoff at the idea that perfectly transparent neural networks are "explainable" in a meaningful sense; So it's not visibility that makes something explainable.

Why was that prediction made as a function of our inputs and their interactions? Under what conditions would the outcome differ? These are questions of explainability, questions we may assume are critical for the pursuit of science. Answers to them hold most of the upside of explainability.

There's a running assumption that people are better than trained algorithms by this second definition. I believe it's mainly a matter of semantics.

Let's think of a physicist that takes in hundreds of petabytes of experimental data, reduces them using equations, and claims with a high probability that there exists this thing called a "Higgs Boson", and that it has various properties with implications for gravity and a bunch of other things.

The resulting Boson can probably be defined within a few pages of text via things such as mass, the things it decays into, its charge, its spin, the various interactions it can have with other particles, and so on.

If a Luddite like myself asks the physicists: "Why did you predict this fundamental particle exists?", he may give me a pop-sci story like:

Well, it was one of the missing particles that we've speculated about for years and fits our theories, helping us merge our models of particles with how gravity works. These experiments helped us "see" this boson in a way that's hard to doubt.

But this obviously explains nothing, it just tells me that the physicist is really sure and thinks this is awesome. If I press on and ask for a real explanation, one that requires counterfactuals and the quantitative motivation behind the claim I'd get something like:

Look, the standard model is a fairly advanced concept in physics, so you first have to understand that and why it came to be. Then you ha

...
Keep your protos in one repo
by RobertM — 2022-04-28T15:53:26.803Z
Link: https://docs.protocall.dev/blog/monorepo

This analysis is targeted at organizations that have not yet locked in their strategy for managing protobufs/gRPC, or have the capacity to pivot. I expect the principles involved to generalize to other organizational questions. Some of the tradeoffs are only relevant from a "protos as service interfaces" (gRPC/twirp/etc) orientation, but others are more general.

Meta-level conflict of interest notice: This is a linkpost to a blog for a (commercial) side-project of mine.

Object-level conflict of interest notice: Protocall currently only supports single-repo workspaces, not multi-repo workspaces. This analysis represents an accurate accounting of my view on the tradeoffs involved in this decision, and is not motivated by any desire to make my own life easier. Implementing basic support for multi-repo workspaces would not be an overwhelming technical challenge; the edge cases around e.g. namespace collisions would need solving but I would only need to solve them once.

Most engineering organizations should keep their proto files in one repo. If you're already operating with a single monorepo for your entire codebase, nothing about protos changes the calculus. If you're not, the benefits of keeping your proto files in one place may not be obvious.

Here are some (valid) reasons for not keeping your proto files in one repo.

Locality

If your proto files are primarily used to define service interfaces, keeping them in the same repo as the service code can reduce overhead for developers working on that service. They only need to issue one PR to update the service interface, rather than two. Working across multiple repos is challenging for most build systems, so there might also be less tooling work involved in maintaining a quick development loop.

Differential Processes, Tooling, and Norms

If your org has multiple repos, it's likely that teams own their own repos. Those teams may have wildly different engineering processes, tooling, norms, etc. Depending on the details, those may be possible to maintain in harmony within a single repo, but it can be less overhead to have teams manage their own processes with the repo as the dividing line of responsibility.

Avoiding (Inappropriate) Shared Dependencies

Protos come with the ability to import other protos. This makes for a very tempting footgun. While there are some rare situations where it makes sense to import a proto...