trentmkelly/LessWrong-43k huggingface dataset:
Unit Test Everything
by DirectedEvolution — 2022-09-29T18:12:28.850Z

Inspiration

Specializing in Problems We Don't Understand
You Are Not Measuring What You Think You Are Measuring
Humans are very reliable agents
Epistemic Legibility

The SELEX Experiment

For my MS research, I'm generating an aptamer for IL-10, a crucial anti-inflammatory immune signaling molecule. Aptamers are short pieces of DNA that bind a specific target. We discover them through a process of directed evolution called SELEX. By dunking IL-10 in a pool of around a quadrillion distinct short DNA sequences and amplifying the DNA that sticks, we can select for sequences that happen to bind IL-10.

In our case, the aptamer also needs to change its shape dramatically when it binds. IL-10 is a delicate protein, and normal SELEX protocols risk denaturing it. On top of these interesting engineering challenges are the ordinary pains that come with bringing a new method to a lab. We must adapt the approaches others have taken to the equipment we own, write new protocols, move tools around, make space to store things, coordinate schedules, and come to consensus despite the different forms of expertise each of us brings to the table.

We've made several modifications to existing SELEX protocols. This is novel enough to be a solid first-author publication for me if the experiment works. It's also complicated enough to be practically guaranteed to fail if we don't approach it very carefully at each step.

What a Unit Test Is And Isn't

Coming from a computational rather than a wet-lab background, I've approached the preliminary work for this experiment by concocting what I think of as unit tests. The term has its origin in software engineering.

Unit tests involve breaking up the code, or a real-world process, into granular chunks and designing ways to confirm that each chunk is giving the expected output, based on a predefined and carefully controlled input. I like to apply this concept to other processes and activities beyond software engineering. So we need to expand this concept beyond its software roots, while preserving the spirit intact.

Unit tests are supposed to be separate from the main process they're testing. In software engineering, they are disabled and excluded when the code is compiled for distribution. When the process is deployed for real, it should not depend on the unit tests in any way to work correctly.

Unit Testing Principles, Practices, and Patterns identifies four...
Builder/Breaker for Deconfusion
by abramdemski — 2022-09-29T17:36:37.725Z

This is something of a grab-bag of thoughts I've had about the Builder/Breaker game.

The ELK document had a really nice explanation of its research methodology in terms of an imaginary dialogue between a "Builder" who makes positive proposals, and a "Breaker" who tries to break them. To an extent, this is just the ordinary philosophical method,[1] and also a common pattern in other research areas. However, I felt that the explicit write-up helped to clarify some things for me.

We might think of the Builder/ Breaker game as an adversarial game where either the builder or breaker "wins", like AI debate. However, I find it more fruitful to think of it as a cooperative game. When the game is played by AI safety researchers, the players have a common goal of finding robust plans to avoid catastrophic outcomes. The builder/breaker game merely organizes cognitive work: both Builder and Breaker are trying to map the space of proposals, but each takes primary responsibility for avoiding a different kind of error (false positives vs false negatives).

Security Mindset

I think Builder/Breaker is a good way to understand Eliezer's notion of security mindset (1, 2). The Builder is trying to construct a positive argument for safety, with (at least[2]) the following good properties:

The argument clearly states its assumptions.
Each assumption is as plausible as possible (because any grain of doubt indicates a possibility of failure).
There are as few assumptions as possible (because more assumptions mean more ways the plan can fail).
Each step of reasoning is sound.
The conclusion of the argument is a meaningful safety guarantee.

I will call such a plan robust. We can question whether AI safety research should focus on robust plans. I won't dwell on this question too much. Clearly, some endeavors require robust plans, while others do not. AI safety seems to me like a domain which requires robust plans. I'll leave it at that for now.[3]

In any case, coming up with robust plans has proven difficult. The Builder/Breaker game allows us to incrementally make progress, by mapping the space of possibilities and marking regions which won't work.

Example: Wireheading

I could easily forgive someone for reading a bunch of AI alignment literature and thinking "AI alignment researchers seem confident that reinforcement learners will wirehead.". This confusion comes from ...
Resources to find/register the rationalists that specialize in a given topic?
by tailcalled — 2022-09-29T17:20:19.752Z

As far as I can tell:

It's not possible for me (or most/all other people, for that matter) to individually evaluate claims every topic I hear about, as it takes too long time and too much effort to learn the foundations needed to evaluate things accurately. Therefore I (and all other people?) need to rely on other people to evaluate topics that I am unfamiliar with.
On a lot of topics, common experts are incompetent; for instance most social science research is just bad, and it's mainly bad because the entire fields have no clue what they're doing. Therefore it seems unwise to rely on those fields. It's unclear why they are so bad, but IME rationalist experts are much better, perhaps because they share my biases, perhaps because rationalism teaches useful tricks, but I think more likely because rationalism just tends to attract incredibly intelligent and curious people.
However, one cannot just blindly rely on opinions by rationalists, as rationalists are not automatically good at everything; when rationalists are unfamiliar with a topic, they often say lots of totally clueless stuff.[1]

So in my view there's basically a lot of utility in finding the rationalists most specialized in each topic to make it easy to learn stuff from them. Has anyone worked on this in the past? Is this something someone (Lightcone Infrastructure, perhaps?) would be interested in setting up?

And I guess if you know of any underrated rationalists who specialize in some topic, feel encourage to share in the comments for this post?

^

This also seems to apply to non-rationalists, but that's not as important for this purpose.
Make-A-Video by Meta AI
by P. — 2022-09-29T17:07:15.664Z
Link: https://makeavideo.studio/

Meta AI (Facebook) created a text-to-video model by taking a diffusion text-to-image model, adding temporal convolutional and attention layers, and fine-tuning it with video data (without text). They also use spatial and temporal super-resolution networks. Showing, to the surprise of no one who was paying attention, that our existing mostly homogeneous architectures can be easily extended to understand, to some extent, the structure of everyday reality. It's not the first text-to-video model, but it's much better than what came before.
FDT is not directly comparable to CDT and EDT

by SMK — 2022-09-29T14:42:59.020Z

In

Functional Decision Theory: A New Theory of Instrumental Rationality

and

Cheating Death in Damascus

, the authors go through a multitude of decision problems and compare the performance of functional decision theory (FDT), on the one hand, and classical evidential and causal decision theory (EDT and CDT) on the other; and they come to the conclusion that former “outperforms”[1] the latter two. That is, FDT is implicitly treated as a decision theory directly comparable to EDT and CDT. I will argue that this methodology is misguided and that there is a strong sense of incommensurability in terms of their respective action recommendations: FDT belongs in a different ontology (together with all other logical decision theories), and although we might be able to compare ontologies, directly comparing the respective decision theories is not meaningful since we lack a common measurement for doing so.

The 2x2x2 decision theory grid

In the

MIRI/OP decision theory discussion

, Garrabrant points out that we can think about different decision theories as locations in the following three-dimensional grid (simplifying of course):

Conditional probability vs. counterfactuals (EDT vs. CDT).

That is, P(o | a) or P(o | do(a)).[2]

Updatefulness vs. updatelessness (“from the perspective of what epistemic state should I make the decision, the prior or posterior?”).

That is, P(o | ..., obs) or P(o | ...).
Perhaps this dimension should be split up into empirical updatelessness, on the one hand, and logical updatelessness on the other.)

Physicalist agent ontology vs. algorithmic/logical agent ontology (“an agent is just a particular configuration of physical stuff doing physical things” vs. “an agent is just an algorithm relating inputs and outputs; independent of substrate and underlying structure”).

Notationally, this corresponds to using P(o | a) or P(o | "DT(s)=a") if we are evidentialists, and P(o | do(a)) or P(o | do("DT(s)=a")) if we are causalists.
(I am using “ontology” to mean something like “a map/model of the world”, which I think is the standard LW usage.)
Demski (also in the

MIRI/OP discussion

) makes this distinction even more fine-grained: “I am my one instance

...
[Link] "Improper Nouns" by siderea
by Kenny — 2022-09-29T13:28:42.268Z
Link: https://siderea.dreamwidth.org/1773806.html

The post:

siderea | Improper Nouns [p/a/s, Patreon]

Interestingly, 'improper nouns' – as described by the post – is itself an example of the phenomena it describes (as 'improper noun' is itself an existing term):

There's a thing you probably have encountered and maybe haven't really registered as it went by. It usually plays out like this. Somebody else takes a noun phrase – usually there's an adjective or two involved, sometimes multiple nouns – that seems perfectly ordinary and makes sense on the face of it, and which refers to something in the real world, and the speaker makes some bold assertion about it that strikes you as really weird, and possibly highly prejudicial. What they've said leaves you thinking, "wait, how could that possibly be true?"

That's all you get. That's all the sign of what's happening. That's the glitch in the matrix.

There's a lot of reasons that people make ridiculous claims about all sorts of things, and you will always be tempted to write it off as just ridiculous people saying ridiculous things and thinking no more about it.

But if you can catch yourself in that moment, and not reflexively dismiss it as silliness, you could, instead, ask yourself: "Hey, is that noun phrase maybe a technical term of art which means something much more specific and possibly quite different from what the individual words literally mean?"

Doing this? Is a super power. Lots of people can recognize specific technical terms that they are acquainted with. Few people cultivate the more general ability to recognize – or just suspect – that a noun phrase is a technical term they aren't familiar with.

This is a useful enough idea that it's already been referenced in another post here.
Open application to become an AI safety project mentor
by Charbel-Raphaël — 2022-09-29T11:27:39.056Z
Link: https://docs.google.com/forms/d/e/1FAIpQLScghGTf_mMHzllSgXhbWQD91wgY2ckDVnJSbNGMHeuhcBuFFw/viewform

EffiSciences is a French organization committed to a more involved research to address the pressing problems of this century.

We are organizing bootcamps for French university students in AI safety. After teaching an intensive curriculum (PyTorch, ML, interpretability, transformers, conceptual AI safety), students start a research project. We also organize research group et seminars that end on research projects in top French universities.

We are looking for qualified mentors to supervise groups of 1-3 students on a technical or conceptual AI safety project. Supervision consists of a 1 or 2 hours session per week for about 3 months.

Our students come from the best French engineering schools and universities and will be familiar with the concept of the Alignment problem. We think that your contribution with this supervision would have a significant chance to create a new AI Safety researcher, as well as give you some experience in mentoring bright students.

If you have any doubts about your availability or skills, we still encourage you to apply using the following form. We will most likely contact you in any case.

https://docs.google.com/forms/d/e/1FAIpQLScghGTf_mMHzllSgXhbWQD91wgY2ckDVnJSbNGMHeuhcBuFFw/viewform
Should reasonably healthy people still take Paxlovid?
by Sameerishere — 2022-09-29T03:41:02.271Z

Should a non-immunocompromised, fairly healthy person in their late 30s take Paxlovid if they get one of the currently circulating strains of Omicron?

How would that answer change if:

Their symptoms are fairly mild (bad cold)
They have a high enough BMI to qualify for Paxlovid, but not a super high BMI?

My previous understanding was that it was a good idea to reduce the severity of the case and risk of long-term effects.

However, a few friends' advice leads me to think that taking Paxlovid for a mild case does not make sense for a reasonably healthy non-elderly person:

"Paxlovid's usefulness is questionable and could lead to resistance. I would follow the meds and supplements suggested by FLCC https://covid19criticalcare.com/.../i-care-early-covid.../" [not sure how reliable that site is, as it recommends Ivermectin, though it seems to be composed of actual MDs]
[3 anecdotes of nasty rebounds]
Twitter threads suggesting Paxlovid reduces your protection against later reinfection [which doesn't really seem like the most important thing to be optimizing for]

https://twitter.com/michaelzlin/status/1574823881528864768?s=20&t=1W5akHDBMpAY-PboFqe3lA
https://twitter.com/michaelzlin/status/1573835507842551809?s=20&t=1W5akHDBMpAY-PboFqe3lA

On the other hand:

Another friend said: "Eric Topol twitter vaguely seems to suggest that data are broadly indicative of protection; haven't heard of much side effects"
(And 2 friends in their 30s took it and were fine)
Reflection on a Consulting Workshop
by myutin — 2022-09-29T03:04:23.537Z

Epistemic status: This is a summary / review / critique of a workshop I attended today. The workshop's materials are not publically available, so I don't expect any productive discussion regarding the workshop itself. This is mostly for myself to process the new information.

Introduction

I work at a consulting firm. All new hires attend an annual course. The course consists of three full-day workshops.

I'm making this sound like a corporate nightmare, but it's not. The format is very flexible. The organizer has a slide deck, but it mostly serves as a seed for group discussion. The first slide is a Kendrick Lamar song. Hopefully that conveys the tone of the workshop.

From here on, my paragraphs will be strictly alternating paraphrasis and analysis.

Consulting

The workshop's definition of consulting is very broad: consulting is when you help someone do something. In a corporate context, you're fulfilling the client's explicitly defined needs. In other contexts, you're working towards favorable outcomes.

This sounds like optimizing for someone else. But my suspicions may be misplaced, as we discussed only the definition today.

Eight Principles

Most of today's discussion centers on the proper mindset for consulting. This consists of eight principles. Each principle also serves as general advice for interacting with people.

The organizer has read the Tao (or so they say). They arrange the eight principles on an infographic in Taoist imagery. Perhaps this evokes a feeling of Deep Wisdom. I find no intersection with my (albeit limited) knowledge of Taoism. This inspires distrust, but should not detract from the ideas' inherent merits.

The first principle is the ego check. Everyone has two egos (no relation to Freud). Your "big ego" says you're the smartest person in the room. Your "little ego" says you have nothing to contribute. You should remove both of these feelings before a conversation starts, the same way you would check your coat at the door. During group introductions, everyone should comment on how they check their ego.

This seems like reasonable life advice in general. However, I can't detect any change in my mental state when I say my ego check. Verdict indecisive.

The second principle is humanism: treat people nicely, feed their ego, and they'll help you succeed. The third principle is curiosity: ask people questions, and they'll enjoy answering them. The fif...
Better Construction Cost Estimates?
by jefftk — 2022-09-29T02:30:01.456Z

When I wrote about how Somerville's affordable housing "bonus density"
rules made it likely

profitable to build 100%
affordable housing,

Chris
pointed out that I was underestimating construction costs (I was
using $150-$200/sqft, though also finding some parcels that looked
good even at $600/sqft). I looked into this some and had trouble
finding good estimates, and then I forgot about the conversation and
posted about how a for-profit

affordable housing
investment fund could be a good idea. At which point Chris needed
to

again
remind me that my cost estimates were likely too low. Whoops!

Chris gave two examples:

A recently completed 98 unit, 95,000 sqft development in Cambridge had
a total cost of $587,000 per unit (Around $100k of that was land
cost.). Another for-profit affordable housing project started
post-pandemic is expecting to have a total cost—with *zero* land
cost—of $615,000/unit. A public housing development is expecting to
pay $960,000/unit. (That one is especially bad for a lot of reasons
related to public procurement, federal public housing authority
mandates, and cross-subsidy of unfunded spaces like community rooms,
but just giving a sense of where these things sit.)
These are for units that have a maximum sale price of around
$250,000-$300,000, under the legislation.

The first one is $503/sqft excluding land. The second two don't
include unit sizes, but assuming they're also 970sqft they come to
$634/sqft and $990/sqft. These are all depressingly high, and I agree
you can't profitably construct affordable housing at those prices.

On the other hand, these are all much higher than I'd expect?
According to the
census, new construction in the Northeast had a median cost of
$179/sqft in 2021 (xls,
sheet).
This already includes things like lumber being more expensive during
the pandemic; it's 24% more expensive than 2020 and 31% more than
2019. Now, the Boston area is one of the most expensive parts of the
Northeast, Somerville can be hard to work with, and wood-frame is
cheaper than 5-over-2. But people building new houses generally make
them fancy ("luxury") and cheaper finish work should push
things the other way. (You usually don't see this because
'affordable' units need to be as nice as the market-rate units in the
development, but this doesn't apply if you're building 100%
affordable.) I also expect ...
Petrov Day Retrospective: 2022
by Ruby — 2022-09-28T22:16:20.325Z

This is a follow-up to the Petrov Day commemoration of 2022

We died.

Twice.

Maybe three times.

This is how it went down...

September, 21, 5:43 PM Pacific Time: I write the code that determines whether a user is authorized to launch. I was in a rush because I leaving for EA Global DC.

September 25, 8:00 PM: Petrov Day celebration commences. The button is on the frontpage. Any users with greater than 2,300 karma could press the button.

September 25: 9:05 PM: I am reminded by others at LessWrong that I was supposed to publish the launch code in the comments. Without the code, no one can nuke the site. Understandably, no one spoke up to point out the absence of code – that would be hella suspicious. This reveals that incompetence on occasion might even save you from destruction. But don't bet on it.

September 25, 10:55:04 PM: The frontpage is nuked after 2 hours and 55 minutes. The karma threshold is 2,100 and there are 290 users with that much karma or more.

September 25, 11:51 PM: Oliver Habryka is suspicious that there is a bug in the code and he digs into it, uncovering that indeed, due to an error in the logic, any user with 0 karma (and exactly 0 karma) was already able to launch nukes and destroy the frontpage. An anonymity-preserving database check confirmed that it was a user with 0 karma who had launched the missiles.

September 26, 12:36 AM: The decision is made to revive the frontpage and keep going with the Petrov Day commemoration.

September 26, 10:11 AM: The entire LessWrong website (not just the front page) goes down and is returning 502s. This is because I pushed a bad commit to fix hiding the Petrov Opt-Out checkbox without properly testing it. The site is soon restored to functioning.

September 26, 5:33:02 PM: At this time, the karma threshold for the ability to launch is 200, and 1,504 users were able to bring down the front page.

With two hours and 27 minutes remaining, someone presses the initiation button, enters the launch code, and presses fire. The frontpage goes down.

With the anonymity introduced in 2022, we do not know who pressed the button or why.

September 26, 8:00:00 PM: The Petrov Day commemoration is concluded. The LessWrong frontpage is restored.

What a day. What a day indeed.

I will tell you the symbolic lessons I take from these events, though you are welcome to interpret things how you please.

First, we very successfully com...