trentmkelly/LessWrong-43k huggingface dataset:
Should we buy Google stock?
by Random Trader — 2022-05-15T18:38:24.009Z

It seems Deepmind is winning the AGI race. In the past, they negotiated with Google for having a nonprofit legal structure, but the proposal got totally rejected.

They have an agreement that, if AGI is achieved, its control will rely on Deepmind's Ethics Board, which we know pretty much nothing about. One would think it's very likely that Google will end up making a profit.

So... what do you think? Is Google stock the best investment a rationalist can make at this very moment, given the information that we have?
The AI Countdown Clock
by River Lewis — 2022-05-15T18:37:47.659Z
Link: https://heytraveler.substack.com/p/the-ai-countdown-clock?r=178u1&s=w&utm_campaign=post&utm_medium=web

I made this clock, counting down the time left until we build AGI:

It uses the most famous Metaculus prediction on the topic, inspired by several recent dives in the expected date. Updates are automatic, so it reflects the constant fluctuations in collective opinion.

Currently, it’s sitting in 2028, i.e. the end of the next presidential term. The year of the LA Olympics. Not so far away.

There were a few motivations behind this project:

Civilizational preparedness. Many people are working on making sure this transition is a good one. Many more probably should be. I don’t want to be alarmist, but the less abstract we can make the question, the better. In this regard, it’s similar to the Doomsday Clock.
Personal logistics. I frequently find myself making decisions about long-term projects that would be deeply affected by the advent of AGI. Having kids, for example.
The prediction is obviously far from absolute, and I’m not about to stop saving more than 5 years and 11 months of living expenses. But it’s good to be reminded that the status quo is no longer the best model for the future.
Savoring the remainder. Most likely, AGI will be the beginning of the end for humanity. Not to say that we will necessarily go extinct, but we will almost definitely stop being “human,” in the recognizable/traditional sense.
For many years, I’ve used the Last Sunday as my new tab page. It shows you how many Sundays you have left in your life, if you live to an average age. I’ve gotten some strange looks, when it accidentally pops up during a presentation. I know it seems morbid, like a fixation on the end. But I don’t see it that way; it’s not about the end, but the finitude of the middle. That precious scarcity.
I’ve spent a lot of time thinking about the meaning of being human, but this mostly dissolves that angst. It’s like: if I live in San Francisco, my daily decisions about what to do here are impossibly broad. But if I’m a tourist, visiting for a few days, then my priority is to do all of the most “San Francisco” things I can: see the Golden Gate, eat a burrito in the Mission, sit among purple-red succulents on a foggy beach cliff.
So, if I only have X years left of being human, I should focus on doing the most human things I can. Whatever that means to me. Conveniently, this applies to the worlds with and without AGI, since in the latter, I’ll still die. But the shorter time

...
Is AI Progress Impossible To Predict?
by alyssavance — 2022-05-15T18:30:12.103Z

People seem to be continually surprised, over and over again, by the new capabilities of big machine learning models, such as PaLM, DALL-E, Chinchilla, SayCan, Socratic Models, Flamingo, and Gato (all in the last two months!). Luckily, there is a famous paper on how AI progress is governed by scaling laws, where models predictably get better as they get larger. Could we forecast AI progress ahead of time by seeing how each task gets better with model size, draw out the curve, and calculate which size model is needed to reach human performance?

I tried this, and apparently the answer is no. In fact, whether AI has improved on a task recently gives us exactly zero predictive power for how much the next model will improve on the same task. The sheer consistency of this unpredictability is remarkable, almost like a law of statistical thermodynamics. No matter what I plug in, the correlation is always zero! For example, does a task improving rapidly when you go from a small model to a 7B parameter model predict similar improvement when you go from a 7B model to Gopher's 280B? No:

I tried making the same graph with MMLU tasks instead of BIG-bench, same result:

What about DeepMind's new Chinchilla? Did rapid improvement of a task on Gopher predict continued improvement going from Gopher to Chinchilla? Nope:

What about Google's PaLM? The full results of PaLM on BIG-bench don't seem to have been published yet, so I couldn't directly compare to Chinchilla or Gopher, but the PaLM paper described an 8B parameter model, a 62B model and a 540B model. Did fast improvement from 8B to 62B predict improvement from 62B to 540B? Not really, R^2 = 0.04:

PaLM also provides data on 30 different NLU benchmark tasks. Plot those and you get the same thing:

The results here seem pretty clear, but I'm honestly not sure how to interpret them. Before trying this, I assumed you would find that some tasks are "easy" and scale quickly, while others are "hard" and scale slowly. But that would get you high predictability, since fast progress between one pair of models would imply that the task is inherently "easy", and predict (perhaps with some noise) fast progress on the next pair. I didn't see that.

You could also have a theory where tasks scaled similarly (all are of comparable "difficulty"), but there was some noise between model training runs, so that task performance on any given run would bo...
Is it possible to implement switching between sequences from its pages?
by EniScien — 2022-05-15T09:19:20.864Z

(When I first met, I didn’t have an account, so I didn’t publish this question for a long time, I didn’t decide, and then I got used to it, but I’ll write anyway.)
I was VERY pleasantly surprised by the design of the original (this) site compared to the Russian version, which was below average in user-friendliness, while this site is not only well above average, but literally the most user-friendly site I have ever used. With one exception. In the Russian version, you could use the arrows to move both between messages in a sequence, being on one of the messages, and between sequences in chapters and chapters in a book, being on their pages, respectively. Immediately, instead of the page of this sequence, you go to the last message of another if you press the arrow on the last message of the sequence.
Is it possible to implement this functionality as I say? You can continue to go directly to the message, and to the sequence by its link, as now, and simply add arrows to the sequence itself. (It seems to me that it will be much more convenient, but maybe I don’t know something, maybe this functionality was before and according to the results of the polls, it was decided that the current solution is more convenient for other users, or there are some technical reasons, or it’s just me worries.)
Gato as the Dawn of Early AGI
by David Udell — 2022-05-15T06:52:02.264Z

Written in a hurry today at the EA UCLA AI Timelines Workshop. Long and stream-of-thought, and a deliberate intellectual overreach as an epistemic exercise. My first foray into developing my own AGI timelines model without deferring! Please, I beg of you, tell me why I'm wrong in the comments!

Epistemic status: Small-N reasoning. Low confidence, but represents my standing understanding of AGI timelines as of now.

This exchange caught my eye a couple days ago:

Yitz:

Would it be fair to call this AGI, albeit not superintelligent yet?

Gato performs over 450 out of 604 tasks at over a 50% expert score threshold.

👀

Daniel Kokotajlo:

Yes. Sub-human-level AGI.

If true, this is a huge milestone!

Here I'm combining thinking about this with thinking about AGI 10 years hence. The latter forecasting task is totally different if we have a form of AGI as of two days ago, even an admittedly weak form.

Do We Have "Subhuman AGI" as of Two Days Ago?

If I want to forecast AGI 10 years out, I first want to understand current-year AGI. Do we currently have AGI? What does it look like? What hyperparameters, up to and including overall architecture, might we push on and make progress on in the coming decade?

To start, I'll read the Gato paper out of DeepMind, investigating for myself Daniel's above claim that Gato constitutes subhuman AGI:[1]

"A Generalist Agent" (Reed et al., 2022)

We focus our training at the operating point of model scale that allows real-time control of real-world robots, currently around 1.2B parameters in the case of Gato. As hardware and model architectures improve, this operating point will naturally increase the feasible model size, pushing generalist models higher up the scaling law curve. For simplicity Gato was trained offline in a purely supervised manner; however, in principle, there is no reason it could not also be trained with either offline or online reinforcement learning (RL) (p. 2).

Gato is small, parameters-wise. At 1.2 billion parameters, it's 1/100th the size of the largest GPT-3 model and 1/500th the size of PaLM. This is a choice due to hardware constraints; larger models, which definitely could have been used here, would not be able to operate real-world robotic limbs in real time. Additionally, the tasks on which Gato was trained were chosen purely incidentally: they could have easily been otherwise. So Gato is a miniaturized version of models to come...
What's up with the font size in the Markdown text editor?
by Ege Erdil — 2022-05-14T21:12:20.812Z

Since the rollout of the recent updates to LessWrong, the font size in my Markdown editor for new posts or questions has become much smaller. Posts appear as normal once they are saved as drafts or published. Is this normal and is there a way to fix it?

If the problem proves difficult to reproduce, happy to provide pictures as necessary.
[Link post] Promising Paths to Alignment - Connor Leahy | Talk
by  — 2022-05-14T16:01:18.795Z

Date of recording: Tuesday May 10, 2022

Description

AI Safety Support

recently hosted a closed, introductory talk by Connor Leahy on Promising Paths to Alignment.

The talk covers: (i) why Alignment is such a difficult problem, (ii) current approaches to solving it, and (iii) some info on Connor’s new Alignment Research startup,

Conjecture

.

I highly recommend this talk for developing a better understanding of the technical Alignment research landscape - particularly for those considering or pursuing a related career.
Inequality is inseparable from markets 
by  — 2022-05-14T13:39:56.295Z

Cross posted to The Good blog

I think prices are often really good at allocating resources efficiently. I also think that utility is roughly the log of income and we have a duty to make sure that no one lives in poverty. It would be great if there was no tradeoff between these goals - efficient allocation of resources, inequality and poverty. But I think there is and it comes down to how the price mechanism works. I spend the first section of this post explaining the price mechanism, so feel free to skip it if you’re already familiar.

The price mechanism

The price mechanism has some really attractive qualities. When prices go up there’s an incentive to try to produce more of that good. Prices rise, in a competitive market, either because more people want a good and because that good has got more valuable - i.e demand has increased. Alternatively, prices could go up because it’s become more expensive to produce that good. In an uncompetitive market agents have a lot more leeway on how they set prices. But, eventually they’ll want to set them above their marginal cost. They can only do that because they have market power. If they didn’t, then other agents would come in and sell the goods at a lower price. As long as the price is above the cost of producing one additional unit of the good, agents are willing to undercut their competitors and sell the good at a lower price1.

In all three of these cases the price mechanism is doing something really useful. When demand goes up, the price mechanism does the really hard job collecting information from everyone in the economy - weighted by how much money they spend - and turning that into a single number that says, according to this weighted average, how much people value a good or service compared to everything else that can be bought and sold. When prices go up in an uncompetitive market it sends a different, but also extremely socially valuable signal. It says that there’s money to be made in this market, so more agents try to enter it, increasing competitiveness and lowering prices. Finally, if prices rise because marginal cost is going up it creates an incentive to find a way to produce the same goods more cheaply.

The tradeoff

So why does this bake in inequality, even in a perfectly functioning market? The way you know if you’ve followed the price signals correctly is that you’ve made money. The eva...
Predicting the Elections with Deep Learning - Part 1 - Results
by Quentin Chenevier — 2022-05-14T12:54:46.454Z

Hi everyone 👋,

During the last 18 months, I played — maybe too much — with census & elections data and tried to predict the french elections with deep learning. I'm not officially a ML researcher, that's only a side project, so be nice please 🙏 ! I'm sharing this only to have fun discussions with people who have the same kink as me. 😊

Here it is 👉 Predicting the Elections with Deep Learning - Part 1 - Results

You don't have to be a data scientist to read this 1st post which talks only about the results of the experiment.

TLDR of this 1st post:

No I didn't find a way to predict the elections results
But It's surprising how much you can learn about voters behavior (who votes for which party), even using only aggregated public data.
It's only qualitative results so it's not real science.
Still, it's interesting (worrying?) to think about what would be possible with more data (e.g: FAANG)

After that, I'll write 2 more posts: 1 for the model implementation and 1 for the MLOps tooling I've used.
Clarifying the confusion around inner alignment
by Rauno Arike — 2022-05-13T23:05:26.854Z

Note 1: This article was written for the EA UC Berkeley Distillation Contest, and is also my capstone project for the AGISF course.

Note 2: All claims here about what different researchers believe and which definitions they endorse are my interpretations. All interpretation errors, though carefully avoided, are my own.

Over the recent years, there have been several LessWrong posts arguing over the most useful way to define inner and outer alignment. The terms were, as far as I know, initially defined in the “Risks from Learned Optimization” paper by Hubinger et al. Naturally, the terms have since been taken into contexts not anticipated at the time of writing the paper, which has prompted several arguments as well as redefinitions.

In this post, I will attempt to summarize and clarify the main points of confusion behind those arguments. To keep the post concise, I will focus on the definition of inner alignment, as this seems to be the term generating the most confusion. When talking about outer alignment, I will adopt the definition from Evan Hubinger’s post clarifying inner alignment terminology, as that’s the definition I feel like is currently used the most: “An objective function r is outer aligned if all models that perform optimally on r in the limit of perfect training and infinite data are intent aligned.” So, I take outer alignment to roughly mean that humans succeed in choosing a training objective that is perfectly aligned with our own values for the AI system. Note, though, that there have been arguments over this definition as well.

Decomposing the alignment problem

First, it seems useful to go over the distinct steps that we will have to succeed on in order to align an AI system. To me, the following decomposition seems sensible:

We may or may not specify the correct training objective.
The training process may or may not generate a separate optimizer for fulfilling that objective, and that optimizer may or may not be aligned with the base objective.
The result of the training process may or may not generalize to the test environment, whether or not a mesa-optimizer was created. This robustness problem is 2-dimensional: a generalization failure may arise either from the system’s objective failing to generalize from the training distribution to the deployment distribution, or from its capabilities failing to generalize.

Classifying the problem...
Costs and benefits of amniocentesis for normal pregnancies
by braces — 2022-05-13T22:47:03.761Z

Disclaimer: No medical training.

Amniocentesis is a sample of the amniotic fluid to test fetal DNA at 15-20 weeks of pregnancy. It involves inserting a long needle into the amniotic sack, which is potentially risky. But it could catch serious defects.

Arguably the main cost of the procedure comes from an increased risk of miscarriage, which increases by about 1 in 1,000. The best-estimated risks (see this

meta-analysis

) may still be confounded by selection into the procedure, but experts agree the risk is non-zero. Depending on the person, you might also consider the stress entailed in the process, the feeling of not being done testing, and the cost of a false positive.

What are the benefits? This is harder, because pregnancy involves a lot of correlated tests. For instance, Down syndrome can be detected in "cell free" DNA testing (also known as NIPT) at week 8. It's also detected with much higher accuracy in the amnio. But a negative in the cell free DNA test will drastically reduce the chance that an amnio comes back positive for Down, so it certainly reduces the benefit of amnio--enough that current convention is to not recommend amnio after a negative NIPT result.

There are also ultrasounds. These can detect issues that are also caught in genetic tests, like Down. But they can also detect issues that are not currently detectable in DNA. For instance, it's estimated that roughly half of all cases of Noonan syndrome are novel, meaning a genetic test for them wouldn't come back positive based on known mutations.

So the amnio can help catch things that are (i) genetic and (ii) invisible to the ultrasound and other tests.

To complicate things further, DNA testing of the amniotic fluid is currently performed at three levels of detail. From lowest to highest resolution:

Karotyping or cytogenetic. The lowest resolution, this essentially counts chromosomes. It detects the same things as the cell free DNA test, but with greater accuracy.
Microarray. Can catch missing or deleted chromosomal segments.
Prenatal exome sequencing: Genotyping the fetus. Explainer

here

.

Getting an increased risk of Down in the cell free DNA test will greatly increase the benefits of the test. Most medical advice treats the amnio as an obvious choice if there's a finding of increased risk.

What about for the majority of people, who have not had any...
Frame for Take-Off Speeds to inform compute governance & scaling alignment
by Logan Riggs — 2022-05-13T22:23:12.143Z

Figure 1: Something happens at future time t' that causes more resources to be poured into alignment

The argument goes: there will be a time in the future, t’, where e.g. a terrible AI accident occurs, alignment failures are documented (e.g. partial deception), or the majority of GDP is AI such that more people are pouring resources into aligning AI. Potentially to the point that >90% of alignment resources will be used in the years before x-catastrophe or a pivotal act (Figure 2)

Figure 2: potentially the majority of total resources poured into alignment happen after t'

The initial graph (Fig. 1) seems surprisingly useful as a frame for arguing different cruxes & intuitions. I will quickly enumerate a few & would appreciate comments where you disagree.

Compute governance w/o considering hardware/software overhang is net-negative

If we just govern compute usage while advances in hardware/software are continued, this may lead to just shifting t’ to the right w/o slowing down timelines, which implies less resources poured into alignment in total for no benefit.

If we limit compute successfully for many years, but hardware & software improvements continue, an actor can defect and experience a large, discontinuous increase in capabilities. If we (somehow) limit all of them, it will become much much harder to produce transformative AI (intuition: it’s like someone trying to build it today).

Operationalizing the Cause of t’

As mentioned before, t’ could be caused by “a terrible AI accident occurs, alignment failures are documented (e.g. partial deception), or the majority of GDP is AI such that more people are pouring resources into aligning AI.”

There are other potential causes as well, and I would find it beneficial to investigate how true the above 3 (and others) are as far as convincing real AI researchers into switching their research focus to alignment. I mean literally talking to researchers in machine learning and asking what capabilities (negative or positive) would get them to seriously consider switching their research focus.

Additionally, if we find that e.g. showing partial deception in models really would be convincing, then pouring resources into showing that sooner would be shifting t’ to the left, implying more overall resources poured into alignment.

More Resources Don’t Imply More Progress

I expect most AI researchers ...
Alignment as Constraints
by Logan Riggs — 2022-05-13T22:07:49.890Z

In order to find the most promising alignment research directions to pour resources into, we can go about it 3 ways

Constraints all alignment proposals should have
Constraints for current research directions
Constraints for new research directions

Constraints all alignment proposals should have

We can imagine the space of all possible research directions

This includes all possible research directions, including pouring resources into McDonald's HR Department. But we can add constraints to focus on research directions more likely to help advance alignment.

If you can tell a story on how it reduces x-risk from AI, then I am slightly more excited about your proposal. But we can continue to add more constraints like:

By constraining more and more, we can narrow down the space to search and (hopefully) avoid dead ends in research. This frame opens up a few questions to ask:

What are the ideal constraints that all alignment proposals should have?
How can we get better at these constraints? e.g. You can tell a better story if you build a story that works, then break it, then iterate on that process (Paul Christiano commonly suggests this). If we can successfully teach these mental movements, we could have more researchers wasting less time.

Constraints for current research directions

We can also perform this constraint-narrowing on known research agendas (or known problems). A good example is this arbital page on boxed AI, clearly explaining the difficulty of:

Building actually robust, air-type systems that can't effect the world except through select channels.
Still getting a pivotal act from those limited channels

Most proposals for a boxed-ai is doomed to fail, but if the proposal competently accounts for (1) & (2) above (which can be considered additional constraints), then I am more excited about this research direction.

Doing a similar process for e.g. interpretability, learning from human feedback, agent foundations research would be very useful. An example would be

Finding the constraints we want for interpretability, such as leading to knowledge of possible deception, what goal the model is pursuing, it's understanding of human values, etc.
Tailor this argument to research at Redwood, Anthropic, OpenAI and get their feedback.
Repeat and write up results

I expect to either convince them to change their research direction, be convinced myself, ...