trentmkelly/LessWrong-43k huggingface dataset:
Why I am not an AI extinction cautionista
by Shmi — 2023-06-18T21:28:38.657Z

First, to dispense with what should be obvious, if a superintelligent agent wants to destroy humans, we are completely and utterly hooped. All the arguing about "but how would it...?" indicates lack of imagination.

...Of course a superintelligence could read your keys off your computer's power light, if it found it worthwhile. Most of the time it would not need to, it would find easier ways to do whatever humans do by pressing keys. Or make the human press the keys. Our brains and minds are full of unpatcheable and largely invisible (to us) security holes. Scott Shapiro calls it upcode:

Hacking, Shapiro explains, is not only a matter of technical computation, or “downcode.” He uncovers how “upcode,” the norms that guide human behavior, and “metacode,” the philosophical principles that govern computation, determine what form hacking takes.

Humans successfully hack each other all the time. If something way smarter than us wants us gone, we would be gone. We cannot block it or stop it. Not even slow it down meaningfully.

Now, with that part out of the way...

My actual beef is with the certainty of the argument "there are many disjunctive paths to extinction, and very few, if any, conjunctive paths to survival" (and we have no do-overs).

The analogy here is sort of like walking through an increasingly dense and sophisticated minefield without a map, only you may not notice when you trigger a mine, and it ends up blowing you to pieces some time later, and it is too late to stop it.

I think this kind of view also indicates lack of imagination, only in the opposite extreme. A conjunctive convoluted path to survival is certainly a possibility, but by no means the default or the only way to self-consistently imagine reality without reaching into wishful thinking.

Now, there are several (not original here, but previously mentioned) potential futures which are not like that, that tend to get doom-piled by the AI extinction cautionistas:

...Recursive self-improvement is an open research problem, is apparently needed for a superintelligence to emerge, and maybe the problem is really hard.

...Pushing ML toward and especially past the top 0.1% of human intelligence level (IQ of 160 or something?) may require some secret sauce we have not discovered or have no clue that it would need to be discovered. Without it, we would be stuck with ML emulating humans, but not r...
My impression of singular learning theory
by Ege Erdil — 2023-06-18T15:34:27.249Z

Disclaimer: I'm by no means an expert on singular learning theory and what I present below is a simplification that experts might not endorse. Still, I think it might be more comprehensible for a general audience than going into digressions about blowing up singularities and birational invariants.

Here is my current understanding of what singular learning theory is about in a simplified (though perhaps more realistic?) discrete setting.

Suppose you represent a neural network architecture as a map A:2N→F where 2={0,1}, 2N is the set of all possible parameters of A (seen as floating point numbers, say) and F is the set of all possible computable functions from the input and output space you're considering. In thermodynamic terms, we could identify elements of 2N as "microstates" and the corresponding functions that the NN architecture A maps them to as "macrostates".

Furthermore, suppose that F comes together with a loss function L:F→R evaluating how good or bad a particular function is. Assume you optimize L using something like stochastic gradient descent on the function L with a particular learning rate.

Then, in general, we have the following results:

SGD defines a Markov chain structure on the space 2N whose stationary distribution is proportional to e−βL(A(θ)) on parameters θ for some positive constant β>0 that depends on the learning rate. This is just a basic fact about the Langevin dynamics that SGD would induce in such a system.

In general A is not injective, and we can define the "A-complexity" of any function f∈Im(A)⊂F as c(f)=Nlog2−log(|A−1(f)|). Then, the probability that we arrive at the macrostate f is going to be proportional to e−c(f)−βL(f).

When L is some kind of negative log-likelihood, this approximates Solomonoff induction in a tempered Bayes paradigm - we raise likelihood ratios to a power β≠1 - insofar as the A-complexity c(f) is a good approximation for the Kolmogorov complexity of the function f, which will happen if the function approximator defined by A is sufficiently well-behaved.

The intuition for why we would expect (3) to be true in practice has to do with the nature of the function approximator A. When c(f) is small, it probably means that we only need a small number of bits of information on top of the definition of A itself to define f, because "many" of the possible parameter values for A are implementing the function ...
Alaska Trip
by jefftk — 2023-06-18T13:40:02.026Z

A few weeks ago I flew to Anchorage to play for the

Dancing Bears
Dance Camp with the

Free
Raisins. It was really fun to play with Amy and Audrey again, to
dance to

KGB, to meet
people from dance communities all over Alaska, and to visit the
farthest North I've been. Some pictures from the trip, with thoughts
hanging off them.

Now that I work with
wastewater I'm much more interested in sewage treatment plants.
Here's Boston's Deer
Island plant, right next to the airport:

I love looking out the window on flights, and occasionally get decent pictures. Here's Hamilton
and Burlington,
SW of Toronto at
the end of Lake
Ontario:

Looking SE from Park
Ridge Station along the UP-NW
Line towards downtown Chicago:

Even though I left Chicago at 7pm it stayed light the entire way
there, making for pretty good views. Lots of clouds approaching
Anchorage, but they broke as we flew West along Turnagain Arm,
following the road (and train) West from
Whittier
(internet-famous for its density, with almost everyone living in a
single apartment
building). In the lower right you can see Beluga
Point, which we later visited terrestrially:

The Anchorage suburbs felt a lot like other parts of the US, except
from almost anywhere there was some direction you could look to see
snow-covered mountains:

Once we got to the camp and there was less evidence of human
habitation this made it feel a lot like a Bob Ross painting:

I brought my 5x5 and 7x7 go boards
and got at least a dozen people to give it a try. You can teach it
quickly, games are fast, and then people can play each other and get a
feel for it:

One of the other dancers brought Ecologies,
a tableau-building card game. I enjoyed it, but with six players it
was much too long. This was partly that five of us were new to the
game, but it doesn't help that it takes time proportional to the
number of players. It has a draw-decide structure, which I think nearly always should be rotated to a
decide-draw structure so people can think more on their opponents'
turns. It also wasn't very well balanced (review)
but I enjoyed the trading, card art, and company:

While teaching a session on playing music to support the dancers a
moose wandered by:

While the most common Alaskan contra dancer's car is probably a
Subaru, perhaps the most literal combination of the...
UK Foundation Model Task Force - Expression of Interest
by ojorgensen — 2023-06-18T09:43:27.734Z
Link: https://twitter.com/soundboy/status/1670343527723679744

Ian Hogarth has just been announced as the Chair of the UK's AI Foundation Model Taskforce. He's the author of the FT article "We must slow down the race to God-like AGI", and seems to take X-risks from AI seriously.
To quote his twitter thread:

And to that end I put out a call to people across the world. If you are an AI specialist or safety researcher who wants to build out state capacity in AI safety and help shape the future of AI policy then get in touch:
We have £100m to spend on AI safety and the first global conference to prepare for. I want to hear from you and how you think you can help. The time is now and we need more people to step up and help.

The google form to leave an expression of interest is here.

(I am in no way affiliated with Ian or the UK Foundation Model Task Force)
Cryonics Career Survey (more jobs than you think)
by Mati_Roy — 2023-06-18T02:13:11.159Z

(x-post LessDead)

Survey presentation

Link to survey: Cryonics Career Survey

Background. During the annual Teens and Twenties Cryonics Gathering (a fully funded yearly gathering for young cryonics members), there is usually a survey that is distributed to all participants to gauge their interests around getting involved in various roles in cryonics, professionally or as a volunteer. This has been useful, and I think it would be even more valuable if it were accessible for anyone to fill out–so I made my own version.

Goals. The goals of this survey are to gather people that are either:

(potentially) interested in utilizing their skills/knowledge/experience to support cryonics endeavors – there are many work opportunities you probably don’t know about;

(potentially) interested in learning and seeking guidance/mentorships/information about having a career in cryonics – we’re already in touch with cryonics researchers willing to mentor.

Note: Ramping up to being able to do meaningful cryonics research might be easier than you think, so I invite you to err on the side of signaling your interest to us.

Email use. We will email the list with work opportunities, as well as (if you opt-in) share your information with interested and approved cryonics(-relevant) organizations for the purpose of helping them find people to work for them. We will also email the list with any information relevant to working in cryonics, such as relevant interviews of cryonics researchers or mentorship opportunities. We expect to send emails infrequently—less than once per month. You can unsubscribe at any time of course.

Related opportunities:

If you’re an MD and/or PhD and support the Scientists’ Open Letter on Cryonics, please also let us know.

If you’re involved in cryonics and willing to mentor others, please reach out!

Fill out the survey here: Cryonics Career Survey

Currently open roles

After an extensive web search of cryonics organizations we know of, here is a list of some current job postings we were able to find (when the country isn’t specified it means it’s in the USA or remote) – some of those roles might have recently close, but I’m leaving them up to show the diversity of options (most or all should still be open at the time of posting though). Fill out the survey to get information about future such opportunities.

Advanced Neural Biosciences (Portland, OR)

Research

...
Solomonoff induction still works if the universe is uncomputable, and its usefulness doesn't require knowing Occam's razor
by Christopher King — 2023-06-18T01:52:25.769Z

Note: I don't think this idea is original, but I couldn't find a good post going over the implications.

I used to think that Solomonoff induction was a bit arbitrary for the following reason: it assigned a 100% probability to the universe being computable. I'm pretty sure the universe is computable (ignoring randomness), but nowhere near 100% sure. Who's to say we won't find a halting oracle floating in space for no reason? That seems like a pretty simple hypothesis. Why the focus on recursive languages? You have to make some choice of how descriptions work (you can't assign positive probability to every infinite bit string), but that didn't change the feelings of arbitrariness.

But then I realized this understanding of why to use Solomonoff induction is incorrect. We do not use it because of the physical church-turing thesis, we use it because of the original church-turing thesis:

L.C.M.s [logical computing machines: Turing’s expression for Turing machines] can do anything that could be described as ‘rule of thumb’ or ‘purely mechanical’. - Alan Turing

Because what matters is not whether the universe is computable, but whether our methods of reasoning are computable. Or in other words, whether the map is computable. Solomonoff's induction is at least as "good" as any computable inference method (up to a constant), regardless of the complexity of the universe. So if you, as a human, are trying to come up with a systematic way to predict things (even uncomputable things), Solomonoff's induction is better. Here is the precise statement:

Theorem: Let D be some probability distributions on infinite sequences of bits such that inferring the next bit from a prefix is computable. The likelihood ratio from D to Solomonoff induction's prior is bounded above by some finite constant (despite the sequence containing infinitely many bits), and this constant is independent of the sequence of bits.

Proof sketch: (Note: this is already a well-known result.) There is a program m that is a decoder for an entropy code over finite bit strings based on D. Given any finite bit string x, we can find a y such that m(y)=x. The length of y is approximately −log2(px) (where px is the probability of a bit string starting with x according to D). The probability of a string starting with x under Solomonoff induction's prior is greater than 2−|m(y)|=2−|m|2log2(px)=px2|m|...
DSLT 2. Why Neural Networks obey Occam's Razor
by Liam Carroll — 2023-06-18T00:23:14.725Z

TLDR; This is the second main post of Distilling Singular Learning Theory which is introduced in DSLT0. I synthesise why Watanabe's free energy formula explains why neural networks have the capacity to generalise well, since different regions of the loss landscape have different accuracy-complexity tradeoffs. I also provide some simple intuitive examples that visually demonstrate why true parameters (i.e. optimally accurate parameters) are preferred according to the RLCT as n→∞, and why non-true parameters can still be preferred at finite n if they have lower RLCT's, due to the accuracy-complexity tradeoff. (The RLCT is introduced and explained in DSLT1).

It is an amazing fact that deep neural networks seem to have an inductive bias towards "simple" models, suggesting that they obey a kind of Occam's Razor:

Plurality should not be posited without necessity.

or in modern parlance,

If two models of the world are similarly accurate, the simpler explanation should be preferred.

This allows them to achieve exceptionally low generalisation error despite classical statistics predictions that they should overfit data:

Neural networks seem to obey a kind of double descent where bigger is better, breaking classical statistics predictions.
(Source: OpenAI's Double Descent blogpost).

This fact has come to be known as the generalisation problem and has been discussed at length in Zhang et. al 2017 (and a 2021 supplement), and in Bengio et al., amongst countless others.

Remarkably, Singular Learning Theory can help explain why neural networks, which are singular models, have the capacity to generalise so well.

The degeneracy of the Fisher information matrix is actually a feature of singular models, not a bug. This is because different regions of parameter space can have different complexities as measured by the RLCT λ, unlike regular models where the complexity is fixed to the total number of parameters in the model d. This is the implicit content of Watanabe's profound free energy formula, called the Widely Applicable Bayesian Information Criterion (WBIC), which quantifies a precise asymptotic tradeoff between inaccuracy and complexity,

WBIC=nLn(w(0))inaccuracy+λcomplexitylogn,

giving a mathematically rigorous realisation of Occam's Razor, since λ≤d2 in singular models.

In this post we will explore Watanabe's fr...
The foundations of knowledge.

by archeon — 2023-06-18T00:05:43.707Z

We all have knowledge, ideas and beliefs that rest on a solid foundation and those that sit on shifting sand. The trouble is they all look the same to us, if we knew which was sound and which not we would change things. Before we reach a age of reason, well meaning adults hot wire us full of their biases and that is especially hard to shift.

Debating an issue with our peers is a good way to sort out the wheat from the chaff. If we can defend a position, at least to our own satisfaction, we have more confidence in the foundation underpinning it. But when someone grabs us by the scruff of the neck and shakes until we realize the position we hold is indefensible, then we were wrong. And now we know it. Bruised ego aside, we now have a clearer view of the world, new avenues of knowledge open up. Win, lose or draw, we always end a debate with more knowledge than we started with.

For this process to work we need protagonists, people of opposing views. Those who agree with us may give confidence in our foundations but do not challenge them. A good debate has us strain every neuron, raid every memory bank and finding reserves we never knew we had. Passion, sound and fury, within reason, give the brain a good workout and keep the audience engaged.

I have bad karma for a post stating that AI was going to prove that humanity and science were incompatible by ending us. Bad karma is a good sign, I have found my protagonists, lots of them it would seem. They may help me prune back the old dead growth of false data to a few healthy branches, sun and air can circulate and new growth and fruits will appear.

On lesswrong bad karma is punished by restricting posts to one a week which is more than I will ever need. More troubling, comments are restricted to one a day which is a good way to kill debate stone dead, not that there has been any. Edit, bad karma of more than 15 restricts comments to 1 every 3 days. Access to the drafting facility is restricted which is self defeating, the more time spent in draft the better the quality is likely to be, but it is a minor inconvenience.

Why would a blog dedicated to reason and rationality restrict their protagonists ability to tell them they are wrong? It is irrational to restrict information about how our opponents view our position, how will we know they are wrong if we do not listen to their objections?

Of course it is normal...
A few more ants and grasshoppers
by c.trout — 2023-06-17T23:38:20.579Z

Literary status: lacking in grace, hopefully compensated by its information density. Inspired by, and in reply to Richard Ngo’s post of parables.

Disclaimer: Though I cite ecological economics and degrowth literature, I don’t have strong credence in all their conclusions (e.g. the high probability of a catastrophic economic climacteric). I do however find solid macro-economic insights in their work (e.g. various formulations of a “growth imperative”), and I find systems dynamics to be a very powerful conceptual framework.

One ant-made winter a grasshopper, starving and frail, approaches the remaining colony of ants drying out their grain in the sun, to ask for food.

“Did you not store up food during the summer?” the ants ask.

“Of course,” says the grasshopper, “but not so much as to survive the millennia-long winter you have brought upon us. You thought that by ever increasing your power, by collectively forbidding yourselves from ever fully enjoying the gains in productivity you won, by always reinvesting and pushing off singing and dancing to next year, then you or your descendants would eventually enjoy an eternal summer. Instead all this has rewarded you with is a great collapse and a small, precarious existence. My carnival wasn’t just shrewd cashing in in the face of risk and uncertainty: it was a key release valve for surplus, essential to maintaining summers for the long-run. With regards to laughter and love I am the efficient one; you, the inefficient.”

(“But what about the gains in health, wealth and material luxury we've accumulated ? Surely those increase laughter and love?”

“Very questionable. And at what cost?”)

On a planet which is resource rich enough for the ant-colonization process to reach escape velocity before burning the house down, and the ants have managed to get themselves on a trajectory to do so…

… “No,” says the grasshopper. “In the past, your colonies diseased, robbed and murdered us. Now you convert us to your way of life by gentler means – interest bearing loans, economic dumping, shiny consumer products, pretty graphs that go up. And any who continue to reject your way are eventually hemmed in, such is your colony’s voracious hunger for resources. Thus I find myself obliged to join your ranks or perish.”

“Apologies for the intrusion, but this universe can only ever reward flawless efficiency in coloniza...
The "Loss Function of Reality" Is Not So Spiky and Unpredictable
by Thoth Hermes — 2023-06-17T21:43:25.908Z
Link: https://thothhermes.substack.com/p/the-loss-function-of-reality-is-not

In other words, there are no "rabbit holes" or "dark tunnels of the mind."

The great success of neural networks is due in large part to the surprising ease in which gradient-based methods are used to train them. Neural networks are often trained using a “loss function” which computes the error rate between predicted and observed values from its environment. The predictions are calculated based on an extremely large and complex function of millions or billions of parameters (the network itself). Since the loss function is merely a simple function between these predictions (outputs) and the values received from the environment, the loss function itself is an extremely high-dimensional function.

Gradient descent, the training method used with neural networks, can be intuitively thought of as “hill climbing” (actually hill descending, but this is isomorphic to the other). In this metaphor, the “parameters” of the model would be represented by the X and Y axes, and the “loss” would be represented by the Z axis, on a typical 3-dimensional graph. The problem, of course, is that the parameters of the model are typically far, far greater than two in number.

That doesn’t stop people from being able to visualize the network training using this metaphor, anyway.[1] Researchers have wanted to understand why and how gradient descent works so well, given that gradient updates are characterized by small, incremental “steps” down the hill. What if there are many hills? What if taking a step down the hill is in the overall wrong direction, if there is a longer, but less steep way to move down in the opposite direction?

Most gradient-descent methods make use of momentum, which means that if the incremental updates have been in largely the same direction for the past several iterations, the speed (magnitude) of the update is allowed to increase. This essentially allows for lots of hills, but assumes that the overall correct direction would mainly be in the same general direction - imagine walking down a large crater with lots of little bumps and rocks all over the slope.

In the early days of neural networks, they were widely thought to be difficult to train.[2] This was largely due to the belief that optimization processes would converge to “local minima” with error rates far higher than the global minimum. This belief persisted for quite some time, until empirical evidenc...