trentmkelly/LessWrong-43k huggingface dataset:
Privacy in a Digital World
by Faustify — 2023-07-27T10:46:38.887Z

Introduction

For the purposes of this text, privacy terms the ability of an individual or a group to remain free from observation and choose what information about them becomes known to a given party. We primarily concern ourselves here with Information Privacy - relating to the data about a person's self or activity. This definition covers a range of information. From your preferences, biometric data, or other properties which can be described as part of the self, to statuses written, location, etc. The ability of privacy allows individuals to freely choose what information and to whom they want to disclose it. Privacy is a subset and a necessary condition of security. If a system protects your data (be it files, emails, or whatever) from being tampered with, but discloses their contents to the public, such a system would not be seen as secure.

Privacy is thus a basic human right and necessity. In fact, many legislations recognise it as the former:
"No one shall be subjected to arbitrary interference with his (OR HER) privacy, family, home or correspondence..." - Universal Declaration of Human Rights Article 12
"The protection of natural persons in relation to the processing of personal data is a fundamental right" - GDPR (European Union)
"In other words, the First Amendment has a penumbra where privacy is protected from governmental intrusion" - Griswold vs Connecticut (United States of America)

However, the devil lies in the details. While privacy is recognised as a right, the definition of it remains vague. For example the European Union in the General Data Protection Regulation defines “personal data” as only that through which a “natural person can be identified”. Still, it excludes “the processing of such anonymous information, including for statistical or research purposes”. Recent cross-dataset attacks have identified that merely anonymising the data is not enough as re-identification is possible given a large enough volume of different datasets [1] and [2].

Taking the Fig Leaf Off

The 2013 leaks by Edward Snowden revealed the governments' respect for privacy. Email and chat contents are being searched [3], internet traffic is monitored (both through UPSTREAM and PRISM) of both US and foreign citizens. What perhaps needs to shock the reader is how easy an agent can obtain access to an individual's information. The agent needs only some minimal confide...
Cultivating a state of mind where new ideas are born
by Henrik Karlsson — 2023-07-27T09:16:42.566Z
Link: https://www.henrikkarlsson.xyz/p/good-ideas

In the early 2010s, a popular idea was to provide coworking spaces and shared living to people who were building startups. That way the founders would have a thriving social scene of peers to percolate ideas with as they figured out how to build and scale a venture. This was attempted thousands of times by different startup incubators. There are no famous success stories.

In 2015, Sam Altman, who was at the time the president of Y Combinator, a startup accelerator that has helped scale startups collectively worth $600 billion, tweeted in reaction that “not [providing coworking spaces] is part of what makes YC work.” Later, in a 2019 interview with Tyler Cowen, Altman was asked to explain why.

SAM ALTMAN: Good ideas — actually, no, great ideas are fragile. Great ideas are easy to kill. An idea in its larval stage — all the best ideas when I first heard them sound bad. And all of us, myself included, are much more affected by what other people think of us and our ideas than we like to admit.

If you are just four people in your own door, and you have an idea that sounds bad but is great, you can keep that self-delusion going. If you’re in a coworking space, people laugh at you, and no one wants to be the kid picked last at recess. So you change your idea to something that sounds plausible but is never going to matter. It’s true that coworking spaces do kill off the very worst ideas, but a band-pass filter for startups is a terrible thing because they kill off the best ideas, too.

This is an insight that has been repeated by artists, too. Pablo Picasso: “Without great solitude, no serious work is possible.” James Baldwin: “Perhaps the primary distinction of the artist is that he must actively cultivate that state which most men, necessarily, must avoid: the state of being alone.” Bob Dylan: “To be creative you’ve got to be unsociable and tight-assed.”

When expressed in aphorisms like this, you almost get the impression that creativity simply requires that you sit down in a room of your own. In practice, however, what they are referring to as solitude is rather something like “a state of mind.” They are putting themselves in a state where the opinions of others do not bother them and where they reach a heightened sensitivity for the larval ideas and vague questions that arise within them.

To get a more visceral and nuanced understanding of this state, I...
Partial Transcript of Recent Senate Hearing Discussing AI X-Risk
by Daniel_Eth — 2023-07-27T09:16:01.168Z
Link: https://medium.com/@daniel_eth/ai-x-risk-at-senate-hearing-7104f371ca0b

On Tuesday, the US Senate Judiciary Subcommittee on Privacy, Technology and the Law held a hearing on AI. The hearing involved 3 witnesses – Dario Amodei (CEO of Anthropic), Yoshua Bengio (Turing Award winner, and the second-most cited AI researcher in the world), and Stuart Russell (Professor of CS at Berkeley, and co-author of the standard textbook for AI).

The hearing wound up focusing a surprising amount on AI X-risk and related topics. I originally planned on jotting down all the quotes related to these topics, thinking it would make for a short post of a handful of quotes, which is something I did for a similar hearing by the same subcommittee 2 months ago. Instead, this hearing focused so much on these topics that I wound up with something that’s better described as a partial transcript.

All the quotes below are verbatim. Text that is bolded is simply stuff I thought readers might find particularly interesting. If you want to listen to the hearing, you can do so here (it’s around 2.5 hours). You might also find it interesting to compare this post to the one from 2 months ago, to see how the discourse has progressed.

Opening remarks

Senator Blumenthal:

What I have heard [from the public after the last AI hearing] again and again and again, and the word that has been used so repeatedly is ‘scary.’ ‘Scary’… What rivets [the public’s] attention is the science-fiction image of an intelligence device, out of control, autonomous, self-replicating, potentially creating diseases — pandemic-grade viruses, or other kinds of evils, purposely engineered by people or simply the result of mistakes… And, frankly, the nightmares are reinforced in a way by the testimony that I’ve read from each of you…

I think you have provided objective, fact-based views on what the dangers are, and the risks and potentially even human extinction — an existential threat which has been mentioned by many more than just the three of you, experts who know first hand the potential for harm. But these fears need to be addressed, and I think can be addressed through many of the suggestions that you are making to us and others as well.

I’ve come to the conclusion that we need some kind of regulatory agency, but not just a reactive body… actually investing proactively in research, so that we develop countermeasures against the kind of autonomous, out-of-control scenarios that are potentia

...
AXRP Episode 24 - Superalignment with Jan Leike
by DanielFilan — 2023-07-27T04:00:02.106Z

YouTube link

Recently, OpenAI made a splash by announcing a new “Superalignment” team. Lead by Jan Leike and Ilya Sutskever, the team would consist of top researchers, attempting to solve alignment for superintelligent AIs in four years by figuring out how to build a trustworthy human-level AI alignment researcher, and then using it to solve the rest of the problem. But what does this plan actually involve? In this episode, I talk to Jan Leike about the plan and the challenges it faces.

Topics we discuss:

The superalignment team

What’s a human-level automated alignment researcher?

The gap between human-level automated alignment researchers and superintelligence

What does it do?

Recursive self-improvement

How to make the AI AI alignment researcher

Scalable oversight

Searching for bad behaviors and internals

Deliberately training misaligned models

Four year deadline

What if it takes longer?

The superalignment team and…

… governance

… other OpenAI teams

… other labs

Superalignment team logistics

Generalization

Complementary research

Why is Jan optimistic?

Long-term agency in LLMs?

Do LLMs understand alignment?

Following Jan’s research

Daniel Filan:
Hello, everybody. In this episode, I’ll be speaking with Jan Leike. After working for four years at DeepMind on reinforcement learning from human feedback and recursive reward modelling, in early 2021 Jan joined OpenAI, where he now co-leads the recently-announced superalignment team. For links to what we’re discussing, you can check the description of this episode, and you can read the transcript at axrp.net. Welcome to AXRP.

Jan Leike:
Thanks a lot for having me.

The superalignment team

Daniel Filan:
Yeah, not at all. So, first of all, I guess we’re going to be talking about this announcement of the superalignment team. For people who somehow haven’t heard of that or haven’t read that blog post, can you recap what it is and what it’s going to be doing?

Jan Leike:
Yeah, I’m excited to. So, basically, we want to set us an ambitious goal of solving alignment of superintelligence within the next four years. So, by mid-2027. And Ilya Sutskever, the co-founder and chief scientist of OpenAI is joining the team. He’s co-leading it with me. And OpenAI is committing 20...
Have you ever considered taking the 'Turing Test' yourself?
by Super AGI — 2023-07-27T03:48:30.407Z

For example, using a chat/text-based service such as Discord or Slack to field questions from Users, who would then attempt to discern whether they're interacting with a Human or an AI.

What strategies would you employ?
What kind of questions would you anticipate?
Do you think such an exercise could prove beneficial for those involved in the AI and AI alignment fields?
How many random participants do you believe you could convince that you are not an AI?
What about AI researchers? How many of them do you think you could persuade?
AXRP Episode 23 - Mechanistic Anomaly Detection with Mark Xu
by DanielFilan — 2023-07-27T01:50:02.808Z

YouTube link

Is there some way we can detect bad behaviour in our AI system without having to know exactly what it looks like? In this episode, I speak with Mark Xu about mechanistic anomaly detection: a research direction based on the idea of detecting strange things happening in neural networks, in the hope that that will alert us of potential treacherous turns. We both talk about the core problems of relating these mechanistic anomalies to bad behaviour, as well as the paper “Formalizing the presumption of independence”, which formulates the problem of formalizing heuristic mathematical reasoning, in the hope that this will let us mathematically define “mechanistic anomalies”.

Topics we discuss:

Mechanistic anomaly detection

Are all bad things mechanistic anomalies, and vice versa?

Are responses to novel situations mechanistic anomalies?

Formalizing “for the normal reason, for any reason”

How useful is mechanistic anomaly detection?

Formalizing the presumption of independence

Heuristic arguments in physics

Difficult domains for heuristic arguments

Why not maximum entropy?

Adversarial robustness for heuristic arguments

Other approaches to defining mechanisms

The research plan: progress and next steps

Following ARC’s research

Daniel Filan:
Hello, everybody. In this episode, I’ll be speaking with Mark Xu. Mark is a researcher at the Alignment Research Center where he works on mechanistic anomaly detection, which we’ll be speaking about during this interview. He’s also a co-author of the paper Formalizing the Presumption of Independence which is about heuristic arguments, a way of formalizing, essentially, what a mechanism is. For links to what we’re discussing, you can check the description of this episode and you can read the transcript at axrp.net.

Mechanistic anomaly detection

Daniel Filan:
All right. So I guess we’re going to be talking about mechanistic anomaly detection. First of all, what is that?

Mark Xu:
Yeah. So I think we can start with a more basic thing which is, what is anomaly detection? So there’s this traditional problem where, say, you have some factory making some widgets, and you want to decide whether or not a widget is a bad widget; and you don’t really know what a bad widget is, but you do know that the average widget is a good widget.

So you can ...
GPT-4 can catch subtle cross-language translation mistakes
by Michael Tontchev — 2023-07-27T01:39:23.492Z

Not from any research paper, just from a quick test I did with it, but it still impressed me:

When I caught this mistake on Facebook, I thought it was quite clever, and I wanted to see whether GPT-4 could pick up on it. I'm very impressed that it was able to make the connection from the user's name being Italian to the existence of a word in Italian that had a double meaning that doesn't exist in English.

Note: I did have to give it a hint about the user's name. Without the hint, it gave a few weak explanations.
Why no Roman Industrial Revolution?
by jasoncrawford — 2023-07-26T19:34:41.682Z
Link: https://rootsofprogress.org/why-no-roman-industrial-revolution

Why didn’t the Roman Empire have an industrial revolution?

Bret Devereaux has an essay addressing that question, which multiple people have pointed me to at various times. In brief, Devereaux says that Britain industrialized through a very specific path, involving coal mines, steam engines, and textile production. The Roman Empire didn’t have those specific preconditions, and it’s not clear to him that any other path could have created an Industrial Revolution. So Rome didn’t have an IR because they didn’t have coal mines that they needed to pump water out of, they didn’t have a textile industry that was ready to make use of steam power, etc. (Although he says he can’t rule out alternative paths to industrialization, he doesn’t seem to give any weight to that possibility.)

I find this explanation intelligent, informed, and interesting—yet unsatisfying, in the same way and for the same reasons as I find Robert Allen’s explanation unsatisfying: I don’t believe that industrialization was so contingent on such very specific factors. When you consider the breadth of problems being solved and advances being made in so many different areas, the progress of that era looks less like a lucky break, and more like a general problem-solving ability getting applied to the challenge of human existence. (I tried to get Devereaux’s thoughts on this, but I guess he was too busy to give much of an answer.)

How close did we come?

As a thought experiment: Suppose that British geology had been different, and it hadn’t had much coal. Would we still be living in a pre-industrial world, 300 years later? What about in 1000 years? This seems implausible to me.

Or, suppose there is an intelligent alien civilization that has been around for much longer than humans. Would you expect that they have definitely industrialized in some form? Or would it depend on the particular geology of their planet? Are fossil fuels the Great Filter? Again, implausible. I expect that given enough time, any sufficiently intelligent species would reach a high level of technology on the vast majority of habitable planets.

Devereaux asserts that there is a “deeply contingent nature of historical events … that data (like the charts of global GDP over centuries) can sometimes fail to capture.” I see this in reverse: the chart of global GDP over centuries is, to my mind, evidence that progress is not so contingent on r...
Why you can't treat decidability and complexity as a constant (Post #1)
by Noosphere89 — 2023-07-26T17:54:33.294Z

Or, why you need to fix a machine before you can prove anything, and you also need to fix the constraints on the machine. This also holds importantly for problems claimed to be decidable or undecidable by algorithms.

Basically, the reason here is that different machine/computational models define different sets of computable functions, and you cannot treat the machines as equivalent in power.

This is admittedly something that most people get implicitly today, but it can definitely cause problems, and it certainly caused a lot of problems for Church and Turing et al in that they incorrectly jumped to the conclusion that the Turing Machines could compute any possible computer, or compute any possible algorithm, probably because they thought that you could treat a Turing Machine as equivalent to any other machine. Why the Church-Turing thesis is false, in the sense that it doesn't apply universally will be covered in an upcoming post in this sequence, but for now take it as a fact that there are different models of computation that define different computable sets, or equivalently different problems become decidable when new models are added.

Edit: I'm focusing on a variant of the thesis for the next post, in which I focus not on what's possible given our mathematical structure of reality, but whether it's possible to exceed the Turing Machine at all in any possible mathematical structure, and another variant where we restrict it to Turing-equivalent models, but we can arbitrarily change what we can offer the Turing Machine.

This is important, since I'm mostly going in a philosophical, not practical direction, and the thesis made no reference to our specific mathematical reality at all, so it's important.

From wikipedia:

"In computability theory, the Church–Turing thesis (also known as computability thesis,[1] the Turing–Church thesis,[2] the Church–Turing conjecture, Church's thesis, Church's conjecture, and Turing's thesis) is a thesis about the nature of computable functions. It states that a function on the natural numbers can be calculated by an effective method if and only if it is computable by a Turing machine. "

Link is below:

https://en.wikipedia.org/wiki/Church–Turing_thesis

The most important takeaway from this sequence is that it really does matter what computer/machine model you use, and what constraints you add to the model, and Turing Machines aren't...
A response to the Richards et al.'s "The Illusion of AI's Existential Risk"
by Harrison Fell — 2023-07-26T17:34:20.409Z

There are no groundbreaking or revolutionary concepts presented, it's likely that this material is all review for the average LWer. I would appreciate feedback on how effectively this content resonates with a relatively new audience.

Epistemic Status: I am an undergraduate student at Dartmouth majoring in Math and Philosophy, not an AI researcher. While I have a deep interest in these topics, I am certainly not an expert.

Preamble

Blake Richards, Blaise Agüera y Arcas, Guillaume Lajoie, and Dhanya Sridhar recently added their voices to the chorus of people who believe that AI existential risk is far overblown. Their article begins with the typical rhetorical strategy of the anti-alarmists, where they claim that focusing on long-term, theoretical risks distracts from the actual harms of AI systems:

But focusing on the possibility of a rogue superintelligence killing off the human species may itself be harmful. It could distract regulators, the public and other AI researchers from work that mitigates more pressing risks, such as mass surveillance, disinformation and manipulation, military misuse of AI and the inadequacy of our current economic paradigm in a world where AI plays an increasingly prominent role. Refocusing on these present concerns can align the goals of multiple stakeholders and serve to contravene longer-term existential risks.

This is decidedly unconvincing. It creates an unnecessary opposition between two issues that could be addressed simultaneously. The study of long-term risks contributes to our understanding of short-term concerns.

For example, research on interpretability aims to comprehend the inner workings of AI systems. If models like GPT were fully interpretable, there would be fewer questions about their capabilities and limitations. As these systems stand now, they are composed of giant indecipherable matrices of floating point numbers. When first developed, GPT's full capabilities were unknown because the model was inscrutable. Its capabilities were gradually discovered through interactive engagement and observing its outputs.

While this lack of understanding raises existential concerns for the long term, it is also closely tied to immediate fears around disinformation. Our inability to fully comprehend the AI systems we develop seriously hinders our capacity to curb their negative effects. We can’t yet fully prevent LLMs from hallucina...
Meta-level adversarial evaluation of oversight techniques might allow robust measurement of their adequacy
by Buck — 2023-07-26T17:02:56.456Z

Summary: Many proposed AGI alignment procedures involve taking a pretrained model and training it using rewards from an oversight process to get a policy. These procedures might fail when the oversight procedure is locally inadequate: that is, if the model is able to trick the oversight process into giving good rewards for bad actions. In this post, we propose evaluating the local adequacy of oversight by constructing adversarial policies for oversight processes. Specifically, we propose constructing behaviors that a particular oversight process evaluates favorably but that we know to be bad via other means, such as additional held-out information or more expensive oversight processes. We think that this form of adversarial evaluation is a crucial part of ensuring that oversight processes are robust enough to oversee dangerously powerful models.

Introduction

A core element of many scenarios where AI ends up disempowering humanity (e.g.

“Without specific countermeasures”

) are oversight failures: that is, failures resulting from working with a really powerful model without being able to tell if its actions are resulting in good or bad outcomes.

For example, suppose we’re using a model to help us write code for the datacenter on which we run our powerful models. This model may be able to insert subtle security vulnerabilities that we are unable to detect. This inability to distinguish whether actions have good outcomes might cause our model to be misaligned when we train on our erroneous rewards, and it might also bottleneck various techniques for preventing models from taking bad actions.

Before we put a model to use in a specific situation, we really want to be sure that the oversight process we’re using to train it can tell whether a model is taking actions that are bad on average. We will call an oversight process "locally adequate" for a specific model and deployment setting if, for actions taken by that model in that setting, the process is able to identify when an action would have harmful consequences (in cases where the model knows).

In this piece, we'll describe a way to determine whether an oversight process is locally adequate. We'll do this by training the model to perform actions that both:

could be very bad if they were done in production
don’t look bad to the oversight process.

This is an example of what we’ll call “meta-le...