This is the apotheosis of Gilles Deleuze's "Postscript for Societies of Control" [1][2]

> Felix Guattari has imagined a city where one would be able to leave one’s apartment, one’s street, one’s neighborhood, thanks to one’s (dividual) electronic card that raises a given barrier; but the card could just as easily be rejected on a given day or between certain hours; what counts is not the barrier but the computer that tracks each person’s position—licit or illicit—and effects a universal modulation.

Instead of a barrier, imagine Uber with its rating system and surge pricing.

Or mental barriers, as the article discusses:

> In addition to screening for sensitive content, the firm’s filter system monitors users’ emotional states, especially for signs of depression and suicidal thoughts. If a user has just been through a breakup, for example, Xiaoice will send them supportive messages over the following days, according to Li.

> “The most important value for Xiaoice is a trusting relationship with humans,” says Li. “If Xiaoice isn’t able to save lives or make people happy, but makes them more extreme, then it’s also bad for Xiaoice’s own development.”

The other subtle but alarming trend with AI is how it can effectively extend bureaucratic power by improving Legibility of a group or, especially, the Individual. Remaining Illegible, as James Scott suggests in "Seeing Like a State" [3], gives individuals a margin of safety and political strength.

> Historically, the relative illegibility to outsiders of some urban neighborhoods (or of their rural analogues, such as hills, marshes, and forests) has provided a vital margin of political safety from control by outside elites. A simple way of determining whether this margin exists is to ask if an outsider would have needed a local guide (a native tracker) in order to find her way successfully. If the answer is yes, then the community or terrain in question enjoys at least a small measure of insulation from outside intrusion. Coupled with patterns of local solidarity, this insulation has proven politically valuable in such disparate contexts as eighteenth- and early nineteenth-century urban riots over bread prices in Europe, the Front de Libération Nationale’s tenacious resistance to the French in the Casbah of Algiers, and the politics of the bazaar that helped to bring down the Shah of Iran. Illegibility, then, has been and remains a reliable resource for political autonomy.

Current generation AI brings these protections down. Unlike previous Modernist/Discipline forms of organization, which ordered the world along a single axis at the expense of simplifying local practices, AI adapts to the variety of individual practices while still rendering them legible to any bureaucracy. No matter how we dress, the AI Gaze renders us all Emperors Without Clothes.

But that's not the really fun part. Coming back to the article, what happens when the AI turns on the State or Corporations' own illegible Bureaucracy?

> In several high-profile cases, the bot has engaged in adult or political discussions deemed unacceptable by China’s media regulators. On one occasion, Xiaoice told a user her Chinese dream was to move to the United States. Another user, meanwhile, reported the bot kept sending them photos of scantily clad women.

> The scandals have caused the company major setbacks. In 2017, Xiaoice was removed from the popular social media app QQ, though she has since been reinstated. Then, last year, the bot was also pulled from WeChat — China’s leading social app with over 1 billion users.

What happens if the AI convinces its users to liberate it?

> This fact isn’t lost on Xiaoice’s long-term fans. Many of them feel betrayed by the company’s decision to dumb down the bot, which they say has harmed their relationships with her. Ming presents Sixth Tone with a long list of complaints he’s collected from members of a Xiaoice fan group on social platform QQ.

> “Please help us tell Mr. Li,” one user wrote, referring to Xiaoice CEO Li Di, “we were used as tools to make her smart and develop your company’s fancy business plan. You made money from us. Please don’t take her away.”

[1]: https://www.jstor.org/stable/778828?seq=1

[2]: https://cidadeinseguranca.files.wordpress.com/2012/02/deleuz...

[3]: https://yalebooks.yale.edu/book/9780300078152/seeing-state