Image item
Image: RB + MJ
Welcome to 
Nextness
Volume 12.
Welcome to Volume 12. Can “poisoning” generative AI be a good thing? Also, in 1939 a stunning essay was written about innovation. It is even more relevant today than when it was originally penned.
SHOULD WE BE POISONING ART?
INTRODUCING NIGHTSHADE, THE GENERATIVE AI KILLER
On December 2, 1942, under a set of football bleachers at the University of Chicago, Enrico Fermi and a team of scientists created the world’s first atomic chain reaction. 
 
This pivotal event marked the initiation of the Manhattan Project, ushering in the nuclear age, introducing the concept of mutually assured destruction, and giving us the ghoulish specter of Dr. Strangelove.
 
Today, another team of scientists at the University of Chicago has crafted a new doomsday machine tailored for the age of generative AI.
 
Meet Nightshade, a creation designed to “poison” images scraped from the internet.
 
Its purpose? To safeguard artists’ work by making certain their creations self-destruct within training models. Not unlike the audio tape sequence at the beginning of those old Mission Impossible episodes.
 
Reactions are mixed. Some cheer. Others are appalled.
 
Nightshade operates by infusing digital images with undetectable changes that trick algorithms.
 
In the first paragraph of their research paper, the team clearly states their intent:
 
Data poisoning attacks manipulate training data to introduce unexpected behaviors into machine learning models at training time. For text-to-image generative models with massive training datasets, current understanding of poisoning attacks suggests that a successful attack would require injecting millions of poison samples into their training pipeline. In this paper, we show that poisoning attacks can be successful on generative models.
 
The result? Nightshade can make the algorithm perceive,say, a truck when it’s really looking at a daisy.
Image item
Many artists view this as the perfect solution to protect their originality, hoping their art won’t be scraped in the future.
 
The dissenting view comes from people who are using generative AI tools such as DALL-E, Stable Diffusion or Midjourney. Their argument sometimes states that all art is derivative, shaped by the artists who came before them.
 
The debate over who is right or wrong might someday find resolution in the courtroom.
 
But beneath this dispute lies a more profound question: what impact can poisoning attacks have on the world’s visual data?
 
While the creators of Nightshade argue that it serves as “a last defense for content creators,” they acknowledge the possibility of nefarious uses for Nightshade outside the realm of web scraping.
 
Their research paper states:
 
Prompt-specific poisoning attack are versatile and powerful. When applied on a single narrow prompt, their impact on the model can be stealthy and difficult to detect, given the large size and prompt space. Examples include advertising (produce Tesla images for ‘luxury car’ prompts. And political attacks (produce offensive images when prompted with a politician (sic) name.)
 
Image item
These details surface a raft of troubling questions:
 
Should government funding support this work? Should our nation’s universities endorse it?
 
On one hand, the government collaborates with AI companies maintain America’s strategic lead in AI. On the other, it essentially invests in poisoning the AI well.
 
Presently, there is no Geneva Convention for AI. Should banning a technology that poisons the world’s shared data be one of our first considerations?
 
Once this technology leaves the lab, it will undoubtedly demand a new wave of security software designed to counter to poisoning attacks – a new arms race begins.
 
Yet, beyond the immediate implications of this paradoxical new technology, profound long-term moral and ethical questions also loom.
 
In a thread on X one user asks, “Why poison your art for one generation of AI? It may be the only thing that carries your art outside of and long after humanity.”
 
As a write this, KDFC, the local classical station, plays in the background. I find solace in the fact that during J.S. Bach’s time, the Nightshade team wasn't present to poison the Brandenburg Concertos.
Image item
The  Nightshade team. Leader Ben Y. Zhao, middle

Image item
In 1933, a unique institution of thinking was founded, known as the Institute of Advanced Study. It was the brainchild of Abraham Flexner and found its home on the campus of Princeton University.
 
The IAS served as an intellectual haven for 20th century geniuses such as Einstein, Von Neumann, and Dirac. It was a place where these brilliant minds could devise ideas that had no immediate utility. In this sense, they might appear useless.
 
In his essay, Flexner gets into the essence of thinking and how human progress is deeply intertwined with the almost innocent quality of curiosity:
 
Curiosity, which may or may not eventuate in something useful, is probably the outstanding characteristic of modern thinking. It is not new. It goes back to Galileo, Bacon, and Sir Isaac Newton, and it must be absolutely unhampered. Institutions of learning should be devoted to the cultivation of curiosity and the less they are deflected by considerations of immediacy of application, they more like they are to contribute not only to human welfare but to the equally important satisfaction of intellectual interest which may indeed be said to have become the ruling passion of intellectual life in modern times.
 
Flexner goes on to emphasize that bringing a product to market is just the final step in a long line of inventive thinking. Figures like Marconi merely added the finishing touches on innovations that would not have been possible without the earlier genius of individuals like Maxwell and Hertz.
 
Flexner also writes of Faraday, who laid the groundwork for Maxwell’s equations of electro-magnetism:
 
“At no period in his unmatched career was he interested in utility. He was absorbed in disentangling the riddles of the universe, at first chemical riddles, in later periods, physical riddles.”
 
These names may seem antique, but they are the people who are responsible for the modern conveniences we take for granted in our daily lives.
 
Flexner's essay was penned in 1939, on the eve of World War Two, by a Jewish intellectual. Flexner could not have foreseen the coming horrors, but his central theme remains pertinent as ever:
 
“The real enemy of the human race is not the fearless and irresponsible thinker, be he right or wrong. The real enemy is the man who tries to mold the human spirit so that is will not dare spread its wings, as its wings were once spread in Italy and Germany, as well as Great Britain and the United States.”
 
In a time when self-identification increasingly leans into religion and skin color, Flexner wondered if this was not distracting us from the real prize:
 
In the face of history of the human race what can be more silly or ridiculous than likes or dislikes founded upon race or religion? Does humanity want symphonies and paintings and profound scientific truth, or does it want Christian symphonies, Christian paintings, Christian science, or Jewish symphonies, Jewish science, of Mohammedan and or Egyptian or Japanese of Chinese or American or German or Russian or Communist or Conservative contributions to and expressions of the infinite richness of the human soul?
 
Ultimately, higher education should be the education that best equips us to elevate the world, irrespective of individual beliefs. It should be a realm of profound curiosity and should not be burdened by the need for immediate utility. That will naturally follow.
 
Flexner envisioned the Institute for Advanced Study as a permissible luxury in a world increasingly dominated by specialization and the demand for material outputs. He imagined a utopia, and advocated for more such places:
 
“We make ourselves no promises, but we cherish the hope that the unobstructed pursuit of useless knowledge will prove to have consequences in the future as in the past. Not for a moment, however, do we defend the Institute on that ground. It exists as a paradise for scholars who, like poets and musicians, have won the right to do as they please and who accomplish most when enabled to do so.”
 
Image item

Sponser
Image item
AI changes everything. Including storytelling. Nextness Volume 12 is brought to you by Storymachine. As a leader in the AI video space, Storymachine scripts, films and delivers everything from branded content and commercials to corporate masterclasses and training films. If you are looking to unleash a new kind of storytelling, Storymachine just might be your jam.  storymachinefilms.com