“There’s an old saying in the international law world,” Gary Brown, an associate professor at the Bush School of Government and Public Service at Texas A&M University, told me. “If you’re not at the table, you’re on the menu.” We were at the opening reception for a conference sponsored by the University of Pennsylvania’s Center for Ethics and the Rule of Law (CERL) called “Ethical and Legal Dilemmas of Autonomous Weapons in War and National Security.” Outside, the weather had regressed to an oppressive damp after a brief period of spring beauty. The keynote address was the only portion of the conference open to the media—the working groups fell under the Chatham House Rule, which dictates that speakers cannot be identified by name or affiliation.
At the reception, attendees hovered around the drinks table and seemed broadly disinterested in speaking to the media. But Brown, a former strategy analyst for the Department of Defense, senior legal adviser for combat air operations in Afghanistan and Iraq, and senior legal counsel to U.S. Cyber Command, was chatty. He admitted that the rise of automated intelligence raises a number of civil liberties issues but argued that its ascension is inevitable. Although their more frightening applications were the focus of this particular conference, robots, he pointed out, will not uniformly be used for lethal activity. They could perform reconnaissance missions and search and rescue, essentially assuming the risk of first responders, at least until they become sentient—at which point, Brown believes, we’ll have to start treating them better. “Since we’re trying to program out prejudice, bias, subjectivity, and all that, honestly, it would be sort of crazy for us to continue to try to be in charge. We should turn the keys over to them,” he said, sounding slightly wistful.
Elsewhere on Penn’s campus, a campaign had just been launched against Ghost Robotics, a company that manufactures robot dogs. It was founded in 2015 by two graduates of Penn’s General Robotics, Automation, Sensing, and Perception (GRASP) Labs and is housed at the Pennovation Center, in what was once a DuPont laboratory. Companies based out of the Pennovation Center receive significant tax breaks as part of a Keystone Innovation Zone, one half—along with the Keystone Opportunity Zones—of a tax program meant to attract businesses to Philadelphia and spur improvement in “blighted areas.” The program has become a flash point for supporters of Philly’s much beleaguered and severely underfunded public school system.
The dogs have been a fixture at Penn for some time now. Unlike Boston Dynamics, which signed an (admittedly flimsy) agreement pledging that they would not allow their four-legged robots to be weaponized, Ghost Robotics has actively courted the military. In December 2023, South Korean weapons manufacturer LIG Nex1 acquired a 60 percent controlling stake in the company for $239 million. Both of the Ghost Robotics founders were mentored by the Penn roboticist Daniel Koditschek, who publicly disavowed the company in 2021, asking them to remove his name and any affiliation with his lab from their website. He warned that the company was treading dangerously close to an international human rights violation. Swarms of autonomous robots, Koditschek cautioned, could amount to a weapon of mass destruction.
His words do not appear to have dissuaded them. Ghost Robotics has formed more than twenty-five national security partnerships, and the Department of Homeland Security is currently testing the dogs’ efficacy at tracking migrants along the southern border of the United States. In March, Haaretz reported that the dogs were being equipped with a small, flying robot called the Rooster in order to navigate Gaza’s system of tunnels. This report was what spurred the protests against the company on Penn’s campus, which would eventually be assimilated into the larger demands of the university’s Palestine solidarity encampment, as students across the country pressured their schools to boycott and divest from Israel in particular and weapons manufacturers in general. At Penn, which does not disclose how it invests its $21 billion endowment, the encampment’s immediate demands were straightforward: protesters wanted increased transparency, and they wanted the school to cut ties with Ghost Robotics.
“Ghost Robotics’s list of so-called ‘solution partners’ makes it clear that they are, yet again, seeing market potential precisely where those with a working conscience see moral hazard,” a representative from Shut Down Ghost Robotics said during a speech at the Penn encampment in late April. A helicopter circled overhead as she spoke; she looked up at the sky briefly. “We do not need innovations that allow for more efficient slaughter.” The camp had sprung up in a rush just a few hours earlier, a collaborative effort between Penn, Drexel, and Temple students. Six days before, the university had revoked Penn Students Against the Occupation’s status as a registered student group. By the time I arrived on the scene, tents covered the green in front of College Hall, banners were draped across the Ben Franklin statue, acoustic guitars had been unsheathed, and a group of counterprotesters had secured a space for themselves on the margins, holding up an Israeli flag. Penn’s campus stretches for nearly three hundred acres across the mouth of West Philadelphia, and the school generally keeps a wary eye trained on the rest of the neighborhood, but during this brief moment of lawlessness, it felt like public land. Supporters surrounded the encampment linking arms, and the mood was hopeful, if not cheerful.
Directly across from the protest stood a thirteen-foot steel peace sign. It had been erected to commemorate the massive demonstrations held by Penn faculty and students against the university’s involvement in the Vietnam War nearly sixty years before. Last summer, when I was working as an editor at a museum magazine, I stumbled across a strange photo while I was sourcing art for a story. It was grainy, clearly a scan of newsprint, showing a figure in a gas mask, wearing a graduation cap and gown. The photo was attributed to the University of Pennsylvania’s student newspaper, and the caption made mention of a DoD contract. By the time students set up camp on College Green, I’d been sorting through archival material about this older antiwar movement for months. Listening to the speaker from Shut Down Ghost Robotics, it was difficult not to think back to an earlier generation of students who’d challenged the university on their investment in lethal technologies six decades prior.
The Anarchist’s Grandson
The Penn archives are housed in a large building on Market Street, down the hall from a sperm bank. This area was once known as the Black Bottom, a working-class black neighborhood that was razed to the ground in the early 1960s. Penn, Drexel, and the University of the Sciences had joined together as the “West Philadelphia Corporation” in an effort to expand their territory and create a shared science research center. To accomplish this, they declared eminent domain and displaced somewhere between five and ten thousand people.
At one point, the Institute for Cooperative Research looked into lacing rice with birth control, planning to air drop drugs across Vietnam to decrease the population.
Shortly after this land grab, in the summer of 1965, a young man named Robin Maisel was hired as a clerk at Penn’s university bookstore. Part of his job entailed processing special orders and delivering them across campus. Slowly, he noticed that one department with a conspicuously nondescript name—the Institute for Cooperative Research (ICR)—was placing some suspicious requests. They were interested in rice crop diseases and aerosol dispersal; they ordered countless books on Vietnamese history. When Maisel went to drop off the material to an office above a Mercedes-Benz dealership, he saw bars on the window and combination locks on all the cabinets.
Robin was still quite young, but he was the wrong person to let near those book orders. “Apart from being superstitious and red, we were a very strange family,” his sister, Merry, told me over the phone. Their grandfather, Max Maisel, had emigrated from Latvia to the United States at eighteen and opened an anarchist bookstore, quickly becoming friends with such figures as Emma Goldman and Alexander Berkman. He also began to act as a publisher—a severely overextended one. He put out Margaret Sanger’s What Every Girl Should Know; held close ties to the Jewish Labor Bund and published numerous books and pamphlets in Yiddish; and even briefly employed Leon Trotsky as a writer at the Russian émigré magazine Novy Mir for twenty dollars a week. To afford this, Robin once explained, his grandfather had taken out four mortgages on his house in Sea Gate, Brooklyn.
Robin’s parents became politically active in the Communist Party during the Great Depression, but they were told to go underground when they moved to the New York City suburbs. For some time, they were holding money for the Comintern. Robin, meanwhile, struggled socially in school and was repeatedly harassed and beaten up by his classmates, who objected to the picket lines he organized outside Woolworth’s in solidarity with civil rights activists. “They broke his jaw,” said Merry, who is three years older than her brother. “He never told me or our parents until years later . . . I went in and out of [politics] depending on what was going on in my life. But he was always a stalwart. Immovable, you know.”
Philadelphia was a relief for Robin. He joined the Young Socialist Alliance and helped organize a rent strike in North Philly. By the time the ICR aroused his suspicions, he’d dealt with enough left sectarianism and public pushback to know that if he brought the information to light himself, it would not be well received. Penn is a fairly conservative campus even today, but in 1965, Penn’s Committee to End the War in Vietnam (CEWV) had been growing steadily, and Maisel turned to them for help. Together, they began to unravel the story behind the secretive lab.
In 1943, President Franklin D. Roosevelt had declared that the United States would never make use of chemical weapons “unless they are first used by our enemies.” In response, a small special interest group of military, industrial, and chemical company executives emerged, hoping to spread the gospel of defoliants, nerve gas, and airborne arsenic. They called themselves the Chemical Warfare Association but were advertised like a strange offshoot of an Elks Lodge. A 1947 issue of the Chemical Corps Journal called for a “spiritual mobilization” toward war among the American people; every issue began with a short quote declaring the authors’ commitment to “perpetuate the friendships, memories, and traditions growing out of their service with the Chemical Warfare Service.” In 1951, these men brought a Chinese video they’d obtained through Air Force Intelligence to Penn. The video, which showed the aftermath of chemical and biological attacks and indicated the Chinese were stockpiling such weapons, had an immediate effect on scientists at the university. Professor Knut Krieger was particularly excited by the idea of participating in weapons research, and shortly thereafter, he was put in charge of a project with the Air Force that was given the code name “Big Ben” in a classified contract. Named after Ben Franklin, the school’s founder, it was, according to the university trustees’ minutes, “a study of biological and chemical warfare from all standpoints—social, political, technological, scientific.”
The ICR was founded in a 1954 partnership with the U.S. Army Chemical Corps to house this new classified initiative and to court larger projects. Soon, the Institute was estimating the potential casualties various chemical weapons might cause, investigating how weather would affect such weapons’ utility, and doing bacteriological research. At one point, the Institute looked into lacing rice with birth control, planning to air drop drugs across Vietnam to decrease the population. By the time the ICR attracted Maisel’s attention, they were primarily engaged with Project Spicerack and Project Summit, which Krieger later told reporters were focused on “carrying out field research on the effectiveness of chemical warfare in Vietnam” and “developing delivery systems” for aerosol dispersal of arsenic, cyanide, and chemical defoliants.
Maisel took his information to Gabriel Kolko, a charismatic young history professor who organized faculty through the University Committee on Problems of War and Peace. A graduate student sent the information off to the press, and through CEWV, sent a letter to President Gaylord Harnwell, calling the contracts “immoral, inhuman, and unbefitting to an academic institution.” Maisel was quickly fired from his job at the university bookstore in retaliation for blowing the whistle, but his cause was aided by ICR researchers’ own clear lack of media training. Krieger in particular spoke to the press quite a bit. He was steadfast in his belief that science stood at an Olympian remove from the world—no research, no matter how horrific, should be off-limits—and he seemed to be laboring under the delusion that chemical weapons provided a kinder, gentler way to put down an enemy army. “Maybe it’s a dream that will never come true,” Krieger told the Charlotte Observer in 1966, as protests of the Spicerack and Summit projects attracted national attention. “But you’d like to be able to fight a war without killing anybody.”
Deadly Lavender
Krieger’s fantasy of humane chemical weapons was shared by an incoherent Penn Comment editorial that appeared in December 1965. “Chemical warfare agents can make a man no more dead than can a rifle bullet,” wrote the author.
And, in fact, they present a far more enjoyable (or rather, less enjoyable) way to die. Incapacitating agents provide new ways to overcome bunkers, with no casualties on either side. . . . War is a dirty subject, unpleasant to all concerned. . . . The question of morality cannot justify unnecessary death; to kill an enemy, it is better to use a chemical agent and lose one soldier than not to use one and lose ten.
The decades-long aftermath of chemical warfare would prove this notion to be patently insane. Agent Orange is still causing birth defects in Vietnam, Laos, Cambodia, and in the children of American veterans. Forty years of chemical weapons use in Iraq resulted in a massive web of environmental devastation. What these weapons did do effectively was increase the psychic distance between the military and the people it was killing. Defoliant sprayed through a fire hose is not a bomb, is not a missile, is technically nothing more than an agricultural accelerant. It may be poison, but it offers plausible deniability for the person with their finger on the trigger. Dow Chemical’s website still denies any causal links between Agent Orange and cancer or birth defects.
In 1993, the United States signed a treaty prohibiting the manufacture or stockpiling of chemical weapons, which came into effect in 1997. The country’s last sarin gas missile was finally destroyed in July 2023 at the Blue Grass Army Depot in Richmond, Kentucky. But today, apologia for autonomous weapons sounds eerily similar to those who touted chemical warfare as a more humane alternative to traditional warfare in the Vietnam era. Roboticist Ronald Arkin has argued that “autonomous robots in the future will be able to act more ‘humanely’ on the battlefield for a number of reasons, including that they do not need to be programmed with a self-preservation instinct, potentially eliminating the need for a ‘shoot first, ask questions later’ attitude.” The scale of destruction in Gaza has undermined such claims in real time.
Much of the recent CERL keynote conversation I attended at Penn focused on Lavender, an artificial intelligence program used by the Israeli military to select targets. A week before the conference, +972 Magazine reported that implementation of the Lavender system played a key role in the brutal pace of bombardment in Gaza that has killed more than thirty-eight thousand people as of this writing, with thousands more presumed to be buried under rubble and over eighty-seven thousand injured. (A recent study by The Lancet states that “it is not implausible” to estimate a death toll as high as 186,000.) According to +972, rules for acceptable levels of collateral damage were loosened; even individuals identified by the algorithm as potential low-level Hamas operatives were targeted for assassination, along with their entire families; and IDF soldiers faced constant pressure to produce new targets. “There’s something about the statistical approach that sets you to a certain norm and standard,” a source told the magazine. “There has been an illogical amount of [bombings] in this operation. This is unparalleled, in my memory. And I have much more trust in a statistical mechanism than a soldier who lost a friend two days ago. Everyone there, including me, lost people on October 7. The machine did it coldly. And that made it easier.”
In conversation at the keynote address were General James Cartwright, former commander of U.S. Strategic Command; Craig Wiener, a former analyst for the Department of Energy’s Office of Intelligence and a technical fellow at MITRE (a sprawling, opaque, federally funded not-for-profit corporation that operates secretive government-sponsored research and development programs); and Dawn Meyerriecks, currently at MITRE and formerly the deputy director of Science and Technology for the CIA.
The speakers were quick to clarify that they had no inside knowledge of the Lavender system specifically; that the United States did not depend on automated intelligence in this way; that we would have checks and balances and multiple lawyers signing off on every family killed in their beds by a drone strike. “We bolstered the intelligence community with sixteen thousand analysts when we started with Iraq and Afghanistan,” Cartwright said, in a less than reassuring exegesis on the military’s kill chain protocol. “The speed of what’s occurring out there [on the battlefield] is not well understood by most, okay?” he told the crowd. “It is difficult to put this in terms people are comfortable with—but understand that your brain is giving you the world already gone by. The fastest reflex out there for a second baseman in professional baseball is about a quarter of a second. I mean, fire control is twenty milliseconds. In the end, it’s faster than a human can comprehend.”
Moderating the conversation were professors Claire Finkelstein and Daniel Koditschek—the GRASP Lab professor who had so firmly distanced himself from Ghost Robotics. Throughout the talk, Koditschek occasionally scanned the crowd with a pained look on his face. “The tempo,” he kept saying. “Could you comment on the worry that the rapid, amazingly accelerated tempo at which these decisions were made . . . is detrimental to war and the law of armed conflict?” His concerns were largely brushed off. The tempo is the point, Cartwright explained.
When the floor opened up for questions, I asked how concerned the public should be about this type of system being used domestically. Craig Wiener from MITRE rolled his eyes, while Cartwright reassured me that the algorithm would be trained differently for civilian law enforcement than for military activity. Finkelstein pushed back on that, saying she’d asked the same thing at a dinner with an unnamed “senior leader in the intelligence community,” who had given a less than soothing response.
Benjamin Kuipers, a computer science professor and AI researcher at the University of Michigan who I spoke with a few weeks after the conference, is also skeptical of assurances like Cartwright’s. “Have you watched the two short videos about Slaughterbots?” he asked me, referring to a nearly eight-minute video on weapons control advocacy made by Berkeley computer science professor Stuart Russell and the Future of Life Institute in 2017. The video depicts a grim near future in which small weaponized drones are dispatched as assassins. “It’s pretty disturbing,” he said. Professor Kuipers, who became a Quaker during the Vietnam War, does not take any military funding for his research. In 1978, he completed a thesis on cognitive maps to find that the only money available to him was coming from the Defense Advanced Research Projects Agency, which was interested in building smart cruise missiles. “This was not what I wanted my life’s work to support,” Kuipers wrote in a statement on his decision. Military research is a slippery slope, he explained. One day you’re working on something completely innocuous; the next, funding dries up, you have a lab full of graduate students and postdocs to support, and you’re railroaded into a weapons project.
School for Spies
When students left Penn for the summer in 1966, the administration clearly hoped that the protests against the Institute for Cooperative Research would die down. Instead, an article by Sol Stern appeared in a small left-wing magazine called Ramparts. “Ben Franklin’s School for Spies” the cover declared, above a smirking cartoon of Penn’s founder. The article seized on the university’s close and confusing relationship with the CIA, accusing Penn of being “not so much a university on the make, as a university that has been had. It has been kept solvent so long by federal contracts, that it would have difficulty operating on its present scale if it had to forego Defense Department financing. Government grants provide the largest single source of the University’s total income—approximately $25 million out of a university budget of almost $90 million.”
When I found the dog, it was quite literally shaking its ass on the dance floor at Philadelphia’s Fitler Club.
This was difficult to refute. At the time, Penn was openly offering a class intended as field training for CIA agents: Poli Sci 551. It was taught by George and Charlotte Dyer, a married couple who both worked in intelligence and graduated with doctorates from Penn after writing a thesis together. The Dyers believed strongly in the practical application of counterintelligence, instructing their students in military maneuvers around their home in Bucks County and a creek in Mays Landing, which they claimed “found a real resemblance to the Mekong Delta.” Another exercise had students reenact the Warsaw Ghetto Uprising in some abandoned buildings in South Jersey; yet another had them recreate the 1965 April Revolution in the Dominican Republic’s civil war.
The Ramparts piece brought a great deal of unwanted attention to Penn’s embattled administration, which is reflected acutely in the Donald Shipley Murray papers in the Penn archives. Murray served as the university’s Assistant to the President for Federal Relations, functioning as a liaison between Penn and D.C. from 1964 to 1976. He arrived at a rough period, just in time for the classified projects to blow up in his face, and his notes reflect that stress, becoming increasingly bitchy and pedantic. Murray even created his own annotated version of the Ramparts article. “In light of the grisly work that goes on at the Institute for Cooperative Research,” the piece read, “[the president’s] laissez-faire attitude toward university research suggests that with a PhD and a foundation willing to provide support, Al Capone could have found a hospitable climate at Penn for a research project into the security systems of various Philadelphia banks.” Al Capone, Murray scrawled, would have to “have a faculty appointment not just a PhD.”
After twenty months of protest, a six-day occupation of College Hall by a thousand people, and threats by a group of faculty members to wear gas masks to graduation, Penn finally caved. (As the photo I stumbled across suggests, some students wore them anyway.) The ICR was forced off campus and sold to Booz Allen Hamilton. More significantly, classified research became broadly seen as inappropriate for all universities. The Wall Street Journal reported that by October 1967, Stanford, Cornell, Columbia, Princeton, NYU, and many other schools were either canceling their classified contracts or severely tightening restrictions on them.
As for the personalities involved in the ICR scandal, they are almost entirely gone now. Sol Stern broke quite forcefully from the left, became a fellow at the Manhattan Institute, and now writes screeds against the Palestinian people for Commentary. Knut Krieger continued to teach at Penn and served on the National Defense Research Committee until his death in 1975. The FBI kept tabs for years on Gabriel Kolko, who died in 2014. Robin Maisel went on to work as a printer, unionizing each shop as he jumped from job to job, and ran for office, unsuccessfully, on the socialist ballot several times. He died of multiple sclerosis in 2020 at the age of seventy-six.
Booting the ICR from campus was a rare victory, but military research kept churning on at Penn. The Holmesburg Prison Experiments, in which a professor tested chemicals such as dioxin—a major component of Agent Orange—on Philly inmates continued until 1974, and declassified documents later revealed that the school also participated in the CIA’s MK-ULTRA mind control program. Today, Penn no longer throws its students into condemned buildings in order to make them spies, but the university does still rely heavily on federal funding for scientific research—Penn received $936 million in federal research funding in 2023, $53 million of which came from the DoD.
This is not to say that everything the DoD funds is destructive. Just last year, Penn Medicine received $10 million for breast cancer research from the military. But in some ways, it feels like the main difference between now and the early 1960s is that military research no longer occurs under the cover of darkness. Ghost Robotics does not disguise its relationship to the national security apparatus but boasts about it proudly on its website. And the dogs themselves, unlike the experiments at the ICR, are not kept under lock and key: they can frequently be seen loping around the yard that surrounds the Pennovation Center, frightening passing cyclists.
Dancing with Dogs
Ghost Robotics did not respond to multiple requests for comment, so I went to Philly Tech Week, where I’d heard a robot dog was being led around to various events. When I found the dog, it was quite literally shaking its ass on the dance floor at the Fitler Club—a members-only club doing its best to become Philadelphia’s answer to Soho House. This was Tech Week’s signature event. The dog was surrounded by considerably more innocuous forms of technology—students from Central High School who had won a robotics contest; Drexel University’s Gaming Lab; and a machine that 3D-printed chocolate. One booth was advertising a video game called Trouble in Tornado Town—you played as the tornado, and the goal was to cause as much destruction as possible.
He warned that Ghost Robotics was treading dangerously close to an international human rights violation.
When I asked representatives at the Ghost Robotics table how they felt about the protests against the company at Penn, they demurred. A woman named Christina told me that this was really something their CEO should speak to, despite the fact that their CEO is impossible to reach. She showed me the controller she used to move the dog. When not being actively directed, it roams around on its own. Watching it dance with the DJ, it was difficult not to think that Marx had it backwards—history occurs first as farce, then as tragedy.
As the night began to wind down, protesters from Shut Down Ghost Robotics came to Tech Week. “Shout out to one of our sponsors, Ghost Robotics, who’s building weapons of mass destruction,” a representative for the campaign said, commandeering a mic. “You might think it’s cute and innocuous, but that fucking robot dog is a fucking machine gun on legs. A machine gun on legs! And it’s made right here in Philadelphia in collaboration with UPenn and Pennovation Center on Gray’s Ferry Ave. . . . Shame on all you motherfuckers, fuck you!” They walked offstage and were pushed out of the event by security. A woman stepped up on stage to retake the microphone. “I am actually a supporter of protests, but not while I’m trying to have a good time,” she said. “So, uh, I hope everyone can get back on track.”
In the early hours of the following day, the Penn encampment was raided by the Philadelphia Police Department. Although the administration accused demonstrators of making the campus “less safe,” police initially refused to clear the tents, insisting that the encampment was peaceful and did not pose any danger to the community. But after Governor Josh Shapiro spoke against it, PPD changed its tune. By the time I biked over on the morning of May 10, the site was cordoned off completely, and thirty-three people had been arrested. A few protesters lingered across the street, chanting at the phalanx of cops standing guard. The head of the faculty senate resigned in protest, but the university went on to issue guidelines for demonstrations that would effectively allow them to throttle large-scale, public pro-Palestine activism on campus.
Despite the forced closure of encampments at colleges and universities across the country, it would be premature to assume that campus protesters will not eventually extract some concessions from their universities. “You are not getting rid of the cause by getting rid of the tents,” a Drexel graduate student told the Philadelphia Inquirer, even if administrators are more comfortable siccing the police on their students than they were in decades past. The campaign to move Ghost Robotics off Penn’s campus continues; in June, representatives from Shut Down Ghost Robotics were issued citations while trying to deliver a petition to the company. It’s important to remember that the fight against the ICR did not succeed overnight, after all—it took two years of sustained pressure from students and faculty alike. But what the legacy of the ICR also makes clear is that academia’s relationship with the military industrial complex is always difficult to sever. Not long after the lab was moved off campus, students began protesting again—this time against Dow Chemical, which was supplying the DoD with napalm and maintained a recruiting office at Penn.
New Kinds of Death
Shortly before writing this piece, I started reading The Names, Don DeLillo’s novel about a risk analyst named James Axton loafing around Greece in the late 1970s and trying to reconcile with his ex-wife, until he becomes fascinated by a murderous cult. “Bank loans, arms credits, goods, technology. Technicians are the infiltrators of ancient societies,” Axton says to his ex at one point. “They speak a secret language. They bring new kinds of death with them. New uses for death. New ways to think about death.” Initially, I felt this dialogue was heavy-handed and overblown; after the CERL conference, I changed my mind.
There was general agreement among the conference’s keynote speakers that any AI targeting system was likely to make regrettable errors, but those errors fell within an acceptable margin. Any strike that was later determined to be beyond the pale could be traced back to a human actor in the kill chain. That soldier would, theoretically, be reprimanded, and justice would be served. Borrowing an automobile analogy, researcher and anthropologist Madeleine Clare Elish refers to this as the “moral crumple zone” in human-robot systems. She wrote in a 2019 article for the journal Engaging Science, Technology, and Society: “While the crumple zone in a car is meant to protect the human driver, the moral crumple zone protects the integrity of the technological system, at the expense of the nearest human operator.”
This concept goes some way in explaining why Penn’s GRASP Lab has maintained an air of moral neutrality that the chemical researchers at the ICR were never able to achieve. As early as 2008, when the lab secured a $22 million DoD grant to make robotic ants, flies, and cockroaches, a security analyst named John Pike sounded the alarm about the armor of willful ignorance protecting GRASP. “No one’s really thought through where this might be heading, and you’re certainly not going to get those nice engineering school people to talk about it,” he told the Inquirer. “They only make nice robots.”