Authors who are worried they'll soon be replaced by computer algorithms can breathe a sigh of relief — for now. But dance club turntablists may have more reason to feel anxious about being supplanted by digital DJs.
The competition is named for Alan Turing, the British mathematician and computer scientist who's considered the father of AI. In 1950, Turing devised a method of assessing a computer's ability to mimic human behavior. A machine "passes" the Turing Test by performing activities that are indistinguishable from those of a human.
The Creative Turing Tests are the brainchild of Dan Rockmore, a professor of mathematics and computer science and director of the Neukom Institute. In 2015, he put out a call to computer programmers to enter their best AI software in one of three competitive categories. "AlgoRhythms" tested the software's ability to generate danceable music mixes and pitted them against human DJs. "PoetiX" focused on the ability to produce Shakespearean or Petrarchan sonnets. And "DigiLit" sought the best software for generating short stories.
What did the contest reveal about the current state of AI creativity? As Rockmore explained in an interview last week, while AI technology has advanced to the point where its musical mixes can "fool" the human ear, neither the poetry nor the short stories were convincing enough to trick human judges.
"Narrative is still really difficult" for machines to pull off, Rockmore explained. Which is not to suggest that the algorithms aren't advancing rapidly on their flesh-and-blood counterparts. As he pointed out, it's relatively easy for software designers to construct sentence-generating machines that can build simple narratives based on specific formulas. For example, algorithms are already being used to convert baseball box scores into game summaries and raw financial data into company earnings reports.
"But if you want to make a story with a character that people care about," Rockmore said, "or if you try to make a logic diagram of what a narrative is, you get stopped pretty quickly."
The contest's poetry generators had no trouble reproducing the sonnet form, Rockmore said, often in complex and clever ways, such as using near rhymes as well as full rhymes. The poems had a problem, though: "In the words of one of the [National Public Radio] judges, Robert Siegel, 'They didn't seem to be about anything.'
"They sounded beautiful," Rockmore went on with a laugh, "so if you weren't trying to make any sense of them, you would have thought you were listening to wonderful poetry being declaimed. But as you looked a little bit more closely, they didn't really have a story."
The narrative hurdle may explain why Rockmore received only three entries in the DigiLit category and four in the PoetiX category.
"It does seem to be a hard problem," he added. "At the end of the day, I think [the result] says a lot about how complicated people are, rather than how easy it is to mimic them."
Judy Malloy, a Middlebury College graduate and now digital studies fellow at Rutgers University-Camden, won a $1,000 prize in the DigiLit category for her algorithm that creates variations on a story called "Another Party in Woodside." She described the Creative Turing Test as "an important competition, in that it brought to the forefront different ways of creating generative literature and music.
"For me, a large part of the learning was considering different approaches to both the creation and the reception of generative work," Malloy added in an email.
Because of the "hard problem" that narrative presents, Rockmore altered the format of the 2017 DigiLit category. Next time, organizers will award prizes to contestants who create algorithms that best complete a short story. Given a 1,000- to 2,000-word story prompt, they will need to generate a 300- to 500-word conclusion.
The dance-track competition was another story. There, three human DJs and three algorithm finalists, all hidden from a live audience, served up 10-minute dance mixes. Audience members cast their votes on each: A 50 percent or greater vote of "human" meant the AI passed the Turing Test.
Two algorithms ending up getting about 40 percent "human" votes, and one fooled half of all listeners in a separate online poll. Rockmore called those "highly sophisticated technical achievements."
The divergent results of tests in the two art forms, Rockmore theorized, may have something to do with the role that computer-generated elements already play in today's pop music. Modern listeners' tastes have evolved to the point where they're more accepting of music created by machines.
"Look, if we'd been reading computer-generated novels for the last 100 years," Rockmore said, "then maybe the ones we got [in the DigiLit contest] wouldn't have looked so weird."
The original print version of this article was headlined "Apparently, Software Can't Write Superior Sonnets — Yet"
Ken Picard has been a Seven Days staff writer since 2002. He has won numerous awards for his work, including the Vermont Press Association's 2005 Mavis Doyle award, a general excellence prize for reporters.