Imagine you’re a biologist with camera traps in the savanna or samples of algae from a bloom in Florida waterways. If you pinpoint endangered species’ migration corridors, your research could draw the boundaries of a new preserve. Determining which algae are blooming — and why — could have massive public health implications. Either way, the challenge is the same: It takes hundreds of hours to sort through tens of thousands of nighttime photos of animals or microscopic images of water samples with thousands of species of algae. But now help is at hand, and you don’t need to be a biologist to access it.
Rapid advances in artificial intelligence (AI) over the past five years are spawning a wave of technologies and apps that are allowing biologists to leave some of their most tedious tasks of identifying species to machines trained to do that instead. This explosion is also spilling over into the public domain, expanding resources that enable curious citizen scientists, foraging backpackers, landscape architects and farmers to, on their own, also identify rare species of plants and animals they come across.
These technologies — collectively referred to as Species-Identifying Artificial Intelligence (SIAI) — rely on machine learning, where statistical methods are used to train machines to perform tasks, to do for plants and animals what tools like the British app Shazam do with songs: identify them. Deep neural networks, or deep learning, which can analyze multiple layers of an image, are particularly useful in identifying wildlife.
So helpful that Pl@ntNet, a plant-identification app that launched in 2010 but switched to deep learning in 2017, has been downloaded 5 million times on Google Play. The Cornell Lab of Ornithology’s Merlin Bird ID app, launched in 2014, can identify 650 American bird species and has been downloaded half a million times on Google Play. Another app, launched by the image database iNaturalist in 2017, uses deep learning and boasts a genus-identification accuracy rate of 86 percent. And earlier this year, two German institutions launched Flora Incognita, an app that draws on expert botanists, AI and geographic data to identify more than 2,700 types of German plants.
They [traditional computer vision experts] mostly worked on recognition of everyday objects that the man on the street would be able to identify.
Serge Belongie, Cornell professor
These technologies rooted in deep learning are fundamentally superior to traditional computer vision tools, where computers analyze images, says Cornell professor Serge Belongie, part of the team that built Merlin Bird ID.
“A lot of people in computer vision before deep learning were already working on object recognition. But they mostly worked on recognition of everyday objects that the man on the street would be able to identify,” says Belongie. The programs could tell you a photo was of a mushroom, but not whether it was poisonous.
For sure, deep learning in many ways represents brute force AI. It requires a massive data set to train. Luckily, over the last few years, open access databases have emerged that provide reams of images of thousands of labeled plants and animals. The largest of these include Zooniverse with 1.2 million images, iNaturalist with 675,000 photos, and iDigBio with 1.8 million snapshots.
Armed with those databases, deep learning can do what ordinary computer vision can’t. Deep learning tools first analyze the most simple parts of an image, say, a leaf’s edge. Then, says Max Planck Institute software engineer Patrick Mäder, they scrutinize more complex image characteristics — for example, a leaf’s texture — before differentiating between the leaf, stem and flower, until you get a full classification of the species. Then, in a process called backpropagation, the program self-corrects errors. It first gauges how great the misses were and penalizes larger mistakes. It then passes instructions on how to change its analytical process for future analysis.
AlexNet, an AI platform that won the 2012 image-identification challenge ImageNet — making half as many errors as the next best contender — represented the first major breakthrough for deep learning capabilities in recognizing photos. Soon, a race to hire deep learning experts began between Google, Baidu and Facebook. Graphics processing power kept improving and funding swelled. In 2012, Google had two deep learning projects. By 2016, it had 1,000. Several tech companies began releasing open access deep learning architectures. Then, the deep learning program AlphaGo beat the human champion at the board game Go in 2016, further bolstering the cache of image-identifying AI initiatives.
By then, the first efforts at using machine learning for identifying species had taken off. In the U.K., researchers in 2015 launched the app Warblr, which uses recordings of bird songs to identify the bird. But while sound-recognition tools are valuable for birds, images remain the best bet for plants and the millions of species — many of them very tiny — whose sounds can’t be recorded.
The growing number and popularity of apps that help identify plants and animals didn’t all start in their present avatars. Pl@ntNet, for instance, began in 2010 as a media research project headed by four French institutions. The intention was to bolster citizen science, says Alexis Joly, a researcher at the National Institute for Research in Computer Science and Control and a Pla@ntNet collaborator. Back then, it had 200 contributors and managed to identify 70 European species. By 2017, after turning to deep learning, it also identified Indian, South American, North African and American plants. The iNaturalist app also initially relied solely on a community of users to identify plants and animals before recognizing the power of deep learning.
At the Cornell Lab of Ornithology, Belongie — inspired in part by his ornithologist sister — began by developing a bird-identifying program before his team switched to Google’s open access deep learning tool TensorFlow to create the Merlin Bird ID app.
These downloadable AI naturalists aren’t flawless. For one, the number of species on Earth is simply massive, and there’s a dearth of data on species for SIAI in the southern hemisphere. Oftentimes, the most identifiable characteristics of species — such as flowers — are seasonal. The photos used to inform these models might not always have the resolution to distinguish between wild carrot and poison hemlock. And foreign plants, such as the Oriental trees popular in American landscaping, can throw off these AI tools.
Nevertheless, apps like Flora Incognita, iNaturalist, Merlin Bird ID and Pl@ntNet are only the vanguard of a revolution in AI-enabled species recognition that includes deep learning research currently underway in dozens of labs worldwide. For biologists and ordinary folks alike, identifying exotic species is increasingly just about pointing the smartphone and clicking a photo.