What Computer Vision Can Tell Us About the Natural World

A Caltech-led program works to give ecologists computer-vision tools to analyze large sets of data.

By Brian Smith

It should not be hard to find the biggest creature in the world. Yet, amid all the noise and activity across a vast ocean, even the calls of the mighty blue whale can be vanishingly hard to detect.

Michaela Alksne, a PhD student at the Scripps Institution of Oceanography, investigates how to detect and classify the social and foraging calls of the Northeastern Pacific blue whale. Her new approach is to use a large dataset of spectrograms, or visual representations of their sound waves. Last year, a three-week intensive workshop at Caltech, called Computer Vision Methods for Ecology (CV4Ecology), gave Alksne the computer-vision (CV) tools—methods by which machines “see” images to reproduce human analytical abilities through artificial intelligence—she needed to move that project forward.

“CV4Ecology helped jump-start my ability to build, fine-tune, and interpret computer vision models,” Alksne says. “I was able to successfully build a model that can classify blue whale calls in spectrograms. This model will be a foundational, invaluable tool in my overall data-processing pipeline and will allow me to efficiently, and precisely, analyze more data.”

Ecologists around the world are harnessing the power of computer vision to study and improve relationships among living organisms, humans, and their physical environments. Scientists can now use computer vision to analyze satellite imagery that reveals how walrus populations respond to changes in the Arctic, monitor sonar cameras in Alaskan riverbeds to track salmon spawning rates, and predict wind speeds by scanning footage of swaying trees to support firefighting efforts.

The CV4Ecology summer workshop, which Caltech hosted for the second time in 2023 and will host again this summer, is meant to further foster this interconnection between the natural sciences and advanced technologies. It provides ecologists with the knowledge, skills, and collaborative opportunities to accurately and efficiently analyze large image, audio, or video datasets using machine learning and artificial intelligence.

The workshop was created by Sara Beery (PhD ’23), Homer A. Burnell Career Development Professor in AI and Decision-Making at MIT; and Pietro Perona, Caltech’s Allen E. Puckett Professor of Electrical Engineering. CV4Ecology is hosted on Caltech’s campus and supported by the Resnick Sustainability Institute and the nonprofit EleutherAI in partnership with the Caltech AI4Science initiative. Co-sponsors include Microsoft AI for Earth, Amazon Web Services, and Wild Me, with data sponsorship from Planet Labs.

Beery says one of the major bottlenecks in using computer vision for ecology—or, more broadly, using AI to advance science—is the need for scientists to master the skills required to take best advantage of this rapidly evolving technology.

“We created CV4Ecology to teach the practical skills needed to use machine learning for ecological research to the people who are formulating and working to solve problems in their respective fields,” she says, “empowering them with the knowledge to do this work on their own.”

Beery knows plenty about integrating ecology with machine learning. Before joining MIT, she was a visiting faculty researcher at Google who worked on Auto Arborist, a dataset of 2.6 million trees across 23 North American cities. The project used machine learning to analyze street-level and aerial imagery of the trees’ growth and health, which allowed for more efficient, less expensive urban-forest monitoring.

Perona is also no stranger to machine learning. He received the 2013 Longuet-Higgins Prize and the 2010 Koenderink Prize, both of which recognize significant contributions in computer vision. His Visipedia project (a visual interface to Wikipedia) produced two apps, iNaturalist and Merlin Bird ID, which allow anyone with a smart device to scan their photos to identify plant and animal species.

Now, Beery says, she is proud to return to Caltech to help support the next generation of ecologists in building computer vision methods that enable global-scale environmental and biodiversity monitoring to tackle real-world challenges.

At CV4Ecology last August, 18 students joined three mentors, four instructors, and four teaching assistants from esteemed science institutions around the world. They convened at Caltech to immerse themselves in classroom training and hands-on projects with one-on-one mentorship opportunities. Students received foundational lessons on the principles of computer vision that focused on training and evaluating CV models specific to their own ecological research data and projects. Each student ended the experience with a working tool related to their research and a firm understanding of the concepts of CV and its real-world ecological applications, best practices, and potential risks.

“Sara and her team cultivated an engaging, collaborative, and supportive learning environment where we all felt comfortable to be ourselves and ask questions,” Alksne says. “CVE4Ecology has been the best educational experience I’ve had during my academic career, and I am immensely grateful to Sara for making this workshop possible.”

Michael Procko, a contractor working with the Tulalip Tribes of Washington state and a former research scientist at the University of Washington, was another student at CV4Ecology 2023. Procko’s work seeks to mitigate human–wildlife conflicts in real time by developing predator-detection models linked to camera traps. The system processes images and sends relevant alerts to land and wildlife managers.

“In addition to offering me tools to streamline my research via expediting photo and video processing, the experience offered me a really interesting perspective on data management that fundamentally altered the way I’ll approach research projects in the future,” Procko says.

In addition to gaining technical skills and fresh perspectives, Procko particularly appreciated the interdisciplinary nature of the workshop.

“It was really amazing to see how bridging gaps between disciplines provides an entirely new perspective,” he says. “To see the weaving of ecological research and computer science into something novel, conservation technology, was a really poignant example of how so much of scientific progress relies on effective collaboration.”

Shir Bar, a PhD candidate at Tel Aviv University who joined CV4Ecology as a teaching assistant, uses computer-vision technology to study the feeding behavior of freely swimming fish larvae. Such small, fast-moving creatures can be difficult to study; by refining computer vision methods for tracking their behavior, Bar says, this work can benefit other similar machine-learning efforts in ecology.

“The experience provided a sense of community. There are not many people at the intersection of these disciplines, and spending three weeks with people who share your geeky interests is a powerful experience,” Bar says. “The workshop brings together engineers who have been in this space for a while with ecologists and conservationists who know how to acquire, interpret, and reason about data. That we were all able to learn from one another how to create better applied computer-vision tools for ecology, in a hands-on manner, is the most valuable part.”

Apply for the third CV4Ecology workshop, hosted at Caltech January 6–24, 2025. Students must apply by March 22, 2024, to be considered, and will be notified of acceptance on April 19, 2024.