GW’s Foggy Bottom Campus is just a few blocks from the Smithsonian Institution, the world’s largest museum, education and research complex, allowing GW students and faculty unprecedented access to some of the most captivating collections in the world. Not all people are lucky enough to have such physical access to the collections, however, which inspired the Smithsonian to launch the Smithsonian Open Access initiative to give individuals digital access to over 5.1 million items from the collections.
A similar passion for search accessibility inspired Professor Robert Pless to propose a collaboration with Dr. Alex White from the Smithsonian Data Science Lab to use computer vision tools to make it easier for users to search for images in the Smithsonian Open Access Image Collection.
Image-text search engines can be particularly challenging to navigate, Pless explained.
“For example, you might make a query: ‘Find a picture of a red butterfly with a green head sitting on a branch.’ When you don’t get any matching results, is it because your dataset doesn’t have any green headed red butterflies on a branch, or is the language component not able to understand one of the concepts or how they relate, or is the image component not able to recognize something?” he clarified.
Pless, with CS PhD students Kevin Robbins and Alper Çetinkaya, is using new computer vision tools like “CLIP” that tie images and languages together, like with Dall-E, where you type in a description and the system generates an image to match that description. They use the same underlying function to help users find the image in the Smithsonian collection that best matches their query, allowing for sophisticated searching.
Much of the collection is a herbarium collection, which documents where and when leaves were collected and preserved. Analysis of these images can show when insects started eating the leaves during the year and how that timing has changed over time.
The Smithsonian was founded with the goal of increasing and diffusing knowledge, and Pless’s work plays an integral role in furthering this mission in our modernized, digitized world.