Quantifying the human visual exposome with vision l... | AI Research

Key Takeaways

Quantifying the human visual exposome with vision language models This research introduces a new way to understand how our daily visual surroundings influenc...
The visual environment is a fundamental yet unquantified determinant of mental health.
While the concept of the environmental exposome is well established, current methods rely on coarse geospatial proxies or biased self reports, failing to capture the first person visual context of daily life.
We addressed this gap by coupling ecological momentary assessment with vision language models (VLMs) to quantify the semantic richness of human visual experience.
Across 2674 participant generated photographs, VLM derived estimates of greenness robustly predicted momentary affect and chronic stress, consistent with established benchmarks.

Paper AbstractExpand

The visual environment is a fundamental yet unquantified determinant of mental health. While the concept of the environmental exposome is well established, current methods rely on coarse geospatial proxies or biased self reports, failing to capture the first person visual context of daily life. We addressed this gap by coupling ecological momentary assessment with vision language models (VLMs) to quantify the semantic richness of human visual experience. Across 2674 participant generated photographs, VLM derived estimates of greenness robustly predicted momentary affect and chronic stress, consistent with established benchmarks. We then developed a semi autonomous large language model (LLM) based pipeline that mined over seven million scientific publications to extract nearly 1000 environmental features empirically linked to mental health. When applied to real world imagery, up to 33 percent of VLM extracted context ratings significantly correlated with affect and stress. These findings establish a scalable objective paradigm for visual exposomics, enabling high throughput decoding of how the visible world is associated with mental health.

Quantifying the human visual exposome with vision language models

This research introduces a new way to understand how our daily visual surroundings influence our mental health. While scientists have long known that our environment affects our well-being, current methods for measuring this—such as using broad geographic data or relying on people to self-report their experiences—are often imprecise. This study proposes a more objective, scalable approach by using artificial intelligence to analyze the actual visual context of people's daily lives, effectively "decoding" how the visible world relates to stress and mood.

Bridging the gap with AI

To capture the "visual exposome"—the sum of all visual experiences a person encounters—the researchers combined ecological momentary assessment with vision language models (VLMs). By analyzing 2,674 photographs taken by participants, the team was able to quantify the semantic richness of these images. This allows researchers to move beyond coarse data and look at the specific visual elements that individuals interact with in their day-to-day environments.

Mining scientific knowledge

A key component of the study was the development of a semi-autonomous pipeline powered by large language models (LLMs). The researchers used this system to scan over seven million scientific publications to identify nearly 1,000 environmental features that have been empirically linked to mental health in existing literature. This massive knowledge base provides a structured framework for the AI to interpret what it "sees" in real-world imagery.

Key findings and impact

The results demonstrate that this AI-driven approach is both effective and accurate. When the researchers tested their model on real-world photos, they found that up to 33 percent of the context ratings extracted by the VLM showed a significant correlation with the participants' reported affect and stress levels. Furthermore, the model’s ability to estimate "greenness" in images successfully predicted stress and mood in a way that aligned with established scientific benchmarks.

A new paradigm for mental health

These findings establish a scalable, objective method for visual exposomics. By enabling high-throughput analysis of visual data, this research provides a new tool for scientists to study the complex relationship between our environment and our mental health. This approach offers a path toward a more nuanced understanding of how the visible world shapes our internal experiences, moving away from subjective reporting toward data-driven insights.