Geospatial Innovation for Food Security
Fields of The World
Features of The World
Benchmarks of The WorldGeospatial foundation models (GFMs) promise reusable starting points for disaster response, land-cover mapping, food-security monitoring, and other Earth-observation tasks. But if you’re choosing a model, our audit of 152 papers found that the published research often doesn’t provide the evidence you need to compare them. We propose six changes to help practitioners decide which models to test on their own data.
The field has grown from early self-supervised approaches such as Tile2Vec, released in 2018, and SatMAE and SSL4EO, both released in 2022. It now includes multimodal models like Ai2’s OlmoEarth, IBM’s TerraMind, enterprise models like Esri’s ArcGIS foundation models, and Google’s AlphaEarth which produces global geospatial embeddings.
In mid-2025, Taylor Geospatial Engine, a predecessor to Taylor Geospatial, conducted a qualitative market assessment of GeoAI, focusing on geospatial foundation models and embeddings. We wanted to understand what was holding back operational use and where investment, research, and product development could help.
The assessment combined in-depth interviews with more than 15 practitioners at industry’s leading organizations and the team’s own analysis. Interviewees represented geospatial data providers, solution firms, cloud service providers, research organizations, and data users. We asked about the promise and limits of GFMs and embeddings, deployment bottlenecks, and missing capabilities.
The practitioners we interviewed generally saw promise in foundation models and embeddings, but disagreed on which approaches would prevail and how they would be used. With no clear consensus among our interviewees, we turned to the research literature.
Turning to the research
We discussed these findings with Dr. Hannah Kerner, Chief Scientist at Taylor Geospatial, and decided to review the evidence in research papers.
Dr. Kerner and Dr. Isaac Corley, Director of AI/ML Research at Taylor Geospatial, worked with the paper’s coauthors to assess which models perform best at which tasks. Practitioners need those answers before building GFMs into their products.
What did we find?
Our audit covered 152 papers from 2019–2025. We found that the published results do not support reliable comparisons across the GFM literature.
Consider Scale-MAE on RESISC45, an aerial-image classification benchmark. Our audit found the original authors report an accuracy of ~90% which is reasonable, while another paper reimplemented their evaluation recipe and reported 33.0%. The paper did not specify enough details to explain the 56 percentage-point gap and never open-sourced any code. This example is an extreme, but many other paper comparisons can vary anywhere from 2-10 percentage points. These types of discrepancies leave a practitioner unable to tell which score to trust or what result to expect from their own evaluation.
This is not an isolated case. It is the central finding in the paper, “No One Knows The State Of The Art In Geospatial Foundation Models,” accepted to the 2026 Conference on Neural Information Processing Systems (NeurIPS).
Why does no one know?
The papers we audited often lack the shared evaluation procedures, released model weights, and controlled comparisons needed to interpret their results.
The table below lists four audit findings and what each means for anyone picking a model.
| Audit findings | What it means if you are choosing a model |
|---|---|
| Roughly 40% of papers release no model weights | You cannot test the model on your own data, and neither can anyone else |
| 45+ cases where papers disagree by 10 points or more on the same model, benchmark, and recipe | A published score is not a reliable guide to which model performs better |
| Over 400 different benchmarks across the audited papers, and 35% of papers use none of the ten most common | Two models you want to compare may never have been tested on the same task |
| About 75% of papers with identifiable pretraining data use a setup no other paper uses | When a new model scores higher, no one can tell whether the model design or the training data made the difference |
We are careful about what these findings do and do not show. “The methods may be useful,” we write, “but the GFM literature does not standardize evaluations, training and testing protocols, released weights, or pretraining controls well enough for anyone to compare or rank them.”
The findings were unsurprising, said Corley. “Anyone who has written a foundation model paper has run into this. You have to compare against other models, and the numbers you need either don’t exist, can’t be reproduced, or were run under a different setup. You just do the best you can.”
The problem is not unique to geospatial. Lipton and Steinhardt’s 2018 essay Troubling Trends in Machine Learning Scholarship, which inspired this audit, criticized the failure to identify what causes reported gains in machine learning. We wanted to measure how often problems with attribution and comparability appeared in the GFM literature.
The paper calls this “a coordination failure, not a fault of any individual lab,” and we count ourselves among those who contributed to it. Addressing it takes work across organizations.
“Honestly, a lot of this paper was just doing the homework,” Corley said. “We went through 152 papers, pulled together their reported results, and got a group of busy researchers working toward the same goal. It’s not exactly glamorous work, and it doesn’t get the same attention as developing a new model. But someone has to do it, and it’s part of our mission at Taylor Geospatial to drive forward the state of the art.”
Can we ever know?
Yes! We can make comparisons more useful. Large language model researchers use shared tools such as EleutherAI’s lm-evaluation-harness, an open-source package with reusable task definitions and evaluation code that also powers public leaderboards. Shared tools make it easier to compare models under a specified setup. Geospatial benchmarking tools already exist; the gap our audit identifies is a common evaluation procedure used consistently across papers.an open-source and independent package that all model creators can run, with fixed task definitions and public leaderboards. Geospatial has many benchmark datasets but no similarly standardized toolkit. In their position statement, the authors encourage the community to come together to “fix this.” They include themselves in the call to action
The paper makes six recommendations, each tied to a troubling trend identified during the audit. They apply to our own work as much as anyone else’s.
| Troubling trend | Recommendation |
|---|---|
| Model weights are often not published | 1. Release weights under a named license, or state what prevents it |
| The field has no shared set of core benchmarks | 2. Report results on a shared core set of evaluations |
| Reported scores for the same model diverge widely | 3. Label every baseline number as rerun or copied. 4. Report how much results vary between runs |
| Benchmark collections bundle datasets but do not run the same test for everyone | 5. Build one shared evaluation harness that every model owner can run |
| Changes to model design and training data are bundled together | 6. Include controlled comparisons that isolate what changed |
The goal is practical. We describe a future in which a user who needs a model “for crop mapping, flood response, or building detection can open a maintained leaderboard, choose the task, sensor, region, and date range they care about, and trust that the top entries were run under the same protocol.” Those tests also need to reflect the intended deployment. A reproducible score on one benchmark does not establish performance in a different region, season, or sensor.
What comes next
Taylor Geospatial has brought together researchers and industry partners to build an open benchmarking system for geospatial AI. Dr. Kerner leads this work through our Innovation Program. We want teams to be able to rerun evaluations, understand how models compare, and spend less time rebuilding the same evaluation code.
Stay tuned for announcements in the coming weeks, and join us at SatSummit in St. Louis, November 18–19 to learn more.
