RQ 2.1
Geographic coherence
Do agents make geographically and temporally coherent decisions within reconstructed wildfire conditions?
PhD research · LLM agent evaluation · Wildfire evacuation
Beyond
plausible
decisions.
LLM agents are increasingly used to stand in for people in disaster simulations. SPARC-Evac tests whether they respond to the right causes (fire, warnings, neighbors, transport), not just whether their decisions sound reasonable.
Mary SalamiPhD Candidate, GIScience · UC Santa BarbaraEaton Fire · January 2025West Altadena, CAPaired counterfactual testsProtective Action Decision Model
Status: Experiment phase
Conceptual hazard / household geometrySchematic · not a real map
My role
Lead researcher: research questions, study design, evaluation protocol, and pipeline
Methods
Paired counterfactual tests, geospatial analysis, uncertainty estimation
Tools
Python · GeoPandas · OSMnx / NetworkX · LLM APIs with structured JSON output · Git
Outputs
Dissertation chapter, then a workshop paper and a journal article (planned, 2027)
01 / Why it matters
Evaluate what agents respond to, not only what they decide.
A simulated household can make the right choice for the wrong reason. If an agent's decision looks sensible but ignores the warning it was given, or shifts because of details that should not matter, the simulation cannot be trusted to inform emergency planning.
RQ 2.2
Behavioral cue response
Do controlled changes in hazard, warnings, social information, and transportation produce theoretically defensible decision changes?
RQ 2.3
Grounding & robustness
Does geographic grounding improve response consistency under changing or held-out hazard conditions?
02 / Approach
Hold everything constant. Change one cue.
Each synthetic household scenario is built from public data on the January 2025 Eaton Fire: fire perimeters, evacuation zones, the road network, and census-based household characteristics. The same scenario is then run in pairs that differ in exactly one decision-relevant cue.
The expected direction of every change is written down before any model is run, using the Protective Action Decision Model from hazards research.
A decision can be geographically feasible without being a valid model of human behavior. This study measures feasibility and consistency with theory; it does not claim to predict what real Altadena households did.
03 / Interactive demo
One changed condition. Did the decision move the right way?
Pick a cue and change only that feature. A simple, transparent toy rule stands in for an agent so you can see how each paired test is scored against its expectation.
Scenario controls
InteractiveReference household: moderate fire proximity, no order received, no neighbors seen leaving, vehicle available.
Paired comparison
Toy ruleReference caseFixed context
Changed caseOne variable
Show the logged scenario pair (JSON)
The toy rule is a few lines of JavaScript, written to illustrate the scoring. It is not an LLM, not a calibrated behavioral model, and not a prediction for the Eaton Fire. It fails some tests on purpose.
04 / The researcher
Mary Salami
PhD Candidate in GIScience, UC Santa Barbara · expected 2027
Spatial Climate Solutions Lab · advised by David Lopez-Carr and Grace C. Wu
I study whether LLM agents respond to the right causes when they stand in for people in human–environment systems. My background spans GIScience, machine learning, and applied AI: I built NLP and computer-vision datasets and models at Data Science Nigeria, designed a generative-AI siting-risk tool at ENGIE North America, and built post-wildfire recovery datasets at Direct Relief.
- Reviewer, NeurIPS 2026 Evaluations & Datasets Track
- Workshop organizer, CLRLC at NeurIPS 2025
- Systematic review of generative AI across socio-environmental systems (in review)
- Earlier work at NeurIPS, CVPR, and AAAI workshops