Accuracy 2010

I’ve mentioned uncertainty several times on this blog in the past (examples one and two), so it seems appropriate to highlight the next International Symposium on Spatial Accuracy Assessment in Natural Resources and Environmental Sciences. The ninth occurrence of this biennial meeting will be hosted by the University of Leicester, UK, 20th – 23rd July 2010. Oral presentations, posters and discussion will address topics including:

Semantic uncertainty and vagueness
Modelling uncertainty using geostatistics
Propagation of uncertainty in GIS
Visualizing spatial uncertainty
Uncertainty in Remote Sensing
Spatiotemporal uncertainty
Accuracy and uncertainty of DEMs
Modelling scale in environmental systems
Positional uncertainty

The deadline for abstract submission is 28th September 2009.

Ensemble Modelling

Uncertainty is an inherent part of modelling. Models by definition are simplified representations of reality – as such they are all wrong. Being wrong doesn’t necessarily make them useless, but it helps to have some idea about how wrong they are for them to be most useful. That’s why we should always try to provide some means to assess the uncertainty in our models output. Producing multiple realisations of a model and its results – a model ensemble – is one way to do this.

Depending on our model we can use ensemble methods to examine four different sources of modelling uncertainty:

a) Model Structure: how appropriate are our model variables, relationships and rules?

b) Model Parameters: what numerical values appropriately represent the strengths of relationships between variables?

c) Initial Conditions: how does our uncertainty in the initial state of the system we are modelling propagate through the model to the output?

d) Boundary Conditions: how do alternative (uncertain) scenarios of events that perturb our model influence output?

In their recent paper, Arujo and New show how ensemble modelling might be used to assess the impacts of these different kinds of uncertainty on model output. They advocate the use of multiple models within an ensemble forecasting framework and argue that more robust forecasts can be achieved via the appropriate analysis of ensemble forecasts. This is important because projections between different models can be so variable as to compromise their usefulness for guiding policy decisions. For example, upon examining nine bioclimatic models for four South African plant species, Pearson et al. found that for different scenarios of future climate predicted changes in species distribution varied from 92% loss to 322% gain between the different models! It’s uncertainty like this that stifles debate about the presence and impacts of anthropogenic climate change. Araujo and New go on to discuss the uses and limitations of ensemble modelling for supporting policy decisions in biodiversity conservation.

In a previous post I discussed how Bayesian methods can be used to examine uncertainty in model structure. I’ve been using Bayesian Model Averaging to help me identify which are the most appropriate predictors of local winter white-tailed deer density for our UP Forest project. Using the relationships developed via that modelling process I’ve produced spatial estimates of deer density in northern hardwood stands for a section of our study area (example below).

Hopefully forest managers will find this sort of modelling useful for their planning (I’ll ask them sometime). However, I think this sort of product will be even more useful if I can provide the managers with a spatial estimate of uncertainty in the deer density estimates. This is important not only to emphasise that there is uncertainty in the model results generally, but also to highlight where (in the landscape) the model is more or less likely to be correct. Here’s the uncertainty map corresponding with the deer density estimate map above.

In this map the lighter colours (yellows and greens) indicate less certainty in the deer density estimate at that point. If managers were to take action in this landscape to reduce deer densities they could use a combination of the maps to find locations where deer densities are estimated to be high with low uncertainty.

To be more specific, the uncertainty map above is the standard deviation of 1,000 deer density estimate maps (correspondingly the deer density map is the mean of these 1,000 models). For each of the 1,000 deer density estimates I used slightly different model parameter values, each chosen with a certain probability. These 1,000 realisations are my model ensemble. The probability a value would be chosen for use as a parameter in any of the 1,000 models was specified by a (normal) probability distribution which came from the mean and standard deviation provided by the original Bayesian regression model. To produce the 1,000 models and sample their parameter values from a probability distribution I wrote my own program which made use of the standalone R math libraries built by Chris Jennings.

Appropriately representing and communicating uncertainty in model output is vital if models and modelling is to be useful for non-modellers (e.g., policy-makers, natural resource managers, etc.). Spatial ensemble modelling helps us to do this by identifying locations where we are more or less confident about our model output.

New Models for Ecosystems Dynamics and Restoration

Recently I’ve been working on a review of the latest contribution to The Science and Practice of Ecological Restoration book series, entitled New Models for Ecosystems Dynamics and Restoration (edited by Hobbs and Suding). Here’s an outline of what I’ve been reading and thinking about – the formal review will appear in print in Landscape Ecology sometime in the future.

The Society for Ecological Restoration defines ecological restoration as an “intentional activity that initiates or accelerates the recovery of an ecosystem with respect to its health, integrity and sustainability”. Restoration ecology is a relatively young academic field of study that addresses problems faced by land managers and other restoration practitioners. Young et al. suggest that models of succession, community assembly and state transitions are an important component of ecological restoration, and that seed and recruitment limitation, soil processes and diversity-function relationships are also important.

The ‘new’ models referenced in the title of the book are ‘threshold’ or ‘regime shift’ ecosystem models. These models are ‘new’, the editors argue, in the sense that they contrast gradual continual models and stochastic models. Gradual continuous models are described as those that assume post-disturbance ecosystem recovery follows a continuous, gradual trajectory and are associated with classical, Clementsian theory that assumes steady, uni-directional change towards some single equilibrium state. Stochastic models assume exogenous drivers dominate the behavior of ecosystems to the extent that non-equilibrium and unstable systems states are the norm. Threshold models assume there are multiple (in contrast to the Clementsian view) stable (in contrast to the stochastic view) ecosystem states and represent changes from one relatively distinct system state to another as the result of small changes in environmental (driving) conditions. Thresholds and regime shifts are important to consider in restoration ecology as there may be thresholds in system states beyond which recovery to the previous (healthy) state is not possible.

Two types of threshold model are considered in New Models;

i) state-and-transition (S-T) models that represent multiple (often qualitative) stable states and the potential transitional relationships between those states (including the rates of transition), and

ii) alternative stable state (ASS) models which are a subset of S-T models and generally represent systems with fewer states and faster transitions (flips) between the alternative states.

For example, S-T models are often used to represent vegetation and land cover dynamics (as I did in the LFSM I developed to examine Mediterranean landscape dynamics), whereas ASS models are more frequently used for aquatic systems (e.g. lake ecosystems) and chemical/nutrient dynamics.

New Models focuses on use of these models in ecological restoration and provides an excellent introduction to key concepts and approaches in this field. Two of the six background chapters in this introduction address models and inference, two introduce transition theory and dynamics in lake and terrestrial ecosystems (respectively), and two discuss issues in social-ecological and rangeland systems. These background chapters are clear and concise, providing accessible and cogent introductions to the systems concepts that arise in the later case studies. The case studies present research and practical examples of threshold models in a range of ecosystems types – from arid, grassland, woodland and savanna ecosystems, though forest and wetland ecosystems, to ‘production landscapes’ (e.g. restoration following mining activities). Although the case study chapters are interesting examples of the current state of the use and practice of threshold modeling for ecological restoration, from my perspective there are certain issues that are insufficiently addressed. Notably, there is limited explicit consideration of spatial interactions or feedbacks between social and ecological systems.

For example, in their background chapter King and Whisenant highlight that many previous studies of thresholds in social-ecological systems have investigated an ecological system driven by a social system, ignoring feedbacks to the social components. Explicitly representing the links between social and ecological components in models does remain a daunting task, and many of the case studies continue in the same vein as the ‘uni-directional’ models King and Whisenant hint at (and I’ve discussed previously). The editors themselves highlight that detailed consideration of social systems is beyond the scope of the book and that such issues are addressed elsewhere (including in other volumes of the Ecological Restoration book series – Aronson et al.). However, representing human-environment feedbacks is becoming increasingly vital to ensure appropriate understanding of many environmental systems and their omission here may prove unsatisfactory to some.

A second shortcoming of the book, from the perspective of a landscape ecologist, is the general lack of consideration for spatial pattern and scaling and their influences on the processes considered in the case studies. In their background chapter on resilience theory and rangelands, Bestelmeyer et al. do highlight the importance of a landscape perspective and considering land as being a ‘state mosaic’, but only a single case study really picks up on these concepts in earnest (Cale and Willoughby). Other case studies do indirectly consider spatial feedbacks and landscape context, but explicit representation of relationships between spatial patterns and ecosystems processes is lacking.

However, these criticisms do need to be considered in light of the objectives of New Models. At the outset, the editors state that the book aims to collectively evaluate threshold modeling approaches as applied to ecological restoration – to examine when and where these models have been used, what evidence is used to derive and apply them, and how effective they are for guiding management. In their synthesis chapter the editors highlight that the models presented in the book have been used heuristically with little testing of their assumptions and ask; “Does this indicate an obvious gap between ecological theory and restoration practice?” For example, in their chapter on conceptual models for Australian wetlands, Sim et al. argue that the primary value of threshold models is to provide a conceptual framework of how ecosystems function relative to a variety of controlling variables. The editors’ suggestion is that restoration practitioners are applying models that work rather than “striving to prove particular elements” (of system function or ecological theory), and that maybe this isn’t such a bad approach given pressing environmental problems.

Potentially, this is a lesson that if landscape ecologists are to provide ecosystem managers and stewards with timely advice they may need to need to scale-back (i.e., reduce the complexity of) their modeling aims and objectives. Alternatively, we could view this situation as an opportunity for landscape ecologists to usefully contribute to advance the field of ecological restoration. Most likely it is indicative that where practical knowledge is needed quickly, simple models using established ecological theory and modelling tools are most useful. But in time, as our theoretical understanding and representation of spatial and human-environment interactions advances, these aspects will be integrated more readily into practical applications of modelling for ecological restoration.

Buy at Amazon

Bayesian Multimodel Inference

The abstract we submitted to the ESA Meeting was accepted a while back. Since we submitted it, Megan and I have been back in the field for some additional data collection and I’ve been doing some new analyses. Some of these new analyses are the result of my attendance at the Bayesian statistics workshop at US-IALE in Snowbird. Since then I’ve been learning more by picking the brain of a former advisor, George Perry, and doing a fair bit of reading (reading list with links at bottom). And of course, using my own data has helped a lot.

One of the main questions I’m facing, as many ecologists often do, is “which variables should be in my regression model?” This question lies at the core of model inference and assumes that it is appropriate to infer ecological process from data by searching for the single model that represents reality most accurately. However, as Link and Barker put it:

“It would be nice if there were no uncertainty about models. In such an ideal world, a single model would be available; the data analyst would be in the enviable position of having only to choose the best method for fitting model parameters based on the available data. The choice would be completely determined by the statistician’s theory, a theory which regards the model as exact depiction of the process that generated the data.

“It is clearly wrong to use the data to choose a model and then to conduct subsequent inference as though the selected model were chosen a priori: to do so is to fail to acknowledge the uncertainties present in the model selection process, and to incestuously use the data for two purposes.”

Thus, it usually more appropriate to undertake a process of multi-model inference and search for the ‘best’ possible model (given current data) rather than a single ‘true’ model. I’ve been looking into the use of Bayesian Model Averaging to address this issue. Bayesian approaches take prior knowledge (i.e., a probability distribution) and data about a system and combine them with a model to produce posterior knowledge (i.e., another probability distribution). This approach differs from the frequentist approach to statistics which calculates probabilities based on the idea of a (hypothetical) long-run of outcomes from a sequence of repeated experiments.

For example, estimating the parameters of a linear regression model using a Bayesian approach differs from a frequentist ordinary least squares (OLS) approach in two ways:

i) a Bayesian approach considers the parameter to be a random variable that might take a range of values each with a given probability, rather than being fixed with unknown probability,

ii) a Bayesian approach conditions the parameter estimate probability on the sample data at hand and not as the result of a set of multiple hypothetical independent samples (as the OLS approach does).

If there is little prior information available about the phenomena being modelled, ‘uninformative priors’ (e.g., a normal distribution with a relatively large variance about a mean of zero) can be used. In this case, the parameter estimates produced by the Bayesian linear regression will be very similar to those produced by regular OLS regression. The difference is in the error estimates and what they represent; a 95% confidence interval produced by a Bayesian analysis specifies that there is a 95% chance that the true value is within that interval given the data analyzed, whereas a 95% confidence interval from a frequentist (OLS) approach implies that if (hypothetical) data were sampled a large number of times, the parameter estimate for those samples would lie within that interval 95% of those times.

There has been debate recently in ecological circles about the merits of Bayesian versus frequentist approaches. Whilst some have strongly advocated the use of Bayesian approaches (e.g., McCarthy 2007), others have suggested a more pluralistic approach (e.g., Stephens et al. 2005). One of the main concerns with the approach of frequentist statistics is related to a broader criticism of the abuse and misuse of the P-value. For example, in linear regression models P-values are often used to examine the hypothesis that the slope of a regression line is not equal to zero (by rejecting the null hypothesis that is equal to zero). Because the slope of a regression line on a two-dimensional plot indicates the rate of change of one measure with respect to the other, a non-zero slope indicates that as one measure changes, so does the other. Consequently it is often inferred that a processes represented by one measure had an effect, or caused, the change in the other). However, as Ben Bolker points out in his excellent book:

“…common sense tells us that the null hypothesis must be false, because [the slope] can’t be exactly zero [due to the inherent variation and error in our data] — which makes the p value into a statement about whether we have enough data to detect a non-zero slope, rather than about whether the slope is actually different from zero.”

This is not to say there’s isn’t a place for null hypothesis testing using P-values in the frequentist approach. As Stephens et al. argue, “marginalizing the use of null-hypothesis testing, ecologists risk rejecting a powerful, informative and well-established analytical tool.” To the pragmatist, using whatever (statistical) tool available seems eminently more sensible than placing all one’s eggs in one basket. The important point is to try to make sure that the hypotheses one tests with P-values are ecologically meaningful.

Back to Bayesian Model Averaging (BMA). BMA provides a method to account for uncertainty in model structure by calculating (approximate) posterior probabilities for each possible model (i.e., combination of variables) that could be constructed from a set of independent variables (see Adrian Raftery’s webpage for details and examples of BMA implementation). The ‘model set’ is all possible combinations of variables (equal to 2n models, where n is the number of variables in the set). The important thing to remember with these probabilities is that it is the probability that the model is the best one from the model set considered – the probability of other models with variables not measured or included in the model set obviously can’t be calculated.

The advantage over other model selection procedures like stepwise regression is that the output provides a measure of the performance of many models, rather than simply providing the single ‘best’ model. For example, here’s a figure I derived from the output BMA provides:

The figure shows BMA results for the five models with highest posterior probability of being the best candidate model from a hypothetical model set. The probability that each model is the best in the model set is shown at top for each model – Model 1 has almost 23% chance that it is the best model given the data available. Dark blocks indicate the corresponding variable (row) is included in a given model – so Model 1 contains variables A and B, whereas Model 2 contains Variable A only. Posterior probabilities of variables being included in the best model (in the model set) are shown to the right of the blocks – as we might expect given that Variable A is present in the five most probable models it has the highest chance of being included in the best model. Click for a larger image.

BMA also provides a posterior probability for each variable being included in the best candidate model. One of the cool things about the variable posterior probability is that it can be used to produce a weighted mean value from all the models for each variable parameter estimate, each with their own Bayesian confidence interval. The weight for each parameter estimate is the probability that variable is present in the ‘best’ model. Thus, the ‘average’ model accounts for uncertainty in variable selection in the best candidate model in the individual parameter estimates.

I’ve been using these approaches to investigate the potential factors influencing local winter white-tailed deer density in in managed forests of Michigan’s Upper Peninsula. One of the most frequently used, and freely available, software packages for Bayesian statistics is WinBUGS. However, because I like to use R I’ve been exploring the packages available in that statistical language environment. Specifically, the BRugs package makes use of many OpenBUGS components (you actually provide R with a script in WinBUGS format to run) and the BMA package provides functionality for model averaging. We’re in the final stages of writing a manuscript incorporating these analyses – once it’s a bit more polished (and submitted) I’ll provide an abstract.

Reading List
Model Inference: Burnham and Anderson 2002, Stephens et al. 2005, Link and Barker 2006, Stephens et al. 2007

Introduction to Bayesian Statistics: Bolker 2007 [webpage with chapter pre-prints and exercises here], McCarthy 2007

Discussion of BMA methods: Hoeting et al. 1999, Adrian Raftery’s Webpage

Examples of BMA application: Wintle et al. 2003, Thomson et al. 2007

Criticisms of Stepwise Regression: James and McCulloch 1990, Whittingham et al. 2006

What is the point… of social simulation modelling?

Previously, I mentioned a thread on SIMSOC initiated by Scott Moss. He asked ‘Does anyone know of a correct, real-time, [agent] model-based, policy-impact forecast?. Following on to the responses to that question, earlier this week he started a new thread entitled ‘What’s the Point?:

“We already know that economic recessions and recoveries have probably never been forecast correctly — at least no counter-examples have been offered. Similarly, no financial market crashes or recoveries or significant shifts in market shares have ever, as far as we know, been forecast correctly in real time.

I believe that social simulation modelling is useful for reasons I have been exploring in publications for a number of years. But I also recognise that my beliefs are not widely held.

So I would be interested to know why other modellers think that modelling is useful or, if not useful, why they do it.”

After reading others’ responses I decided to reply with my own view:

“For me prediction of the future is only one facet of modelling (whether agent-based or any other kind) and not necessarily the primary use, especially with regards policy modelling. This view stems party from the philosophical difficulties outlined by Oreskes et al. (1994), amongst others. I agree with Mike that the field is still in the early stages of development, but I’m less confident about ever being able to precisely predict future systems states in the open systems of the ‘real world’. As Pablo suggested, if we are to predict the future the inherent uncertainties will be best highlighted and accounted for by ensuring predictions are tied to a probability.”

I also highlighted the reasons offered by Epstein and outlined a couple of other reasons I think ABM are useful.

There was a brief response to mine then and then another, more assertive, response that (I think) highlights a common confusion of the different uses of prediction in modelling:

“If models of economic policy are fundamentally unable to at some point predict the effects of policy — that is, to in some measure predict the future — then, to be blunt, what good are they? If they are unable to be predictive then they have no empirical, practical, or theoretical value. What’s left? I ask that in all seriousness.

Referring to Epstein’s article, if a model is not sufficiently grounded to show predictive power (a necessary condition of scientific results), then how can it be said to have any explanatory power? Without prediction as a stringent filter, any amount of explanation from a model becomes equivalent to a “just so” story, at worst giving old suppositions the unearned weight of observation, and at best hitting unknowably close to the mark by accident. To put that differently, if I have a model that provides a neat and tidy explanation of some social phenomena, and yet that model does not successfully replicate (and thus predict) real-world results to any degree, then we have no way of knowing if it is more accurate as an explanation than “the stars made it happen” or any other pseudo-scientific explanation. Explanations abound; we have never been short of them. Those that can be cross-checked in a predictive fashion against hard reality are those that have enduring value.

But the difficulty of creating even probabalistically predictive models, and the relative infancy of our knowledge of models and how they correspond to real-world phenomena, should not lead us into denying the need for prediction, nor into self-justification in the face of these difficulties. Rather than a scholarly “the dog ate my homework,” let’s acknowledge where we are, and maintain our standards of what modeling needs to do to be effective and valuable in any practical or theoretical way. Lowering the bar (we can “train practitioners” and “discipline policy dialogue” even if we have no way of showing that any one model is better than another) does not help the cause of agent-based modeling in the long run.

I felt this required a response – it seemed to me that difference between logical prediction and temporal prediction was being missed:

“In my earlier post I wrote: “I’m less confident about ever being able to precisely predict future systems states in the open systems of the ‘real world'”. I was careful about how I worded this [more careful than ensuring correct formatting of the post it seems – my original post is below in a more human-readable format] and maybe some clarification in the light of Mike’s comments would be useful. Here goes…

Precisely predicting the future state of an ‘open’ system at a particular instance in time does not imply we have explained or understand it (due to the philosophical issues of affirming the consequent, equifinality, underdetermination, etc.). To be really useful for explanation and to have enduring value model predictions of any system need to be cross-checked against hard reality *many times*, and in the case of societies probably also in many places (and should ideally be produced by models that are consistent with other theories). Producing multiple accurate predictions will be particularly tricky for things like the global economy for which only have one example (but of course will be easier where experimental replication more ogistically feasible).

My point is two-fold:
1) a single, precise prediction of a future does not really mean much with regard our understanding of an open system,
2) multiple precise predictions are more useful but will be more difficult to come by.

This doesn’t necessarily mean that we will never be able to consistently predict the future of open systems (in Scott’s sense of correctly forecasting of the timing and direction of change of specified indicators). I just think it’s a ways off yet, that there will always be uncertainty, and that we need to deal with this uncertainty explicitly via probabilistic output from model ensembles and other methods.Rather than lowering standards, a heuristic use of models demands we think more closely about *how* we model and what information we provide to policy makers (isn’t that the point of modelling policy outcomes in the end?).

Let’s be clear, the heuristic use of models does not allow us to ignore the real world – it still requires us to compare our model output with empirical data. And as Mike rightly pointed out, many of Epstein’s reasons to model – other than to predict – require such comparisons. However, the scientific modelling process of iteratively comparing model output with empirical data and then updating our models is a heuristic one – it does not require that precise prediction at specific point in the future is the goal before all others.

Lowering any level of standards will not help modelling – but I would argue that understanding and acknowledging the limits of using modelling in different situations in the short-term will actually help to improve standards in the long run. To develop this understanding we need to push models and modelling to their limits to find our what works, what we can do and what we can’t – that includes iteratively testing the temporal predictions of models. Iteratively testing models, understanding the philosophical issues of attempting to model social systems, exploring the use of models and modelling qualitatively (as a discussant, and a communication tool, etc.) should help modellers improve the information, the recommendations, and the working relationships they have with policy-makers.

In the long run I’d argue that both modellers and policy-makers will benefit from a pragmatic and pluralistic approach to modelling – one that acknowledges there are multiple approaches and uses of models and modelling to address societal (and environmental) questions and problems, and that [possibly self evidently] in different situations different approaches will be warranted. Predicting the future should not be the only goal of modelling social (or environmental) systems and hopefully this thread will continue to throw up alternative ideas for how we can use models and the process of modelling.”

Note that I didn’t explicitly point out the difference between the two different uses of prediction (that Oreskes and other have previously highlighted). It took Dan Olner a couple of posts later to explicitly describe the difference:

“We need some better words to describe model purpose. I would distinguish two –

a. Forecasting (not prediction) – As Mike Sellers notes, future prediction is usually “inherently probabalistic” – we need to know whether our models can do any better than chance, and how that success tails off as time passes. Often when we talk about “prediction” this is what we mean – prediction of a more-or-less uncertain future. I can’t think of a better word than forecasting.

b. Ontological prediction (OK, that’s two words!) – a term from Gregor Betz, Prediction Or Prophecy (2006). He gives the example of the prediction of Neptune’s existence from Newton’s laws – Uranus’ orbit implied that another body must exist. Betz’s point is that an ontological prediction is “timeless” – the phenomenon was always there. Einstein’s predictions about light bending near the sun is another: something that always happened, we just didn’t think to look for it. (And doubtless Eddington wouldn’t have considered *how* to look, without the theory.)

In this sense forecasting (my temporal prediction) is distinctly temporal (or spatial) and demands some statement about when (or where) an event or phenomena will occur. In contrast, ontological prediction (my logical prediction) is independent of time and/or space and is often used in closed system experiments searching for ‘universal’ laws. I wrote more about this in a series of blog posts I wrote a while back on the validation of models of open systems.

This discussion is ongoing on SIMSOC and Scott Moss has recently posted again suggesting a summary of the early responses:

“I think a perhaps extreme summary of the common element in the responses to my initial question (what is the point?, 9/6/09) is this:

**The point of modelling is to achieve precision as distinct from accuracy.**

That is, a model is a more or less complicated formal function relating a set of inputs clearly to a set of outputs. The formal inputs and outputs should relate unambiguously to the semantics of policy discussions or descriptions of observed social states and/or processes.

This precision has a number of virtues including the reasons for modelling listed by Josh Epstein. The reasons offered by Epstein and expressed separately by Lynne Hamill in her response to my question include the bounding and informing of policy discussions.

I find it interesting that most of my respondents do not consider accuracy to be an issue (though several believe that some empirically justified frequency or even probability distributions can be produced by models). And Epstein explicitly avoids using the term validation in the sense of confirmation that a model in some sense accurately describes its target phenomena.

So the upshot of all this is that models provide a kind of socially relevant precision. I think it is implicit in all of the responses (and the Epstein note) that, because of the precision, other people should care about the implications of our respective models. This leads to my follow-on questions:

Is precision a good enough reason for anyone to take seriously anyone else’s model? If it is not a good enough reason, then what is?

And so arises the debate about the importance of accuracy over precision (but the original ‘What is the point’ thread continues also). In hindsight, I think it may have been more appropriate for me to use the word accurate than precise in my postings. All this debate may seem to be just semantics and navel-gazing to many people, but as I argued in my second post, understanding the underlying philosophical basis of modelling and representing reality (however we might measure or perceive it) gives us a better chance of improving models and modelling in the long run…

US-IALE 2009: GLP Agent-Based Modelling Symposium

The second symposium I spent time in at US-IALE 2009, other than the CHANS workshop, was the Global Land Project Symposium on Agent-Based Modeling of Land Use Effects on Ecosystem Processes and Services. My notes for this symposium aren’t quite as extensive as for the CHANS workshop (and I had leave the discussion part-way through to give another presentation) but below I outline the main questions and issues raised and addressed by the symposium (drawing largely on Gary Polhill’s summary presentation).

The presentations highlighted the broad applicability of agent-based models (ABMs) across many places, landscapes and cultures using a diverse range of methodologies an populations. Locations and subjects of projects ranged from potential impacts of land use planning on the carbon balance in Michigan and rangeland management in the Greater Yellowstone Ecosystem, through impacts of land use change on wildfire regimes in Spain and water quality management in Australia, to conflicts between livestock and reforestation efforts in Thailand and the resilience of pastoral communities to drought Kenya. It was suggested that this diversity is a testament to the flexibility and power of the agent-based modelling approach. Methodologies used and explored by the projects in the symposium included:

  • model coupling
  • laboratory experiments (with humans and computers)
  • approaches to decision-making representation
  • scenario analysis
  • visualisation of model output and function
  • approaches to validation
  • companion modelling

Applied questions that were raised by these projects included:

  • how do we get from interviews to agent-behaviours?
  • how well do our models work? (and how do we assess that?)
  • how sensitive is land use change to planning policies?
  • how (why) do we engage with stakeholders?

In our discussion following the presentation it was interesting to have some social scientists join in the discussion that was dominated by computer scientists and modellers. Most interestingly was the viewpoint of a social scientist (a political scientist I believe) who suggested that one reason social scientists may be skeptical of the power of ABMs is that social science inherently understands that ‘some agents are more important than others’ and that this is not often well reflected (or at least analysed) in recent agent-based modelling.

Possibly the most important question raised in discussion was ‘what are we [as agent-based modellers] taking back to science more generally?’ There were plenty of examples in the projects about issues that have wider scientific applicability; scale issues, the intricacies of (very) large scale simulation with millions of agents, the integration of social and ecological complexity, forest transition theory, edge effects in models, and the presence of provenance (path-dependencies) in model dynamics. Agent-based modellers clearly deal with many interesting problems encountered and investigated in other areas of science, but whether we are doing a good job at communicating our experiences of these issues to the wider scientific community is certainly something open to debate (and was in the symposium).

A related question, recently raised on the SIMSOC listserv (but not in the GLP sumposium) is ‘what are ABMs taking back to policy-making and policy-makers’? Specifically, Scott Moss asked the question; ‘Does anyone know of a correct, real-time, [agent] model-based, policy-impact forecast? His reasoning behind this question is as follows:

“In relation to policy, it is common for social scientists (including but not exclusively economists) to use some a priori reasoning (frequently driven by a theory) to propose specific policies or to evaluate the benefits of alternative policies. In either case, the presumption must be that the benefits or relative benefits of the specified policies can be forecast. I am not aware of any successful tests of this presumption and none of my colleagues at the meeting of UK agent-based modelling experts could point me to a successful test in the sense of a well documented correct forecast of any policy benefit.

The importance of the question: If there is no history or, more weakly, no systematic history of successful forecasts of policy impacts, then is the standard approach to theory-driven policy advice defensible? If so, on what grounds? If not, then is an alternative approach to policy analysis and an alternative role for policy modelling indicated?”

The two most interesting replies were from Alan Penn and Mike Batty. Penn suggested [my links added]:

“… the best description I have heard of ‘policy’ in the sense you are using was by Peter Allen who described it “at best policy is a perturbation on the fitness landscape“. Making predictions of the outcome of any policy intervention therefore requires a detailed understanding of the shape of the mophogenetic landscape. Most often a perturbation will just nudge the system up a wall of the valley it is in, only for it to return back into the same valley and no significant lasting effect will be seen. On occasion a perturbation will nudge the trajectory over a pass into a neighbouring valley and some kind of change will result, but unless you have a proper understanding of the shape of this landscape you wont necessarily be able to say in advance what the new trajectory will be.

What this way of thinking about things implies is that what we need to understand is the shape of the fitness landscape. With that understanding we would be able to say how much of a nudge is needed (say the size of a tax incentive) to get over a pass. We would also know what direction the neighbouring ‘valleys’ might take the system,
and this would allow predictions of the kind you want.”


“I was at the meeting where Scott raised this issue. Alan Wilson said that his company GMAP was built on developing spatial interaction models for predicting short term shifts in retailing activity which routinely produced predictions that were close to the mark. There are no better examples than the large retail units that routinely – every week – run their models to make predictions in retail markets and reportedly they produce good predictions. These are outfits like Tesco, Asda, M[orrisons] and S[ainsbury’s] and so on. I cant give you chapter and verse of where these predictions have been verified and documented because I am an academic and dont have access to this sort of material. The kinds of models that I am referring to are essentially land use transport models which began in the 1960s and are still widely used today. Those people reading this post who arent familiar with these models because they are not agent based models can get a quick view by looking at my early book which is downloadable

I think that the problem with this debate is that it is focussed on academia and academics don’t traditionally revisit their models to see if longer term predictions work out. In fact for the reasons Alan [Penn] says one would probably not expect them to work out as we cant know the future. However there is loads of evidence about how well some models such as the ones I have referred to can fit existing data – ie in terms of their calibration. My book and lots of other work with these models shows that can predict the baseline rather well. In fact too well and the problem has been that although they predict the baseline well, they can often be quite deficient at predicting short term change well and often this arises from their cross sectional static nature and a million other problems that have been raised over the last 30 or more years.”

In response to Batty, Moss wrote:

“It is by no means unusual for model-based forecasts to be sufficiently accurate that the error is less than the value of the variable and perhaps much less. What systematically does not happen (and I know of no counterexample at all) is correct forecasting of volatile episodes such as big shifts in market shares in retail sales, macroeconomic recessions or recoveries, the onset of bear or bull phases in financial markets.

Policy initiatives are usually intended to change something from what has gone on before. Democratic governments — executive and legislative branches — typically investigate the reasons for choosing one policy rather than another or, at least, justify a proposed policy before implementation. Sometimes these justifications are based on forecasts of impacts derived from models. Certainly this is happening now in relation to the current recession. So the question is not whether there are ever correct forecasts. Certainly on the minimal criteria I suggested, there are many. The question is strictly about forecasts of policy impacts which, I conjecture, are rather like other major shifts in social trend and stability.

I believe this particular question is important because I don’t understand the point of policy modelling if we cannot usefully inform policy formation. If the usefulness we claim is that we can evaluate policy impacts and, in point of fact, we systematically (or always) produce incorrect forecasts of the direction and/or timing of intended changes, then it seems hard to argue that this is a useful exercise.”

But is focussing on the accuracy of forecasts of the future the only, or indeed best, way of using models to inform policy? In recent times some policy-makers (e.g. Tony Blair and New Labour) have come to see science (and it’s tools of modelling and predictions) as some kind of a ‘policy saviour’, leading to what is known as evidence-based policy-making. In this framework, science sits upstream of policy-making providing evidence about the real state of the world that then trickles down to steer policy discourse. This may be fine when the science is solving puzzles, but there are many instances (climate change for instance) where science has not solved the problem and rather has merely demonstrated more clearly our ignorance and uncertainty about the material state of the world.

Thus, when (scientific) models are developed to represent ‘open’ systems, as most real world systems are (e.g. the atmosphere, the global economy), I would argue that model forecasts or predictions are not the best way to inform policy formation. I have discussed such a perspective previously. Models and modelling are useful for understanding the world and making decisions, but they do not provide this utility by making accurate predictions about it. I argue that modelling is useful because it forces us to make explicit our implicitly held ‘mental models’ providing others with the opportunity to scrutinise the logic and coherence of that model and discuss its implications. Modelling helps us to think about potential alternative futures, what factors are likely to be most important in determining future events, how these factors and events are (inter)related, and what the current state of the world implies for the likelihood of different future states.

Science, generally, is about finding out about how the material world is. Policy, generally, is about deciding and making how the world how it ought to be. In many instances science can only provide an incomplete picture of how the world is, and even when it is confident about the material state of the world, there is only some much it can provide to an argument about how we think the world should be (which is what policy-making is all about). Emphasising the use of scientific models and modelling as a discussant, not a predictor, may be the best way to inform policy formulation.

In a paper submitted with one of my PhD advisors we discuss this sort of thing with reference to ‘participatory science’. The GLP ABM symposium is planning on publishing a special issue of Land Use Science containing papers from the meeting – in the manuscript I will submit I plan to follow-up in more detail on some of these participatory and ‘model as discussant’ issues with reference to my own agent-based modelling.

Peter Orszag and Economic Models

A while ago I heard this interview with Peter Orszag, Director of the US Office of Management and Budget and one of President Obama’s key economic advisors. Interestingly to me, given what I’ve written previously about quantitative models of human social and economic activity, Orszag is interested in Behavioural Economics and is somewhat skeptical about the power of mathematical models:

“Too many academic fields have tried to apply pure mathematical models to activities that involve human beings. And whenever that happens — whether it’s in economics or health care or medical science — whenever human beings are involved, an approach that is attracted by that purity will lead you astray”

That’s not to say he’s not going to use some form of model forecasting to do his job. When Jon Stewart highlights (in his own amusingly honest way) the wide range of economic model projections out there for the US deficit, Orszag points out that he needs at least some semblance of a platform from which to anchor his management of the US economy. But it’s reassuring for me to know that in managing the future this guy won’t be seduced by quantitative predictions of it.

US-IALE 2009: Overview and Fire

Last week I was at 2009 US-IALE in Snowbird, Utah. It was a great meeting; my presentations went down well, I participated in two stimulating symposia and a statistics workshop, heard interesting presentations that spanned a range of subjects, made new friends, talked about potential collaborations and even found time at the end of the week for a spot of Spring snowboarding. There was so much going on that I’m going to devote two other blog posts to the ‘Complexity in Human-Nature Interactions across Landscapes’ symposium and the ‘Global Land Project Symposium on Agent-Based Modeling of Land Use Effects on Ecosystem Processes and Services’.

The conference plenary, entitled ‘Facilitating the Conduct of Naturally Humane and Humanely Natural Research’, was given by Thomas Baerwald, Senior Science Advisor at the National Science Foundation. In-keeping with his position, Baerwald dealt with several issues related to the execution of coupled human-natural type research, from the scientific or policy questions that need to be addressed to the mechanics of putting together a research team or proposal. Broadly, his comments could be interpreted (respectively) as i) CHANS research needs to provide a better understanding of the processes underlying observed dynamics, and ii) that effective teamwork (including developing a common language between researchers from diverse backgrounds) are required in the interdisciplinary research projects his department funds. Many questions and issues raised in the plenary were later addressed in the Complexity in Human-Nature Interactions symposium.

Two areas of research caught my attention in the Fire and Landscapes session. First was the ongoing work of Don McKenzie and his PostDoc Maureen Kennedy at USFS. Don has been examining the mechanisms behind scaling laws in wildfire regimes such as those I worked on during my Masters with Bruce Malamud. In particular, Don and Maureen are trying to determine whether scaling relationships like the power-law frequency-area wildfire distribution arise from physical mechanisms or are numerical artifacts of the way data are quantified.

In his presentation Don proposed that topographic controls on fire spread are the underlying driver for more proximate mechanisms that govern the observed scaling relationships. Maureen then demonstrated how they used a raster-based neutral model for fire history to generate fire history patterns to examine this. Using the neutral model, Maureen has found the expected value of Sorensen distance (a metric for fire co-occurrence between pairs of trees) depends both on the probability two trees are both in a given fire, and on the probability a tree that is in a fire records that fire with a scar [this is important given much wildfire regime data come from paleorecords of wildfire scars]. In turn, this is related to the topographic complexity of the simulated landscape.

In conclusion, Don suggested that “the search for mechanisms behind scaling laws in landscape ecology may be fruitful only when the scope of observed phenomena is sufficiently local to be in the domain of a contagious process… Power laws and other scaling relationships at broader scales, even if not simply numerical artifacts, are likely to be phenomenological in nature rather than governed by identifiable mechanisms.” Thus, Don is arguing against trying to find mechanisms driving broad-scale patterns in wildfire regimes like those Bruce Malamud, George Perry, and I found for the ecoregions of the conterminous USA. The neutral model approach is certainly appealing and provides a definitive way to test the importance of a variety of variables. We’ve stalled lately on following-up on our PNAS paper, but the work Don and Maureen are tackling definitely provides some food for thought.

The second area of fire research that interested came from a distinctly different background. Francisco Seijo Maceiras discussed the governance of wildfire regimes. Following-up on previous work, Francisco developed the idea that the disruption of ‘Pre-Industrial Anthropogenic Fire Regimes’ (PIAFRs) – and the livelihoods and lifestyles of the social groups that generated them – is an important factor in changes in wildfire regimes in recent decades. Using Spain as an example, Francisco argues that changes in understanding regarding the ecological role of wildfire in landscapes (e.g. see Perry 2002) “provides an excellent opportunity for both re-enfranchising local communities regarding fire use and improving fire management.” I am no expert in the history of Spanish wildfire policy but I can certainly see potential uses of my Landscape Fire Succession Model I to examine potential consequences of a change in wildfire management strategies from top-down, state-organised management towards those favoured by local community fire practitioners.

In another session I happened to drop in on, Virginia Dale gave an interesting presentation on climate change, land-use change, and energy use. What specifically caught my attention was her discussion of the use of the net environmental benefit framework for landscape ecologists to explore the land and water resource effects climate change and different energy options might bring. Papers will be appearing with more on that soon I believe.

On the final day of the meeting I attended the bayesian statistics workshop led by Mevin Hooten from Utah State University. The introduction looked at hierarchical models and the difference between forward models (e.g. forest simulation modelling: set the parameters, run the model, look at the data produced) and inverse model (e.g. linear regression: collect the data, think about how the process works, fit the parameters). Bayesian modelling is inverse modelling that uses conditional probability: first we specify a stochastic model that explains where the data come from (i.e. a likelihood) and a stochastic model for the parameters (i.e. a prior), then we fit the model by finding the posterior distribution of the parameters give the data. That’s a very simplified explanation of the approach and the workshop proceeded to get technical. What re-affirmed my determination to experiment with this approach in the future were the examples Mevin’s graduate students provided: Ephraim Hanks presented his work and a tutorial on the prediction of dwarf mistletoe incidence in Black Spruce stands of Northern Minnesota using Bayesian methods, and Ryan Wilson presented his work and a tutorial that used Bayesian methods to examine uncertainty, and multi-scale clustering in core area (habitat) characterisation of a variety of mammals (hopefully forthcoming in Ecology).

Even without my notes on the comments on the ‘Complexity in Human-Nature Interactions across Landscapes’ symposium and the ‘Global Land Project Symposium on Agent-Based Modeling of Land Use Effects on Ecosystem Processes and Services’ this has turned into a long blog post. There really was a lot on at the US-IALE this year. I hope to post on those symposia very soon.

Environmental Modelling and Software paper In Press

It took a while (first submitted late February 2008) but the manuscript I submitted with colleagues to Environmental Modelling and Software has now been accepted for publication. The paper describes the bio-physical component of the integrated socio-ecological simulation model I developed during my PhD. I don’t envision it changing it much so the abstract is copied below. When it’s in print I’ll holler again…

Modelling Mediterranean Landscape Succession-Disturbance Dynamics: A Landscape Fire-Succession Model
James D.A. Millington, John Wainwright, George L.W. Perry, Raul Romero-Calcerrada and Bruce D. Malamud

We present a spatially explicit Landscape Fire Succession Model (LFSM) developed to represent Mediterranean Basin landscapes and capable of integrating modules and functions that explicitly represent human activity. Plant functional types are used to represent spatial and temporal competition for resources (water and light) in a rule-based modelling framework. Vegetation dynamics are represented using a rule-based community-level modelling approach that considers multiple succession pathways and vegetation ‘climax’ states. Wildfire behaviour is represented using a cellular automata model of fire spread that accounts for land-cover flammability, slope, wind and vegetation moisture. Results show that wildfire spread parameters have the greatest influence on two aspects of the model: land cover change and the wildfire regime. Such sensitivity highlights the importance of accurately parameterising this type of grid-based model for representing landscape-level processes. We use a ‘pattern-oriented modelling’ approach in conjunction with wildfire power-law frequency-area scaling exponent beta to calibrate the model. Parameters describing the role of soil moisture on vegetation dynamics are also found to significantly influence land-cover change. Recent improvements in understanding the role of soil moisture and wildfire fuel loads at the landscape-level will drive advances in Mediterranean LFSMs.

Winter White-Tailed Deer Density Paper

First week back in CSIS after the holiday and I got cracking with the winter white-tailed deer density paper we’re working. Understanding the winter spatial distribution of deer are important for the wider simulation modelling project we’re working on as the model needs to be able to estimate deer densities at each model timestep. We need to do this so that we might represent the impacts of deer on tree regeneration following timber harvest in the simulation model. The work the paper will present is using data from several sources:

  1. data we collected this summer regarding forest stand composition and structure,
  2. similar data kindly shared with us by the Michigan DNR,
  3. estimates of deer density derived from deer pellet counts we also made this year,
  4. other environmental data such as snow depth data from SNODAS.

Here’s my first stab at the opening paragraph (which will no doubt change before publication):

Spatial distributions of wildlife species in forest landscapes are known to be influenced by forest-cover composition and pattern. The influence of forest stand structure on the spatial distribution of wildlife is less well understood. However, understanding the spatial distribution of herbivorous ungulate species that modify vegetation regeneration dynamics is vital for forest managers entrusted with the goal of ensuring both ecological and economic sustainability of their forests. Feedbacks between timber harvest, landscape pattern, stand structure, and herbivore population density may lead to spatial variation in tree regeneration success. In this paper we explore how forest stand structure and landscape pattern, and their interactions with other environmental factors, can be used to predict and understand the winter spatial distribution of white-tailed deer (Odocoileus virginianus) during in the managed forests of the central Upper Peninsula (U.P.) of Michigan, USA.

I’ll update the status of the paper here periodically.