Model selection strategies

Forum for discussion of general questions related to study design and/or analysis of existing data - software neutral.

Model selection strategies

Postby simone77 » Thu May 30, 2019 11:08 am

A popular model selection strategy in capture-recapture is what Lebreton and colleagues (1992)* defined step-down approach where “the model structure on the survival parameter is fixed at a high dimensionality and a model for capture probabilities is selected” (Doherty, White and Burnham 2012)**.

What do you expect to happen if, instead of fixing the other parameters (i.e. those that are modelled next, survival in the previous example) at a high dimensionality (e.g. that of the global model), it would be fixed at a low dimensionality (e.g. constant, no effect)? Actually there would not be a global model here...

So, for instance, instead of this:
1.{S(time + sex) P(time + sex)}
2.{S(time + sex) P(time)}
3.{S(time + sex) P(sex)}# this the lowest AIC model
4.{S(time + sex) P(.)}
5.{S(time) P(sex)}
6.{S(sex) P(sex)}
7.{S(.) P(sex)}

This:
1.{S(.) P(time + sex)}
2.{S(.) P(time)}
3.{S(.) P(sex)}# this the lowest AIC model (would it be?)
4.{S(.) P(.)}
5.{S(time + sex) P(sex)}
6.{S(time) P(sex)}
7.{S(sex) P(sex)}
8.{S(.) P(sex)}


* Lebreton JD, Nichols JD, Barker RJ, Pradel R, Spendelow JA (1992) Modeling individual animal histories with multistate capturere capture models. Adv Ecol Res 41:88–173
** Doherty, P. F., White, G. C., & Burnham, K. P. (2012). Comparison of model building and selection strategies. Journal of Ornithology, 152(2), 317-323.
Last edited by simone77 on Thu May 30, 2019 4:33 pm, edited 1 time in total.
simone77
 
Posts: 197
Joined: Mon Aug 10, 2009 2:52 pm

Re: Model selection strategies

Postby cooch » Thu May 30, 2019 3:15 pm

Step-up or step-down both suffer from the same problem -- its easy to 'take the wrong branch, simply because of which parameter you fix first. Meaning, you get the wrong answer...this has been demonstrated any number of times using simulation.

Either of these approaches is based on this idea that 'there are so many models that you need to use a step-wise procedure to reduce the model set to something manageable'. I find this a fairly specious argument -- if it takes you a day or so to construct/build/fit your models, so what? Pretty cheap compared to the time/energy and money that went into collecting the data in the first place.

Two additional comments:

1\ trying all possible models is a euphimism for 'not thinking hard enough about the models'. [queue comments about data dredging...]

but, if you must

2\ MARK has some capabilities for build a lot of models, very quickly....see 'subset models and the designm matrix' in Chapter 6.
cooch
 
Posts: 1628
Joined: Thu May 15, 2003 4:11 pm
Location: Cornell University

Re: Model selection strategies

Postby simone77 » Fri May 31, 2019 10:07 am

Thank you for your explanation, it is much in line with the arguments made by Doherty, White and Burnham (2012) (cited in the previous post). Model selection strategy is a topic that should be of much interest to anybody who works with CR analyses (and, more in general, to anyone working with models with hierarchically-related parameters). Doherty and colleagues compared ad-hoc (step-down and step-up) with all-combinations strategies and found that the latter are more recommendable especially for variable selection whereas parameter estimation was not much affected. They also recommended to use model-averaging for parameter estimation.

Their simulations were done with CJS models, with two parameters (phi and p). My opinion is that when you have a model with, say, 6-7 parameters, the all-combinations strategy is very difficult / infeasible to users who cannot run models over several machines in parallel. For instance, I have been running models of a single analysis on 12 CPU (with remote control) for more than a week 24h/day something that would have been really difficult if I had no access to all those machines.

Anyway, this is certainly an interesting topic but I would like to focus on a specific aspect. In the example I made before I showed two kinds of step-down strategies. The first is the “classical” one, the second looks pretty unusual to me. I am wondering if you see some additional pitfall in the second strategy. I have not a clear idea about that. My feeling is that, by doing so, one is giving to p a kind of unwanted priority to explain a part of the variance that in reality is explained by phi (the parameter of biological interest). I would appreciate to hear opinions on this aspect.
simone77
 
Posts: 197
Joined: Mon Aug 10, 2009 2:52 pm


Return to analysis & design questions

Who is online

Users browsing this forum: No registered users and 15 guests