AIC is unacceptable

Forum for discussion of general questions related to study design and/or analysis of existing data - software neutral.

AIC is unacceptable

Postby brid0030 » Tue Oct 13, 2009 12:09 pm

I just got a paper rejected by Proceedings largely because of a lack of what the reviewers referred to as "formal statistical testing." The study was a pretty straightforward experiment, with effects clearly evident in the AIC scores of the different models. This comment from one reviewer more or less sums it all up:

"the method of Model selection according to AIC is interesting, but one still needs to show that the main factors have a significant effect of the dependent variables."

My interpretation of this is: "OK, so you found the best model, but are the factors in that model significant at the p < 0.05 level."

Both reviewers seemed to home in on the fact that there is no p-value associated with analyses. Do any of you have any sage advice for making MARK results more palatable to those who crave p-values? I consulted section 4.6 of the MARK book in preparing my manuscript, and as I reread it, I didn't see a succinct way (i.e. something that fits into a couple of sentences) to relate the notion of "significance" in AIC. I am considering likelihood ratio tests, but then not all of the models are nested. Thanks for any wisdom you can impart.
brid0030
 
Posts: 11
Joined: Sat Jun 27, 2009 11:20 am
Location: University of Oklahoma

Postby jlaake » Tue Oct 13, 2009 1:09 pm

You could add a table of betas, std errors and ratio of beta/se which can be treated as an approximate z-test for null that beta=0, so any value z >2 or z < -2 would be important main effects. I'd stop short of adding p-values and point the reviewer to some of the literature on the subject. Doing something like the above is helpful to understand what parts of your model are most important but be careful to understand that you have to interpret the betas and the ratio in terms of your design matrix. For example, an effect specified as an identity matrix is testing whether each value is different from 0 which does not mean that the effect is important. The same effect specified as a treatment contrast (intercept + effect) does provide an assessment of the importance of the main effect values. For example, if you were to use time as an effect for survival where average survival was 0.9 then with an identity matrix all of the betas for each time will have a large beta/se ratio (if data are sufficient) even if the true model is a constant survival. Typically this will not be a problem if you use AIC to assess best model and then look at the se/beta ratios. B&A discuss an example of where this is relevant. You have 2 models where the second model contains one more parameter and the AIC difference is just about 2 where the first model is "better". If you looked at the beta/se ratio you would see that it would be small because the added parameter was not important. Another place where this can be useful is to assess what part(s) of a main "factor" effect are contributing most. For example, if time was the best model, you may want to know which times are the most important. Just make sure you understand your DM and how the parameters relate to one another.

From where I sit, there is nothing "wrong" with doing the above and it helps you understand the model for your data because it provides a measure of your effect sizes and their certainty and that is what is really important. Providing a measure and confidence interval for an effect size is different than using p-values for model selection, although some will be tempted to treat your conf interval as a hypothesis test.

Also, you'll need to educate your reviewers delicately.

--jeff
jlaake
 
Posts: 1417
Joined: Fri May 12, 2006 12:50 pm
Location: Escondido, CA

Re: AIC is unacceptable

Postby cooch » Tue Oct 13, 2009 1:15 pm

brid0030 wrote:Both reviewers seemed to home in on the fact that there is no p-value associated with analyses. Do any of you have any sage advice for making MARK results more palatable to those who crave p-values? I consulted section 4.6 of the MARK book in preparing my manuscript, and as I reread it, I didn't see a succinct way (i.e. something that fits into a couple of sentences) to relate the notion of "significance" in AIC. I am considering likelihood ratio tests, but then not all of the models are nested. Thanks for any wisdom you can impart.


Whether you get pulled over by the brown-shirts of the P-value army, or the AIC police (I could list journals where if you don't use AIC as the basis for inference you're likely to get rejected), is somewhat of a realization of some annoying stochastic bits of the 'review' process (i.e., it all depends on the journal, the 'ass editor', and his/her selection of reviewers - at least for journals that actually allow the 'ass editor' to select reviewers. I digress...).

More to the point, your question touches at an important, and fairly fluid issue. As noted at the end of section 4.6 (current online version)

"While this approach (summing cumulative AIC weights) seems to have some merit, there is by no means consensus that this is the ‘best approach’ - see for example Murray & Conner (2009: ‘Methods to quantify variable importance: implications for the analysis of noisy ecological data. Ecology 90:348-355). Stay tuned - assessment of ‘relative importance’ of variables in complex models is a ‘work in progress’."


There are are several quick (very quick - I'm already late) points to make:

1. first, I'd be somewhat uncomfortable thinking people were using 'the MARK book' as a canonical reference on anything more than 'how to use MARK'. The issue you're addressing goes far beyond what 'the book' is 'qualified' (or intended) to address. That being said, the 'book' does at least mention in broad terms some of the approaches, ideas, and concepts that relate to multi-model inference, since such inference is very much at the conceptual heart of MARK, and it would make little sense not to address 'big issues' to at least some degree. But, 'the book' is *not* a substitute for the primary literature (which is why we ask people not to cite it as such).

2. model averaging in some fashion yields parsimonious estimates of things of interest - be they parameter values, or estimates of effect size. There is general agreement that in this context, multimodel inference is a superior paradigm. There are high level debates about just how to do this averaging, but in broad principle, people agree (generally) that it is a robust framework.

3. there is far less agreement, however, on how to handle 'factor importance'. The Murray & Connor paper (noted above) addresses this issue, but it is clearly not the final answer to the problem. There isn't even general agreement on how to construct very large candidate model sets to avoid model redundancy over various factors. This is all (as noted) a work in progress.

4. probably the best thing you can do that will not offend the 'purists' on either side (which is clearly of practical interest when you're trying to get something published) is to consider effect size - this is the basic admonition of B&A in the first place. Specifying a priori how big of an effect of some set of factors is *biologically* meaningful is arguably the way you/we should proceed - when the interest is on 'effect', and not simply 'parsimonious estimation of a parameter' (although clearly estimation of effect size is related to the latter).

5. you mention 'experiments'. B&A acknowledge the role of experiments, and a properly conducted experiment is always encouraged. Whether you address the data collected during the experiment using classical approaches (like a LRT - very planned experiments often yield a classical nested structure), or using an AIC approach is not particularly important (or shouldn't be). Using the AIC approach, if you have a carefully constructed set of candidate models with a balance between those that include and exclude the factor(s) of interest, then you can in fact, make inference that can be relatively robust. Or, you could model average effects of those factors using the same candidate model set and work from there.

In short, the use of the word 'significant' by the reviewers is (hopefully) intended to get you to provide a robust summary of the importance of a factor. They may want it in terms of P=values, but you could make the argument that this is inappropriate for many of the reasons discussed in B&A. Read the literature, and go from there.

Hope this helps.
cooch
 
Posts: 1628
Joined: Thu May 15, 2003 4:11 pm
Location: Cornell University

Postby cooch » Tue Oct 13, 2009 1:30 pm

jlaake wrote:From where I sit, there is nothing "wrong" with doing the above and it helps you understand the model for your data because it provides a measure of your effect sizes and their certainty and that is what is really important. Providing a measure and confidence interval for an effect size is different than using p-values for model selection, although some will be tempted to treat your conf interval as a hypothesis test.

Also, you'll need to educate your reviewers delicately.

--jeff


Thanks to Jeff for expanding on the issue I was rushing through. I have at various times thought about expanding some of the verbiage related to this issue in 'the book', but have resisted - in part in light of the first point I raised in my first reply. I may need to re-think - this issue does come up with some frequency (not surprisingly).
cooch
 
Posts: 1628
Joined: Thu May 15, 2003 4:11 pm
Location: Cornell University

Excelent responses -- thanks

Postby brid0030 » Tue Oct 13, 2009 2:00 pm

Thanks Jeff and Evan (and Michael) for you attention. These are very helpful replies. I will definitely look into effect sizes as a means of establishing the importance of my model parameters, and from what I remember of the analysis (it's been a few months) this should be straightforward.

Note that I did not and would not use "the book" as a formal citation to justify my methods. But I did (and will) use it to help decide on my approach and to implement it.
brid0030
 
Posts: 11
Joined: Sat Jun 27, 2009 11:20 am
Location: University of Oklahoma

Re: AIC is unacceptable

Postby bacollier » Tue Oct 13, 2009 2:27 pm

Slightly OT, but inline with this post:


As one of the 'ass editors' for a couple of journals, this is a problem that I frequently come across and I have had to tried to come up with some simplistic rules for what I expect to see when I am reading/interpreting a results section. Obviously a nekked AIC table (e.g., a table with naught but AIC values and associated w_i/delta AICs and K's) when used in a manuscript is fairly useless, but so are nekkid p-values, or any nekkid statistic regardless of the flavor of the test used. Jeff's point that providing a table that includes beta's, SE, etc. is right on the mark (no pun intended) as that is the information that helps with inference and evaluation. I think this topic will be included/addressed in part in an upcoming editorial by the current JWM editor as it was one topic of discussion at the AE meeting last month.

Anderson et al. had a paper in 2001 that addressed in very general terms how data should be presented under various scenarios where they laid out some very general guidelines and I think (but don't have the reference handy) that Tacha et al. had a paper in 1982 in the WSB on how to use statistics in wildlife journals if you need some references outside B&A.

I would think that it might be time that someone/group of folks (that is hopefully following this board) takes the initiative to write up a "How to present results in ecological papers" specific to presentation/interpretation of CMR-based analysis, maybe following the same idea as done in

Thomson et al. 2009: Standardizing terminology and notation for the analysis of demographic processes in marked populations. In Modeling Demographic Processes in Marked Populations. Environ and Ecol. Stat 3.

That would take the onus off of folks wanting to cite "the MARK book" (see Faq No. 7) and perhaps provide at least an initial standard for data presentation, which would help with future 'reviewer education'.
bacollier
 
Posts: 230
Joined: Fri Nov 26, 2004 10:33 am
Location: Louisiana State University

right on the mark?

Postby brid0030 » Thu Oct 15, 2009 11:32 pm

Dear bacollier,

Would you (or someone) have a suggestion for a paper that does indeed have the AIC table right on the mark? I fear that mine as well as most of the ones I encounter might qualify as nekked. I have the paper by Anderson et al ("Suggestions for presenting..."), which is very concise and helpful. But a good example would be quite welcome.
brid0030
 
Posts: 11
Joined: Sat Jun 27, 2009 11:20 am
Location: University of Oklahoma

nuts and bolts

Postby brid0030 » Fri Oct 16, 2009 2:04 pm

Quick follow up question. In the beta table output from my analysis there are no labels for the parameter estimates. The parameters are just "1:," "2:," "3:," etc. I can guess what is what, but I would rather not. Any way to fix this?
brid0030
 
Posts: 11
Joined: Sat Jun 27, 2009 11:20 am
Location: University of Oklahoma

Re: nuts and bolts

Postby cooch » Fri Oct 16, 2009 2:26 pm

brid0030 wrote:Quick follow up question. In the beta table output from my analysis there are no labels for the parameter estimates. The parameters are just "1:," "2:," "3:," etc. I can guess what is what, but I would rather not. Any way to fix this?


Spoke too soon (thats what happens when you try to answer while doing something else). Yes, just label columns in design matrix.
cooch
 
Posts: 1628
Joined: Thu May 15, 2003 4:11 pm
Location: Cornell University

Re: right on the mark?

Postby bacollier » Mon Oct 19, 2009 9:53 am

brid0030 wrote:Dear bacollier,

Would you (or someone) have a suggestion for a paper that does indeed have the AIC table right on the mark? I fear that mine as well as most of the ones I encounter might qualify as nekked. I have the paper by Anderson et al ("Suggestions for presenting..."), which is very concise and helpful. But a good example would be quite welcome.



Honestly, I cannot think of one off the top of my head. I guess that the way I look at it is that the table of AIC-type information are fairly useless from a interpretive standpoint, is the parameter estimates that are associated with the posited variables from the set of models. The really important part is what do the model parameter estimates look like for the various models, what are the model averaged estimates, SE, CI, etc as several others have suggested.

I guess I would argue that AIC tables, that just show models should be regulated to appendices or otherwise not used within the manuscripts as they really don't provide a how lot of info other than a list of models and some model relationship statistics, and I think more focus should be put on the parameter estimates from those tables.

Obviously I have been guilty of using a nekkid AIC table, so keep that in mind. I will look around this week and see if I can find any examples of papers I like (noting that my perspective may differ from others in what is good or not good).

Bret
bacollier
 
Posts: 230
Joined: Fri Nov 26, 2004 10:33 am
Location: Louisiana State University

Next

Return to analysis & design questions

Who is online

Users browsing this forum: No registered users and 11 guests