Valid comparison of AIC of models with different structure?

questions concerning analysis/theory using programs M-SURGE, E-SURGE and U-CARE

Valid comparison of AIC of models with different structure?

Postby Roasty247 » Fri Jul 08, 2022 5:18 am

I have set of capture histories with some information on breeding status (non-breeders/successful breeders/failed breeders) which I can choose to model as separate states when estimating survival, or combine breeders, or adults, into a single event and/or state, for example. I understand this choice will also depend on the exact parameters requiring estimation and the hypothesis that might be tested, but assuming this makes no difference and I could model either way, would comparing AIC of two models with different fundamental structures be valid? (i.e. to see which might be the better approach)

To my understanding, AIC is normally compared between two models that differ in their GEMACO model definition, for example with time variant or time constant survival. At what point does it become an invalid comparison of AIC? If only GEPAT matrices have parameters/transitions specified differently? If GEPAT matrices have different steps? If only the events are coded differently from the data? If underlying states are defined differently?

Given that the AIC derives from the likelihood, which is the probability of the data given the model parameters, are comparisons valid in all variations of parameter definition and model structure except where the raw data changes? And would therefore coding a set of capture histories differently in two models constitute changing the raw data and make AIC not comparable, in the same way as changing the set of capture histories (which is surely not comparable)?

Thanks for any opinions or advice!
Roasty247
 
Posts: 5
Joined: Mon Mar 14, 2022 6:31 am

Re: Valid comparison of AIC of models with different structu

Postby simone77 » Fri Jul 08, 2022 11:53 am

I like your question! According to my understanding, you can perfectly compare AIC in all of the situations you've mentioned. You cannot if the raw data of observations are different. To me, changing the raw data means changing the binary nature of the encounter histories, where 0 is not captured and 1 is captured. Of course, changing the raw data means also changing the set of encounter occasions or individuals.

Consider the following scenario: if breeders have lower apparent survival than non-breeders and the data is messed up so that some breeders are now coded as non-breeders (i.e. different events in multievent speaking), the AIC of such a model should be much worse than the AIC of an identical model with same data and correct events' assignment.

I find it incredibly appealing how easily you can build different models in E-SURGE (by changing the GEPAT or the event assignments of the non-zeros) because it gives you a lot of flexibility while remaining in the frequentist field and leveraging its model selection strength. For this reason, I frequently find myself running even non-multievent models in E-SURGE...
simone77
 
Posts: 188
Joined: Mon Aug 10, 2009 2:52 pm

Re: Valid comparison of AIC of models with different structu

Postby Roasty247 » Tue Jul 12, 2022 12:38 pm

Thanks @simone77, so altered coding of events should also be comparable with AIC, except where underlying presence or absence is changed, or the set of capture histories.

For example, non-breeders, failed, and successful breeders coded as 1, 2, and 3 (e.g. 010102332), could be compared with the same histories coded as non-breeders and breeders 1 and 2 (e.g.010102222), so long as 0s are not substituted in or out of capture histories anywhere.

Seems to make sense to me and along the lines I was thinking :).

Thanks
Roasty247
 
Posts: 5
Joined: Mon Mar 14, 2022 6:31 am

Re: Valid comparison of AIC of models with different structu

Postby simone77 » Wed Jul 13, 2022 10:51 am

Roasty247 wrote:For example, non-breeders, failed, and successful breeders coded as 1, 2, and 3 (e.g. 010102332), could be compared with the same histories coded as non-breeders and breeders 1 and 2 (e.g.010102222), so long as 0s are not substituted in or out of capture histories anywhere.

You certainly can! Indeed, it is part of the fun of using multievent models...
simone77
 
Posts: 188
Joined: Mon Aug 10, 2009 2:52 pm


Return to analysis help

Who is online

Users browsing this forum: No registered users and 2 guests