Model averaging and individual covariates

[mmcleod]

In section 11.8, the MARK manual says, "...take the reconstituted values of Phi for each model, for a given value of the covariate, then average them using the AIC weights as weighting factors (after renormalizing the weights to reflect only those models with the individual covariates)." Does this mean you can average only over models containing the covariate? This seems to defeat the purpose of model averaging, if not all of the candidate models contain the covariate as a factor. It seems analagous to, say, if you're model averaging to get real estimates of Phi, and some models have Phi(t) while others have Phi(.), excluding all the Phi(.) models because they don't contain t. Maybe I'm just missing the mathematical boat here. Elucidation appreciated!

Thanks,
Mary Anne

[cooch]

In section 11.8, the MARK manual says, "...take the reconstituted values of Phi for each model, for a given value of the covariate, then average them using the AIC weights as weighting factors (after renormalizing the weights to reflect only those models with the individual covariates)." Does this mean you can average only over models containing the covariate? This seems to defeat the purpose of model averaging, if not all of the candidate models contain the covariate as a factor. It seems analagous to, say, if you're model averaging to get real estimates of Phi, and some models have Phi(t) while others have Phi(.), excluding all the Phi(.) models because they don't contain t. Maybe I'm just missing the mathematical boat here. Elucidation appreciated!

Thanks,
Mary Anne

Generally, if you're interested in models with individual covariates, you're interested in average survival (say) over a given interval for an individual with a particular covariate value (for purposes of making the point, lets assume we're talking about some relationship between survival and body bass - as in the book). So, the glib answer if you're interested in survival for an individual of a given mass, then you model average only over models that contain mass (or functions of mass), since estimates from models without the covariates will assume (essentially) that the beta for mass (and functions thereof) is zero, and thus, provides no information on survival for a given mass.

So, one view is that you model average over those models which contain the covariate of interest. What is unfortunate about the example in the book (which I'll correct) is that the proposed model set has only 2 models, both with the mass covariate. But, suppose you had a model set with multiple models, some with, some without the covariate. What you do is

1. identify the subset of those models which contain the covariate

2. take a weighted average of the reconstituted estimates evaluated at the desired covariate value, using AIC weights re-normalized to include only those models in the original candidate model set which contained the covariate. You have to do that by hand.

3. calculate the SE of the model averaged value - again, by hand, but the basic idea is straightforward: see section 4.5.1 in Chapter 4.

The example in the book was convenient - in that because both models contained the covariate - you didn't need to do anything by hand. The 'automatic' model averaging is what was described in the book. While convenient, this was in hindsight an unfortunate example to present, since it provides no guidance on what to do if the candidate model set contains a mixture of models with and without the covariate.

Your question was a good one - it is worth noting that in one of the places in Burnham & Anderson where this is addressed (indirectly - not specifically speaking to individual covariates) is section 8.8 (in second edition). The basic idea I outline (above) is laid out there. However, they also not that 'Investigation of this idea, and extensions of it, is an open research question'.


I'll add more here once I get a chance to dive back into Chapter 11 and extend the example(s) of this problem. Hopefully this will suffice for now.

One different approach I have thought about is model averaging beta terms for the covariate, on the assumption that beta=0 for models without the covariate, and so you could average beta over all models - not just those with the covariate. You then take your model averaged estimate of beta, and then reconstitute the estimate and SE for some value you choose. I think this would work, and have some advantages over the approach Burnham & Anderson describe (above), but again - this is a somewhat open question. For the moment, the B&A approach has at least their tacit support, even though it is clearly somewhat cumbersome.

[darryl]

But ....

Those models without the covariate of interest (eg mass) still provide you an estimate of survival (say) for an individual of a particular mass, it just happens to be the same survival probability as for individuals with any other mass. I'd argue for model averaging over the entire set of models and I believe it's more akin to the idea of averaging the beta's and setting beta=0 for those models without the covariate of interest.

Darryl

[darryl]

PS, Meant to add that ...

I would expect you'd get a bias if you only focused on models with the covariate of interest, particularly if you're interested in estimating survival for individuals with a covariate value away from the mean value, and support for whether the covariate was even important was ambiguous.

[cooch]

PS, Meant to add that ...

I would expect you'd get a bias if you only focused on models with the covariate of interest, particularly if you're interested in estimating survival for individuals with a covariate value away from the mean value, and support for whether the covariate was even important was ambiguous.

As I recall, B&A acknowledge that bias, and suggest a correction. I'll look into it tomorrow when I can lay my hands on the book (same section/page I mentioned in my earlier response).

[cooch]

But ....

Those models without the covariate of interest (eg mass) still provide you an estimate of survival (say) for an individual of a particular mass, it just happens to be the same survival probability as for individuals with any other mass. I'd argue for model averaging over the entire set of models and I believe it's more akin to the idea of averaging the beta's and setting beta=0 for those models without the covariate of interest.

Darryl

Good point...

[cooch]

As it turns out (perhaps inevitably), the situation is more complicated than we'd probably like. Turns out the sections of B&A (2nd Edition) that seem most relevant to this issue are on pp. pp. 252-255. In particular, the section on p. 252 starting with 'The model-averaged prediction (estime) of E(y|x) s

What B&A describe immediately below that is the approach I originally posted. However, at the top of p. 253, they propose an 'alternative', which I also described, and which Darryl thinks makes more sense (superficially, I'm inclined to agree). On p. 254 is the description of the full model-averaged estimator over all models wherein if predictor x(i) is not in model g(r), we simply set beta(i,r)=0. They comment on this being analogous to a shrinkage estimator. In fact, they seem to conclude in the middle of p. 253 that this second approach is preferred. Will have to consult with B&A.

What may be more of an issue, though, is not the simple calculation of the model-averaged beta, but the sampling variance and associated CI for beta. When a parameter appears in every model, then there is support for the estimator they show at the top of p. 254. The most difficult conclusion they come to on p. 254: "We do not yet know a reliable analytical estimator for the SE of model averaged beta..".

So, there seems no 'consensus' approach to model averaging beta, which makes the second approach I described (and the alternative B&A refer to) somewhat more difficult than it might first appear. I'm sufficiently intrigued by this to go bug B&A (assuming they don't have a spam filter set up to trigger on email from me

[darryl]

Last time I chatted with A of B&A he was saying he thought the approach of setting beta=0 if the covariate is not in that particular model, was the better approach. In A's 2008 primer the model-averaged SE equation is also slightly different to B&A 2002 (and it's not a misprint), but don't know if there's been any resolution on the issue Evan noted above. A also said though that he knew of a few groups that were looking at the issues via simulation.

One other point about model averaging beta's is that you have to be very careful that their interpretation is consistent across the set of models, which may not happen if you have interactions between covariates for example. There can also be issues if you have correlated covariates, as there you probably need to model-average the entire variance-covariance matrix, not just SE's. Depending on the question of interest, I often think it safer to model average the resulting real parameters (ie the predictions) rather than the betas, particularly if you're just model averaging the beta's to build an equation that you're then going to use for prediction of real parameters.

Cheers
Darryl

[bmitchel]

Just to follow up on this thread...

Burnham and Anderson went into this issue in more detail in 2004 (Sociological Methods and Research 33:261-304). They explicitly state that when model averaging a parameter, the parameter should be considered to be zero for models that lack the parameter. The variance formula in this paper is probably the same as in the 2008 book. I followed up with David Anderson about variances, and he suggested using 0 for the variance in the variance formula, when the parameter does not appear in the model. As Darryl and Evan noted, including models that don't have the parameter you are interested in ensures that the parameter estimate and the variance estimate are not biased high.

Darryl's point about making sure you understand what the parameter means in all models is a very important one... ALL models in the set MUST be nested (or potentially nested under some global model that is not necessarily in your set) for averaging parameters to make sense. If the model set is not nested, you can not model average the betas (although you can still average your real parameters).

Brian Mitchell

[jlaake]

Just a note of caution. Usual definition of nesting is not sufficient to ensure that all beta's are conforming. The design matrices also have to be set up in the same manner. For example, a shift in the reference level of a factor variable between models will make betas change definitions.

When I've done this I've found that using the real parameters like Darryl said is the easiest. It is the approach that I used in covariate.predictions in RMark.

--jeff