Model averaging and summary statistics

posts related to the RMark library, which may not be of general interest to users of 'classic' MARK

Model averaging and summary statistics

Postby Bryan Hamilton » Sun Jan 22, 2017 3:53 pm

Hello,

I'm trying to get model averaged parameter estimates (mean, standard error, and confidence intervals). I've spent some time reading the markbook, trying to understand the delta method, and using the design matrix to code the intercept for mean values. I had been taking the mean of the parameter estimate and standard error from the model average table. In plain sight appendix 3 of the MarkBook, I just found that:

Code: Select all
summary(rattle.mod.avg$estimate)


of the model results table (which includes the vcv matrix), provides everything I needed, including confidence intervals. The mean and standard errors from the summary, are very similar to the values I calculated "by hand".

My question is, what if I want mean values for groups or times? Can I just take the mean of the parameter estimate and standard error by groups? What if I want to get the confidence interval? Just taking the mean of the estimate and the standard error seems too easy but perhaps it really is that easy....

Thank you.
Bryan Hamilton
 
Posts: 111
Joined: Thu Jun 15, 2006 11:36 am
Location: Great Basin National Park

Re: Model averaging and summary statistics

Postby jlaake » Mon Jan 23, 2017 2:47 pm

Not sure I can respond sufficiently to your questions because I'm not exactly certain what you are asking, In particular, I don't think summary(....$estimate) is going to give you want you want. Parameter estimates almost always have non-zero covariances, so you can just use summary to get the variance of a mean of a set of parameter estimates. Re-read the section on delta method and this will be clear. You can get the variance-covariance matrix of the model averaged estimates.

--jeff
jlaake
 
Posts: 1417
Joined: Fri May 12, 2006 12:50 pm
Location: Escondido, CA

Re: Model averaging and summary statistics

Postby Bryan Hamilton » Mon Jan 23, 2017 7:41 pm

Sorry I wasn't very clear. I would like to get model averaged values for Phi.

Simply taking the mean of Phi and its se from a model averaged dataset (this is the swift data), gives the same value as the summary of the model average table.

Code: Select all
mean(aa.model.avg.Phi$estimates$estimate)#0.6671
mean(aa.model.avg.Phi$estimates$se)#1.33414
summary(aa.model.avg.Phi$estimate)
#Estimate: Mean   :0.6671
#se:  Mean   :1.33414


If this is true, then:

Code: Select all
tapply(aa.model.avg.Phi$estimates$estimate, aa.model.avg.Phi$estimates$group,mean)
     Good      Poor
0.7618894 0.5723049
tapply(aa.model.avg.Phi$estimates$se, aa.model.avg.Phi$estimates$group,mean)
    Good     Poor
1.320591 1.347678


Should give me model averaged, means and se of phi for the groups. But I don't think this is correct.

I'll read Appendix B on the delta method again. It might sink in this time.

Edit: Went through the Delta Method appendix again. Its really hard for me to understand. I'll revise my question.

Is the delta method necessary, when using real parameters? Can I take means of parameter estimates and their standard errors if I am not involving the betas? My impression is yes, if you perform a function or transformation, then to get valid variance estimates, the delta method is necessary.
Bryan Hamilton
 
Posts: 111
Joined: Thu Jun 15, 2006 11:36 am
Location: Great Basin National Park

Re: Model averaging and summary statistics

Postby jlaake » Mon Jan 23, 2017 9:22 pm

Sure you'll get the same thing with the mean function and the mean from summary but you are missing the point. The mean estimate is fine but you cannot compute the se of that mean as the mean of the se values. The se of a mean is the sqrt of the variance of the mean. The variance of the mean is the sum of all the elements of the v-c matrix which is the same as the sum of the variances and 2x covariances because they are in upper and lower portion of the v-c matrix and then you divide by the number of means squared. That is by definition. In your beginning stats class you may learned this for a mean of independent and identically distributed measurements. In that case, it is sum of sigma^2 over n statistics divided by n^2 which is the same as n*sigma^2/n^2=sigma^2/n. There were no covariances because they were assumed to be independent measurements. When you estimate a set of parameters in a model, they will rarely have 0 covariance values.
jlaake
 
Posts: 1417
Joined: Fri May 12, 2006 12:50 pm
Location: Escondido, CA

Re: Model averaging and summary statistics

Postby cooch » Mon Jan 23, 2017 10:50 pm

Jeff is entirely correct (no surprise there). You can't simply take the average of the SE's and use that. The basic details for calculating 'mean estimates' over some sampling frame is disscussed in Chapter 6. Short of calculating the SE's by hand using the Delta method (doable, but a fair bit of work), the more direct (and perhaps easier) approach is to use either the 'random effects' approach (based on the method of moments -- as described in short form in Chapter 6, and in *much* detail in Appendix D), or, using MCMC. In some respects, using MCMC is the more flexible approach, because you can set up hyperdistributions for whatever 'groups' you want (this is not so tractable using the variance components approach discussed in Appendix D). So, for the swift data, for exaple, you could calculate \mu and process variation \sigma^2 for each of the two colonies. MCMC is discussing *much* detail in Appendix 5. I suppose I should add a 'short form' demo of same to the appropriate section of Chapter 6.

The only real issue is, this approach calculate a mean over time (say), for a given model. It is not the 'average mean' over all models, if you follow my point. So, using either random effects (Appendix D), or MCMC (Appendix E), it is relatively straightforward to derive a mean estimate of some parameter -- and the correct SE of same -- over some time interval. But, such a mean could be derived for any of your candidate models. I suspect all you would need to do then is a 'by hand' model averaging of those means -- weighting by the AIC weights, and using the appropriate formula for constructing the SE for that average (said formula first discussed in detail in chapter 4).
cooch
 
Posts: 1628
Joined: Thu May 15, 2003 4:11 pm
Location: Cornell University

Re: Model averaging and summary statistics

Postby cooch » Mon Jan 23, 2017 10:56 pm

I change my mind, slightly. With some further thought, I wouldn't suggest trying to calculate the mean over time, and then model averaging over those mean estimates, using all your candidate models. This is because the approach to estimating the mean (based on either a VC demoposition -- appendix D -- or MCMC -- appendix E) really only works will for the fully time-dependent model (although I could hold my nose and work with a reduced parameter model, so long as the parameter I was interested in was time-dependent).
cooch
 
Posts: 1628
Joined: Thu May 15, 2003 4:11 pm
Location: Cornell University

Re: Model averaging and summary statistics

Postby jlaake » Tue Jan 24, 2017 1:22 pm

What about model averaging the parameters and then using variance components? Presumably you only want to average over time because the pre-dominant model has time dependence. You can use var.components to do what is done in MARK or use var.components.reml to use a REML approach.
jlaake
 
Posts: 1417
Joined: Fri May 12, 2006 12:50 pm
Location: Escondido, CA

Re: Model averaging and summary statistics

Postby cooch » Tue Jan 24, 2017 1:36 pm

jlaake wrote:What about model averaging the parameters and then using variance components? Presumably you only want to average over time because the pre-dominant model has time dependence. You can use var.components to do what is done in MARK or use var.components.reml to use a REML approach.


Simple -- because (i) you don't (or, shouldn't) model average beta's (being, the parameters), except under particular circumstances, and (ii) most folks are less interested in an interval-specific estimate of model averaged reals than they are in answering the question 'what is the average?', which i take to mean, if I were to pick a year (interval) at random, what is my best prediction of the parameter? Answer - the mean.

Think about people building a (very simple) projection matrix model. They want (typically) a sinlge value for the mean and the variance for some parameter in the projection model, which they use to specify a distribution from which they will sample stochasticially with each iteration of the projection model. So, what value should they use? One asnwer is possibly 'the mean'. Leading to -- how do you get the mean? Leading to - various approaches as discussed.

Something like that.

One unavoidably issue, though, is that unless you have a decent run of data (Burnham recommends at least 10 intervals worth), you might be wasting your time.
cooch
 
Posts: 1628
Joined: Thu May 15, 2003 4:11 pm
Location: Cornell University

Re: Model averaging and summary statistics

Postby jCeradini » Tue Jan 24, 2017 4:44 pm

Cooch said:
You can't simply take the average of the SE's and use that. The basic details for calculating 'mean estimates' over some sampling frame is disscussed in Chapter 6. Short of calculating the SE's by hand using the Delta method (doable, but a fair bit of work), the more direct (and perhaps easier) approach is to use either the 'random effects' approach (based on the method of moments -- as described in short form in Chapter 6, and in *much* detail in Appendix D), or, using MCMC.


Somewhat on topic, although I'm not thinking about it in the context of model averaging: could you accomplish the same thing - getting an overall average estimate with error across groups - via bootstrapping? So, for each iteration of the bootstrap 1) fit a model that includes the group and estimate reals for each group level and 2) average the means of the reals across the group levels. Then use the vector of averaged means to estimate CI's.

Joe
jCeradini
 
Posts: 72
Joined: Mon Oct 13, 2014 3:53 pm

Re: Model averaging and summary statistics

Postby Bryan Hamilton » Tue Jan 24, 2017 9:06 pm

Thank you all for your help. After reading the section in Chapter 6 on mean values, I think I have a path forward with the variance components. I also have a sore brain but that's nothing unusual. I've already used variance components to get shrinkage estimates for time periods so I just need to pull the mean and se from that.

Using the variance components is limited to a single model though right? I won't be able to use average values from model averaging?

Thank you again.
Bryan Hamilton
 
Posts: 111
Joined: Thu Jun 15, 2006 11:36 am
Location: Great Basin National Park


Return to RMark

Who is online

Users browsing this forum: No registered users and 12 guests

cron