Numerical underflow/overflow and profile CI

questions concerning analysis/theory using program MARK

Numerical underflow/overflow and profile CI

Postby Eurycea » Wed Jan 04, 2017 7:58 pm

I'm running into a problem with Link-Barker JS models whereby the default estimation options run fine, but when I attempt profile confidence intervals, I get errors such as "Error number 3 from VA09AD optimization routine" as well as "underflow" errors. Typically I have a couple of phi and f parameters estimated at the boundary, and the data are somewhat sparse, so I'm having to put some constraints on the estimates. That is, I'm not running fully time-dependent models here.

I'm wondering if I can "trust" the standard estimates, and simply report the boundary estimates without confidence limits, or if the Profile likelihood results are alerting me to a deeper problem in the model. I'm using data cloning to explore any problems, but this doesn't work so well when the profile likelihood routine throws errors, because I don't think the program is returning the proper profile estimates.

Any advice would be appreciated. The estimates I am getting are generally consistent with what I expect based on other features of the data set.

This is output from the default estimation options:

Code: Select all
 
    30:f    0.2624716       0.1697179       0.0824103       0.8359561                           
    31:f    0.7578689       0.5119190       0.2278903       2.5203588                           
    32:f    0.1085695E-007  0.7446060E-005  0.9107025E-011  0.1294313E-004                     
    33:f    0.1413096E-011  0.0000000       0.1413096E-011  0.1413096E-011                         


And this is what happens with the profile likelihood when it throws underflow errors (with SA in this case):

Code: Select all
 
    30:f    0.2604790       0.1673543       0.1914167E-011  0.8360978E+020 
    31:f    0.7691676       0.5085021       0.1511681E-019  0.2468910E+021
    32:f    0.1015309E-019  0.2596587E-017  0.1015309E-019  0.2353853E+018 
    33:f    0.1511490E-011  0.0000000       0.1511490E-011  0.2300445E-011



I put a seasonal constraint on f, and typically the latter half of the year we don't see much recruitment, so zeroes make biological sense to me.

Is it ok to report a boundary estimate without a CI? (I've seen it here and there...)
Eurycea
 
Posts: 99
Joined: Thu Feb 25, 2010 11:21 am

Re: Numerical underflow/overflow and profile CI

Postby cooch » Wed Jan 04, 2017 8:34 pm

With sparse data (as you describe it) there are always limits to what you can expect. But, in short order, try the following.

1\ establish as best you can if the boundary estimates are really boundary estimates or not (i.e., are they extrinsically nonidentfiable because of the data). See Appendix F - data cloning.

2\ try running the profile CI using simulated annealing, rather than ythe default optimization (be warned -- it can take some time).

3\ if you don't have the data, nothing wrong with a constrained model. If you find (step 1) that parms really are 0 or (more commonly) 1, simply fix them to same, and move on.
cooch
 
Posts: 1628
Joined: Thu May 15, 2003 4:11 pm
Location: Cornell University

Re: Numerical underflow/overflow and profile CI

Postby Eurycea » Wed Jan 04, 2017 9:47 pm

Thank you for your reply Cooch.

1\ It seems like a catch-22, because, to identify a boundary estimate with data cloning, I need to compare the cloned data CI the the profile likelihood CI. Since the profile likelihood returns errors (while the default routine does not), I'm at a loss.

2\ The simulated annealing (SA) didn't do much for me besides taking longer. I used it on the result above. Strangely, when I use SA, the deviance can change. No so if I do profile likelihood using the default method.

3\ Roger that. So estimates at boundaries should be fixed rather than simply left alone (and counted to adjust K)?
Eurycea
 
Posts: 99
Joined: Thu Feb 25, 2010 11:21 am

Re: Numerical underflow/overflow and profile CI

Postby cooch » Wed Jan 04, 2017 10:19 pm

Eurycea wrote:Thank you for your reply Cooch.

1\ It seems like a catch-22, because, to identify a boundary estimate with data cloning, I need to compare the cloned data CI the the profile likelihood CI. Since the profile likelihood returns errors (while the default routine does not), I'm at a loss.



You're missing the point, somewhat. Look at the example starting on p.2 of the appendix -- parameter 14 (encounter probabilty) is not well estimated. It isn't the original CI that is the issue, but the SE and CI of the analysis of the cloned data.

However, if you don't have enough data to even estimate a profile CI without numerical errors, then you could fix the paraameters (and raise the wrath of a lot of folks who wouldn't agree with you), or, simply accept you don't have the data to do anything more than fit a constrained model. Sorry, but to quote John Tukey:

The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data.
cooch
 
Posts: 1628
Joined: Thu May 15, 2003 4:11 pm
Location: Cornell University

Re: Numerical underflow/overflow and profile CI

Postby Eurycea » Wed Jan 04, 2017 11:06 pm

to quote from that chapter, this is what I was referring to:

...except parameter 14,which happens to have been estimated at its boundary,
i.e. 1.0. We return here to the question we noted before – is this parameter truly estimable as 1.0, or is
it being estimated at 1.0 because (i) it is near the boundary, and (ii) the data, and the link function, are
insufficient to resolve the parameter. To differentiate between the two (i.e., to confirm that this parameter
is truly estimable),we need to compare its profile confidence intervals for the original and cloned data sets.
A parameter at a boundary, e.g., a survival estimate equal to 1, will generally have a zero (or at least
unrealistically small) standard error. Cloning the data does not change this small standard error.

However, if you have computed profile likelihood confidence intervals for this parameter, the profile
likelihood confidence intervals for the cloned data will be considerably shorter (assuming you clone a
100 copies) than the original data. So, data cloning is also useful for verifying that a parameter estimated
at the boundary is also estimable.

....

You can now see that the profile interval for parameter 14 has shortened considerably for the cloned
data, with the lower bound changing from 0.732 to 0.997, indicating that this parameter was actually
being estimated. In other words, parameter 14 was extrinsically non-identifiable. In contrast, the 4
intrinsically confounded parameters (6, 12, 18 and 24) still show a relatively wide profile likelihood
confidence interval for the cloned analysis, only slightly reduced from the original values.


So as I understand it, to determine whether the parameter 14 is non-identifiable because it's actually 1 or because of the data/link function requires calculating the profile CI. If the interval shrinks with cloned data, it's actually estimated, and I can fix it to 1 (or zero in my case) and move on. If it does not shrink then it's an artifact and I must/should (further) constrain the model. Do I understand this correctly?
Eurycea
 
Posts: 99
Joined: Thu Feb 25, 2010 11:21 am

Re: Numerical underflow/overflow and profile CI

Postby cooch » Thu Jan 05, 2017 8:10 pm

Correct.
cooch
 
Posts: 1628
Joined: Thu May 15, 2003 4:11 pm
Location: Cornell University

Fixing extrinsically inestimable parameters

Postby jbauder » Tue Nov 21, 2017 5:58 pm

I know it has been awhile since this thread was active but I wanted to ask a couple follow up questions.

My analysis that raised this question has a p(time+group) model as being the best supported model (i.e., >0.80 of the model weight). In my study, sampling effort declines during the last few occasions such that on the last occasion only four individuals were recaptured (out of a total of 114 individuals marked during the study over 10 occasions). I was surprised to see a p(time) model rise to the top but the output for this model is:
Code: Select all
                                     estimate        se          lcl       ucl fixed                 note
Phi gFMixedPIT c1 a0 t1             0.6017315 0.1028030 3.945809e-01 0.7778993                           
Phi gFMixedTelemetry c1 a0 t1       0.9562854 0.0427469 7.467114e-01 0.9938772                           
p gFMixedPIT c1 a0.252 t1.252       0.1610930 0.0566340 7.788080e-02 0.3039116                           
p gFMixedPIT c1 a1 t2               0.2082473 0.0699519 1.027469e-01 0.3766067                           
p gFMixedPIT c1 a1.252 t2.252       0.0791437 0.0399791 2.849220e-02 0.2011922                           
p gFMixedPIT c1 a2 t3               0.0487796 0.0358970 1.113170e-02 0.1893716                           
p gFMixedPIT c1 a2.252 t3.252       0.0362653 0.0263230 8.525400e-03 0.1413934                           
p gFMixedPIT c1 a3 t4               0.0520243 0.0399202 1.110560e-02 0.2114679                           
p gFMixedPIT c1 a3.252 t4.252       0.0576836 0.0450460 1.191870e-02 0.2370214                           
p gFMixedPIT c1 a4 t5               0.0393392 0.0427157 4.448300e-03 0.2728879                           
p gFMixedPIT c1 a6 t7               0.9408154 0.7759375 2.184801e-11 1.0000000                           
p gFMixedTelemetry c1 a0.252 t1.252 1.0000000 0.0000000 1.000000e+00 1.0000000 Fixed


The parameter
Code: Select all
estimate        se          lcl       ucl fixed
p gFMixedPIT c1 a6 t7               0.9408154 0.7759375 2.184801e-11 1.0000000

is the parameter I was concerned about.
I used data cloning and profile CI and I think this parameter is being estimated. But then I wondered:

1) Does the profile likelihood CI shrinking from 0.12-0.999999 (original model) to 0.999996-0.999999 (cloned data) "count" as sufficient shrinkage to assume that the parameter estimate is being estimated? This seems like too much shrinkage (although I did 500 clones and SE ratio is 20.8 when the ideal is 22).

2) In the above case, it seems I should fix this parameter to 1 and proceed with my analyses, but I can't believe that we had 100% recapture probability on that last occasion...? I would almost believe p=0 more, even though we recaptured individuals. Would it be better to fit a more constrained model instead?

Thanks,
Javan
jbauder
 
Posts: 56
Joined: Wed May 25, 2011 12:01 pm

Re: Numerical underflow/overflow and profile CI

Postby Bryan Hamilton » Sat Nov 25, 2017 8:00 pm

You could fix the parameter at 1 and see how the rest of the analysis plays out. You could also combine the last and second to last sampling occasions (since you only had the 4 recaptures). This might absorb some of the uncertainty in that last occasion. The estimate is 1 but the confidence interval is effectively from 0-1.

Are you going to have any additional recapture intervals? That could help boost the precision of the last interval. The last intervals can be kind of wonky even if they're not technically confounded.
Bryan Hamilton
 
Posts: 111
Joined: Thu Jun 15, 2006 11:36 am
Location: Great Basin National Park

Re: Numerical underflow/overflow and profile CI

Postby jbauder » Thu Nov 30, 2017 12:32 pm

Thanks for chiming in Bryan. I did try combining the last two occasions but that did not change anything. Unfortunately the study has been concluded so I won't have any additional occasions. I did run through the analysis without fixing any parameters and turns out the model with this "Year" effect had low support and the model-averaged reals were virtually identical with and without "Year" models. So maybe I lucked out of having to deal with this issue...? For now!
jbauder
 
Posts: 56
Joined: Wed May 25, 2011 12:01 pm

Re: Numerical underflow/overflow and profile CI

Postby Bryan Hamilton » Sat Dec 02, 2017 7:43 pm

My experience with combining occasions is like yours, it hasn't done much. Worth a try though I suppose.

Hey I say on another of your posts that you had some really low survival values for rattlesnakes. Did you get that resolved? They were lower than some of the values I have for neonate survival in Great Basin rattlesnakes.
Bryan Hamilton
 
Posts: 111
Joined: Thu Jun 15, 2006 11:36 am
Location: Great Basin National Park


Return to analysis help

Who is online

Users browsing this forum: No registered users and 11 guests

cron