GOF (RDsurviv) and c-hat for Mark

[cooch]

Is this partial convergence failure or nothing to worry about? I would not think I'd have such an issue with a fairly simple model, compared to the robust design stuff.

Thanks,

Nate

Why are you working with a known-fate model? If your interest is in GOF for a RD, the CJS test for the open periods (as described) is sufficient. I'm guessing you're up to something else that you haven't explained in your original post.

[Eurycea]

Why are you working with a known-fate model? If your interest is in GOF for a RD, the CJS test for the open periods (as described) is sufficient. I'm guessing you're up to something else that you haven't explained in your original post.

No, I'm using a CJS model (of the reduced RD dataset). Page 5-29 sidebar states that the logistic regression for the estimate of c-hat is done by a known fate model, so the results posted farther down in the text output are supposedly from this known fate model.

Thanks,

Nate

[cooch]

Why are you working with a known-fate model? If your interest is in GOF for a RD, the CJS test for the open periods (as described) is sufficient. I'm guessing you're up to something else that you haven't explained in your original post.

No, I'm using a CJS model (of the reduced RD dataset). Page 5-29 sidebar states that the logistic regression for the estimate of c-hat is done by a known fate model, so the results posted farther down in the text output are supposedly from this known fate model.

Thanks,

Nate

Now I see. Yes, the known fate model is built around a logistic regression, so MARK simply 'borrows' those routines to do the logistic regression. So, if you're not getting convergence, that typically means either something weird with your data, or (more often) that the bounds on the range of c under for which the data are simulated aren't appropriate for your data. Hard to be more specific than that. There are some suggestions in Chapter 5 on how to deal with a few pathologies (and others are noted on this forum), but beyond that. One approach that I use periodically is to make a rough estimate of c-hat for a time-specific CJS model using RELEASE, then use the median c-hat by simulating over a range above/below the estimate from RELEASE (in other words, giving the median c-hat a better 'starting point' than might occur by default).

Something like that.

[Eurycea]

Now I see. Yes, the known fate model is built around a logistic regression, so MARK simply 'borrows' those routines to do the logistic regression. So, if you're not getting convergence, that typically means either something weird with your data, or (more often) that the bounds on the range of c under for which the data are simulated aren't appropriate for your data. Hard to be more specific than that. There are some suggestions in Chapter 5 on how to deal with a few pathologies (and others are noted on this forum), but beyond that. One approach that I use periodically is to make a rough estimate of c-hat for a time-specific CJS model using RELEASE, then use the median c-hat by simulating over a range above/below the estimate from RELEASE (in other words, giving the median c-hat a better 'starting point' than might occur by default).

Something like that.

So I estimated a huge c-hat from release, around 12. You are saying fix the bounds to something like 1 to 20, for the median c-hat analysis? Do I have that right?

I tried this and i get the same type of result result- a sensible c-hat, intercept and slope, but weird "parameter 1" estimate, whatever that happens to be:

Code: Select all


  Logistic Regression Estimate of c

     Estimated c-hat = 2.7784860 with sampling SE = 0.0262903

>snip
INPUT ---    icovariates Truth;
  INPUT ---         10 100    1.00000;
  INPUT ---         11   0    1.00000;
  INPUT ---         10 100    1.90476;
  INPUT ---         11   0    1.90476;
  INPUT ---         10  28    2.80952;
  INPUT ---         11  72    2.80952;
  INPUT ---         10   0    3.71429;
  INPUT ---         11 100    3.71429;
  INPUT ---         10   0    4.61905;
  INPUT ---         11 100    4.61905;
  INPUT ---         10   0    5.52381;
  INPUT ---         11 100    5.52381;
  INPUT ---         10   0    6.42857;
  INPUT ---         11 100    6.42857;
  INPUT ---         10   0    7.33333;
  INPUT ---         11 100    7.33333;
  INPUT ---         10   0    8.23810;
etc.......

LOGIT Link Function Parameters of {c-hat logistic regression}
                                                              95% Confidence Interval
 Parameter                    Beta         Standard Error      Lower           Upper
 -------------------------  --------------  --------------  --------------  --------------
    1:Intercept Truth       84.559014       69.603269       -51.863395      220.98142     
    2:Slope Truth           -30.433485      24.774078       -78.990680      18.123709     
 
                 Real Function Parameters of {c-hat logistic regression}

      Following estimates based on unstandardized individual covariate values:
          Variable   Value         
          ---------  -------------
          TRUTH      1.0000000     
                                                              95% Confidence Interval
 Parameter                  Estimate       Standard Error      Lower           Upper
 -------------------------  --------------  --------------  --------------  --------------
    1:                      1.0000000       0.0000000       1.0000000       1.0000000   

If i did this right, I guess my data is, for lack of a better phrase, 'effed'?

Thanks for all your help thus far Cooch.

Nate

[cooch]

I tried this and i get the same type of result result- a sensible c-hat, intercept and slope, but weird "parameter 1" estimate, whatever that happens to be...

Don't worry about that -- those values have to do with calculations that are relevant if in fact you were doing a known-fate analysis. In this case, for the median c-hat, the slope and intercept estimate of interest are

Code: Select all

LOGIT Link Function Parameters of {c-hat logistic regression}
                                                              95% Confidence Interval
Parameter                    Beta         Standard Error      Lower           Upper
-------------------------  --------------  --------------  --------------  --------------
    1:Intercept Truth       84.559014       69.603269       -51.863395      220.98142     
    2:Slope Truth           -30.433485      24.774078       -78.990680      18.123709     


which are what are used to derive the estimate of c-hat of 2.778, which is marginally acceptable (anything >2 is a bit suspicious, anything >3 is borderline 'big problems').

[Eurycea]

oh FANTASTIC!!!!!!!!

The median c-hat values have no affect on the order of the best models under QAICc compared to AICc. They do juggle around the models with partial convergence problems, but I'm ignoring those since there is nothing I can do about it.

I'm half wanting just to send the study out to publication only to see what kind of feedback I get from referees, what with all the convergence issues and limited model set as a result.

Thanks so much for your help Cooch!

Nathan

[dhewitt]

I haven't been following this thread closely, so I'm not sure what you mean by "partial convergence" problems. There is a great deal of information on this forum about dealing with sparse data problems, boundary estimates, overfitting, etc. In any case, I'd sort this out before sending a manuscript to a journal. As an editor and someone regularly asked to review manuscripts on this topic for fisheries, I dislike the idea of "wasting" reviewer time on partly finished work. Just my opinion, but it's your job to sort that stuff out before submission.

[Eurycea]

I haven't been following this thread closely, so I'm not sure what you mean by "partial convergence" problems. There is a great deal of information on this forum about dealing with sparse data problems, boundary estimates, overfitting, etc. In any case, I'd sort this out before sending a manuscript to a journal. As an editor and someone regularly asked to review manuscripts on this topic for fisheries, I dislike the idea of "wasting" reviewer time on partly finished work. Just my opinion, but it's your job to sort that stuff out before submission.

Partial convergence, meaning that boundary estimates for some parameters of some models.

I have read a great deal of information on this forum about dealing with the issues you mention, and asked people "in the field" their opinion. The consensus I got was that you remove models that have issues with boundary estimates from the model selection process (as they give misleading AIC values), and report what you did (if all else fails, which it has to my knowledge). I could not find very much "official" information on this topic, but perhaps I'm looking in the wrong places?

As far as I can tell, the issue is beyond sorting out at this point. Either this approach will be accepted or not. I certainly don't want to waste anyone's time, but if it is asking whether the study and results are acceptable, despite being far from perfect, I don't think that is a waste of time. Maybe the work is partly finished in some people's view, but I don't know that- and would like to. Does that make sense?

[dhewitt]

So you'd remove a model from consideration because it has one boundary estimate? In many cases that would be nonsensical. What if the parameter was really very close to 0 or 1? We have estimates that boundary because annual survival (in CJS) is very high on a long-lived animal and even thousands of animals tagged and tens to hundreds recaptured in a year cannot pull an estimate off of 1 (SE=0) [at least without a Bayesian prior!]. In these cases I am not convinced that the degree to which the model selection statistics (e.g., AICc) are "misleading" makes it necessary to toss the model (I haven't in papers I've published). How can the log likelihood value be "importantly" (significantly?) affected (for model selection purposes) when essentially the boundary estimate is telling you that there is little information in the likelihood for that parameter to begin with? I suppose one way to assess this would be to look at the model with the boundary estimate and then also a model with the parameter fixed (to 0 or 1, whichever is appropriate) and see what the -2logL difference is between the two. Hadn't thought of this before but it sounds like a good idea (pending lashings from the "big wheels").

You're correct that this topic needs more work, and I'd love to hear the big wheels weigh in, but it's probably better under a new thread on the forum if there's interest.

In the end, if you have exhausted all reasonable avenues for understanding "issues" in your analysis, all you can do is be complete in describing them and their potential effects on your inference and then see whether it flies.

[Eurycea]

Here's an older thread I had on my issues:

viewtopic.php?f=1&t=1484

I'll post a response there.

Cheers,

Nate