Problem when selecting "Assess model fit"

announcements (new versions, changes, bugs, installation problems...) related to program PRESENCE

Problem when selecting "Assess model fit"

Postby jonnyc » Tue Mar 01, 2022 7:06 pm

Hello,
I am running a single season model in Presence 2.13.18. I am running a model with no covariates first to test my detection history data (610 survey occasions, 40 sites, 22658 missing obs). When I choose "Assess model fit" the model results show in the results browser, however, when I view the model output (in either notepad, Word, or notepad ++) the results are cut-off. Prior to this I get a Windows error message: "presence.exe has stopped working. A problem caused the program to stop working correctly. Windows will close the program and notify you if a solution is available". I click the "close program" button, but the program doesn't close and I am still presented with the "append to results" button, and the results of the model run are presented.

When I do not choose "Assess model fit" I get the full results output and no error message. If I want to assess model fit (with the global model) without the problem of the program "crashing" and presenting cut-off results, must I reduce the number of missing observations? For example, I could organize my detection history data differently. It is currently set up by calendar day, with a subset of sites surveyed almost daily and the remainder surveyed weekly or less frequently, which is the reason for all the missing obs.

Thanks!
jonnyc
 
Posts: 2
Joined: Fri Feb 25, 2022 5:18 pm
Location: Windsor, Ontario, Canada

Re: Problem when selecting "Assess model fit"

Postby darryl » Tue Mar 01, 2022 7:33 pm

PRESENCE is possibly running out of memory at some stage. With 610 survey occasions there are 2^610 possible detection histories presuming no missing values. PRESENCE will then be trying to create a separate cohort for each unique combination of observed missing combinations, probably leading to another few million, or trillion, possible detection histories. The model fit assessment is then trying to determine how well the 40 detection histories you have compare to the (insert a very very large number here) possible detection histories (if it hasn't already ran out of memory trying to allocate RAM for the storage of the required objects).

In short, it's not going to work well. With 40 sites I doubt you're going to get very much power to assess model fit regardless of how you reformat your data as there will be a lot of possible detection histories that are never observed, unless you can meaningfully reduce the data down into 3-4 survey occasions.

Cheers
Darryl
darryl
 
Posts: 494
Joined: Thu Jun 12, 2003 3:04 pm
Location: Dunedin, New Zealand

Re: Problem when selecting "Assess model fit"

Postby jhines » Tue Mar 01, 2022 8:29 pm

I suggest pooling survey days into weeks or months, where you enter a "1" if there is a detection on any of the 7 days for the week, "0" if no detections for the week. Since you have missing data, perhaps you should also create a covariate which keeps track of how many days you had non-missing data for the week. That would allow you to model detection as a function of the number of days with surveys.

Another thing to consider, with 610 days (= 1 year, 8 months), should you really be using the single-season model? Does the species occupancy state remain constant over that period? You may need to consider the multi-season model, which means you would have to decide on sets of days to call a "season" as well as some interval of time between seasons.
jhines
 
Posts: 596
Joined: Fri May 16, 2003 9:24 am
Location: Laurel, MD, USA

Re: Problem when selecting "Assess model fit"

Postby jonnyc » Tue Mar 08, 2022 11:54 pm

Thanks Darryl and Jim for your answers,

In response to both of your points about reducing the number of surveys, I am hesitant to do this because some of my research questions pertain to the effect of certain survey-specific covariates on detection probability (dp) (e.g., survey team size, cloud cover, etc.). If I were to reduce the actual number of surveys into a smaller number of representative surveys (e.g., by pooling many surveys into a weekly "survey") then I might lose the ability to meaningfully incorporate those survey-specific covariates (or at best, have re-think how to code each weekly survey to properly model heterogeneity in the daily surveys).

That said, I did attempt to re-run my original detection history following a reformat of the data. Instead of each survey period representing a strict calendar day, I changed it so that each survey period simply represented a sequence of site surveys (with all the missing data added to the "back end", so to speak). I do not intend to model survey-specific dp, so I don't think there would be an issue with the surveys at each site not actually lining up closely in time (the first survey at each site, although each considered survey 1, occurred on different dates). This resulted in a reduction from 610 survey occasions to 255, and from 22658 missing obs to 8458. After running the null model (with "assess model fit", c-hat =6.58) I no longer experienced my previous problem with the incomplete model output. I have also tested a couple other models with covariates, but have not yet tried assessing model fit with a model with more than 2 parameters. In any event, it’s still over 250 surveys, and given Darryl's suggestion of aiming for 3-4 survey occasions (!), would it be problematic to pursue the analysis in this way?

Finally, Jim mentioned using a multi-season model. My data was collected over 6 calendar years, which I have considered thus far as 1 “season” based on the assumption that the occupancy status of my target species (a rattlesnake) would not change at the scale of the 2ha study sites over that time. Also, my aim was to invest a target number of surveys at each site over the 6 years to have high confidence in absence of the species at sites where it was not found. If I analyzed my data using a multi-season model, reducing the number of survey occasions to 30-50 in each season (i.e, calendar year), would that increase the power required to assess model fit?

Thanks so much!
jonnyc
 
Posts: 2
Joined: Fri Feb 25, 2022 5:18 pm
Location: Windsor, Ontario, Canada

Re: Problem when selecting "Assess model fit"

Postby jhines » Thu Mar 10, 2022 11:26 am

If you're comfortable with the assumption of closure of occupancy status for that period of time, then the single-season model is the one to use. At the moment, there is no gof test for the multi-season model anyway, mostly due to the problem you have - so many surveys that many detection histories have tiny probabilities and expected values. I'm not sure what you can do, other than pool surveys down to a reasonable number, although with the number of missing data points, it will be a problem.
jhines
 
Posts: 596
Joined: Fri May 16, 2003 9:24 am
Location: Laurel, MD, USA

Re: Problem when selecting "Assess model fit"

Postby cooch » Thu Mar 10, 2022 11:38 am

jhines wrote:... At the moment, there is no gof test for the multi-season model anyway, mostly due to the problem you have - so many surveys that many detection histories have tiny probabilities and expected values. I'm not sure what you can do, other than pool surveys down to a reasonable number, although with the number of missing data points, it will be a problem.


Despite that fact, the 'unmarked' folks make frequent (chronic?) use of an approach based on bootstrapping the MacKenzie-Bailey method (there is a function in the unmarked package called mb.gof.test -- for 'MacKenzie-Bailey GOF test).

Details here: https://www.rdocumentation.org/packages ... b.gof.test

Having said that, to my knowledge, there is no canonical paper which demonstrates that doing so is correct or robust (there are conceptual reasons to think it might be reasonable, but...as Jim points out, there are potential problems with tiny observed and expected cell frequencies).
cooch
 
Posts: 1628
Joined: Thu May 15, 2003 4:11 pm
Location: Cornell University


Return to software problems/news

Who is online

Users browsing this forum: No registered users and 1 guest