Checkpoint Mechanism

[horchard]

Hi there,

From my understanding, secr does not inherently support checkpointing/saving model progress before fitting is complete. I am wondering if there is a possible workaround for this? I am currently trying to fit secr models without spatial covariates. They are not able to finish before the allocated runtime is up (5-10 days). The detector type is proximity, so I am using fastproximity = FALSE to maintain the three occasions. This seems to have increased the runtime a lot. There are likely some other factors increasing runtime, and I'm going to try some other adjustments to speed up the models. However, in the meantime, I'd really like to be able to pickup where the unfitted model left off, if this is at all possible.

Thank you!

[murray.efford]

The direct answer to your question is that there is no canned code for saving model progress (I never thought of that), but you can manually take the link-scale coefficients (beta parameters) of the last likelihood evaluation (output to screen when trace = TRUE, the default) and provide them as a vector of starting values in a new call. Maybe I'll look into a restart option, but that will take weeks.

However, there are few datasets large enough or complex enough that an adequate model takes days to fit. Of course there are many ways to add spurious precision or complexity that inflate the run time (excessive mask size is the obvious one). I would work on that.

Murray

[horchard]

Hi Murray,

Thanks for your prompt reply! I'll try setting those starting values.
My mask has a 30km buffer around the study area, as the study species is woodland caribou with a lot of captures at the edge of the area. We had some individuals moving 20km in a day. I imagine this mask size is slowing things down as you mentioned, but I also don't want it to be too small. Would increasing mask spacing help runtime as well?
Some further information: There are 245 individual samples, with a total of 35 spatial recaptures. 3 sampling occasions. The study area is 21,300 km^2. Flight transects were flown every 3 kms, so detectors are spaced in a 3x3km grid, for a total of 2379 detectors. Many of the detectors did not have any captures.

If you have any other suggestions for how I could speed up runtime, I would really appreciate it!

Here is what my script looks like:

maskarea = st_read(dsn = "/home/hsorchar/scratch/Churchill/", layer = "ChurchillRange_30kmMask")

traps <- read.traps(file = "/home/hsorchar/scratch/Churchill/detectors_final.txt", detector = "proximity", header = TRUE, sep = "\t")

caps <- read.capthist(captfile = "/home/hsorchar/scratch/Churchill/detection_history.txt", trapfile = "/home/hsorchar/scratch/Churchill/detectors_final.txt", fmt = "trapID", detector = "proximity", sep = "\t", header = TRUE)

mask <- make.mask(traps(caps), buffer = 30000, type = "polygon", spacing = 500, poly = maskarea, poly.habitat = T)

model_nocovariates.1 <- secr.fit(capthist='caps', model = list (D~1, sigma~K, g0~1), details = list(fastproximity = FALSE), buffer = 30000, trace = TRUE, method = 'BFGS', detectfn = 'HEX', CL = FALSE, verify = FALSE, mask = mask, ncores = 4)

Thank you again!

[murray.efford]

That is a lot of 'detectors', so I suppose there is a good reason that fitting is slow. I hope some of the tricks in the documentation are helpful. I would work on getting down the size of the mask. The default type in make.mask is 'traprect' when I think you need 'type = trapbuffer'. Individual mask buffers may help (details argument 'maxdistance'). I'm not clear why you override fastproximity. Can you increase the number of cores?

[horchard]

I overrode fastproximity because without it the output only had 1 occasion instead of 3. There were three ~1 week sampling occasions each a month apart, so I didn't want to collapse the data into one occasion. Or does this not end up having a significant effect on the outputs? I had simpler models without overriding fastproximity run in about 9 hours, that are now taking over 3 days. So if the benefit of overriding fastproximity doesn't outweigh the longer runtimes, I won't override it anymore.
I can increase the number of cores, I had tried 8 cores and the memory usage was less efficient, but I will try it again.
I'm going to try all these other changes you mentioned. Thank you again!

[murray.efford]

'fastproximity' collapses multi-occasion binary proximity data (0/1 for each animal x detector x occasion) to binomial data (integer count for each animal x detector). So long as there is no desire to model temporal effects (between occasions) the results are identical and the model fits much faster, as you found.

I suggest timing a single likelihood evaluation (details = list(LLonly = TRUE)) to find fast settings and mask.

[murray.efford]

Further to your original post - I have added a details 'saveprogress' option to the development version. Also a new function 'secr.refit' that continues a previous fit from the saved progress RDS file. Check ?saveprogress and ?secr.refit. You can install the development version with

install.packages("secr", repos = "https://MurrayEfford.r-universe.dev")

[horchard]

Hi again Murray,

Thanks so much for the update and taking the time to add those features!
I have been working on speeding up the model runtime. Based on the large home range size of caribou I decided to keep the buffer size the same, and I did want to model temporal effects so I didn't want to collapse the occasions.
The data is relatively sparse so I tried using 'multi' detectors instead of 'proximity' and the models ran in minutes instead of days. They also produced more realistic outputs than previous attempts with proximity detectors. However, there are 8 samples in the data where an individual was captured at two different detectors within a single occasion. Is this too many to use multi-catch detectors instead of proximity? If so, I will continue with proximity and use the saveprogress and secr.refit options!

Thank you

[murray.efford]

On the one hand, my suggestion to use 'multi' was completely ad hoc and unsupported (someone should do some simulations). On the other, why not? Of course the effect will depend on how important those 8 discarded locations are in relation to the whole, but I guess you lose very little. Discarding the second observation might be problematic if they are in a particular sequence (e.g. sorted by detectors east to west), so I'd keep an eye on that.

I hope someone eventually uses saveprogress & secr.refit. Now in version 5.2.1 on CRAN.

[horchard]

Once I add spatial covariates I'll definitely use saveprogress and secr.refit.
I decided against using multi detectors, I didn't want to lose 8 of the 35 total spatial recaptures. There is still the issue with the proximity detector models producing high density estimates, but I made a new post about this. Thanks again, this has been really helpful!