problem when using hcov

questions concerning anlysis/theory using program DENSITY and R package secr. Focus on spatially-explicit analysis.

problem when using hcov

Postby matobler » Wed Sep 02, 2020 2:02 am

I am having some issue with models not working in secr version 4.3.1 (and also 4.2.x) when using hcov="sex" (where sex is a factor with F, M and NA). This is a large multi-survey dataset that was working fine in secr 3.1.5.

Code: Select all
> secr.0<- secr.fit (capthist ,model = list(g0~1,sigma~1,D~1), hcov='sex',  buffer=buffer)
Checking data
Preparing detection design matrices
Preparing density design matrix
Finding initial parameter values...
Initial values  D = 2e-05, g0 = 0.00527, sigma = 6574.44594
Maximizing likelihood...
Eval     Loglik        D       g0    sigma pmix.h2M
   1         NA -10.6871  -5.2411   8.7909   1.5000
   2         NA -10.6871  -5.2411   8.7909   1.5000
   3         NA -10.6871  -5.2411   8.7909   1.5000
  .....
  19         NA -10.6871  -5.2411   8.7918   1.5002
  20         NA -10.6871  -5.2411   8.7909   1.5003
Completed in 69.84 seconds at 22:36:17 01 Sep 2020


If I remove hcov everything works fine:

Code: Select all
> secr.0<- secr.fit (capthist ,model = list(g0~1,sigma~1,D~1),  buffer=buffer)
Checking data
Preparing detection design matrices
Preparing density design matrix
Finding initial parameter values...
Initial values  D = 2e-05, g0 = 0.00503, sigma = 6574.44594
Maximizing likelihood...
Eval     Loglik        D       g0    sigma
   1  -2219.929 -10.7166  -5.2868   8.7909
   2  -2219.929 -10.7166  -5.2868   8.7909
   3  -2219.928 -10.7166  -5.2868   8.7909
   .....
  65  -2086.951  -9.7860  -4.4418   8.4692
  66  -2086.953  -9.7860  -4.4419   8.4700
Completed in 99.19 seconds at 22:39:02 01 Sep 2020


Digging a bit deeper I found that the issue only occurs if at least one of the surveys in capthist only has individuals of one sex. Any suggestions on how to fix this? The sex covariate is already coded as a factor with two levels (F and M).

Here code to reproduce the issue:

Code: Select all
temptrap <- make.grid(nx = 6, ny = 6, spacing = 20, detector ="proximity")
capthist1<-sim.capthist (temptrap, detectpar = list(g0 = 0.2,sigma = 20))
capthist2<-sim.capthist (temptrap, detectpar = list(g0 = 0.2,sigma = 20))
covariates(capthist1)<-data.frame(sex=c("F","M")[rbinom(dim(capthist1)[1],1,0.5)+1])
covariates(capthist2)<-data.frame(sex=rep("M",dim(capthist2)[1]))

capthist<-MS.capthist(capthist1,capthist2)

secr.fit(capthist,hcov="sex")
secr.fit(capthist)


Thanks,

Mathias
matobler
 
Posts: 19
Joined: Fri Nov 03, 2006 9:44 pm
Location: Peru

Re: problem when using hcov

Postby murray.efford » Fri Sep 04, 2020 4:52 pm

Hi Mathias
I've found the bug and will provide a fix soon. Thanks for your report.
Murray
murray.efford
 
Posts: 686
Joined: Mon Sep 29, 2008 7:11 pm
Location: Dunedin, New Zealand

Re: problem when using hcov

Postby murray.efford » Fri Sep 04, 2020 6:19 pm

OK. The bug was fixed by replacing 'factor' with 'as.factor' in internal function getknownclass().

The new windows binary is available at secr_4.3.2.zip, and the source package at secr_4.3.2.tar.gz. It will be a month or more before 4.3.2 appears on CRAN, and this 'pre-release' version will change. There's another issue to sort out (vignettes not compiling to pdf). See secr 4.3.2 NEWS.txt for recent changes.

Murray
murray.efford
 
Posts: 686
Joined: Mon Sep 29, 2008 7:11 pm
Location: Dunedin, New Zealand

Re: problem when using hcov

Postby matobler » Sun Sep 06, 2020 3:50 pm

Hi Murray,

thanks for the quick fix. It works for my case. The only thing to keep in mind with this fix is that users will need to manually define the correct factor levels for each capture history in the multi-session capture history list. If they just provide a string or the default factor levels (see code below) it won't work.

Best,

Mathias

Code: Select all
temptrap <- make.grid(nx = 6, ny = 6, spacing = 20, detector ="proximity")
capthist1<-sim.capthist (temptrap, detectpar = list(g0 = 0.2,sigma = 20))
capthist2<-sim.capthist (temptrap, detectpar = list(g0 = 0.2,sigma = 20))

#still does not work
covariates(capthist1)<-data.frame(sex=c("F","M")[rbinom(dim(capthist1)[1],1,0.5)+1])
covariates(capthist2)<-data.frame(sex=rep("M",dim(capthist2)[1]))

#this works now
covariates(capthist1)<-data.frame(sex=factor(c("F","M")[rbinom(dim(capthist1)[1],1,0.5)+1],levels=c("F","M")))
covariates(capthist2)<-data.frame(sex=factor(rep("M",dim(capthist2)[1]),levels=c("F","M")))

capthist<-MS.capthist(capthist1,capthist2)

secr.fit(capthist,hcov="sex")
matobler
 
Posts: 19
Joined: Fri Nov 03, 2006 9:44 pm
Location: Peru

Re: problem when using hcov

Postby murray.efford » Mon Sep 07, 2020 6:34 am

That's a good point. The function shareFactorLevels() was meant to fix inconsistencies between sessions, and it does so with a couple of minor adjustments in a new version accessible by the links given previously. Following your first example, this now works:
Code: Select all
CH <- MS.capthist(capthist1, capthist2)
CH <- shareFactorLevels(CH)
secr.fit(CH, hcov = 'sex')

I have also amplified the warning message to suggest using shareFactorLevels.
Murray
murray.efford
 
Posts: 686
Joined: Mon Sep 29, 2008 7:11 pm
Location: Dunedin, New Zealand

Re: problem when using hcov

Postby dipodomys » Thu Oct 08, 2020 1:13 am

Hello,

I am having the same issue where all the log likelihood values are NA when I include hcov = "Sex", but using version 4.3.2 and shareFactorlevels() haven't helped.

For context, I am using a dataset with 224 sessions over 12 years. There are some sessions with very few individuals captured (total capture occasions ranges from 1 to 46), which I would think might cause some issues. However, other models I've tried have converged and seemed to work well, including a finite mixture model.

Is there something I'm missing or am I using it incorrectly?

Code: Select all
reducedcapthist = shareFactorLevels(reducedcapthist)
m_sex_hcov =  secr.fit(reducedcapthist,
                   ncores = 4,
                   model = list(g0 ~ 1,
                   sigma ~ 1, D ~ 1),
                   masks_reduced,
                   verify = FALSE,
                   CL = FALSE,
                   details =list(autoini = 220),
                   hcov = "Sex"
                   )


Code: Select all
Preparing detection design matrices
Preparing density design matrix
Finding initial parameter values...
Initial values  D = 7.66332, g0 = 0.10035, sigma = 27.49728
Maximizing likelihood...
Eval     Loglik        D       g0    sigma pmix.h2M
   1         NA   2.0364  -2.1933   3.3141   1.5000
   2         NA   2.0364  -2.1933   3.3141   1.5000
   3         NA   2.0364  -2.1933   3.3141   1.5000
   4         NA   2.0364  -2.1933   3.3141   1.5000
   5         NA   2.0364  -2.1933   3.3141   1.5000
   6         NA   2.0364  -2.1933   3.3141   1.5000
   7         NA   2.0366  -2.1933   3.3141   1.5000
   8         NA   2.0364  -2.1932   3.3141   1.5000
dipodomys
 
Posts: 4
Joined: Sat Oct 03, 2020 2:21 pm

Re: problem when using hcov

Postby murray.efford » Thu Oct 08, 2020 2:26 am

This may have nothing much to do with hcov or factor levels - is there direct evidence of that? Sometimes it's hard to find a set of parameter values that 'work' across all sessions of a multisession dataset (and yours has way more sessions than any I've handled). If other models fitted OK you can provide one in the 'start' argument to avoid the autoini algorithm altogether.
murray.efford
 
Posts: 686
Joined: Mon Sep 29, 2008 7:11 pm
Location: Dunedin, New Zealand

Re: problem when using hcov

Postby dipodomys » Fri Oct 09, 2020 1:14 am

Hi Murray,

I'm not quite sure what you mean by finding parameter values that work across all sessions. Do you think allowing for flexibility in values for pmix would help? For instance, including year or site as a covariate in pmix or something similar?

Thanks,
Ryan
dipodomys
 
Posts: 4
Joined: Sat Oct 03, 2020 2:21 pm

Re: problem when using hcov

Postby murray.efford » Fri Oct 09, 2020 2:41 am

Your very first likelihood evaluation fails. My guess is that the starting values are somehow very highly improbable with respect to the data for at least one session. Hence my suggestion you fish around for better starting values. I doubt messing with the model structure (including covariates or other variation) itself will fix the initial-NA problem.

If all else fails I would attempt some debugging by separately evaluating the likelihood for each of your numerous sessions with the initial parameter values:

for (i in 1:224) {
cat('i = ', i, ' LL = ', secr.fit(reducedcapthist[[i]], start = c(2.0364, -2.1933, 3.3141, 1.5000),
hcov = 'Sex', details = list(LLonly = TRUE)), '\n')
}

Which ones fail?

(I can't remember how hcov handles sessions with no detections - that may be the problem.)
murray.efford
 
Posts: 686
Joined: Mon Sep 29, 2008 7:11 pm
Location: Dunedin, New Zealand

Re: problem when using hcov

Postby dipodomys » Fri Oct 09, 2020 8:43 pm

Hi Murray,

I took at a look at the capthist dataframes for the eight sessions it failed for, and they had only one capture occasion with 1 or 2 females. The sessions where there were any males worked, although the coefficient estimates for pmix.h2M were ridiculously large (36 in one session). I've played around with changing the starting value for pmix.h2M but also had no luck.

I currently have all the empty sessions removed because it was the only way I could include individual covariates without getting errors.

Any thoughts? I'm thinking of just including sex as a group, because the majority of the values are known.
dipodomys
 
Posts: 4
Joined: Sat Oct 03, 2020 2:21 pm

Next

Return to analysis help

Who is online

Users browsing this forum: Google [Bot] and 10 guests