individual covariates-EM is not avaliable for this model

questions concerning analysis/theory using programs M-SURGE, E-SURGE and U-CARE

individual covariates-EM is not avaliable for this model

Postby Ruijiao_Sun » Wed Jun 17, 2020 10:30 pm

Hi all,
(I first post this problem in the software session and then relize I should post it here.)
My question is:

I am currently running a model to test the individual covariates. And I have two groups, one of the groups has no individual covariates, which is group 1. Another group has individual covariate which is group 2. My GEMACO code is

from.t(1:45).g(1)+from.t(1:45).g(2).[i+xind]

The quantitative individual covariate is personality scores.

Using the solver EM+quasi Newton, an error message occurs when I try to run the model "EM is not available for this model"/(EM is not available for random effect).

Can anyone help me with that? Really appreciate.

Looking forward to the reply.
Ruijiao_Sun
 
Posts: 6
Joined: Wed Jun 17, 2020 7:00 pm

Re: individual covariates-EM is not avaliable for this model

Postby simone77 » Thu Jun 18, 2020 8:53 am

Is it an error message, or is it just a warning? Is the model running? If it's running, it's just a warning telling you that you can run it with the "EM-Quasi Newton" solver, but you can't do it with the "EM" solver. Parallel question: do you have any reason to use the EM-Quasi Newton instead of the (by-default) Quasi Newton solver?

A few suggestions/commentaries (that you may already know, just in case!):
1) remember you have to set the number of parameters manually, if in the same identical model without the "[i+xind]" you had x parameters, then in the model with the xind effect you should have x + n parameters where n is the number of states (except the dead state) in the transition parameter (I suppose the GEMACO sentence applies to a transition parameter, e.g. apparent survival).
2) you need to know how to keep track of the betas because that's all you will get in the model output (you do not get the estimates of the reduced parameters and parameters in the excel file for models with individual effects). This is important because you will need to know what the betas are in order to calculate the biological parameters. You need to pay much attention to the IVFV window to know what corresponds to what.
3) it is a good idea to run this kind of models using the "From last Model" option in "Initial Value". I think it may help the solver to find the right path to the maximum likelihood and it certainly will save time. In you case you should first retrieve the model with the GEMACO sentence (I guess is for a transition parameter, e.g. apparent survival) without the ".[i+xind]" part.
4) if you have 45 intervals (46 sessions), you can omit ".t(1:45)" in GEMACO. It corresponds to say you have no time effect. In that case it would be the same using
from.g(1)+from.g(2).[i+xind]
Also, although in this case it should not make any difference, it is always a good idea to isolate/separate the sentences with the [], like:
[f.g(1)]+[f.g.[i+xind]]
5) You've only got a single combination in "from.to" for that parameter, right? Otherwise, it's not the same thing "from" and "from.to".
simone77
 
Posts: 197
Joined: Mon Aug 10, 2009 2:52 pm

Re: individual covariates-EM is not avaliable for this model

Postby Ruijiao_Sun » Thu Jun 18, 2020 9:24 am

Hi Simone,
Many thanks for your quick reply.
simone77 wrote:Is it an error message, or is it just a warning? Is the model running? If it's running, it's just a warning telling you that you can run it with the "EM-Quasi Newton" solver, but you can't do it with the "EM" solver. Parallel question: do you have any reason to use the EM-Quasi Newton instead of the (by-default) Quasi Newton solver?

So the thing is, I tried different solver. Using EM-Quasi/Newton solver, after the warning message "EM is not available for this model", then the model should be fit using the Quasi/newton solver. But in my case, the command terminal stopped there and nothing popped out. The same thing happened when I only used the default Quasi/Newton solver.

For now, I have no idea what's wrong with the model. The solver works after I removed the effects of individual covariates.

I am wondering is there any requirements for the input values of individual covariates?

simone77 wrote:A few suggestions/commentaries (that you may already know, just in case!):
1) remember you have to set the number of parameters manually, if in the same identical model without the "[i+xind]" you had x parameters, then in the model with the xind effect you should have x + n parameters where n is the number of states (except the dead state) in the transition parameter (I suppose the GEMACO sentence applies to a transition parameter, e.g. apparent survival).
2) you need to know how to keep track of the betas because that's all you will get in the model output (you do not get the estimates of the reduced parameters and parameters in the excel file for models with individual effects). This is important because you will need to know what the betas are in order to calculate the biological parameters. You need to pay much attention to the IVFV window to know what corresponds to what.
3) it is a good idea to run this kind of models using the "From last Model" option in "Initial Value". I think it may help the solver to find the right path to the maximum likelihood and it certainly will save time. In you case you should first retrieve the model with the GEMACO sentence (I guess is for a transition parameter, e.g. apparent survival) without the ".[i+xind]" part.
4) if you have 45 intervals (46 sessions), you can omit ".t(1:45)" in GEMACO. It corresponds to say you have no time effect. In that case it would be the same using
from.g(1)+from.g(2).[i+xind]
Also, although in this case it should not make any difference, it is always a good idea to isolate/separate the sentences with the [], like:
[f.g(1)]+[f.g.[i+xind]]
5) You've only got a single combination in "from.to" for that parameter, right? Otherwise, it's not the same thing "from" and "from.to".

Thanks for all these suggestions. They are super helpful.
Ruijiao_Sun
 
Posts: 6
Joined: Wed Jun 17, 2020 7:00 pm

Re: individual covariates-EM is not avaliable for this model

Postby simone77 » Thu Jun 18, 2020 10:36 am

There are a lot of things that can go wrong. It's hard to help you without knowing more about your study. Perhaps it would help if you could copy your GEPAT here, how are the data formatted? Headed format? inp? What is the range of values of your individual covariate? are they numerical values? what are the GEMACO sentences you are using for the other parameters?

It's not clear to me if you have empty spaces in your dataset for a group (which would create problems).
In general, it is recommended that the values of the individual covariates be standardized (0 mean, unit variance).
simone77
 
Posts: 197
Joined: Mon Aug 10, 2009 2:52 pm

Re: individual covariates-EM is not avaliable for this model

Postby Ruijiao_Sun » Thu Jun 18, 2020 11:05 am

Hi Simone,
Thanks again for your reply.
simone77 wrote:There are a lot of things that can go wrong. It's hard to help you without knowing more about your study. Perhaps it would help if you could copy your GEPAT here, how are the data formatted? Headed format? inp? What is the range of values of your individual covariate? are they numerical values? what are the GEMACO sentences you are using for the other parameters?


In this study, I am trying to estimate some vital rates and pair-bond disruption rates. In the transition part, Step 1 models survival, Step 2 models widowhood, Step 3 models breeding probability and Step 4 models divorce (marked in red color with questions).
In Step 4, I am trying to test whether the divorce rate is associated with individual personality. So I use personality score as individual covariates.
All other steps work well since this is the best model after a step down model selection. The problem happens when I tried to test the individual covariates in step 4 of transition.
I pasted my GEPAT and GEMACO code here and mark the session with questions in red color.
-------- INIT -------------
1
= STEP 1 =
1
to(1,2,3,4:7)
0
1 7
p p p p p p *

--------- TRANSITION --------
4
= STEP 1 =
1
from(1 4, 2 3 5, 6 7).t(1:9,10:19,20:29,30:39,40:45)
0
8 8
s - - - - - - *
- s - - - - - *
- - s - - - - *
- - - s - - - *
- - - - s - - *
- - - - - s - *
- - - - - - s *
- - - - - - - *
= STEP 2 =
1
from(1,2,3, 4 5).t(1:9,10:19,20:29,30:39,40:45)
0
8 8
* - - - - - w -
- * - - - - w -
- - * - - - w -
- - - * - - w -
- - - - * - w -
- - - - - * - -
- - - - - - * -
- - - - - - - *
= STEP 3 =
1
from.t(1:19,20_45)
0
8 12
b - - - - * - - - - - -
- b - - - - * - - - - -
- - b - - - * - - - - -
- - - b - * - - - - - -
- - - - b - * - - - - -
- - - - - - - b - * - -
- - - - - - - - b - * -
- - - - - - - - - - - *
= STEP 4 =
1
from.t(1:45).g(1)+from.t(1:45).g(2).[i+xind]
0
12 8
* d - - - - - -
* d - - - - - -
* d - - - - - -
* d - - - - - -
* d - - - - - -
- - - * - d - -
- - - - * d - -
- * - - - - - -
- - * - - - - -
- - - - - * - -
- - - - - - * -
- - - - - - - *

---------- CAPTURE -----------
1
= STEP 1 =
1
firste.[from(4 5).to(9)+to(1 12 13, 2 3 7,4,5,6,10,11)+from(6,7).to(9)+from(1,2,3).to(8)]+nexte.[from(4 5,6,7).to(9)+from(1,2 3).to(7)+to(2,3,4,6,10,11,12,13)+from(1,2).to(5)+from(1,2,3).to(8)]+nexte.[from(4 5,7).to(9)+from(1,2 3).to(7)+to(2,3,4,6,10,11,12,13)+from(1,2).to(5)+from(1,2,3).to(8)].t(2_46)*x(1)+nexte.[from(4 5,6,7).to(9)+from(1,2 3).to(7)+to(2,3,4,6,10,11,12,13)+from(1,2).to(5)+from(1,2,3).to(8)].t(2_46)*x(2)
0
8 13
* p - - p - p p - - - - -
* - p - p p p p - - - - -
* - - p - - p p - - - - -
* - - - - - - - p - - - -
* - - - - - - - p - - - -
* - - - - - - - p p - p -
* - - - - - - - p - p - p
* - - - - - - - - - - - -

simone77 wrote:It's not clear to me if you have empty spaces in your dataset for a group (which would create problems).
In general, it is recommended that the values of the individual covariates be standardized (0 mean, unit variance).


I am using the head format for data input and I pasted part of my input data here.

Code: Select all
H:  H:  H:   H:  H:  H: S:               $COV:per    COV:personality
0   1   4   6   3   0   1                  2         6.648117
0   1   0   1   0   0   1                  2         4.684725
2   0   0   1   0   0   1                  1                 0
1   1   0   1   0   1   1                  2         4.519807
1   0   1   0   1   0   1                  2         4.519807
0   1   0   1   0   1   1                  1              0


$COV:per is used to define groups. And for individuals in group 1, we don't have personality scores, so I put 0 in COV:personality. I tried to standerdize my personality score, and it did not help. Would be appreciate if you can help me with that.

Looking forward to the reply. Many thanks in advance
Ruijiao_Sun
 
Posts: 6
Joined: Wed Jun 17, 2020 7:00 pm

Re: individual covariates-EM is not avaliable for this model

Postby simone77 » Thu Jun 18, 2020 4:32 pm

I haven't seen it thoroughly yet, but I think I've detected a few problems. It would make it much easier for me to help you if I knew what the 7 live states (in the order in which they appear in the Initial State columns) and the 13 events are. Is that possible?
By the way: do you fix any parameters in the IVFV? The model is "conditioned on 1st encounter" (this is what comes by default), isn't it?
simone77
 
Posts: 197
Joined: Mon Aug 10, 2009 2:52 pm

Re: individual covariates-EM is not avaliable for this model

Postby Ruijiao_Sun » Thu Jun 18, 2020 6:02 pm

Hi Simone,
Many thanks for your reply.
simone77 wrote:I haven't seen it thoroughly yet, but I think I've detected a few problems. It would make it much easier for me to help you if I knew what the 7 live states (in the order in which they appear in the Initial State columns) and the 13 events are. Is that possible?


We modeled males and females seperately. So the initial 7 states are individual pair-bond status, which contains breeders with mate retention (1) and breeders changing partners (2-3), and non-breeders (4-7). The transitions between two different states condition on individual survival, widowhood, breeding, and divorce, which are 4 steps in the transition section. The 13 events are observation events according to capture histories, but it's can be too complex to explain it here because we included uncertainty in the capture record.

I am happy to take any suggestions addressing that. But I am quite sure about the part without the effects of individual covariates.

simone77 wrote:By the way: do you fix any parameters in the IVFV? The model is "conditioned on 1st encounter" (this is what comes by default), isn't it?


Yes, I fixed the last parameter of INT and the first two parameters of EVENT.

Hope this can help!

Many thanks in advance
Ruijiao_Sun
 
Posts: 6
Joined: Wed Jun 17, 2020 7:00 pm

Re: individual covariates-EM is not avaliable for this model

Postby simone77 » Fri Jun 19, 2020 3:22 am

I suppose you fixed to 0 the probability of first capture (INITIAL CAPTURE) in the state non-breeder widowed (I suppose the seventh state is something like that). Actually, with the information you have provided until now, I cannot understand why (and how - to which value) you fixed the first two parameters in CAPTURE. In general, if you have uncertainty in the state assignment (that's why you are using the multievent approach), it is a good idea to split the EVENT parameter type into steps. In general, you start the GEMACO sentence for CAPTURE with something like "firste + nexte.[whatsoever happens in recapture]" and fix to 1 the first parameter meaning that at the first encounter the probability of detection of all the individuals, is 1. You may find details on why that in literature (e.g. Pradel 2005).

Once again, without knowing in detail what is each row and column of your probability matrices, I think the GEMACO syntax you are using is unnecessarily complex (redundant?) and this could be the model difficult to interpret and understand. Just as an example, your first two parameters for CAPTURE, those you are fixing, should be:
1) the first encounter of the non-breeders (state) 4 and (state) 5 that are captured and recorded with a (event) 9 (this is the only event possible for these states apparently)
2) the first encounter of non-breeders (state) 6 and (state) 7 that are the only one that can be captured with the event 12 and 13. The "to(1" does not make sense to me as there is not such a parameter in the CAPTURE matrix. That's the complementary probability in all the rows, which is the probability of a zero (non-detection). In the IVFV, you cannot fix a complementary probability.

Unfortunately, I cannot say why you had such an error when you included the individual covariate. I have the impression that you have some other problem with the analysis that, for some reason, becomes evident (by giving an error) when you include the individual covariate.
If you need more detailed support, please feel free to contact me off-list. Otherwise, I would wait to see if anyone comes up with some other ideas/suggestions.
simone77
 
Posts: 197
Joined: Mon Aug 10, 2009 2:52 pm

Re: individual covariates-EM is not avaliable for this model

Postby Ruijiao_Sun » Fri Jun 19, 2020 10:11 am

Hi Simone,
Thanks a lot for your reply.
simone77 wrote:I suppose you fixed to 0 the probability of first capture (INITIAL CAPTURE) in the state non-breeder widowed (I suppose the seventh state is something like that). Actually, with the information you have provided until now, I cannot understand why (and how - to which value) you fixed the first two parameters in CAPTURE. In general, if you have uncertainty in the state assignment (that's why you are using the multievent approach), it is a good idea to split the EVENT parameter type into steps. In general, you start the GEMACO sentence for CAPTURE with something like "firste + nexte.[whatsoever happens in recapture]" and fix to 1 the first parameter meaning that at the first encounter the probability of detection of all the individuals, is 1. You may find details on why that in literature (e.g. Pradel 2005).


Yes, as you said and according to Pradel 2005, the first parameter of capture event is fixed to 1. I fixed the second to 0 as we have the information of the parameter value.

simone77 wrote:Once again, without knowing in detail what is each row and column of your probability matrices, I think the GEMACO syntax you are using is unnecessarily complex (redundant?) and this could be the model difficult to interpret and understand. Just as an example, your first two parameters for CAPTURE, those you are fixing, should be:
1) the first encounter of the non-breeders (state) 4 and (state) 5 that are captured and recorded with a (event) 9 (this is the only event possible for these states apparently)
2) the first encounter of non-breeders (state) 6 and (state) 7 that are the only one that can be captured with the event 12 and 13. The "to(1" does not make sense to me as there is not such a parameter in the CAPTURE matrix. That's the complementary probability in all the rows, which is the probability of a zero (non-detection). In the IVFV, you cannot fix a complementary probability.


Honestly, there should be three steps in the CAPTURE part, but I combined them all together, which makes that step redundant. I need to check that later. Thanks for bringing that up!

simone77 wrote:Unfortunately, I cannot say why you had such an error when you included the individual covariate. I have the impression that you have some other problem with the analysis that, for some reason, becomes evident (by giving an error) when you include the individual covariate.
If you need more detailed support, please feel free to contact me off-list. Otherwise, I would wait to see if anyone comes up with some other ideas/suggestions.


It might be the case. I will go through it again. Many thanks for your help! I know it's a lot of work to go through all of these steps. I really appreciate that.

Best wishes
Ruijiao
Ruijiao_Sun
 
Posts: 6
Joined: Wed Jun 17, 2020 7:00 pm

Re: individual covariates-EM is not avaliable for this model

Postby simone77 » Fri Jun 19, 2020 11:56 am

Right. Are you sure that your individuals can start their encounter history (by "start" I mean their first non-zero event) in any of the seven states? For example, can an individual be recorded for the first time as a non-breeder? I ask because very often the non-breeder state is a ghost state (you can't record it - it's a zero in the history of encounters).

In that case, your Initial State should be:
p p * - - - -

I 'm wondering if when you put the individual covariate in the model, it crashes because, to put it in some way, the model is trying to calculate things for individuals who can't be there.

If this is the case, you should get the model working with the individual covariate after correcting the Initial State parameterization. However, I would suggest that you do a careful revision of the GEMACO syntax in any case.
Hope it helps!

Simone
simone77
 
Posts: 197
Joined: Mon Aug 10, 2009 2:52 pm


Return to analysis help

Who is online

Users browsing this forum: No registered users and 14 guests