Error: "Cannot allocate vector of length 1262807296"

posts related to the RMark library, which may not be of general interest to users of 'classic' MARK

Error: "Cannot allocate vector of length 1262807296"

Postby jbauder » Mon Jul 31, 2017 5:43 pm

Hello,

I have a data set I would like to fit with a multi-state model in RMark. I have 103 individuals and encounter histories with 189 entries. I ran a model with constant parameters and got the following error message:

Error in diag(nrow(complete.design.matrix)) :
cannot allocate vector of length 1262807296

Does this suggest that my data set is too large for RMark/MARK? Some colleagues analyzed this data set back in 2011 and had to use SURVIVE because MARK could not accommodate such long encounter histories but I was hoping that the newest version of MARK would allow for larger data sets. Are there any other options available for fitting multi-state models to a data set of this size?

Thank you very much for your help,
Javan
jbauder
 
Posts: 56
Joined: Wed May 25, 2011 12:01 pm

Re: Error: "Cannot allocate vector of length 1262807296"

Postby jlaake » Tue Aug 01, 2017 1:56 pm

You didn't say how many strata you have but I'm guessing it is large. If you have k strata then you have k*(k-1)×(nocc-1) entries in DM just for Psi. Also if you have g groups then that is multiplied by g. What you are specifying is too much for the available memory. Are you using R64 and how much memory do you have?
The error is occurring before it tries to run MARK so you still don't know whether it will accept the longer capture histories. You may want to try using MARK interface with your data to see if it accepts the longer capture histories.
jlaake
 
Posts: 1417
Joined: Fri May 12, 2006 12:50 pm
Location: Escondido, CA

Re: Error: "Cannot allocate vector of length 1262807296"

Postby jbauder » Wed Aug 02, 2017 8:57 pm

Hi Jeff,
Thanks for responding. I have two strata but it sounds like even that would make for a very large DM. I tried running the model 64-bit R (had previously been using 32-bit) but got the following error. I will give it a try directly with MARK though.

Error : cannot allocate vector of size 9.4 Gb
In addition: Warning messages:
1: In diag(nrow(complete.design.matrix)) :
Reached total allocation of 8106Mb: see help(memory.size)
2: In diag(nrow(complete.design.matrix)) :
Reached total allocation of 8106Mb: see help(memory.size)
3: In diag(nrow(complete.design.matrix)) :
Reached total allocation of 8106Mb: see help(memory.size)
4: In diag(nrow(complete.design.matrix)) :
Reached total allocation of 8106Mb: see help(memory.size)
Error in mark(mstrata.processed, mstrata.ddl) :
Misspecification of model or internal error in code

Thanks,
Javan
jbauder
 
Posts: 56
Joined: Wed May 25, 2011 12:01 pm

Re: Error: "Cannot allocate vector of length 1262807296"

Postby gwhite » Wed Aug 02, 2017 9:04 pm

I'm confident that MARK can handle an encounter history of 189 occasions. However, before you spend a lot of time developing a design matrix, create the simplest model you can using the PIMs and then run it to be sure that your encounter histories file is valid.
gwhite
 
Posts: 329
Joined: Fri May 16, 2003 9:05 am

Re: Error: "Cannot allocate vector of length 1262807296"

Postby jlaake » Thu Aug 03, 2017 8:48 am

If you only have 2 strata try the logit link for Psi. However, something seems wrong here. A problem this size should not need that much memory.
jlaake
 
Posts: 1417
Joined: Fri May 12, 2006 12:50 pm
Location: Escondido, CA

Re: Error: "Cannot allocate vector of length 1262807296"

Postby jbauder » Fri Aug 04, 2017 2:59 pm

Hi Gary and Jeff,

Thanks for your responses. That's encouraging to hear that MARK should be able to handle my data. I will review my data and design matrix formats to see if I can find out what is causing the error.

Thanks,
Javan
jbauder
 
Posts: 56
Joined: Wed May 25, 2011 12:01 pm

Re: Error: "Cannot allocate vector of length 1262807296"

Postby bmarcek » Tue Jun 08, 2021 12:33 pm

Hi all,
I am new to Mark and RMark and I'm trying to run a multistrata model. I replied to this post because this is exactly the issue that I'm having and I'm curious if this was resolved and if so, how. In my data, I have 642 individuals, 91 occasions, and 11 strata. When I try to run the model, I get 2 errors:

Error : cannot allocate vector of size 1511.9 Gb
Error in mark(eh_2013_2020_2.processed, eh_2013_2020_2.ddl, model = "Multistrata") :
Misspecification of model or internal error in code

I am running 64-bit R and have 32Gb of RAM.

Any insights or suggestions are greatly appreciated.
Thanks,
Ben
bmarcek
 
Posts: 10
Joined: Tue Jun 08, 2021 12:04 pm

Re: Error: "Cannot allocate vector of length 1262807296"

Postby darryl » Tue Jun 08, 2021 5:07 pm

Hi Ben,
You have 32 Gb of RAM, and MARK is trying to allocate of vector of size 1511.9 Gb (which requires memory to do so)....

You need to reduce the dimension of your problem. 11 strata and 91 occasions is going to involve a lot of parameters to estimate, especially with 642 individuals.

Cheers
Darryl
darryl
 
Posts: 495
Joined: Thu Jun 12, 2003 3:04 pm
Location: Dunedin, New Zealand

Re: Error: "Cannot allocate vector of length 1262807296"

Postby bmarcek » Tue Jun 08, 2021 9:06 pm

I figured this would be a long shot but was hoping there might be a way around this issue without reducing my data set. Sounds like that may not be possible.
Thanks
bmarcek
 
Posts: 10
Joined: Tue Jun 08, 2021 12:04 pm

Re: Error: "Cannot allocate vector of length 1262807296"

Postby jlaake » Tue Jun 08, 2021 9:56 pm

Try setting pim.type="time" for Psi in make.design.data. That will reduce the size of the design data for Psi. Also if many of your transitions between the 11 states are not feasible (set to 0) then use fixed=0 in the Psi design data for those transitions that are 0 and use mlogit0=TRUE in the call to mark or mark.wrapper. But Darryl is correct that you are going to have to limit the complexity of the model with n=642 and n=91 occasions.

--jeff
jlaake
 
Posts: 1417
Joined: Fri May 12, 2006 12:50 pm
Location: Escondido, CA

Next

Return to RMark

Who is online

Users browsing this forum: Google [Bot] and 10 guests

cron