RMark 2.12 for new MARK.exe

[jlaake]

If you are using RMark and want to upgrade to the newest versions of mark.exe then download this newest version of RMark at:

https://github.com/downloads/jlaake/RMark/RMark_2.1.2.zip

You can upgrade RMark and it will work with the prior Compaq version of mark.exe. However the newer mark.exes will not work with RMark 2.1.1. If you are used to monitoring the progress of mark fitting with invisible=FALSE argument, pay particular attention to the changes listed below. Also, if you are using 32 bit R it will automatically use mark32.exe and 64bit R will use mark64.exe. You can override this by renaming the file to mark.exe. I've not tested Evan's suggestion of a symbolic link but that may be another way.

The main benefit of moving to the new mark executables is their multiple cpu capability. See item below to see how to use that with RMark. Note that I did find numerical differences in results with the old compaq version but most were very minor unless the parameter was at a boundary. I did find differences in the results of mark's parameter counting with the newer version. I found the fewest differences between mark64.exe and the compaq version.

Also, there is no need to set MarkPath if your mark executables are in C:/Program Files (x86)/Mark or c:/Program Files/Mark. It will look in both places by default but any setting of MarkPath will override.

I won't be posting this version of RMark to CRAN for awhile as it and the newer mark exes undergo further testing. Let me know if you experience any problems. --jeff

NEW FEATURES

* Added function create.mark.mcmc to read in mcmc binary file from MARK for analysis with coda package. If you download this from the github site, make sure to install coda separately. Also, to use codamenu to analyze the results you'll need to install lattice package.

* Made changes for new gfortran versions of mark which include mark32.exe and mark64.exe. If there is a mark.exe it will use it so you can force the use of 32 or 64 bit by renaming mark32 or mark64 to mark; otherwise if running from 64 bit R it uses mark64.exe and from 32 but R mark32.exe

* For Windows will search both Program Files and Program Files (x86) for mark executables so no need to set MarkPath unless it is not in one of those directories and not in your Path.

* The addition of mlogit0 argument in 2.1.1 was useful to reduce number of real parameters when many mlogit parameters are fixed at 0; however, it was retaining betas that were no longer needed. These have been removed in this version.

* An argument threads was added to mark and make.mark.model to take advantage of the multiple cpu capabilities of the new mark exes. If positive, it specifies the number of cpus to use and if negative the number of cpus to remain idle. The default is -1 which uses all but one cpu.

* One of the side effects of the new mark executables is that setting invisible=FALSE in the Rgui will no longer work to show the iteration counter in a window. Instead regardless of the value of invisible, it creates a file fort.0 which contains the output and you can check on its progress. The fort.0 file is deleted after completing the run. I recommend using Rterm if you are going to use mark.wrapper.parallel because I'm not certain what will happen with parallel runs all using fort.0. If you run it from rterm then using invisible=FALSE will allow you to monitor the progress.

* Default formula for CRDMS was changed to ~session from ~1 for N.

[Eldar]

I recommend using Rterm if you are going to use mark.wrapper.parallel because I'm not certain what will happen with parallel runs all using fort.0. If you run it from rterm then using invisible=FALSE will allow you to monitor the progress.

mark.wrapper.parallel works with approximately the same speed as new mark and it seems that problems with fort0 are not critical. The only issue is that fort0 will not be deleted in the end.
for the I7 with hyperthreading turned off the best time I've got was from the combination of two techniques:

Code: Select all

mark.wrapper.parallel(..., threads=2, parallel=T, cpus=2)

General difference is that Mark is doing parallel estimations for one model while RMark is running different models on different cores - so I would suspect that with a lot of cores (e.g. 12) paralleled estimations for one model would be ineffective and mark.wrapper.parallel will be much more effective.
So for me the main difference between new and old Mark (6.2) is addition mlogit0 option that improves the estimation time for multistrata models a lot!
Thanks to Gary and Jeff!

[tommyg]

Hi,

I'm having some trouble getting the the new version of RMark (2.1.2) to run with the new version of Mark (6.2). I've uninstalled my previous version of MARK and downloaded / installed the new version of Mark from and the new RMark from https://github.com/downloads/jlaake/RMa ... _2.1.2.zip. I have a 64-bit machine (MARK is installed here: C:\Program Files (x86)\Mark), but can't get anything to run on both 32-bit R and 64-bit R - version 2.15.0.

RMark seems to load properly

Code: Select all

> library(RMark)
Loading required package: plotrix
Loading required package: snowfall
Loading required package: snow
Loading required package: coda
Loading required package: lattice
This is RMark 2.1.2

So I assume that it is finding the correct *.exe

When I try and run even the most basic CJS model I get

Code: Select all

> mod.1 <- mark(dat.proc, dat.ddl)
Error in extract.mark.output(out, model, adjust, realvcv, vcvfile) :
  MARK did not run properly.  If error message was not shown, re-run MARK with invisible=FALSE


********Following model failed to run : Phi(~1)p(~1) **********

I know this isn't a lot of specific information, but are there any typical 'smoking guns' I should be checking for here ? Thanks

[jlaake]

I'm running with the specification that you give and it is working fine. Try this

Code: Select all

data(dipper)
debug(run.mark.model)
mark(dipper)


Then go through the code until it defines MarkPath and see what it is defined as and whether that exe exists.

--jeff

[tommyg]

The slashes and quotes look pretty jumbled in the MarkPath that was defined in debug mode

Code: Select all

Browse[2]> print(MarkPath)
[1] "\"c:/Program Files (x86)/Mark//mark32.exe\""

Is that what you expected to see ?

[tommyg]

Jeff helped resolve this problem offline and had nothing to do with any version incompatibility, so by all means go get it !

[jlaake]

Unless there are changes in MARK or you install MARK outside the default location, there shouldn't be any connection problems as in previous versions. For the sake of others, the error here is the most common error and is typically due to problems with the data or convergence of the model. If you get this error, look in the most recent .out file which will be linked to the model run that failed and it will contain the error from MARK. I need to add some code to RMark to extract the error and report it rather than this generic error but there are lots of possibilities to consider. --jeff

[jlaake]

V2.1.2 of RMark should be appearing on CRAN soon. Here is a list of all the changes that it contains. You can also get it from my github site.

* Added function create.mark.mcmc to read in mcmc binary file from MARK for analysis with coda package. If you download this from the github site, make sure to install coda separately. Also, to use codamenu to analyze the results you'll need to install lattice package.
* Made changes for new gfortran versions of mark which include mark32.exe and mark64.exe. If there is a mark.exe it will use it so you can force the use of 32 or 64 bit by renaming mark32 or mark64 to mark; otherwise if running from 64 bit R it uses mark64.exe and from 32 but R mark32.exe
* For Windows will search both Program Files and Program Files (x86) for mark executables so no need to set MarkPath unless it is not in one of those directories and not in your Path.
* The addition of mlogit0 argument in 2.1.1 was useful to reduce number of real parameters when many mlogit parameters are fixed at 0; however, it was retaining betas that were no longer needed. These have been removed in this version.
* An argument threads was added to mark and make.mark.model to take advantage of the multiple cpu capabilities of the new mark exes. If positive, it specifies the number of cpus to use and if negative the number of cpus to remain idle. The default is -1 which uses all but one cpu.
* One of the side effects of the new mark executables is that setting invisible=FALSE in the Rgui will no longer work to show the iteration counter in a window. Instead regardless of the value of invisible, it creates a file fort.0 which contains the output and you can check on its progress. The fort.0 file is deleted after completing the run. I recommend using Rterm if you are going to use mark.wrapper.parallel because I'm not certain what will happen with parallel runs all using fort.0. If you run it from rterm then using invisible=FALSE will allow you to monitor the progress. The previous version of mark.exe using the Compaq compiler will work with this version of RMark but setting invisible=FALSE does nothing.
* Default formula for CRDMS was changed to ~session from ~1 for N.
* Functions find.possible.transitions and transition.pairs were added. These can be useful for examining transitions in MS models. See help for an example.
* Added pim.type="age" to allowable values for pims in make.design.data. Now all pim types can be created: all, constant, time and age
* Added functions splitCH and collapseCH which can be used to split a vector of character strings into a matrix and collapse a matrix into a vector of character strings.

[jlaake]

There has been a delay getting 2.1.2 on CRAN but it will get there eventually. In the meantime, I came up with an idea to speed things up in MARK with individual covariates. MARK will accumulate frequencies for capture histories that are the same if there are no individual covariates. However, it does not appear to do that when there are individual covariates probably under the presumption they are all unique. For some situations with individual covariates there are not a lot of unique values so there is room to accumulate the frequencies. So in v2.1.3 on my github site I've added some code that will automatically accumulate the capture histories before it creates the input file it sends to MARK. It accumulates using the data needed for each specific model that you build, so the savings in execution time will be dependent on the number of unique combinations of individual covariates in the particular model. Often the accumulated histories are the ones where it is released and never resighted so the gain will be highly data dependent but I expect most runs will be quicker.

Bret Collier and I have both run some tests and it appears the accumulation is working but I'd appreciate folks testing the new version and reporting back any problems. With the changes that Gary has made in mark.exe to use multi-threads and the recent changes I've made in RMark with 0 mlogit parameters, Bret's one run went from over 24 hrs to 1.5 hrs. I don't expect this new change to make a similar jump but it should help speed models with individual covariates. The one test I conducted with example.data in RMark where I used example.data$weight=floor(example.data$weight) so it made weights into integer values it cut the time in half.

--jeff