by jlaake » Mon Jun 14, 2021 5:21 pm
I communicated with Ben offline and my suggestion reduced the memory requirements such that he was able to run the jobs on his computer. Keep in mind that the default for each parameter is an all different PIM which means there is a different triangular PIM for each cohort (release for each occasion). For Psi, if there are k occasions there are k*(k+1)/2 design data records for each possible transition. Then if there are s strata then there are s*(s-1) transitions so there are s*(s-1)*k*(k+1)/2 records. For Ben's example k=91 and s=11 so there were 460460 - nearly 1/2 million records. So this can get big fast although that is only 134 Mb for the default design data. That gives me some pause as it should not have required 1511 Gb. So I may have to dig around to see what it is requiring so much memory. Regardless when you set pim.type="time" for Psi, it would only be s*(s-1)*k records which is only 10,010 records - 1/46.