We learned today from an NIH announcement about a new plan by the federal agency to relocate and transfer all of the NIH-owned chimpanzees to the federally-funded sanctuary, Chimp Haven, by 2021 or later. The announcement was quickly the subject of announcements and proclamations of victory by PETA, HSUS, and some associated with Chimp Haven.
For others, in light of the concerns raised about the death of 9 of 13 chimpanzees transferred to Chimp Haven recently and subsequent calls for a thoughtful examination of these cases—at least a review of what might be done to minimize future risk—the announcement was troubling.
It does appear that NIH either shared, or was at least responsive to the need to address, the concerns that were expressed about the consequences of relocation on the chimpanzees’ health and welfare. That is evidenced by the fact that NIH did undertake an analysis of mortality rates at Chimp Haven and the research centers that house NIH chimpanzees. That is as it should be – scientists use data to inform decisions. No problem there. NIH conducted the analysis on the basis of data requested from each of the centers. It also appears that they referenced the findings of the analysis in their decision. So what’s the problem?
It appears that the only evidence of the mortality analysis is a non- reviewed paper that was posted just yesterday to a website (Biorxiv) by the study author, NIH’s Dr. Michael Lauer. 
That paper may be viewed here. After even a cursory review and analysis of the Lauer paper, many questions are raised about both the methods and the conclusions drawn from the results. Just a few of the issues or potential problems that an academic reviewer might raise are listed below. Others may read the paper and have different impressions or questions.The data includes 764 chimpanzees; 314 died during the 7 year median follow-up. The author states that: “The analyses were conducted to inform NIH’s plans to retire its surviving chimpanzees.”
To see NIH use data from an unpublished, non-peer reviewed manuscript as a basis for their decisions is incredibly disheartening. It defies the very premise and basis on which the scientific process works. Science doesn’t accept as fact those data and findings that are presented on the internet and that have not been properly vetted through the peer review process. 
But before turning to questions about the paper, let’s be clear on a critical point: The questions and critiques raised here would be raised regardless of the conclusions of the paper and regardless of the direction of NIH’s decision. The questions raised here are at the heart of how science is used to inform decisions and judgments.
In other words, what would we conclude if NIH had used a non-reviewed paper to suggest that relocation was a threat to chimpanzee well-being and that the chimpanzees should be retired in place? The same criticism would hold. The issue is about the conduct of science and how it should be shared and viewed in decision-making. In this case, it is particularly important because of the close relationship between the findings and decisions that have immediate and real impact. Furthermore, in a time where scientific rigor and reproducibility are the subject of a great deal of concern and discussion, it is even more troubling to see that the results of an unreviewed paper posted only yesterday in public view are the basis for an announcement made today.
That means that there was no opportunity for a broader public consideration, for thoughtful analysis, for viewing the data, for asking questions about the approach, methods, analysis, interpretation of results, and conclusions. Thus, we post here some initial questions and comments about the unpublished and unreviewed paper from several scientific reviewers. We hope others read the paper (here) and offer their comments, or offer additional insight into the approach, analysis, and conclusions.
To be clear, these comments are not designed to advocate for or against the transfer of chimpanzees to Chimp Haven. Nor are they designed to judge the quality of care the animals receive after arriving there. Rather, they are designed to illustrate the fact that decisions about the welfare of captive chimpanzees that are being made by NIH appear to be based on data and analyses that are arguably flawed, at least as presented in the current draft of the Lauer paper. Dr. Lauer might have excellent responses and answers to these critiques, which may then validate the claims in the current paper.
And, that is the point: the data will then have been subjected to critical peer review, the bedrock of the scientific method. It is disappointing, and frankly, stunning, that NIH appears to have accepted these results without proper peer review. Making captive chimpanzee retirement and movement decisions based on these findings seems premature and foolish. Sadly, that may lead to unnecessary deaths of chimpanzees. NIH is clearly committed to sending their chimpanzees to Chimp Haven; if that is the mandate, then why try to justify the decision based on methods and analyses that have not been subjected to the normal scientific peer review process? That ultimately raises more questions than answers and stands to further confuse the public view of how science works and how claims should be evaluated.
Finally, we would also note that the data does not appear to be publicly available. In other words, while the un-reviewed article is in public view and its conclusions appear to have informed the decisions the data is not in view and cannot be evaluated or analyzed by scientists or others who are independent of the decisions and the centers involved.
Below are just a few issues, or potential problems, that any reviewer might point out.
- The first part of the study was aimed at addressing mortality rates in chimpanzees housed at Chimp Haven compared to other facilities (Bastrop, Southwest Foundation and Alamogordo Primate facility). The author reported that age and sex had strong effect on mortality rates whereas location had only moderate effects. In point of fact, the influence of location was not a trivial effect based on the results presented in Table 2 but rather a statistically significant one. The author seems to want to minimize the significance of the location effect because the overall p-value (p=.0173) was close to the significance level adopted in their analysis. The effect for Chimp Haven was far below that, at p=.005. The problem, however, is that the argument for adjusting alpha as reported by the author was because they had 6 predictor variables, they therefore they increased alpha to control for possible Type I error. There are a number of issues here. First, it is not clear how the authors dummy coded the location variable. Second, even if there were 6 predictor variables, there were also more than 700 subjects in the study and thus whether the author had adequate power to guard against Type I error (and thus needed to adjust alpha below the traditional < .05) is not entirely clear without presentation of effect size or further rationale. In turn, to state that sex and age had strong effects and location had a moderate effect on mortality is simply not supported by any statistics other than the p-levels.
Maynard at MD Anderson. The paper reports: “The strongest predictor, by far, of mortality was
age (as calculated to be on January 1, 2005), followed by male sex and location. Older age predicted higher mortality (adjusted hazard ratio comparing animals 30 years versus 17 years 2.23, 95% CI 1.91 to 2.61); males also had higher mortality (adjusted hazard ratio 1.50, 95% CI 1.20 to 1.88). Location was only marginally associated with mortality (Wald c2=10, df=3, P=0.017). Compared to Chimp Haven, mortality was lower at APF (adjusted hazard ratio 0.65, 95% CI 0.48 to 0.88), while it was similar at Bastrop (adjusted hazard ratio 0.84, 95% CI 0.60 to 1.16) and almost identical at SNPC (adjusted hazard ratio 1.00, 95% CI 0.71 to 1.39).”
- The second part of the paper was designed to examine the influence of relocation/transfer on mortality rates in the different chimpanzee populations. This aspect of the study is likely in response to recent reports of higher-than-normal rates of mortality in chimpanzees transferred to Chimp Haven but sadly neither the design nor data analysis allow for any meaningful conclusions to be drawn. Specifically, there is no control or comparison group. According to the author, at Chimp Haven (CH), chimpanzees die as they get older and this isn’t due to factors such as when they arrived at Chimp Haven, the season of year, etc….but these analyses are irrelevant. What one would want to know is what the mortality rate is for chimpanzees that get transferred to CH compared to either: 1) chimpanzees that stay at their original facility and don’t transfer; or 2) mortality in chimpanzees that transfer INTO CH from a lab compared to mortality rates of chimpanzees that are transferred FROM CH to another facility or 3) mortality rates of chimpanzees that have been transferred INTO another sanctuary (e.g., Save the Chimps). The second situation does not occur; however, the 3rd situation could be analyzed. Furthermore, there is a 4) mortality rates of chimpanzees that transferred FROM other facilities and INTO Bastrop. None of these comparisons were made in the paper though they are necessary to make inferences about the effect of transfers on the quality of care and mortality. Thus, this entire part of the paper addressing the effects of transfer on mortality is fundamentally flawed. Of course, it is also recognized that in addition to the analyses, appropriate balancing of covariates that relate to the mortality for each of these four comparisons may be difficult, post hoc; however, the alternatives and limitations should be a feature of a carefully considered conclusion and discussion.
- For the Chimp Haven sample, why were non NIH-owned chimpanzees excluded from the mortality analyses? Chimp Haven has taken chimpanzees from other facilities such as Ohio State, Yerkes and New Iberia. If the question is not about mortality rates at a given facility, but rather the effect of transferring individuals from established housing conditions, why exclude any individuals? Further, were the non NIH-owned chimpanzees included in the sample size census within Chimp Haven? In other words, in Table 1, it indicates that the Chimp Haven had 273 chimpanzees. Is that all the chimpanzees at CH, or only those that are NIH-owned? A reviewer might guess is that these numbers are based on the entire sample of chimpanzees at Chimp Haven but it is far less clear on the mortality numbers (107). Moreover, peer review would surely point out that the methods are not sufficient for reproducibility.
- In Table 2, the most relevant comparison (at least in relation to the current issue, the transfer of NIH chimpanzees to CH) is starkly missing. Specifically, for location, Chimp Haven needs to be the reference group, so that comparisons of transfer from all other sites can be made. This is particularly strange as the text lists Chimp Haven as the reference group and interprets the data in this regard. If the point of this analysis is to inform the decision to transfer the animals to CH, vs retire-in-place, then the comparisons should be made with CH as the reference group so that we can see how it truly stacks up against leaving the animals where they are.
Speaking of Research 
MMS:
I agree with the overall point reflected in the post and commentaries that NIH needs to analyze and analyze each transfer case to fullfill the main goal and mandate of choosing the best situation possible for the animals’ welfare.
Dear ICARE and others,
You have made a very reasoned argument but you are missing an important fact. NIH has corrupted the entire process by which they have come to these final decisions. It started with the Institute of Medicine and subsequent NIH Working Group reports. Dr. Steve Ross was on the board at Chimp Haven but got off to be a member of the NIH Working Group, then returned to the Chimp Haven board at the end of the NIH deliberations. The NIH Working group is the body of individuals that recommended the cessation of all biomedical research with chimpanzees and their retirement. NIH has since decided that Chimp Haven will receive all their chimpanzees. Not to disparage Dr. Ross or Chimp Haven, but most people and organizations, including NIH, would see a potential conflict of interest in that arrangement and would normally prevent this from happening. Moreover, most scientists would recuse themselves from these committees under those circumstances. Neither of these things happened and look at the outcome. Think about how organizations such as HSUS would have responded to NIH if directors of primate facilities holding chimpanzees would have been put on the NIH Working group and this group concluded that all chimpanzees should be retired at their current facilities? I am sure HSUS would have been outraged and pointed out the conflict of interest.
Give HSUS credit, they finessed and lobbied NIH to get exactly the people they wanted on the NIH Working group who would give them the decision they wanted and NIH the justification for their decision. The stupid scientists were too naive and erroneously assumed that the NIH would be impartial in their deliberations but they perfectly played them to be the fools.
Now we have read and seen the stunt NIH pulled last week. Announcing their plan for retirement in conjunction with the simultaneous posting of a non peer-reviewed paper as an argument in support of their decision is truly sad but I guess we should not be surprised. Would NIH ever publically endorse a new treatment for cancer without the study being properly vetted, particularly if the drug being developed was by a company that was lobbying NIH? The answer is hopefully no and there are many rules in place against this happening but this is exactly what took place. HSUS lobbied NIH to make this decision and they then used non peer-reviewed data to make their case for the plan they have proposed. HSUS then made sure to reward NIH by offering blanket support for the plan, thereby closing the corrupt loop. More than anything, this event clearly shows that the NIH decisions about chimpanzee retirement is all about politics and optics and has nothing to do with some change in heart or ethics about the use of chimpanzees in research. I am also of the opinion that no one at NIH cares about chimpanzee well being and that they have all along wanted to retire the chimpanzee to Chimp Haven. That executive decision could have been made long ago but no one at NIH was willing to take that bold action. Instead, NIH pretended to go through an objective evaluation process to get an outcome they always wanted. That wasted time, energy, and tax payer money and makes NIH officials look no different than any another politician, unwilling to take a stand on principle and bought off by the biggest lobbyist. Shame on NIH.
I have read with bewilderment the recent NIH announcement about their plan to retire the remaining chimpanzees housed at Alamogordo, Bastrop and Southwest Foundation and the press release from HSUS applauding the plan and their appreciation of all the effort by Dr. Collins and NIH have made to implement the plan. Why should anyone be excited about this plan? From what I read, their plan is attrition. For presumably the next 10 years, NIH is going to watch and monitor the chimpanzee mortality rate at Chimp Haven and fill the vacancies with chimpanzees currently residing at APF, Bastrop and then Southwest (the NIH preferred order). How can attrition and replacement be considered a reasonable and humane retirement plan by either the research community or animal welfare advocates? Indeed, based on this plan, how can someone argue that anyone at NIH actually cares about chimpanzee welfare? They seem to be rooting for chimpanzees to die and apparently with the endorsement of HSUS. My mother always told me that actions speak louder than words and, by that standard, NIH has done nothing innovative to solve a problem they created. Here are just a few thoughts that come to mind based on these decisions and their plans. I have tried not to just criticize but offer some real solutions.
Attrition can happen anywhere. Thus, why move the chimpanzees to Chimp Haven at all? This seems particularly relevant in light of the SOR discussions of the paper by Lauer in which he shows that the mortality rates are comparable between Chimp Haven, APF, Bastrop and Southwest? Indeed, mortality rates (as presented in table 1) were highest at Chimp Haven. These data, coupled with the attrition plan, seems to argue against moving the chimpanzees to Chimp Haven. To be clear, I have nothing against Chimp Haven and I have absolutely no doubt that their staff care deeply about chimpanzees and their well being. My comment is not meant to be any type of criticism of Chimp Haven but rather is an indictment of NIH. Why is NIH so committed to sending all the chimpanzees to Chimp Haven? By any objective measure or observer, it makes absolutely no sense.
One might argue that the NIH decision and plan are based on financial motives (note, not ethical or moral). NIH provides 75% of the costs for maintaining the chimpanzees at Chimp Haven and they must match it with 25% support. In contrast, NIH provides 100% of the financial support for APF, Southwest, and Bastrop chimpanzees (I am assuming this but don’t know for sure). This is certainly possible but, if this is the case, why doesn’t NIH offer the same option to APF, Bastrop and Southwest? In other words why not offer the same cost-sharing plan to APF, Bastrop and Southwest and allow them to form nonprofit 501(3) © entities to raise private sector funds to support the chimpanzees? This seems reasonable particularly in light of the fact that NIH has provided no extra funding to build new housing at Chimp Haven or other places.
I will go even one step further with this argument. Why doesn’t NIH open up retirement options to all sanctuaries or facilities that could manage and care for their chimpanzees? For my vantage point, waiting 10 years to retire all the chimpanzees using an attrition plan is too long and costly. IF NIH were to allow other sanctuaries or facilities to house their apes using the same cost-sharing plan that they have with Chimp Haven, my guess is that the chimpanzees could be moved and retired far more quickly. For the life of me, I cannot understand the rationale by NIH to move every chimpanzee to Chimp Haven when it is likely that, at minimum, comparable care and housing could be provided at other places at no extra expense (including in situ retirement). It would also remove all the pressure to raise private sector funds for the chimpanzees from one entity (Chimp Haven). Think about it. Imagine there were 10 non-profit organizations raising funds to take these chimpanzees and improve their lives instead of a single one?
On the opposite side, I think it is also fair to ask why Chimp Haven seems so motivated to be the lone sanctuary to house NIH-owned chimpanzees? Again, to be clear, this is in no way a criticism of the organization and I have nothing but respect for the people running Chimp Haven. By any measure, it is a daunting task to raise funds and manage so many chimpanzees, which leads me to wonder why they prefer to “go it alone”? It may be the simple fact that NIH has told Chimp Haven that they are only entity with whom they want to work but this is not obvious to anyone looking at their arrangement from the outside. More to the point though, if we can all agree that some housing of chimpanzees in laboratories is as good and exceeds those provided at Chimp Haven (in its entirety), then shouldn’t the focus of these discussions be on retiring those chimpanzees that are in poorer housing conditions rather than trying to focus on moving and retiring entire colonies of apes to Chimp Haven? In short, why not advocate moving and retiring those chimpanzees living in less then ideal circumstances either in labs or other situations to sanctuaries (Chimp Haven)? This might actually improve the lives of some chimpanzees in a very tangible and timely manner.
There has got to be a better solution.
Dear BMM-
Thank you for your comments. They are appreciated as I am not an expert on survival statistics.
However, I think you you miss the point on the transfer analysis. The author opted to use the Cox analysis when relevant comparison groups and data were readily available to make more relevant comparisons and presumably better inferences about the effect of this variable on mortality. If we can agree that, all things being equal, this would be a better approach to addressing this specific question, then why ignore that possibility? The author chose to use statistics to address this issue rather than a pseudo- or experimental design. Everyone knows that experimental designs rather than correlational designs improve the ability to make cause-effect inferences. I was disappointed not to see these compassion made, particularly when they had available information to improve the design. That for me would have been enough to reject the paper.
I would also disagree with your conclusion that we should “review the peer reviewers” and that the Lauer paper would have never seen the light of day. As indicated in the original SOR piece, it was explicitly stated that “Lauer may very well have had answer to the questions raised that would bolster the claims and arguments in the paper” and you have provided some. So I don’t think it is fair to say that the paper would have never seen the light of day. The arguments you made were not explicitly made in the draft of the paper posted online and that would have greatly IMPROVED the original submission had the details you provided been included, at least with respect to the first aim.
I also think the point made by Lisa is not trivial. If I understand what is reported in the paper, the sampling period for mortality is shorter in the Chimp Haven population compared to the others but I must admit, I am sure because that variable is not well defined in the manuscript.
“We hope others read the paper and offer their comments.” I’ll add a few thoughts here on the two longest bullets in this post.
Regarding bullet 1:
This post as well as a comment from Bill on BioRxiv seem to belay a misunderstanding of type 1 error.
BioRxiv: “with 6 variables (really only 3) and a N > 700, why adjust alpha to p < .01? Seems like the author had more than adequate power to avoid Type I error”
Post: “In point of fact, the influence of location was not a trivial effect based on the results presented in Table 2 but rather a statistically significant one. "
This is phrased and emphasized as a “gotcha”, but this is exactly what you would expect. For type 1 error the fear isn’t an underpowered analysis, but an overpowered one. With a large sample size factors may easily be statistically significant yet not predictive (type 1 error, false positive, oversensitivity, etc). Therefore, the alpha threshold was lowered and effect size becomes a much more important metric. So, let’s look at an effect size indicator (figure 1). As expected, location not only has a tiny VIMP, but it looks to be some 10x times smaller than age at transfer.
“In turn, to state that sex and age had strong effects and location had a moderate effect on mortality is simply not supported by any statistics other than the p-levels. “
Again, this is not true and we aren’t really interested in p-values anyway. The greatest piece of evidence for this claim is the large differences in VIMP values, hence the point of figure 1. http://projecteuclid.org/download/pdfview_1/euclid.aoas/1223908043
Regarding bullet point 2:
Agreed, a comparison group would have allowed for firmer conclusions however the reviewer seems to be criticizing the analysis as if it was a lopsided ANOVA. A Cox proportional hazards model is a different beast.
See Wikipedia: “Survival models relate the time that passes before some event occurs to one or more covariates that may be associated with that quantity of time.”
Note, this doesn’t mean it is just as good a comparison group but attempting to interpret/critique a hazard model as if it were identical to an ANOVA is a non-starter. To claim the analysis is “fundamentally flawed” due to this ignores all the subtleties of a hazard model.
Lastly since the entire post seems to glorify the peer review process I would just add that had this paper gone to either of the bulleted reviewers it would likely have never seen the light of day. Perhaps peer reviews should be peer reviewed. While one paper shouldn't be the basis of a decision this big (it also wasn't clear to me that this paper was the basis of the NIH decision?) At least with BioRxiv it can be openly discussed.
I guess I don’t understand where in Table 1 the 3.5 years comes from for CH but the other facilities are based off of 7-11 years. There doesn’t seem to be a reason for this, the data should be there, why limit it/count it differently than the other facilities? It’s definitely something a peer review would have flagged.