Categories

AOAV: all our reportsReverberating effects of explosive violenceImpact of explosive violence on civilians

The Peruvian TRC named a surprising group as the primary perpetrator in Peru’s civil war. Professor Mike Spagat talks to Dr Silvio Rendon about how the TRC got it wrong.

Executive Summary: In this critical discussion on the quantification of casualties in the Peruvian Civil War, Mike Spagat, an economics professor at the University of Royal Holloway, with a focus on armed conflict, engages with Dr Silvio Rendon, a former executive director at the Inter-American Development Bank with a Ph.D. in economics. They delve into the methodologies and findings of the Truth and Reconciliation Commission (TRC), which attempted to estimate the death toll from the conflict. Rendon highlights the statistical challenges and methodological shortcomings in the TRC’s work, particularly its indirect estimation methods that led to controversial conclusions regarding the primary perpetrators of violence. By employing direct estimation techniques and statistical methods like kriging, Rendon challenges the TRC’s assertion that the Shining Path was the principal perpetrator, suggesting a lower overall death toll and sparking debate on the accuracy of the TRC’s conclusions and their broader implications.

Mike Spagat: Can you give us an extremely brief overview of the Peruvian Civil War?

Silvio Rendon: In Peru, people are reluctant to call it a civil war. They call it a war against terrorism, if they are sympathetic to the military, or an internal armed conflict, following the international law terminology. I call it an insurgency-counterinsurgency war, a military term that I consider more related to the social sciences. This war was initiated in 1980 by a group called Shining Path (Sendero Luminoso) in one region, Ayacucho, and then it spread out to most of the Highlands, part of the Amazonia and the urban centres. The police and the military reacted very violently, until, in 1992, the leader of the Shining Path was captured. During this time, there were lots of deaths from this war.

Mike Spagat: After the war was over, Peru had a Truth and Reconciliation Commission (TRC). Can you introduce us to the work of this Commission?

Silvio Rendon: The Truth Commission was part of a wave of transitional justice in countries where there were dictatorships or wars. The idea was to reconcile sides, address the underlying causes of wars, and make institutional reforms to make things better going forward. Part of this process was to document what happened, especially how many people died. So, the Truth Commission produced a large document, that synthesized social science research on political violence in Peru with several additional materials, such as interviews and recollections of quantitative evidence.

Mike Spagat: Today we are here to talk about how the TRC tried to quantify the total number of people killed in the war and how this number breaks down by perpetrator. How did the TRC go about this work?

Silvio Rendon: There were lists of victims collected by three sources: Non-Governmental Organizations, the Ombudsman’s Office, and the TRC itself. Altogether the TRC documented roughly 24,000 deaths, but they claimed that these constituted only a part of all the deaths that happened in the country. So, they applied a statistical technique to calculate the total number of deaths.

Mike Spagat: Let me try to summarise. The first step is trying to document deaths one by one. The TRC did some of its own documentation. But there was also documentation available from two other sources. From this basis the TRC went on to formulate statistical estimates. And they reached some surprising conclusions, didn’t they? Can you elaborate?

Silvio Rendon: The first two sources, the NGOs and the Ombudsman office, documented overwhelmingly victims of the State and very few victims of the Shining Path. The TRC had precisely the opposite composition: overwhelmingly more victims by the Shining Path. To disentangle these conflicting patterns, the TRC brought in a team of consultants, led by Dr. Patrick Ball, who proposed to do a projection similar to one he had applied in Guatemala. Ball’s team used a method known as “capture-recapture” or “multi-system estimation”. What’s that? It sounds sophisticated, but it is essentially a rule of three. If there are two sources, A and B, you say that the overlap of A and B is to A like B is to the total population. If you have three sources, there are more subtleties involving something known as “logarithmic estimation”, but the same principle applies. This method has been applied, for example, to estimate sizes of wildlife populations: you capture some birds one time, mark them, capture a new bunch of birds a second time, and then see how many of this second bunch of captured birds were already marked from the first time.

Mike Spagat: Intuitively, you figure that if very few marked birds are captured the second time, then the total population of birds is probably much larger than all the birds you captured on your two tries. One the other hand, if most of the birds caught on the second try were already captured on the first try then then you figure that you’ve already captured most of the birds in the whole population. Right?

Silvio Rendon: That’s right. Then you apply the rule of three formula and produce an estimate of the total population of birds in an ecosystem. The same idea is applied to lists of casualties in armed conflicts. From the overlaps of individuals in lists of deaths, you infer the total number of deaths in a war. The TRC applied this method and from the 24,000 documented deaths, they estimated about 69,000 deaths. That was a surprising result, but it was not the TRC’s biggest surprise. They made separate estimates of people killed by each of three perpetrators: the State, Shining Path, and others. And although the State was the biggest perpetrator among the 24,000 documented deaths, the TRC projected the Shining Path to be the biggest perpetrator. This was a surprising conclusion because it reversed the pattern among the documented deaths but also because the Shining Path was the defeated party in the war. So, people wondered how it could be that the Shining Path killed the most? Well, the team of consultants tried to rationalize their big number and reversal of responsibilities. They claimed that information collected prior to the TRC came from the “western”, better developed part of Peru. However, they argued, the war was most intense in an “ignored” Peru: remote rural areas inhabited primarily by poor peasants, culturally farthest from the “western” world. They claimed that a large number of victims, discovered through estimation rather than documentation, perished under cover of ignorance and even indifference of the official, modern, and “western” Perú. Thus, the team’s report claims that their “Statistical methodology”, in fact, their particular application thereof, “has parted the veil of indifference and ignorance, and the true state of affairs in Perú over the past two decades has begun to emerge.”

Mike Spagat: Again, let me jump in and try to summarize. You can agree or disagree and clarify. There are three sources. Between them, the sources documented about 24,000 unique deaths. For most of these 24,000 documented deaths the government is listed as the perpetrator. The TRC team then analysed overlaps between the three sources, i.e., they figured out how many deaths are on just source 1, just source 2, just source 3, sources 1 & 2 but not 3, sources 1 & 3 but not 2, etc. This overlap analysis led to an estimate of roughly 45,000 deaths that they say happened but that didn’t make it onto any of the three lists. This means that the TRC estimated a total of 69,000 deaths, almost three times the number of documented deaths. This is pretty surprising but not as surprising as the TRC’s dramatic reversal in responsibility from the State being the primary perpetrator among the documented deaths to the Shining Path being the primary perpetrator for the TRC’s estimates. Do you agree with that characterisation?

Silvio Rendon: Yes, that’s correct.

Mike Spagat: So, the TRC unleashed a big surprise. What was the reaction in Peruvian society?

Silvio Rendon: Well, from the beginning, people were very sceptical about the reversal. The Peruvian Army was by no means flattered by the conclusion that their former enemy had killed more people. The especially disliked the high estimated numbers of killings and did not appreciate the qualitative work by the TRC with detailed documentation of killings by the Army, which eventually led to convictions of some army members. But other sectors of society did not believe the conclusions of the TRC either. The TRC members reacted very defensively in trying to justify the validity of their methods as rooted in sound statistics. Very few people could understand what they had done, let alone conduct a proper evaluation of their statistical work.

Mike Spagat: Where do you come in?

Silvio Rendon: I reviewed the TRC’s work. But I’m an economist by training. I do econometrics, not statistics, at least not the way that they had done it. So, it was a learning process for me. The first thing that I found were problems, not with the statistical method itself (capture-recapture), but in the way the consultants applied it. They split the country into 58 strata, to account for geographical heterogeneity. But, as discussed, the method is based on analysing overlaps between sources but in many of their strata the overlaps among Shining Path perpetrated deaths were so few that the method could not yield valid estimates. On the other hand, there were always plenty of overlaps for the State, so I concluded that these estimates for the State were correctly estimated. But for the Shining Path; the data were just too sparse. The consultants recognized this problem and proposed what might seem at first glance like a clever workaround. They applied the usual capture-recapture method to estimate the total number of victims and the number of State victims in each stratum. Then they declared that the number of Shining Path victims was equal to total victims minus State victims. However, nobody had ever done such an indirect estimation, and nobody has done one ever since. And there is no theoretical support for this device of assigning victims to a group as a leftover once another group had been assigned a portion of deaths.

Mike Spagat: Let me interrupt again with a summary. Again, you can approve or correct. The TRC divided the country up into 58 different regions called strata. But when you chop up the country into so many pieces you often find that many of these pieces have hardly any documented Shining Path killings. So, the data is too sparse to allow a decent statistical analysis. The TRC realized that they couldn’t make valid direct estimates for the number of Shining Path killings in these places, so they took an indirect approach. They estimated total killings for the State and Shining Path combined. Then they estimated killings by the State, for which there is always enough data in all 58 strata. Then they set their estimate for killings by the Shining Path equal to killings by the groups combined minus killings by the State.

Silvio Rendon: Yes, that was the first potential shortcoming that I found so I investigated further. I identified 9 strata out of the 58 for which there was enough data to do direct estimation for the Shining Path. The TRC, using its indirect method, had almost tripled the number of deaths attributed in the documented data to the Shining Path. But when you estimate Shining Path deaths directly in the 9 strata where it is possible to do so, then you get lower numbers. So, their application was incorrect. That was the first thing.

Mike Spagat: I see, so you found 9 strata where you could do a head-to-head comparison of consultants’ work-around approach with a traditional, direct approach and you find that directly estimated deaths attributed to the Shining Path are substantially lower than indirectly estimated Shining Path-caused deaths. Did this happen every single time you could compare the TRC’s indirect approach with a direct approach?

Silvio Rendon: That did not happen every single time. It happened in 7 out of 9 strata. But that was very material. These 9 strata accounted for 44% of the whole sample. So, I knew that this was an important discovery.

Mike Spagat: What did you do next?

Silvio Rendon: From these core estimations, I proceeded to estimate for the whole sample using several methods. First, I projected estimates for other sparse-data strata using a geography-based method called kriging. This is a method used, for example, to determine oil reserves under the ground. You drill a few wells over some geographic area and use your findings there to project the oil content for the whole area. I borrowed that method, which is accepted in statistics, to project the number of killings by the Shining Path. A second approach was simply to partition the country into fewer than 58 strata so that data density was always adequate for direct estimation. These two methods validate each other by giving consistent results. And they do not yield estimates that triple the observed number of killings as the TRC does. More importantly, they do not lead to a conclusion that the Shining Path killed more people than the State did. This last statement is very important, because it changes our whole perception of the war.

Mike Spagat: Okay, time for another recap. We’ve learned that here were two surprising conclusions from the quantitative analysis that the TRC did but one is more important than the other. The first was just to say that the number of deaths was nearly three times as high as the number of deaths that had been documented. And that certainly is kind of eye-popping, and it gets attention. But is, perhaps, less important than the second one, which reversed the blame. That is, the TRC reversed the identity of the group that did the most killing. And this change, if true, prompts a fundamental rethinking of the nature of the war. That’s what you’re saying, right?

Silvio Rendon: Yes, that’s correct. Because that weakens the image of a western vs an ignored Peru in which marginalized people are just killed without any attention. I would say that it is worse: relatively integrated people are killed, while the rest of Peru and the world knew, but they did not care. We can see that kind of indifference in several other wars in the world. Indifference does happen without ignorance. The statement “I did not know that that was happening” is usually not true.

Mike Spagat: But you found in your research that this reversal claim doesn’t withstand scrutiny. And you described two different methods you used to make full national estimates, and the reversal claim doesn’t hold up under either one. For your first approach you cut the country into 9 rather than 58 pieces. Then you always have enough data to do direct estimation. That’s very simple and straightforward. [Technical note – these 9 strata cover the entirety of Peru and are different from the 9 strata mentioned higher up in this interview that cover only part of Peru.] The other approach seems more subtle. I gather that you stuck with the 58 strata the TRC used but then you projected numbers into areas where the data is sparse (and into the other areas too) based on what is happening in surrounding areas. Essentially, you’re saying that the experience in one of these small 58 pieces of the country will resemble the experiences in neighbouring pieces.

Silvio Rendon: Correct. Yes, I take 9 strata that can support direct estimates. I use the geographic coordinates to pick the middle point in the map for each of these nine geographic areas. I build a net of these locations which are weighted according to the distance to the midpoint of the area to be predicted, for which I know the observed killings. The total number of killings for the predicted area is projected by a combination of multipliers coming from the available strata times the observed number of killings in the predicted area. That’s how the method works. It’s an accepted method in statistics.

Mike Spagat: Did you do anything else?

Silvio Rendon: Yes, I have a third method which is known as “fixed effect estimation”. This works by pooling groups of strata where direct estimation is possible. Instead of estimating all parameters for each of the 58 strata, I do a joint estimation of the 58 strata, with one parameter free, which is a fixed effect for each of the 58 strata. You can consider this as a robustness check on the previously described work and it works. You get the same message for this method as you get from the others. To summarize, whenever you do a direct estimation, you don’t get the reversal of responsibilities.

Mike Spagat: Great, and as a secondary matter whenever you do direct estimation you get a much higher number for estimated dead compared to the documented number, but you get something like a factor-of- two increase rather than a factor-of-three increase like the TRC got. Is that right?

Silvio Rendon: Yes. From the 24,000 observations, I estimated around 48,000 total deaths. The TRC and I coincide in some estimates. For example, for the State, the TRC did a direct estimation: from 10,000 killings, they estimate 21,000 killings., which is about the same as what I obtain. However, for the Shining Path, from 9,000 killings, they estimate 31,000. That is, their estimated number is two and a half times their documented number. In my case, from these same 9,000 killings, I estimate 15,000. That was my contribution. I tried to move on from raw scepticism about the TRC and to go one step further by seriously interrogating and then improving their methods. I checked out alternatives to estimate the 58 strata for all perpetrators. By the way, their problem starts with their choice of stratification. They split the country into 58 strata, apparently just considering the State’s killings and total killings. They did not seem to consider the feasibility of directly estimating the Shining Path´s killings when they did their stratification. Their result could have been much different if they had done this split thinking of the three perpetrators in the first place. My work was an attempt to address this erroneous decision.

Mike Spagat: Would it be fair to say that you applied a more standard versions of their own capture-recapture methods than they did themselves?

Silvio Rendon: Yes.

Mike Spagat: Did the authors of the TRC quantitative analysis react to your criticism?

Silvio Rendon: Yes, Ball and Manrique- Vallier were very dismissive toward my work. However, interestingly, they didn’t really defend their own indirect estimation work that they had done for the TRC. Instead of defending this unorthodox approach they brought in a new dataset and did another estimation, using a direct method, and obtained results similar to their original TRC results. Well, if you add additional data, then we are not evaluating the same thing. In my work I did the same exercise they did for the TRC with their same method and data and then progressed from there. But perhaps more importantly, these additional data were not good. This was a list of 35,000 victims collected in the Census for Peace by Peruvian authorities, overwhelmingly making the Shining Path responsible for most victims. Crucially, this dataset was created to support reparations for victims, but victims of the State may not receive reparations. In fact, you could place yourself in danger by claiming victimization by the State because this could be taken as evidence of terrorist sympathies. So, it is hardly surprising that this dataset contains overwhelmingly more victims of the Shining Path. Tellingly, these data existed back when the TRC did their estimation in 2003, but the authors themselves did not use them and even questioned them. That’s why they came up with this capture-recapture method in the first place. So, they replied with these same questionable data that they rejected, data that were originally collected with the purpose of reparations only for victims of terrorism. However, they could not undermine the estimates that I found without invoking this dubious dataset.

Mike Spagat: Have there been further developments on this topic since you did your reanalysis?

Silvio Rendon: Well, yes. There is a paper by Das, Kennedy and Jewell published in the Journal of the American Statistical Association, more a theoretical paper with an application to Peru, based on Das’s dissertation work. They don’t find the reversal of responsibilities put forward by the TRC that I refuted. So, I see it as vindicating my work. That said their estimate for the total number of people killed is very close to the TRC’s. I would argue, however, that they are wrong about this. They have three categories of perpetrator: State, Shining Path and Other. “Other” basically translates into undocumented and unidentified killings because there are only a handful of cases for which another group is responsible. That is, “other” is a black box where you don’t know what’s going on. In my paper I proposed a method of imputation for these unidentified killings that must be attributed either to the State or to the Shining Path which I think is a better way to handle this category. Moreover, the attribution of responsibilities to the State and the Shining Path themselves may be unclear. Anyway, it gets very technical, but the bottom line is that these authors don’t find that reversal of responsibilities.

Mike Spagat: Were there some other discussions of your work more in the public domain?

Silvio Rendon: Yes, Andy Gellman in his blog bounced some of these discussions, in particular, that “we shouldn’t automatically accept conclusions just because they come from what sounds like an official source”. Anyhow, I can tell you that recent independent research is not corroborating what the TRC did, but rather questioning it.

Mike Spagat: Thank you very much for a very interesting interview.