Statistical Evidence and Bias

I have posted previously about Sean Rehaag's empirical analysis of immigration decisions. He also authored an analysis of refugee claim data for 2011:

Data obtained from the Immigration and Refugee Board (IRB) through an Access to Information Request reveals vast disparities in refugee claim recognition rates across IRB Members in 2011.

In 2011, some Members very rarely granted refugee status, including Daniel McSweeney (0%, 127 decisions) and David McBean (1.9%, 108 decisions). Others granted refugee status in most of the cases they heard, including Thomas Pinkney (98.0%, 799 decisions) and Deborah Morrish (97.9%, 366 decisions).

This report was relied on by the applicants in Turoczi v. Canada (Citizenship and Immigration), 2012 FC 1423. Member McBean had refused their refugee claim. They argued, based on Rehaag's statistical analysis, that the rate at which Member McBean granted refugee claims raised a reasonable apprehension of bias: a fair-minded observer would not conclude that the decision-maker had decided the case fairly.

Zinn J. rejected the application. Without more, the statistics did not satisfy the test for bias:

[13]           Quite simply, the statistics provided by the applicants are not, without more, sufficiently informative.  Furthermore, one must question what the “informed person” would take from them.

[14]           The applicants submit, and this is the true focus of their submission, that the acceptance and rejection rate data, standing alone, is such that “one must be wilfully blind not to see that there exists a reasonable apprehension of bias” on the Member’s part.  This ignores or overlooks that the acceptance and rejection rate alone says nothing to the “informed person” even if the uninformed person might reach the conclusion that the applicants suggest.

[15]           Although the statistical data presented by the applicants may raise an eyebrow for some, the informed reasonable person, thinking the matter through, would demand to know much more, including:

•                     Were all of the figures, including, importantly, the weighted country origin averages, properly compiled?

•                     Did the RPD randomly assign cases within each country of origin?  If not, how did the RPD assign cases?

•                     Can factors affecting the randomness of case assignment be reliably adjusted for statistically?

•                     If so, what are the adjusted statistics, and what is their significance?

•                     If the RPD did randomly assign cases, what is the statistical significance of the Member’s rejection rate?

•                     Beyond the Member’s relative performance within the RPD, is there anything objective impugning the Member’s decisions (i.e. that suggests they are wrongly decided)?

•                     Accounting for appropriate factors (if that is possible), are the Member’s decisions more frequently quashed on judicial review than would be expected?

•                     Has the Member made recurring errors of a certain type, e.g. on credibility, state protection, etc., that bear a semblance to the impugned decision?

In short, the informed reasonable person, thinking the matter through, would demand a statistical analysis of this data by an expert based upon and having taken into consideration all of the various factors and circumstances that are unique to and impact on determinations of refugee claims before he or she would think it more likely than not that the decision-maker would not render a fair decision

Zinn J. did not shut the door entirely on the possibility that statistical evidence could assist an applicant in a future case. Analysis, rather than assertion, would be necessary. It will be interesting to see whether a future applicant engages a statistician to perform (and explain) a regression analysis of the type suggested by Zinn J. 

There might, in addition, be better cases in which to make arguments based on statistics. As Zinn J. observed, the present case was straightforward:

[18]           The applicants make no attempt to impugn the Member’s decision on their application.  It did not involve the exercise of discretion on his part.  The applicants claimed refugee protection fearing Ms. Karpati’s violent former boyfriend, who could not accept that their relationship was over and that a new one with Mr. Turoczi had begun.  The Member determined that the applicants had a suitable internal flight alternative (IFA) in Budapest, which is 200 kilometres away from the applicants’ home town, and that they had not rebutted the presumption of state protection.  These findings were straightforward applications of binding legal authorities and the relevant burden of proof.  In my view, the fact that the Member was practically obliged, in light of the relevant law and the burden of proof, to decide as he did, is another factor that a reasonable and informed person, examining the issue thoughtfully, would consider.  Indeed, in the instant case, there is every likelihood that an informed person, viewing the matter realistically and practically – and having thought the matter through – would conclude that there was very little likelihood that any member would have decided the claim differently. 

Similar questions have been raised in Ireland. In Nyembo v. Refugee Appeals Tribunal, 2007 IESC 25, the applicant was permitted by the Supreme Court to make an argument based on statistics which demonstrated an elevated rate of refusal, but the case settled before the judicial review concluded.

I am not aware of any case in which a statistical argument based on elevated refusal rates have been successful. If I have missed one, please let me know.