The main aim of “A counterfactual impact evaluation of a bilingual program on students’ grade point average at a spanish university” written by Arco-Tirado et al. (2018) published on Evaluation and Program Planning, is to measure the impact of using English as Medium of Instruction (EMI) in a Primary Education Teacher Degree. This public Spanish University has two different itineraries for the same curricula: a Spanish program and a Bilingual program with a 65% of the hours taught in English. Students can voluntary apply to be enrolled in the EMI program. So, the question the paper would like to answer is if there is a cost in students’ academic performance of enrolling in the EMI program.
The main methodological problem the authors face is the non-equivalence of treatment and control groups as the individuals are not randomly assigned to both groups. Contrary, they have to apply and probably are enrolled in treatment (EMI group) or control (Spanish group) using a cutoff score on a some pretreatment variable (access grade?, English level?,…) These means that “potential differences between treatment group members and potential controls that are likely to affect the decision to participate” exist.
To solve this problem, authors decide to use Counterfactual Impact Evaluation (CIE) approach. This approach compares two potential results. First result is the performance obtained by the individuals under treatment; i.e. the academic performance from members of the EMI group, measured by the Grade Point Average (GPA). The second result is the performance these individuals would have obtained if they would not have been in the EMI group but in the Spanish one. Obviously, this second result is impossible to record as the students have actually followed the treatment and have studied in the EMI group.
Where could the authors get this counterfactual result? The researchers use some members from the control group, which have effectively followed the Spanish itinerary, but are perfectly compatible with the students in the EMI group. To decide which student in the EMI group is compatible with which student in the Spanish group they use different Matching techniques (Genetic, Nearest Neighbor and Coarsened Exact Matching). With these techniques it is possible to identify pairs of students with similar characteristics in the pre-treatment or confounding variables and assume that they have the same probability to apply to the EMI group (propensity score). Now, it would be possible to use the academic performance of these individuals in the control group as counterfactual of the individuals in the treatment group and compare the results.
I find the approach really original and inspiring as the problem related with research designs in education is important. I like a lot this different way (at least till I know) to propose the analysis. Intuitively, it can be reasonable to assume that, even the students in both groups have different characteristics in the covariates, it could be relatively easy to find “similar” students that could have apply to treatment although finally they have not done it. Technically, the matching methods used are enough referenced in the paper and I also find sound the option “to build a credible control group (without the use of randomization) from existing non-participants”. So, my first impression is that, both the aim and the method used in the paper, are worth and compelling.
The conclusions probably are a bit disappointing for those who teach in EMI modules as they find that there is a negative effect of treatment groups; i.e. being enrolled in the EMI group has the cost of worse academic performance than the results the student would have got in the Spanish group. They discuss this result, look for explanations for this, and take into account the role student’s English command at baseline should have in a research like this.
In summary, it is an interesting paper that is worth reading. It gives some clues to make better design of EMI modules and curricula. Here you have the link to the published paper in Evaluation and Program Planning. Volume 68, June 2018, Pages 81-89
