How do we estimate the EVA?
If all schools in Poland cooperated with students with the same level of prior school achievements, the results of the leaving exams would be a good measure of effectiveness. But it is not so. The EVA method enables taking into account the differentiation of schools in terms of the “input” resources. In the case of lower secondary schools, the best available measure of the input resources are students’ results from the sixth grade test. Not only do the results of the test shed light on the level of school achievements of a student at the beginning of the lower secondary school, but they also carry information on the child’s home environment, his or her level of ability and school motivation. It is so, since the test results are determined by factors similar to those that determine the results of the lower secondary school exam.
The EVA indicator for a school reveals how high/low final exam results were achieved by its leavers in comparison to the students in the whole Poland with analogous test results. The indicator is of relative nature, that is it serves to compare schools. At the national scale, the EVA indicator is, ex defintione, equal to zero. An additional value of the EVA indicates an above-average effectiveness of teaching, while a negative value a lower than average effectiveness.
The statistical idea of estimating the EVA indicators is well illustrated by the following graphs.
These are the so-called dispersion graphs. Both show the statistical dependency between the result of the sixth grade test (the horizontal axis) and the results of the lower secondary school exam three years later (vertical axis). As an example, graphs that show the dependency for the test in 2003 and lower secondary school exam in the Maths and Sciences three years later, that is in 2006.
The small dots are graphic representations of the results of more than 400,000 lower secondary school students taking the exam in 2006. The line (called regression line) shows the statistical dependency between the results of the test and the lower secondary school exam at the national scale. On thus prepared graphs, additional, coloured dots that symbolise the results for two schools selected for analysis have been plotted. First, let’s focus on the left-hand graph. The red dots are the results of students in school A, while the green dots are the results of students in school B. Which school teaches better? If we focused only on the final results, we would not hesitate to say that it is school B. The arithmetic mean of the results of the lower secondary school exam in that school is around 36 points, while the mean for school A is just 22 points. But let’s look at the fact that school B worked with students with very high test results, while school A had students with a much lower level of prior achievements. How to take that into account when assessing school performance? It is enough to refer the results of the lower secondary school exam to the average results obtained nationwide by students with a given number of points at the test three years earlier. These points of reference are indicated by the regression line. Let’s note that the results of both school A and school B are located above that line. It means that they both teach more effectively than an average Polish lower secondary school, students in both schools stand the chance to make the same progress. Although in terms of final results the schools are worlds apart, in terms of a different, important – and from the viewpoint of school evaluation, the most important – aspect, they are the same. in both of them, we observe positive educational value added.
Let’s move our focus to the graph on the right-hand side. There are plotted the results of two other school, lower secondary school C in red, and lower secondary school D in green. Again, in terms of final results, one school (C) is evidently worse than the other (D). Yet, an analysis of the EVA shows that it is school C (red dots) which teaches more effectively. The results of students from that school are above the statistically predicted result. On the other hand, the results in school D are below that line, which means that effectiveness of teaching is below the national mean (then, we can talk about a negative EVA).
In practice, the EVA indicators for school are estimated in a slightly more complex way, but the general idea remains the same.
Why three-year indicators?
In estimating the indicators we use the results from three subsequent years. Therefore, we talk about three-year indicators. The way to count three-year indicators is well demonstrated in the following graph.
Each year, data from the oldest previously included class are removed and current results are added.
The calculation of three-year indicators is justified for three reasons. The most important is the fact that exam results are impacted by measurement uncertainty. The greater the pool of data at our disposal, the lesser the impact of uncertainty on estimation of indicators for a school. Using results from three consecutive years, we obtain three times as much data, and a large body of information means greater precision of estimates. School evaluation is, therefore, not dependent on temporary changes, a single “statistical oddity” increasing or decreasing the value of indicators.
Yet three-year indicators also have a disadvantage – they are less sensitive to actual short-term changes. Therefore, a school should not rest with analysis of long-term indicators, and intraschool analyses should include calculation, with the use of the EVA Calculator (see: http://ewd.edu.pl) annual measures.
What data are used to estimate indicators?
To estimate indicators, we use the results of exams in the standard version for students taking them in the main session. Since 2011, the analyses have disregarded the results of students who repeated a class. From the period of 2010 through 2012, also the results of students with prolonged education are included when estimating the EVA. Inclusion of those students is possible because there was developed a special way of estimating the EVA which takes into account the specificity of that category of students.
On what scale are the results presented?
The use of the results from various editions of external exams leads to a significant problem. Raw exam results are not comparable between subsequent editions of the exam. Before commencement of teaching effectiveness analyses, we need to bring them to a comparable form.
In the first phase, the results of each edition of the exam were transformed so that their maximum dispersions would be close to normal dispersion (normalisation). In the second step, normalised exam results from each year were converted into a common scale, so that the national mean would be 100 points, and standard deviation would be 15 points (standardisation). The assume distribution of exam results after normalisation and standardisation looks as follows:
Let’s note that, thanks to standardisation, the results can be easily interpreted quantitatively. It is known that the national average is always 100, 115 is the result exceeding the national average by one standard deviation, we additionally know from the properties of normal distribution that a result higher than 115 was obtained in Poland by about 16% of exam takers. The applied procedure makes it possible to use results from various years in school evaluation.
Normalised and standardised scale of students’ results also serves to present the result of the exam result of a given lower secondary school, as well as presentation of the value of the educational value added indicator. In the case of the EVA, the scale has its centre in point 0. For example, an EVA score equal to 5 points means that the average student from the lower secondary school received at the lower secondary school exam 5 points more than it would result from his or her result of the primary school leaving test.
Since the period 2010 through 2012, the results of the test and the lower secondary school exam are scaled with the use of the modern measurement technique IRT. It enables more effective estimation of the level of skills of students based on raw exam results. The new procedure of scaling does not affect interpretation of presented results.
