Introduction
The external examination system, which has been present in Polish education since 2002, has caused an increasing pressure of the public opinion and policy makers on evaluating schools on the basis of examination scores. The motivation for developing value-added indicators in Poland was the desire to obtain fairer indicators to asses how well schools work. In the period of 2005-2007, research activities were launched in order to develop the Educational Value-Added (EVA) models for Polish schools. The methodological team has created EVA indicators for lower- and upper-secondary schools. It has also trained the potential users in analyzing these indicators.
At the beginning, rather simple solutions were adopted, but with time more sophisticated methods were introduced, improving different aspects of computing the indicators. Since several years the Educational Value-Added indicators for lower-secondary and upper-secondary schools have been publicly available attracting growing attention of teachers, headmasters, policy-makers and parents.
Nevertheless, the methodology of computing the Polish Educational Value-Added is still being developed. Although value-added indicators are far from being perfect determinants of school quality, they provide important information that should be taken into account in school management.
Types of indicators
Polish Educational Value-Added indicators can be classified according to two main criteria: 1) the school type and the subject the indicators refer to, 2) the period covered. Regarding the second criterion there are one-cohort indicators and three-cohorts indicators. One-cohort indicators are based on the results of a single cohort of students taking exam at the end of school in the same year. They are intended mainly for internal evaluation of schools and are not publicly available (they can be analyzed with EVA Calculator application). Three-cohorts indicators take into account the results of three consecutive cohorts of students. As they are more stable and reliable they are publicly available on this website.
Regarding the first criterion, different types of indicators are summarized in the table below. The parts of examinations used to compute indicators are given in the two last columns.
It has to be stressed that for upper-secondary schools of a mixed general-vocational educational track (technikum), the Educational Value-Added indicators take into account only achievements in the field of general education ignoring the vocational training. It is a serious limitation of Polish value-added indicators regarding this type of schools.
Indicators for lower-secondary schools were published in year 2008 for the first time and indicators for upper-secondary schools two years later.
|
school type |
type of indicator (domain covered) |
one / three year |
“exit” exam |
“entrance” exam |
|
gimnazjum (lower-secondary) |
math & science |
O/T |
overall score of math test and science test of EG |
Sprawdzian |
|
humanities |
O/T |
overall score of Polish language test and history & civic educ. test of EG |
||
|
mathematics |
O |
math test of EG |
||
|
science |
O |
science test of EG |
||
|
Polish language |
O |
Polish language test of EG |
||
|
history & civic educ. |
O |
history & civic educ. test of EG |
||
|
liceum* (upper-secondary, general track)
technikum* (upper-secondary, mixed track) |
math & science |
T |
scores of 6 math & science subjects on matura |
overall score of math test and science test of EG |
|
humanities |
T |
scores of Polish language, history and civic educ. subjects on matura |
overall score of Polish language test and history & civic educ. test of EG |
|
|
mathematics |
O/T |
scores of mathematics on matura |
math test of EG |
|
|
Polish language |
T |
scores of Polish language on matura |
Polish language test of EG |
*Indicators for liceum and technikum are calculated and presented separately, but there are analogous types of domains of indicators for both type of schools.
O – one-cohort indicators available; T – three-cohorts indicators available; O/T – both one-cohort and three-cohorts indicators available;
EG stands for egzamin gimnazjalny.
Exclusions
The idea behind the value-added indicators is to show the average achievements of students of a given school compared to achievements of similar students in other schools. But there are some specific types of schools that cannot be reasonably compared to typical schools, so they should be excluded from the computation of the value-added indicators. With regards to Polish Educational Value-Added those are: 1) schools for adults, 2) schools for students with mental disabilities, 3) schools in hospitals, sanatoria, youth detention centers, etc.
In the computation of the Educational Value-Added indicators, only students taking the „exit” exam for the first time are taken into account. Students who repeated a grade once during the given educational stage are included into model, but treated specially. Students who repeated a grade more than once are excluded.
Transformation of examinations scores
Examination results are reported by examination boards as raw scores or a percentage of correct answers. Unfortunately, the distributions of those scores may differ considerably between years. To ensure better psychometric properties and an easier interpretation of scores from different years some methods of examinations scores transformation are applied before computing Educational Value-Added indicators.
Different methods are used with respect to different types of value-added indicators:
- The quantile normalization (Hazen formula) of overall score for one-cohort indicators.
- The IRT 2 Parameter Logistic/Graded Response models for single tests (parts of exams). Those are used to obtain estimates of students’ abilities in the egzamin gimnazjalny for the three-cohorts Educational Value-Added indicators.
- The multigroup IRT 2 Parameter Logistic/Graded Response models of general skills (i.e. affecting scores in different subjects) when students from different types of schools are taking the same exam and not all students take all the subjects . Those are used to obtain estimates of students’ abilities in the matura for the three-cohorts Educational Value-Added indicators for upper-secondary schools.
- The multigroup Rasch/Rasch Graded Response models of general skills (i.e. affecting scores in different subjects). Those are used to obtain estimates of students’ abilities in the matura for the one-cohorts Educational Value-Added indicators for upper-secondary schools.
All examination scores are presented using a normalized scale with a mean of 100 and a standard deviation of 15 (with respect to matura exam there are distinct normalizations for liceum and technikum schools).
Regression models
In the value-added models students’ prior attainments (i.e. scores on “entrance” exam) are used to predict scores achieved on the examination at the end of school. In the Polish models also information on students’ sex and having a dyslexia certificate are included into prediction. Recently, it has been investigated how to properly include information on students’ age and school’s mean score on the „entrance” exam. Unfortunately, there are no other students’ characteristics that could be used for computing the value-added indicators in Poland.
The relationship between “entrance” and “exit” examination scores is described in the Polish models by a polynomial of up to 5th degree as there are considerable deviations from linearity. Different polynomial parameters are estimated for every cohort of examinees (in models used to compute three-cohorts indicators) and for every cohort of grade repeaters. Because there is no vertical equating between Polish examinations, an absolute students’ gain in skills cannot be inferred from the model.
Depending on the type of indicators, one of two methods is used to obtain value-added indicators:
- To compute one-cohort indicators an OLS regression with Educational Value-Added computed as a mean of residuals among group of interest (particularly students of a given school) is employed.
- To compute three year indicators a mixed-effects (multilevel) regression with random effect for intercept connected with division of students into schools is used. The Educational Value-Added indicators from those models are computed as Bayesian posteriori predictions of random effect (so called BLUPs: Best Linear Unbiased Predictors).
More informations about how EVA three-cohorts indicators are estimated and presented can be found on other pages of our website:
- How to read the graphs? Lower-secondary schools and upper-secondary schools.
- How were the indicators calculated? Lower-secondary schools and upper-secondary schools.
