BEHIND THE CHANGE OF THE NATIONAL EXAMINATION POLICY IN INDONESIA

This study aimed to ﬁ nd out the mapping of diﬃ culties in mathematics National Examination (NE) in Senior High School along with changes in NE policy in Indonesia in the last 10 years. Researcher conducted descriptive explorative analysis to ﬁ nd out the scope of material, the absorption, and mapping the diﬃ culties faced by students in mathematics. Data analysis carried out by examining the data of each material and indicators. It grouped into two, namely low (<50%) and high absorption (≥ 50%). The results showed that the overall absorption of the NE had decreased signiﬁ cantly over the past 5 years (2013 to 2017). There did not ﬁ nd out of diﬃ cult material before implementation of Curriculum 2013. However, the last 5 years (after the period of implementation Curriculum 2013) have found several diﬃ culties. Moreover, since 2015, almost all the materials on mathematics NE were categorized as diﬃ cult after the changing in the status of the NE (not a determinant factor of students’ graduation) and implementation the computer-based test. The diﬃ cult materials were probability, geometry transformations, space geometry, calculus, circles, linear programs, and statistics. Generally, the diﬃ culties were in narrative problem-solving questions or using real-world contexts. Also, the results showed non-cognitive impacts (students’ motivation) as the result of the policy changing, especially the status and technical implementation of the NE.


INTRODUCTION
Evaluation is an important aspect to the improvement of education. The success of education and the quality of the education implementation can be refl ected in the results of the evaluation. Ebel and Frisbie (1991) stated that the objective of the evaluation is to determine the students' achievement. Therefore, a benchmark of educational success is seen from the students' achievement. If the evaluation results show a good students' achievement; means that the implementation of an education program has a positive impact. In the other hand, if the evaluation results show a low of students' achievement; means that there are problems in the implementation of education. Here, it needs to review the implementation of education.
Evaluation of primary and secondary education in Indonesia is regulated in Government Regulation No. 19 of 2005. In the regulation, evaluation of primary and secondary education is performed by teachers, educational institutions, and government (regulation number 19 of 2005). The evaluation aims to fi nd out the students' learning outcome of mathematics learning in the class. The learning outcome might be the mastery of concepts or skills. Because the evaluations are limited to one or several classes, then the results of evaluations performed by the teacher can only be used to determine the successes or diffi culties faced by a group of students in a class. Meanwhile, the evaluation is performed by the educational institution through fi nal school examinations aiming to fi nd out the success or diffi culties of students in one school. Furthermore, a national-scale education evaluation conducts to find out the success or diffi culties of students nationally. Evaluations on the national scope are performed by the government through ministries or national education standard board (BSNP). In addition, aiming to know the successes or diffi culties faced by the students, the evaluations on a national scope are also used as material to improve the curriculum or generate other education policies (Nitko & Brookhart, 2011).
A form of national-scale education evaluation in Indonesia is through the national examination (NE). There are gradual changes in government policies regarding this NE, such as changes in names, changes in graduation criteria, and changes in technical implementation. A nationalscale education evaluation has been held in Indonesia since 1965 with the name "Ujian Negara" or state examination. After that, in 1972After that, in -1979, the implementation of the evaluation was no longer held by the central government, but the implementation was conducted by the school or school group and called as "Ujian Sekolah" or school examination. A national level evaluation was held from 1982 to 2002 and called as "EBTANAS" or a national learning evaluation. Furthermore, in 2003, there a changing become "UAN" or national fi nal exams, and it was implemented until 2004. The latest changes were made in 2005, namely "UN" or national examinations.
In the beginning, the result of the NE is the fi rst indicator for a student graduation. If a student meets the standards or criteria, then the students are declared as graduated and they can continue to a higher level of education. Criteria for student graduation from year to year also changes. For example, in 2005, the government determined the graduation based on two criteria, namely the minimum score for each subject and a minimum average score for all subjects. In 2005, the scores on each subject and the minimum average of all subjects were 4.25. In 2006, the minimum score for each subject was 4.25, but the minimum average of all subjects was increased to 5.00. In 2007, the minimum score of each subject and the minimum average score were 5.00. The minimum average score determined by the government in the following years continued to increase until 2014 (the implementation of the Curriculum 2013) which reached 5.50, while the minimum score for each subject decreased to 4.00 in 2014. In 2015 to today, the government issued a new policy by giving mandates to each school to determine the graduation of students. In this case, the results of the NE are no longer used as an indicator of student graduation.
Not only on indicators of graduation but also there changed in the technical implementation of the NE. Until 2014, the examination was paper-based test (PPT). Meanwhile, in 2015, several schools had implemented a computer-based test (CBT). In 2016 and 2017, most of the schools have implemented computer-based national exams. Although there were various problems in the implementation of the CBT (Retnawati, Hadi, et al., 2017), the government attempts to improve and set a target in which all students could take part in a computer-based NE in the next years.
As a follow-up to the implementation of national-level evaluations, the data and evaluation tools can be used as material in improvements and planning in the curriculum (Goodson-Espy et al., 2014;Saylor, 1974). The tools and data of evaluation on the NE might use to obtain information. Some previous study used the evaluation tool, the question in the tasks of the NE to obtain information of the characteristics of the National Exams questions (Farhan & Retnawati, 2014;Huriaty & Mardapi, 2014;Isgiyanto, 2012;Kartowagiran, 2008;Retnawati, 2014). Moreover, the questions also use as a reference in designing similar questions and use in measuring the students' competencies (Azis & Sugiman, 2015). There are also researchers who use the data of NE result to fi nd out the students' characteristics who participated in the NE (Prayitno, Nurjana, & Khasanah, 2017;Rumasoreng & Sugiman, 2014;Tias & Wutsqa, 2015), or to made the clusters of the school in an area (Rosa & Gunawan, 2015).
The evaluation data that might use in the research are the report of the NE score published by the national education standard board (BSNP), the ministry of education and culture. The report of NE over the last 10 years can use to determine the development of the absorption and to fi nd out the problems faced by students in the NE. Absorption of NE is the percentage of students who answer correctly to the material, or a scope of material tested in the NE. Unfortunately, the report published by the BSNP has not been fully used for research purpose. A study by Lumapow (2012) found out the diffi culties of students in the NE using the absorption of the NE in which BSNP report as the main data source. However, the data of the research is limited to 3 years (2007 to 2010). Then, it impacts the information in the conclusion that is still limited only on the period of 3 years. In fact, in the past 10 years, there have been several policies changing toward the implementation of education in Indonesia (as previously described). Therefore, the study tried to fi nd out what happened behind the changes in NE policy in Indonesia through re-analyzing the absorption data of the NE in the last 10 years.
The data of absorption obtained from the report of NE used to determine the material tested at the NE in the period of 2008 to 2017 and find out how the development of the absorption along with the occurring of policy changing. This study also sought to map the difficult of mathematical materials in the period of 2008 to 2017, as well as fi nding out the difficulties. The researchers also documented the questions of the tasks used in the period of 2008 to 2017 aimed to support the analysis results and find out more in detail the diffi culties faced by students.

METHOD
This research used a quantitative a p p r o a c h t h r o u g h a n e x p l o r a t i v e descriptive analysis of the absorption on the mathematics subjects of NE at Senior High School. The data were from the NE report published by BSNP (Balitbang Kemdikbud RI, 2008, 2009, 2010, 2012, 2013, 2014, 2015. In the report, the data presented were annual NE results consisting of 3 levels (junior high school / islamic junior high school, senior high school/islamic senior high school, and vocational high school). The data divided into several types, namely statistics (mean, standard deviation, maximum and minimum scores), list (school ranking, or province), absorption (based on SKL, material, indicators, and items), and the graph of the score distribution. For the purposes of data analysis, researchers documented the absorption on NE results based on indicators of the mathematics subject in senior high school of science in the period of 2008 to 2017.
Analysis carried out on the indicators of mathematical material in the NE in the period of 2008 to 2017. The indicators of material grouped based on the material, and the type of ability. Aiming to determine the development of the absorption in the NE, the absorption of each material and the material indicator was mapped into two, namely absorption <50% and absorption ≥50%. The absorption which is less than 50% is categorized as diffi cult, and absorption which is more or equal to 50% is categorized as not difficult (easy). Furthermore, the absorption of each competency indicator on each year also mapped which aimed to fi nd out the diffi culties in each material more clearly. In addition, the researchers also documented the questions of the task in the period of 2008 to 2017 which aimed to capture the types of diffi cult questions and fi nd out the cause of the diffi culties.
The analysis results of the absorption data were also associated with the policy changes as highlighted in this study. The policy as the focus of the study was curriculum changing, changing in the technical implementation of the NE, and changes in the status of the NE. The analysis result of absorption presented separately for each year to fi nd out the infl uence of the policy, especially in the low of absorption data.

FINDINGS AND DISCUSSION
Based on the analysis results toward the material tested at the NE at a senior high school in the period of 2008 to 2017, it obtained 19 materials in the mathematics subject for the last 10 years. Of the 19 materials, there are 6 mathematical materials that do not appear every year. The materials are algebra, mathematical logic, vectors, probability, polynomial, combinatorics, and composition and inverse functions. Mathematical and vector logic materials were not tested in the last 2 years in the NE. Meanwhile, the probability and polynomial materials were not tested in 2017. On the other hand, there were 5 materials that always appeared in the NE with more than 3 times, and even 10 times. The materials are calculus, trigonometry, exponential and logarithms, equations and quadratic functions, and Sequences and Series. Data on the total of the material distribution and material indicators in the period of 2008 to 2017 is presented in Figure 1 (the data is sorted from the lowest to the most).
Generally, from 2008 to 2012, there were no difficulties for students in the NE on all the mathematics materials. The indicators that showed the students have no diffi culties are seen from the overall absorption rate in 2008 which was 72.62%. The absorption continues to increase until 2012, which is 80.58%. However, there was a signifi cantly decreased to 41.40% (see Figure 2) in 2013 and 2017. Although there was an increase in 2015, the increase was not signifi cant and there a decrease in the next year. The same fact is also seen from the trendline that looks to the left (slope <0) which indicates the decreasing the tendency of absorption.
Based on the absorption rate in Figure  2, further analysis is needed to fi nd out the condition of NE in the period of 2013 to 2017. Then, the absorption of the NE is low. For this purpose, annual absorption data needs to be mapped (absorption <50% and absorption > 50%), then it obtained the details of materials with low absorption. Material with absorption less than 50% is categorized as diffi cult material; meanwhile, the material with absorption more than 50% is categorized as not diffi cult material. The results of material mapping based on the absorption are presented in Figure 3. Based on Figure Figure 4, a signifi cant diff erence in the mean absorption of mathematics material at the NE in the last 10 years. In the period of 2008 to 2012, almost all the material absorption was above 70% (there is only one material that is less than 70%, namely solid geometry). Meanwhile, in the period of 2013 to 2014, there were only 2 materials which the absorption is more than 70% and the rest is less than 70%. Absorption has decreased in the last three years (2015 to 2017), which is mostly less than 60%. Moreover, 7 materials have absorption which is less than 50%. The materials are probability, geometry transformations, solid geometry, calculus, circles, linear programs, and statistics.
The absorption of NE is certainly influenced by the distribution of the question's types tested each year. Therefore, the indicators of questions are mapped into two types: the question of applying the common concepts (standard questions) and problem-solving. The results of mapping the question types tested in the period of 2008 to 2017 at the NE is presented in Table 1.
Based on Table 1, the total indicators of problem-solving often appear from 2012 to 2017 (about 25% to 51.7%). The data is diff erent from previous years which only ranged from 10% to 15% of the total Jailani: Behind The Change of The National Examination ...

Figure 3
Distribution of Mathematics Materials in NE Based on the Absorption Data indicators tested. This data seems to show a correlation between the numbers of problemsolving indicators with several difficult materials in the NE. In other words, the low absorption is caused by problem-solving questions in the NE. However, the number of problem-solving indicators was 51.72% in 2012 and there were no diffi culties in the material of NE. Meanwhile, by 25% the number of indicators of problem-solving in 2016 and 2017 the diffi culty appears from the indicators of the application of concepts in the standard questions. These conditions indicated that the diffi culties are not only caused by the characteristics and distribution of the question types. In other words, there are other factors that also infl uence the low absorption score. Therefore, the analysis continued by looking at the diffi culties in each year.
In 2011, there were 19 materials tested at the mathematics subject in the NE. Table  2 shows that students' diffi culties are in the problem-solving question type. Figure 5 shows one of the calculus questions in the NE on 2011, which is a problem-solving question and the question cannot be solved directly by using the characteristic or rules of diff erential. Based on the data and documentation of the question in 2011 NE, found out that the students' diffi culties in the NE on 2011 are only in one material,    In 2013, there were 2 diffi cult materials, namely probability, and trigonometry. Based on the mean of absorption; the absorption of less than 50% is only seen in the probability material. Meanwhile, the mean of absorption on trigonometry material was more than 50%. Although the mean of absorption has above 50% but there found some indicators with a mean of less than 50%. The details of students' diffi culties in the material are presented in Table 3.
Based on the distribution of material indicators (see Figure 1), there is only one indicator in the probability material tested at the NE in 2013. The material indicators of the probability are related to problem-solving involving the concept of probability. For example, in the calculus material, the low absorption of probability material is also shown in the indicators relating problem-solving. The probability questions in the NE in 2013 are presented in Figure 6.
Although the mean of absorption on trigonometry material in 2014 was more than 50% (51.08%) but the absorption in the indicators of trigonometry problemsolving questions was also low. There is one indicator in trigonometry material which Jailani: Behind The Change of The National Examination ...

Figure 5
The Question of Application The Diff erential in 2011 NE is categorized as diffi cult from the three indicators. The trigonometry questions in the NE in 2014 are presented in Figure 7. The questions in Figure 6 and 7 cannot be solved directly by applying the concept. It requires an understanding of the relationship between concepts and the context (see Figure 6), and how to determine the right rules according to the situation in Figure 7.
In 2014, there found diffi culties on fi ve materials, namely vectors, linear programs, quadratic equations, and functions, exponential and logarithms, and calculus. The details of students' diffi culties in the material are presented in Table 4.
Students' diffi culties in the NE in 2014 are in problem-solving questions or applied concepts. Problem-solving and applied concepts consist of two types, namely involving other concepts or using realworld contexts. The examples of diffi cult questions in 2014 are presented in Figure  8 and 9.
There found similar data on 2015 with data on 2013 and 2014, which there found difficulties on four materials, namely probability, geometry, quadratic equations and functions, and exponential and logarithms. The details of students' diffi culties in the material are presented in Table 5.
In the probability material, quadratic equation, and function, exponential and logarithmic, and calculus still have the JURNAL KEPENDIDIKAN, Volume 5, Nomor 2, 2021, Halaman 158-177    same diffi culties, namely the indicator of competence related to problem-solving or applied concepts. Meanwhile, the diffi culties in the geometry material are in visualizing the geometric shape or the display in narrative questions. The example of geometry question in 2015 is presented in Figure 10. As previously mentioned, there were diffi culties in 2016. The details of diffi cult mathematical material in the NE in 2016 are presented in Table 6. Table 6 shows that diffi culties in the NE in 2016 are in the problem-solving questions such as applying a concept to another concept or application of concepts into real world contexts.
However, unlike to previous years, in 2016, there also find difficulties in several indicators that are not related to problem-solving or the application of concepts of real-world contexts. In Table 6, there are diffi culties in indicators toward the application of rules or using the characteristic or theorems in a concept. In this case, students also have diffi culty in problem-solving of applying the common concepts (standard questions).
There are four diffi cult indicators of the application of rules or using the characteristic or theorems in a concept, such as: determining the results for polynomial calculation, determining the limit score, determining JURNAL KEPENDIDIKAN, Volume 5, Nomor 2, 2021, Halaman 158-177   Figure 11 and 12.
In 2017, students' diffi culties were in all material except in algebra material (see Table 7). From the competency indicators, Jailani: Behind The Change of The National Examination ...

Figure 11
Diffi cult Questions on Polynomial Material in The NE on 2016 there are 29 indicators with absorption of less than 50%. As occurring in previous years, there are 10 diffi cult indicators on the problem solving or applying in a concept in a real-world context.
In addition, there are 19 indicators on the application of rules or using the characteristic or theorems in a concept with the absorption of less than 50%. This number is higher than in the previous year (there were only 4 indicators in 2016) which means the diffi culties experienced by students also increasing. The examples of these diffi cult questions are presented in Figure 13 and 14.
I f w e c a r e f u l l y e x a m i n e t h e characteristics of the questions in Figure  13 and Figure 14, then the questions should be solved directly by applying simple algorithms, as it has learned in the class. For example, a defi nite integral question (see fi gure 14) can be immediately solved by applying the quadratic rule to the integral function. And then, it substitutes the upper and lower limits of the integration results.
If considering the absorption of calculus material from 2008 to 2015, the students should not have diffi culty to the questions in Figure 14 because the questions are common problems faced by students when they study the material of calculus. However, absorption in the sample questions is low (47.51%). This shows a new problem which is the factor of diffi culties in solving the question in the NE on 2017.
Data analysis results showed a trend of decreasing the absorption of the NE from 2013 to 2017. The decreasing was indicated by the low absorption of several the materials tested. Several mathematics materials, such as probability, geometry transformations, solid geometry, calculus, circles, linear programs, and statistics, have low absorption on NE from 2013 to 2017. The score of low absorption in 2013 to 2015 was dominated by material indicators in measuring the problem-solving ability. However, in 2016 and 2017, the low absorption was found in material indicators that only measured the ability in solving the common concepts or standard questions.
Some previous research results showed that the diffi culties in solving mathematical problems are caused by cognitive factors and non-cognitive factors. The cognitive factors make the students' diffi culties are explained as follow. The research conducted by Prayitno, et al (2017)found that, in solving the mathematical problems, students have diffi culty in understanding the problems, connecting with previous concepts, and JURNAL KEPENDIDIKAN, Volume 5, Nomor 2, 2021, Halaman 158-177   (Retnawati, Kartowagiran et al., 2017). The problems are caused, because, the students are not usual in facing problem-solving questions and narrative question or using real-world contexts (Retnawati, Kartowagiran, et al., 2017). Another research discussing the questions of task equivalent to the NE indicated that students have diffi culty in understanding images (Rumasoreng & Sugiman, 2014). Meanwhile, non-cognitive factors that aff ect student diffi culties include students' anxiety, running out of time, and giving up easily (Novferma, 2016;Tias & Wutsqa, 2015). The low absorption of NE from 2013 to 2017, such previous explained in the analysis results, it always related to the policy changing. Such previously explained, there had been a policy change in 2013 related to the implementation of the Curriculum 2013. The changing was followed by the number of indicators of problem-solving in the questions which increases (25-51%).
The conclusion obtained if compared to the result in 2008 to 2011 which was less than 20% of all material indicators. The diffi culties in 2013 to 2015 are in indicators that measure the problem-solving abilities. The result showed that the factor of the students' diffi culties in 2013 to 2015 was that they were less familiar with problemsolving questions. However, the results of the NE in 2012 (before the implementation of Curriculum 2013) showed the opposite result. With the number of 51% indicators of problem-solving questions, it did not fi nd a low absorption. Means, the factors of low absorption in 2013 to 2015 might not only be associated with the number of problem-solving indicators in the NE. Other factors, such as the quality of the question, are also the factors that distinguish the results (Sulistiawan, 2016) before and after the implementation of the new curriculum.

Figure 13
The Example of Questions on Composition Function in The NE on 2017

Figure 14
The Example of Questions on Calculus in The NE on 2017 ability can solve by training the students through the learning process. Various learning activities might be planned by teachers who train students' abilities in solving mathematical problems (Djidu & Jailani, 2017). The changing of Curriculum 2013 has also recommended the implementation of learning with various learning models such as problem-based learning Djidu & Jailani, 2018;Listiani, 2016) or project based learning (Anazifa & Djukri, 2017), or the other activities to train the students with mathematical problems (Djidu & Jailani, 2017). This means that the teachers' understanding of planning mathematics learning  is a key to solve the problem of students' diffi culties.
Meanwhile, the new policies on the changing in status and technical implementation of the NE in 2015 also influenced the absorption of NE on mathematics subjects. In this case, the status changing, and technical implementation of the NE were followed by decreasing the absorption of the NE. In this change, low absorption is not only on indicators of problem-solving abilities but also occurs in the indicators of questions in common concepts or standard questions. This showed that the policies issued in 2015 had a significant impact on decreasing the absorption of the NE, especially on mathematics subjects. I f v i e we d f r o m t h e t e c h ni c a l implementation, the implementation of computer-based exam is actually considered as better for national-scale assessments (Debuse & Lawley, 2016;Logan, 2015;O'Leary, Scully, Karakolidis, & Pitsia, 2018), due to it produces more valid results (Diaz Maggioli, 2018;Shute & Rahimi, 2017;Suwarto, 2017). However, the implementation of the computer-based exam also makes the extraneous cognitive load on students in answering questions, such as, when they are not familiar with using a computer (Logan, 2015). In addition, various technical problems in implementing the NE (Retnawati, Hadi, et al., 2017) also become one of the factors contributing to infl uencing the results obtained by students. This fact showed that the policy changing on the technical implementation of the NE also infl uence the absorption within students. However, aiming to fi nd out in more detail the difficulties through the implementation of computer-based exams, it needs further research.
The policy in the changing of NE status also had a negative impact on the absorption of the NE. Research in 2015 showed that the policy changing on the NE status in 2015 greatly aff ected the readiness of students (Saukah & Cahyono, 2015). The results indicated that schools with low grade are greatly aff ected by the changes in NE status. For example, there decrease the motivation of students in following the exams (Saukah & Cahyono, 2015). The results provide empirical evidence that the low absorption of the NE in 2016 and 2017 does not merely describe the students' diffi culties in completing the questions in the task of the NE. And, also, it supported by the low trend of absorption in the questions which apply simple concepts or common questions. Means, the low absorption is caused by the low motivation or seriousness of students in following the NE.

CONCLUSIONS
The phenomenon in 2013 to 2017 showed that the changes in education policy in Indonesia, especially the implementation of the NE, have an impact on reducing the absorption of NE, especially on mathematics subjects. The decrease in absorption is not only caused by cognitive aspects but also non-cognitive aspects of students following the NE. Generally, some materials such as probability, geometry transformations, solid geometry, calculus, circles, linear programs, and statistics, have low absorption or might categorize as diffi cult material in the NE on 2013 to 2017. The diffi culties, generally, are in the narrative questions of problem-solving or uses real-world context. The analysis results also indicate non-cognitive impacts causing by policy changes, especially the status and technical implementation of the NE, which has implications for the low absorption of the mathematics in the NE.
Even though the technical implementation of the NE also aff ects absorption, but these problems may solve by using computer-based assessments in schools. However, the changes in NE status that are no longer a determinant of student graduation still need to review because they affect non-cognitive aspects such as motivation or seriousness of students in following the NE. Therefore, it needs further research aiming to fi nd out more details about non-cognitive eff ects because of the changing of NE policies in Indonesia.