ITEM ANALYSIS OF A CRITICAL THINKING SKILLS TEST ON PLC PROGRAMMING FOR PROSPECTIVE TEACHERS

This study aimed at developing a test instrument to asses prospective vocational teachers’ critical thinking skills on Programmable Logic Controller (PLC) programming. This research and development study produced the test instrument that had been trialled to 68 Indonesian electrical engineering education prospective teachers at Universitas Negeri Surabaya. The method used was Research and Development by referring to the revised Bloom method and the adjusted Facione model. The instrument consisted of 40 items that were further analyzed using a classical test and item response approaches. Students' responses were analyzed using the computer program of a Test Analysis Program and ConQest. The item difficulty of classical test theory was represented by p value revealed based on prospective teachers’ correct answers. Parameter b referred to location parameter that conveyed the position of the item’s characteristic curve in relation to the ability scales. The benefit of this study was it can measure the level of critical skills, especially on the subject of PLC programming. The existing instruments only measure PLC programming skills in general but have not yet specifically reached the measurement of critical skills level.


INTRODUCTION
The 21 st century challenges young people to become good learners who are persistent to always learn from mistakes they have ever made [1]. They also must be independent not because of the teacher's orders or schoolworks, but due to learning on their own encouragement. They must be familiar with the flexible learning process, use different strategies, and keep abreast of the rapid changes of the world. If they are able to achieve changes quickly and independently, then the teacher must be able to compensate for these changes, of which effort is by becoming a lifetime learner.
Meanwhile, educators who are members of the National Education Association (NEA) [2], based in the USA, see that critical thinking and problem solving, communication, collaboration and creativity, and innovation are the four most sustainable competencies as variables demanded by every company and needed by every professionals. The competencies were then wellknown as the Four Cs. The students must be prepared with those four competencies [2]. Regardless the existence of various other competencies, these four competencies are a much stronger focus for teacher assignments in the 21 st century education. There are four basic competencies that students must have in order to conceive critical thinking skills well, that is, being able to deliver logical arguments effectively, to think systemically, to make conclusions, and to do problem solving effectively.
Graduates of the bachelor program of Electrical Engineering Education at Universitas Negeri Surabaya (Unesa) were prepared to become vocational school teachers. The curriculum used referred to the Rusimamto et  Director of Vocational Development Regulation on the implementation of the vocational secondary education curriculum [3]. One subject named the Program Control System became the required competency in industrial automation technical expertise with Programmable Logic Controller (PLC) material [4]. Given the importance of PLC material to vocational students, the PLC material was included in the curriculum, which covered theoretical and practical learning activities. PLC is also a research that is still being discussed, both its use in the industrial world [5], and the application of PLCs that have been controlled via the internet [6], [7].
With 21 st century demands on teachers who must prepare students to have critical thinking, communication, collaboration, and creativity and innovation competencies, the bachelor program of Electrical Engineering Education must equip prospective teachers to compensate for changes that might occur quickly and became the front line in these changes. Moreover, they must have the knowledge, skills, and support to become an effective 21 st century teacher. One of the knowledge and skills that must be possessed by vocational prospective teachers from the bachelor program of Electrical Engineering Education was PLC programming competence. PLC learning emphasized the prospective teachers' understanding, knowledge, and skills towards the achievement of PLC learning according to the demands of the 21 st century. The process and assessment of PLC learning must be oriented to developing four competencies namely critical thinking, communication, collaboration, and creativity and innovation.
Problems and gaps that arise lead to the need for critical skills in learning PLC Programming thus an instrument that can measure these critical skills with is required.
The existing measurement is only limited to cognitive knowledge and skills that are general in nature. There appears to be lack of studies that investigate the measurement of critical skills in depth, especially on the subject of PLC Programming. By referring to the above case, this study was conducted by aiming to construct a test instrument that assessed the students' critical thinking skills in the bachelor program of Electrical Engineering Education on PLC programming competencies.
Critical thinking skills become the purpose and reason in solving problems, formulating conclusions, calculating possibilities, and making decisions that refer to the use of cognitive skills or strategies to improve desired results [4]. Halpern [8] uses an empirical four-part model to guide, teach, and learn critical thinking, namely the disposition component to prepare students for cognitive work, instruction in critical thinking skills, training in structural aspects of problems and arguments to promote transcontextual transfer of critical thinking skills, and metacognitive components that include checking accuracy and monitoring progress towards goals.
In addition, according to [9], critical thinking has many similarities with scientific thinking. Three core components of critical thinking encompass analyzing and interpreting the text (including consideration of context, genre, source, etc.), evaluating arguments, and presenting arguments. In solving problems, the following elements are needed such as analyzing, evaluating, concluding, justifying, explaining, and developing further arguments. The skills are acquired, enhanced, and assessed based on the performance. Critical thinking skills are likely to be obtained from various sources. It is possible that the source of non-academic critical thinking is the main cause of better decision making. It is also possible that explicit instruction in critical thinking causes students to transfer the skills taught in class into their lives. Finally, it is not known whether critical thinking affects all aspects of decision making equally or whether certain decisions (e.g. financial and health decisions) benefit more from the application of critical thinking skills than other decisions (e.g. interpersonal decisions) [10]. Cognitive skills here that become the core of critical thinking include interpretation, analysis, evaluation, conclusions, explanations, and selfregulation [11], which are shown in Figure 1. Interpretation is aimed to understand and express the meaning of various kinds of experiences, situations, data, events, judgments, conventions, beliefs, rules, procedures, or criteria. Interpretations include sub-categorization skills, decoding significance, and clarifying meanings. Moreover, analysis is essential in identifying inferential and intended relationships between statements, questions, concepts, and descriptions. Analysis is another form of representation, which is intended to express beliefs, judgments, experiences, reasons, information or opinions.
Evaluation is to assess the credibility of other statements or representations or descriptions of one's perceptions, experiences, situations, judgments, beliefs, or opinions. This also aims to assess the logical strength of actual inferential relationships or intended between statements, descriptions, questions or other forms of representation. Further, inference is used to identify and secure the elements needed to draw reasonable conclusions. This also aims to form conjectures and hypotheses, consider relevant information, and decide the consequences that flow from data, reports, principles, evidence, judgments, beliefs, opinions, concepts, descriptions, questions, or other forms of representation. Some subskills of inference involve proof of demand, suspect alternatives, and draw conclusions. Explanation is able to present the results of one's reasoning in a convincing and coherent manner. Several sub-skills of explanation describe the methods and results, justify procedures, propose and defend an event or a point of view with good reasons that are both causal and conceptual, and present systematic reasoning or arguments in the context of seeking the best understanding. Self-regulation refers to selfconsciously monitoring one's cognitive activity, the elements used in the activity, and the results especially by applying skills in analysis and evaluating their own inferential judgments by approaching a view toward questions, confirming, validating, or correcting reasons or results.
The results of the study conducted by [12] resulted on a model of critical thinking skills in ESP engineering courses. It must be pointed out that using class discussions gave students the opportunity to work in collaborative and cooperative group settings. The discussion allowed students to increase their participation, to obtain interpersonal and oral communication skills, and to develop a better understanding of the problems at hand. While the results of the study conducted by [13] used descriptive research approach carried out with survey models. Developed questionnaire consisting of 21 questions with five-points Likert's model was used to measure students' critical thinking skills. The study samples were 722 high school students from four different types of schools in the 2010-2011 education year from Hakkari. The Cronbach's Alpha coefficient from the questionnaire was 0.87. Based on the F-test, the t-test and the mean value of the significance level were 0.05. The survey results revealed that students' critical thinking skills varied between types of schools namely Ana Tolian High School, Science Middle School, and Vocational High School.
The results of the study carried out by [14] showed that the CCTST (California Critical Thinking Skills Test) and some similar instrument extensions actually measured testing abilities and did not measure critical thinking skills. In qualitative studies, engineering scholars were interviewed about how they used critical thinking. It was found that their descriptions were more complex than the typical definitions in the literature. Overall it showed the need for further research on critical thinking. Wechsler said that there was difference and independence of creativity and critical thinking in cognitive performance, even though both build small roles but complement each other in the different creative problem solving phases [15]. Critical thinking can be considered associated with a type of thinking that is less dogmatic and more relativistic, which indicates the need to think flexibly. While creativity prefers inductive reasoning especially in identifying regularities and principles or making conclusions about relationships, which do not occur in relation to logical reasoning, focus on finding the best unique answers.
Critical thinking is generally recognized as an important skill and becomes the main goal of higher education. However, there is very little in the literature about critical thinking skills in the field of engineering. Based on what has been stated above, this study focuses on presenting critical thinking models in the engineering class by applying the Facione model that suggests 6 skills in which each has its subskills. These 6 skills include interpretation, analysis, evaluation, inference, explanation and self-regulation. Table 1 shows the skills and sub-skills of critical thinking [11]. Several studies that have applied importance of critical skills in learning state that students become more understanding and better at decision making [12]- [19] [20]. The test measured the critical thinking ability of PLC programming by referring to revised Bloom and to adjusted Facione models. The test assessed topics according to the description of Lesson Plan also known as RPS, which conducted an assessment and provided understanding of control systems, PLC hardware, internal architecture, PLC input or output, ladder diagram, basic instructions, function block diagrams, CX-Programmer and HMI, PLC programming on a simple automation system, PLC programming simulation on a simple automation system and lighting settings using a PLC. Students' responses to 40 items were then analyzed using the computer program Test Analysis Program [21] and ConQest [22].
The Program Analysis Test was used to estimate the parameters of the difficulty of item (p) based on classical test theory, while ConQest was used to estimate the parameters of the difficulty of item (b) based on item response theory. The test results were analyzed using Rasch model item response theory. Fit analysis was used to check the unidimensionality of the items in the PLC test. Unidimensionality was a very important assumption in item response theory. Unidimensional test items meant that the items measured one ability [23]. One indication of unidimensional test items was the data fit with the Rasch model [24]. To find out whether the Rasch model could predict the response of each respondent using mean-square infit and mean-square outfit statistics [24]. Infit Mean Square (IMS) and Outfit Mean Square (OMS) statistics became a measure of the degree of conformity between observation data and predicted values by the model.
The test items used in this study were in accordance with the Rasch model. Test items were selected based on the value of IMS and OMSs. Linacre compiled a table to interpret the meaning of the values of IMSs and OMSs [25]. Table 2 shows the meaning of the values of IMSs and OMSs. The test items used had IMS and OMS values ranging from 0.5 to 1.5.

RESULTS AND DISCUSSION
The test consists of 40 items in which the responses to the items were analyzed using the computer QUEST program [26]. Table 3 shows the results of estimated parameters of the type of response. There were 40 parameters of the difficulty of item (b).
The item difficulty parameter was the point on the capability scale where the probability of correct answer was 0.5. These parameters were location parameters, which indicated the position of the item characteristic curve in relation to the scale of ability. The greater the item difficulty parameter value, the greater the ability required by the test participant to get 50% chance of answering the item correctly. The leftmost curve had a location of -2.876 and the rightmost curve had a location of 4.288. According to the classic test analysis, the difficulty of the item had a range from 0.234 to 1.873. Figure 2 shows the relationship between the values of items. The correlation between the values of b and p was -0.738.  The important characteristics of a set of test items that measured a construct were the dimensions. In Rasch analysis, if all the coherent items formed a scale, then the items were dimensions. Keeves and Masters proposed that item fit was used to examine unidimensionality [26]. Fit items were analyzed using the Quest program [27]. There were two sizes of item fit, namely Infit Mean Square (IMS) and Outfit Mean-Square (OMS). Table 4 shows the results of the analysis. The IMS and OMS values were close to 1.0 that indicated small distortions for the measurement system. The value of 1.0 was the expectation value. If the observation data was fit with the Rasch model, the expected value was 1.0. The IMS and OMS values listed in Table 5 showed that almost all items of the test were useful for measurement. There was no IMS value above 1.5. All items were fit with no value below 0.5.
The results show that the instrument used can measure students 'critical skills, where the instrument has problems that students must solve, where with these problems where we can access students' abilities in problem solving, communication, collaboration and creativity, and innovation. The making of this instrument is not easy because it has to go through validation of the test questions to be used.
This finding is in line with research [9] where critical thinking has many similarities with scientific thinking. Three core components of critical thinking encompass analyzing and interpreting the text (including consideration of context, genre, source, etc.), evaluating arguments, and presenting arguments. In this instrument students are required to analyze and interpret, communicate with the pather, collaborate, and be creative and innovative to solve tea problems. This is also in accordance with research [10] where interpretation, analysis, evaluation, conclusions, explanations, and self-regulation capabilities are needed. Self regulation in this study is defined as how students look for solutions, time management, choice of tools and materials, selection of algorithms, and risk management of the resulting PLC programming.
Constribution of this study is the development of test instruments that can measure the level of critical skills, especially on the subject of PLC programming. The existing instruments so far still measure PLC programming skills in general but have not yet reached the extent to which to measure the critical skill level.

CONCLUSION
The item difficulty statistic in the classical test theory is the p value, which is calculated based on the proportion of test takers who correctly answer a test item.
Parameter b is the location parameter, which shows the position of the item characteristic curve in relation to the scale of ability. The correlation between the values of b and p is -0.738. Of the 40 PLC test items, everything is fit in terms of the Rasch model. Parameter b has values ranging from -2.876 to 4.288. The parameter difficulty of item p had values ranging from 0.234 to 1.873. The instrument used can measure students' critical skills components, namely the ability of interpretation, analysis, evaluation, conclusions, explanations, and selfregulation. Self regulation in this study is defined as how students look for solutions, time management, choice of tools and materials, selection of algorithms, and risk management of the resulting PLC programming. The implementation in this measurement has a time constraint to access the students' abilities and the ability to do the tests must have very good PLC programming competence. Future research is how to measure students' metacognition in solving problems in the field of PLC programming and define the suppoting component.