PAPS PREDICTIVE VALIDITY IN PREDICTING THE LEARNING SUCCESS OF MASTER OF PROFESSIONAL PSYCHOLOGY STUDENTS

This study aims to determine the ability of Tes Potensi Akademik Pascasarjana (PAPS Test) in predicting the grade point average (GPA) of the graduate students of the Master of Professional Psychology. PAPS test is the graduate entry requirement test in UGM. Graduate students’ GPA includes semester grade point average (semester GPA), cumulative grade point average (cumulative GPA), grade point average of master courses, and grade point average of professional courses. The data were obtained in the form of PAPS test scores (verbal subtest, reasoning subtest, quantitative subtest, and total score) and the GPA of the graduate students of Master of Professional Psychology of Universitas Gadjah Mada in the class of 2013, 2014, and 2015. The research subjects were 214 students. The analysis was carried out using correlation analysis. The results of the analysis generally indicate that the PAPS test is able to become a predictor of learning success for the graduate students in the Master of Professional Psychology of Universitas Gadjah Mada. The learning success consists of semester GPA, cumulative GPA, GPA of master courses, and GPA of professional courses. If analyzed per subtest, the quantitative subtest has the highest predictive power. On the other hand, the verbal subtest is only a good predictor for GPA of professional courses.


Introduction
Nowadays, continuing to graduate education seems to be a necessity for students. It is indicated by the increasing number of master and doctoral degree students in Indonesia from year to year. Based on the data on education development in 1999/2000-2013/2014 from the Ministry of Education and Culture in 2015, the number of students in the master's degree program has the highest growth of 7.15% per year from 40,309 to 65,366. Likewise, the increase also occurred in the doctoral degree program, which has a growth of 1.54% from 4,349 to 4,839 students (Ministry of Education and Culture, 2015).
The increase in the number of graduate students in Indonesia also happens at Universitas Gadjah Mada (UGM). This increase is indicated by the number of UGM's graduate students in 2002 reaching 12% of the total number of UGM's students, and in 2007, it rose sharply to 28% (Public Relations of UGM, 2007).
Similarly, the increase in the number of graduate students also took place in the Master of Professional Psychology of Universitas Gadjah Mada. Table 1   The vast public interest in continuing to the Master of Professional Psychology of Universitas Gadjah Mada (UGM) indeed cannot be separated from the good quality of education at UGM. UGM is a university providing many departments and study programs in the field of science in Indonesia. The Master of Professional Psychology of Universitas Gadjah Mada consists of 18 faculties and one graduate school with a total of 89 master study programs. Besides being extensive in the field of science, UGM also has good achievements in the international arena. It is indicated by UGM as the top 500 best universities in the world, according to QS World University Ranking in 2017/2018(Top Universities, 2017. It makes many people interested in continuing their graduate studies at UGM. Since the even semester of 2008/ 2009, UGM has required the applicants of the Graduate Program to take the Tes Potensi Akademik (TPA) or the Academic Potential Test. TPA is developed and implemented by BAPPENAS (Badan Perencanaan Pembangunan Nasional or National Development Planning Agency)or the Potensi Akademik Pascasarjana (PAPS) test, which is developed and implemented by the Unit Pengembangan Alat Tes Psikodiagnostika (UPAP) or Psychodiagnostic Test Equipment Development Unit, Faculty of Psychology UGM. TPA is used as a graduate entry requirement because it is specifically designed to explore a person's ability to deal with cognitive problems requiring a strategic and fast solution approach. This ability is a general reasoning ability supporting individual performance in solving problems. General reasoning skills are needed in the learning process in universities, where the acquisition of knowledge and skills is not much more determined by the method of delivery and order following the learning syllabus as it occurs at the middle and primary level of education (Azwar, 2016b).
This entry test requirements must be genuinely effective in determining the students predicted to have a high learning success in graduate studies. Moreover, UGM, with many interested applicants, must predict the learning success so that the decisions taken on the results of the tests do not disadvantage some parties. This test has 119 serious consequences for at least one stakeholder or often referred to as a high stakes test (Gregory, 2007;Stobart & Eggen, 2012). The intended consequences are for students and the university. For students, the test determines the future of an individual. If an error occurs, unselected prospective applicants (false negative), it affects the future of the students. The student should be able to enroll in a proper department and university according to his ability, but he should not choose another department or university because of the inability of the test in predicting his potential. Likewise, choosing applicants who have no successful prospects (false positives) will be detrimental to the faculty and students. The disadvantage is that choosing students who have no successful prospects will weaken the quality of the faculty, and the students will also have difficulty in attending lectures that have an impact on students' academic inefficiencies (Kuncel et al., 2001). Thus, the test used both for the entry requirements and graduate selection must be absolutely valid to predict students' success during the study.
Therefore, it is important to ensure that the measuring instruments in the student admission process in the graduate programs at UGM are truly able to predict the learning success of students by examining the predictive validity of the measuring instrument. Validity is the degree to which a test performs its measuring function or measures what intends to measure (Azwar, 2015b;Cohen et al., 2013;Gregory, 2007). Meanwhile, predictive validity is the extent to which test scores can be used to predict measure scores administered at a later date (Cohen et al., 2013;Gregory, 2007;Rudner, 1994).
One of the measuring instruments used in the student admission process at Universiats Gadjah Mada is the Tes Potensi Akademik Pascasarjana (PAPS test). PAPS test is a cognitive ability test specifically designed to reveal the academic potential underlying the possibility of the applicants to be successful if he gets the opportunity to learn more in university, especially in the master and doctoral programs (Widhiarso, 2017). PAPS test is developed using the intelligence theory foundation of Spearman's g factor (general cognitive ability). This general cognitive ability is needed by students to study in higher education. Therefore, the PAPS test is used as one of the graduate entry requirements and is expected to be highly correlated with the indicators of learning success in university (Azwar, 2008).
Related to the use of TPA in the student admission process, the aspects of validity, especially predictive validity, are important for predictive accuracy (Azwar, 2008). Tests that have been examined for validity are effective tests for the student admission process because they have good predictive functions (Azwar, 2015b;Kuncel et al., 2001). An effective acceptance process can maintain qualified candidates and improve the quality of the department (Kuncel et al., 2001). Thus, it is important to look at the PAPS test predictive validity as an admission requirement for the graduate programs at UGM.
The importance of research on predictive validity, especially for tests determining the individual's potential, is also indicated by many studies related to testing of predictive validity. For the PAPS test, several studies had been conducted on predictive validity. The research by Pradipta et al. (2017) shows that TPA scores in both TPA Bappenas and PAPS tests can predict the learning success of the master's degree students at UGM. In addition, the research by Nurhayati and Widhiarso (2017) examines the specific predictive validity of the PAPS test and shows that the test predicts the learning success in the Master of Psychology of UGM. From the research by Nurhayati and Widhiarso (2017), it is suggested to conduct research towards other study programs.
Other research also stated the importance of conducting predictive validity in other study programs. Academic potential test scores, such as Graduate Record Examination (GRE), demonstrate varying pre-dictive validity, depending on the area of study (Goldberg & Alliger, 1992;Sternberg & Williams, 1997;Thornell & McCoy, 1985). Although there are many similarities in fundamental assignments for the students, there are still many differences in curricula, output, success measures, assessment standards, and evaluation criteria in different academic areas (Mupinga & Mupinga, 2005;Wang, 2013). Therefore, the results of predictive validity research produce valid conclusions for several purposes and several contexts, but cannot be generalized for all purposes and contexts (Powers, 2004). Thus, it is necessary to consider the predictive validity research on TPA in different programs, even on closely related study programs (Wang, 2013). Therefore, this research aims to conduct a PAPS predictive validity test on the learning success of the graduate students of the Master of Professional Psychology of Universitas Gadjah Mada.
More specifically, the PAPS test criteria used in this study have two parameters, namely the semester grade point average (semester GPA) and cumulative grade point average (cumulative GPA). These criteria are used to see the trends in the relation between PAPS test and learning success. In particular, the trend analysis aims to look at PAPS predictability over time during the academic semester (Wao et al., 2016). The criteria are also used with the aim of contributing to the literature focusing on long-term performance, while long-term performance is a more informative criterion than short-term performance (Wao et al., 2016). The GPA referred to in this study is not only the overall GPA from the beginning to the end of the semester but the accumulation of each semester. For example, cumulative GPA 2 is an accumulation of the first semester GPA and the second semester GPA.
In addition, other criteria used are the GPA of master courses and GPA of professional courses. In the Master of Professional Psychology, there are master courses that cover Philosophy, Research Method courses, and Thesis, and then, there are professional courses such as Observation, Interview, and Assessment courses. Professional courses require more empathy and skill, while the PAPS test measures cognitive abilities more. Therefore, the researchers tested the ability of PAPS in predicting GPA of Master and GPA of professional courses.
Based on the aforementioned description, this study aims to examine further the extent to which PAPS test scores can predict learning success as indicated by semester GPA, cumulative GPA, GPA of master courses, and GPA of professional courses for graduate students of Master of Professional Psychology at UGM. The research problem is formulated as follows: (1) how far PAPS test can predict semester GPA courses for graduate students of Master of Professional Psychology at UGM; (2) how far PAPS test can predict cumulative GPA courses for graduate students of Master of Professional Psychology at UGM; (3) how far PAPS test can predict GPA of master courses and GPA of professional courses for graduate students of Master of Professional Psychology at UGM.

Research Method
Research regarding the predictive validity involves at least one test variable as a predictor and one other variable as a validation criterion. In this study, the following variables were determined. The predictive variable in this study consists of PAPS test scores of graduate students of the Master of Professional Psychology at UGM. The PAPS test scores are verbal subtest score, quantitative subtest score, reasoning subtest score, and PAPS test total score. Then, for the criterion variables in this study, there are four parameters, namely semester grade point average (semester GPA), cumulative grade point average (cumulative GPA), GPA of master courses, and GPA of professional courses. Semester GPA consists of first semester GPA (semester GPA 1), second semester GPA (semester GPA 2), third semester GPA (semester GPA 3), fourth semester GPA (semester GPA 4), and fifth semester GPA (semester GPA 5). Cumulative GPA consists of cumulative GPA 2 (accumulation of semester GPA 1 and semester GPA 2), cumulative GPA 3 (accumulation of semester GPA 1, semester GPA 2, and semester GPA 3), cumulative GPA 4 (accumulation of semester GPA 1, semester GPA 2, semester GPA 3, and semester GPA 4), and cumulative GPA 5 (accumulation of semester GPA 1, semester GPA 2, semester GPA 3, semester GPA 4, and semester GPA 5).
The study data were in the form of a total PAPS test score and GPA of graduate students of Master of Professional Psychology of UGM in 2013, 2014, and 2015. Data were obtained from the Unit Pengembangan Alat Tes Psikodiagnostika (UPAP) UGM and Master of Professional Psychology UGM.
To determine the relation of predictor variables, namely PAPS test (verbal subtest score, quantitative subtest score, reasoning subtest score, and total score) with GPA of graduate students of Master of Professional Psychology of UGM (semester GPA, cumulative GPA, GPA of master courses, and GPA of professional courses), correlation analysis was used. The predictor variable is the PAPS test score, and the criteria variable is the Students' GPA. Table   2 presents a guideline for interpreting uncorrected correlation coefficients in the study of predictive validity based on simplified guidelines from the US Department of Labor, Employment Training and Administration cited by Emery (2007) in (Azwar, 2016a).

Findings and Discussion
A predictive validity test was done by correlating predictors with the criteria to obtain a predictive validity coefficient (Rudner, 1994). The PAPS predictive validity test is indicated by the linear correlation coefficient between verbal subtests, quantitative subtests, reasoning subtests, and PAPS test total scores as predictors with students' GPA as the criteria. The results of simple linear correlation coefficient computation are presented in Table 3. The analysis shows that the highest correlation is the correlation between reasoning subtests and semester GPA 4 of 0.294. On the other hand, the lowest correlation is the correlation between verbal subtests and semester GPA 2 of 0.025. From the results of the analysis, it can also be revealed that the correlation between the PAPS test score and the grade point average generally has a positive correlation up to the third semester. Nevertheless, in the fourth and fifth semesters, there is a negative correlation. In addition, it can also be seen that general quantitative subtests have a higher correlation number than verbal subtests, reasoning subtests, and also the total score. However, verbal subtests generally have lower correlation rates than quantitative subtests, reasoning, subtests, and also total score.
The results of the analysis also reveal that the grade point average of the master courses compared to the grade point average of the professional courses has a higher correlation with quantitative subtests, reasoning subtests, and total scores. On the other hand, the grade point average of the professional courses compared to the grade point average of the master courses, has a higher correlation with verbal subtests.
From the results of the analysis, the correlation coefficient between predictors (the PAPS test and criteria) can be seen, namely, the students' GPA. The correlation coefficient obtained turns out that some of the numbers are so close to zero that there is no linear relation between the GPA and the PAPS test score. In addition, there is a correlation coefficient that has a negative mark.
Then, the researchers correct the restriction of range in order to increase the accuracy of the estimation of the coefficient of actual predictive validity. Positive correlation coefficients in Table 2 were corrected for restriction of range on the criteria, namely students' GPA. The correlation coefficients marked negative, even though the numbers are large, are not included in the analysis because negative numbers have no meaning in interpreting validity (Azwar & Ancok, 2008). Therefore, in this study, corrections for restriction of range on the criteria were carried out because the criteria were more homogeneous than the predictors. The formula used is as follows:
rR was obtained from correlation coefficients in Table 2. sR was obtained from empirical standard deviations. sU was obtained from a hypothetical standard deviation calculation. The hypothetical standard deviation was calculated by dividing the GPA range by six units of standard deviation (Azwar, 2015a). The range of GPA is 4 divided by six units of standard deviation resulting in a hypothetical standard deviation of the GPA of 0.67. The computational results of the corrected correlation coefficients are presented in Table 4. This study is intended to see the predictive power of the PAPS test on the learning success of graduate students of Master of Professional Psychology of UGM. The PAPS test has been used as an entry requirement for the Graduate Programs at UGM. Therefore, it is important to examine the PAPS test's ability to predict the learning success as reflected in the students' GPA. If PAPS test has good predictive power, it will help attract prospective students who have the potential to succeed so that they can maintain the qualified applicants and improve the quality of study programs (Kuncel et al., 2001).
The small correlation coefficient, as presented in Table 3, can be caused by homogeneous groups. Homogeneous groups can be seen from the small differences in the observed scores, showing the narrow spread of numbers and also the lack of diversity in numbers (Azwar, 1996;Furr & Bacharach, 2014;Kurpius & Stafford, 2006). The correlation coefficient indicates how well the point in the scatters plot runs along the line. If the group is homogeneous, then the point in the scatter plot only collects on a certain line and does not run along the line. It has an impact on the low correlation coefficient (Allen & Yen, 1979).
Seeing whether the research data is homogeneous or not can be identified from its variability. In statistics, there are three measures showing the variability or diversity of numbers in a distribution. The three measures are range, standard deviation, and variance. Range, or distribution distance, is the difference between the biggest score and the smallest score. As a variability measure, the range is actually very unstable and easily provides false information about actual variability. It is because if there is one extreme number, the range will change dramatically while the overall variation in numbers is not much affected by only one extreme number. The standard deviation is the average number deviation from the mean. The variance is the sum of squares of deviations from the mean (Allen & Yen, 1979;Furr & Bacharach, 2014;Kurpius & Stafford, 2006).
In the data, it can be seen that the standard deviation for the PAPS test score and the students' GPA value is relatively small. The standard deviation of the PAPS test is 100. In this study, the standard deviation moves between 52.83 until 86.43. The standard deviation in this study is below the standard deviation of the PAPS test. Then, the standard deviation of students' GPA moves between 0.18 until 0.82. A small standard deviation indicating the group of the sample of this study is homogeneous. This homogeneous group influences the magnitude of the coefficient of the validity of measurement results (Azwar, 2016a;Sackett et al., 2002).
In predictive validity, it is often defined as a restriction of range. Restriction of a range is a decrease in the variance of the sample score in both the independent and dependent variables so that the score distribution becomes more homogeneous compared to the previous variant. This restriction results in a low correlation coefficient or predictive validity coefficient (Allen & Yen, 1979;Sackett et al., 2002).
Such cases usually occur when groups that initially have a wide variety of abilities then become "selected" groups (Azwar, 2015b). Likewise, when a large number of prospective students participating in the admission process of Master of Professional Psychology of UGM were selected, only a small number of them were accepted, namely those who met the entry requirements and had the best scores on aspects of the admission test. When undergoing an admission test, the variation in their scores was large because prospective students consist of students with very diverse or varied abilities. Therefore, if these all could be accepted as students, the GPA would also vary greatly. However, only those with the best abilities were selected in the selection, namely those who met the entry requirements and the highest admission test scores. Hence, groups of students who passed the selection are more homogeneous subject groups and resulting in restriction of range. This restriction of range resulted in underestimating the true coefficient of validity. Thus, we must correct the restriction of range to increase the accuracy of the estimation of the coefficient of actual predictive validity. The result of Coefficient Validity Predictive (Corrected for Restriction of Range on the Criteria) can be seen in Table 4.
Considering the categorization of the validity coefficient meaning according to the US Department of Labor, Employment Training and Administration (Emery, 2007) in Azwar (2016b) which had been simplified, all validity coefficients smaller than r = 0.300 is stated not satisfying and the test is declared unable to produce a valid measure. From the calculation of the correction for restriction of range, it can be seen that there is an increase in the correlation coefficient. It further supports that the group in this study is homogeneous. Therefore, the small number of correlation coefficients or predictive power in this study is not because there is a problem with the predictor but is caused by a very homogeneous group. If the group is more heterogeneous, then it will produce higher PAPS test predictive power towards students' GPA. It is in accordance with previous research by Azwar (2008), which found a low relationship between TPA version 07A and the first-year GPA in Master of Professional Psychology of UGM, Class of 2007. It is because the first-year GPA used as a validation criterion is very homogeneous. This condition resulted in no estimation of true predictive validity because computational validity resulted in underestimation. The low coefficient of validity occurred not because of the problem in the predictor but rather from the problem of reliability of the criterion score that was worth questioning and the variability of the predictor and also the criteria score, which is equally homogeneous.
Thus, it appears from the computational results in Table 4 that, in general, the coefficient of validity obtained is greater than r = 0.300, except for verbal subtests. More specifically, the total PAPS test score serves as a valid predictor of the students' GPA, except for semester GPA 3, semester GPA 4, and semester GPA 5. Verbal PAPS test subtitles only function as valid predictors of the GPA of Professional courses. In addition, it can also be seen that the quantitative subtests on the PAPS test are valid predictors for the students' GPA, except for semester GPA 3, semester GPA 4, and semester GPA 5. The reasoning subtests serve as valid predictors for students' GPA, except for semester GPA 3 and semester GPA 5.
In addition, it can be seen that the PAPS test is more consistent if the cumulative GPA is used as a criterion. It can be explained that the learning success of students is reflected by overall achievement, so the use of Cumulative GPA as a criterion is more precise than use semester GPA (Azwar & Koentjoro, 1985). The results of the analysis also show that quantitative subtests are generally the best predictive power compared to other subtests in predicting semester GPA and cumulative GPA. On the other hand, verbal subtests do not predict semester GPA and cumulative GPA. It can be explained that in previous studies, it was also found that verbal abilities such as those in Indonesian and English test courses did not have predictive power towards learning achievement as well as quantitative abilities (Azwar & Koentjoro, 1985).
The Azwar and Ancok (2008) study of differential predictive validity in the examination in the admission process or Ujian Masuk of UGM (UM UGM) also shows that quantitative subtests in the potential academic test at the Faculty of Psychology have a higher predictive power (r = 0.263), compared to verbal subtests (r = 0.152) and reasoning subtests (r = 0.015). The reasoning ability that is revealed through skills in finding more quantitative solutions is needed to undergo success in psychology studies rather than reasoning skills through understanding the meaning of words.
More specifically, in the Master of Professional Psychology of UGM, in semester GPA and cumulative GPA, there are master courses and professional courses. Quantitative abilities are used both for basic master courses and professional courses. Quantitative abilities are used to understand the results of research, especially in the interpretation of the results of quantitative analysis. Quantitative ability is also used to determine statistical analysis used in accordance with research problems in the experimental research design. Then, in the basic professional courses, the students will accumulate behavior from observation, words from interviews, and calculate scores from psychological tests to make a diagnosis. On the other hand, verbal ability is only needed in professional courses that require the students to write reports for all of their subjects.
In understanding references for master and professional subjects, reasoning skills rather than verbal abilities are needed. It is because, in graduate studies, the students need not only understanding reading but also a more critical attitude towards existing references by drawing conclusions from theory and then used to interpret cases. It requires more reasoning skills, not only the ability to interpret words as measured by verbal abilities. Therefore, quantitative ability predicts semester GPA and cumulative GPA in the Master of Professional Psychology of UGM. In contrast, the verbal ability does not predict semester GPA and cumulative GPA in the Master of Professional Psychology of UGM (Azwar, 2016b;Master Program of Professional Psychology, 2013).
However, when distinguished into a master and professional degree course, the total score has the highest predictive validity. Then, the verbal ability can predict professional courses. Verbal ability can predict the learning success in psychology, especially in the Master of Professional Psychology, only in courses that meet directly with clients. When students meet with clients to assess and provide therapy, and verbal abilities predict their success. On the other hand, when they learn about theories in psychology research and carry out theses, verbal abilities do not predict their learning success. In addition, in the professional courses, the students always make psychological reports in each of their subjects. The students describe the conclusions of the assessment results, observations, interviews, and psychological tests in the form of psychological reports. The students also make reports on the results of psychological interventions that have been given. Therefore, verbal ability is used in professional courses to interact with clients, arrange good sentences, and collect the results of interviews. On the other hand, verbal ability is not needed for master courses because, in the master courses, there tends to be more theory and does not meet with clients to collect the results of interviews (Master Program of Professional Psychology, 2013).
This study found that the initial analysis shows a low correlation between the PAPS test and the GPA of Master of Professional Psychology of UGM. However, after correction for the restriction of range on the criteria shows that the PAPS test has good predictive validity on the grade point average. It shows that the criteria of a homogeneous grade point average have an impact on the low correlation coefficient. The students' GPA of the Master of Professional Psychology of UGM has a relatively high score because, indeed, the students are selected students and have the best scores. Therefore, the low correlation coefficient in the initial analysis is not caused by the inability of the PAPS test to predict students' GPA, but rather due to homogeneous criteria score data. Thus, in general, the PAPS test can be a good predictor for the students of the Master of Professional Psychology of UGM.
The limitation of this study is vulnerability to commission bias. PAPS test is tested both for enrolled applicants and applicants who were not accepted. However, in this study, only the prospective students were accepted. It causes that the predictive validity of the PAPS test cannot be empirically known for all applicants of Master of Professional Psychology of UGM, both those who are accepted and those who are not accepted. In addition, another limitation is that this study only focuses on predictive validity. Therefore, this study can only answer whether the PAPS test is valid or not as a predictor of the learning success of the Master of Professional Psychology of UGM. However, this study did not examine further how far the PAPS test was able to predict the learning success in the Master of Professional Psychology of UGM.

Conclusion
Based on the results of this study, it can be concluded that in general the PAPS test is able to be a predictor of the learning success of the graduate students of Master of Professional Psychology of UGM and can continue its use as an entry requirement for Master of Professional Psychology of UGM. Learning successes referred to in this research include semester GPA, cumulative GPA, GPA of master courses, and GPA of professional courses. If viewed per subtest, quantitative subtest has the highest predictive power. On the other hand, verbal subtests are only good predictors of the GPA of professional courses.
Based on the results of the study, some advices can be given. For the master of Professional Psychology of UGM, as the party using PAPS test as an entry requirement can continue its use because it is proven that the PAPS test is able to be a predictor of the students' learning success of the master of Professional Psychology of UGM. The results of this study also show that quantitative subtests predict the most learning success in the Master of Professional Psychology of UGM. Therefore, in the use of the PAPS test in the admission process in the Master of Professional Psychology of UGM, in addition, to see the total score, it is also seen whether the quantitative subtest score had a high score. Then, for the Unit Pengembangan Alat Tes Psikodiagnostika (UPAP) of the Faculty of Psychology UGM, to maintain the quality of the PAPS test both in terms of test content and test administration. For further researchers, it is expected to conduct research related to the PAPS predictive validity in other study programs because the results of this research can only be used for the Master of Professional Psychology of UGM.