The respondent factors on the digital questionnaire responses

Progress in the field of technology often facilitates human work. One of them is progress in the development of questionnaire modes. Currently, existing questionnaires have been based on a digital platform, which makes evaluators easy to design, disseminate, and conduct scoring. All are computerbased, making them reachable by the respondents no matter how far the location of the respondent is, as long as they are connected to the internet. However, any progress is accompanied by several obstacles. For example, the respondents experienced an error in responding to having the intent to respond 'Yes' option but pressing the 'No' button instead. It is very different from filling in paper and pencil based questionnaires in which they are sure to put a checkmark using a pencil on the answer choices. This problem is what the researchers found when distributing digital questionnaires to participants of the National Questions Writing Program based on the 'SIAP' (Sistem Inovatif Aplikasi Penilaian) application. On conditional questions (if you choose 'No', please stop), some respondents who have chosen 'No' answers still respond to the next questions. It causes the data obtained are unreliable. After conducting a more in-depth analysis, the researchers found that respondents’ factors as psychological factors are the cause, such as the new experience of accessing applications, understanding of applications, stress, and personal health. Uniquely, the respondents who have problems are those in the context of productive age, i.e 30 to 39 years old, more than five years of teaching experience, postgraduate level, and female.


Introduction
Progress in the field of assessment is increasingly rapid. Assessment is no longer carried out in conventional ways, such as paper-based test (PBT) but has been in the form of a computer-based test (CBT). Many experts argue that well-designed computerbased assessments are considered more comprehensive, more accurate, and able to describe the profile of each test participant (Nezami & Butcher, 2000). Computer-based assessment is also the first step in the test model for the future (Yanxia, 2017) by not ignoring other aspects of the application that can cause the computer to make an error in processing the response given (McArthur & Choppin, 1984).
Errors are predictable and deserve more attention than the application of CBT, not only errors from the application side but also those from the respondent's side. Web-based questionnaires, as one of the other forms of application of CBT, as they have been used in Australia (Nulty, 2008), also need to pay attention to these errors. Before being applied, it is better for prospective respondents to be given socialization, training, and how to use the application (Sevillano-García & Vázquez-Cano, 2015).
Web-based questionnaires, also known as digital questionnaires, need to pay attention to some aspects such as questions asked and visual design so respondents are more aware in answering so that there are no mistakes when answering. Excessive use of media, such as quality-check reminders, will take a long time which can make the respondents frustrated and leave the questionnaire prematurely (Reja, Manfreda, Hlebec, & Vehovar, 2003).
It also happened to the web-based questionnaire used by the Center for Educational Assessment or Pusat Penilaian Pendidikan (Puspendik) as an instrument for measuring the level of satisfaction with the implementation of the national question writing application based on SIAP. As a new program that has only been operating since 2017, SIAP needs to be evaluated to see its effectiveness.
Evaluation of the participants' satisfaction level was done using a web integrated with the SIAP application. On the questionnaire's initial page, it was stated that the questionnaire was arranged to increase the capability of SIAP in providing information technology services to users, especially writers, reviewers, and material administrators. There are two types of responses provided in this questionnaire: Yes/No answers and gradation of five Likert scales approval levels (SIAP Application Development Questionnaire, n.d.).
It is interesting to find based on the response that there were respondents who still answered follow-up questions that they should not answer if they chose the response 'No'. An example can be seen in Table 1.  Table 1 is the result of question data regarding communication with the person in charge (PIC) of the subject matter. Of the 200 respondents, 92 people answered that they did not communicate with the PIC of the subject matter. However, of the 92 people, 30 of them (32.60%) still answered the follow-up questions, namely regarding clarity of indicators, improvements, and rejections.
It shows that web-based questionnaires require the ability of 'extensive reading' rather than 'speed reading'. In fact, more than 60% of respondents are postgraduate students. Thus, the habit of delaying the task of reading during education (Onwuegbuzie et al., 2004) or the habit of skimming reading (Cheng & Tsai, 2017) can be the cause. The respondents who graduated from the Bahasa Indonesia major did not show a much different pattern. In addition, most of them are male. Indeed, the results of the study by Parquette (1952) show that men find it difficult to understand a text in the first chance of reading.
Data that are duly not filled, but still get the response, as mentioned earlier, is said to be data outliers or outlier data. These data cause the shrinking parameter to be zero so the model used becomes inappropriate, or there is a bias in the parameters (Martel, 2015). Further, these outliers can also influence subsequent observations which have implications for the resulting parameter estimators (McQuarrie & Tsai, 2003). It may cause fatal errors if the next statistical analysis cannot be performed optimally only because of the data outliers.
This study focuses on the context 'attached' to the outliers: explaining the context behind the outliers' emergence. The context, in this case, refers to the things attached to the respondent, such as gender, age, education level, and length of teaching time. In contrast to studies conducted by psychometrists, such studies on outliers only focused on the outliers location in the data (Ali, 1994), statistical behavior tests (Prescott, 1978), and outliers' patterns in the contingency tables (Rapallo, 2012), this study looks at outliers from a qualitative perspective, which uses statistical data, that are, nonparametric scale.

Method
Data on the outliers and respondents' identity (context) were obtained from secondary data (Harris, 2001) from the web questionnaire (SIAP Application Development Questionnaire, n.d.) in the SIAP application. The respondents are 295 people from national question writers from national selection results in 2016 and 2017. The number of respondents sampled was all respondents who answered 'No', but still answered follow-up questions for three questions, 106 respondents for questions took a long time accessing the SIAP application, 92 respondents for communication questions with the subject matter PIC, and four people for communication questions with the SIAP administrator.
The questionnaire used as the data source is in the form of a digital questionnaire consisting of five parts, namely, the first part containing the identity of the respondents (name, subject, post, city, province, age, gender, the experience of accessing SIAP applications, length of teaching, level of education, suitability for subjects taught), the second part in the form of questions about the time needed to access the SIAP application, the third part in the form of questions about communication with the PIC of the subject matter, the fourth part which is a question regarding communication with the SIAP administrator, and the last part in the form of suggestion. The stages of research are guided by evaluative methods (Ross & Cronbach, 1976) and (Fíncher, 1981), namely by reading-intensive results of the response, especially responses related to questions that have further ques-tions; recording into an identity data card (in this case the context in the form of gender, recent education, and length of teaching time) of respondents who are outliers; doing coding based on context; grouping the data; giving meaning to the context of outliers; drawing a conclusion; and compiling reports.

Findings and Discussion
The analysis was done on questionnaires distributed to national question writers about the satisfaction level with the SIAP application, communication with the subject matter PIC, and communication with the administrator. The questions on the SIAP application include the need to access the SIAP application for a long time, interactive features, and overall satisfaction with the SIAP application. The level of satisfaction with regard to communication with the PIC of the subject matter contains questions about the explanation of the indicators, remedial comments, and explanations of rejection. Communication with the administrator contains questions related to the call center, WhatsApp, and email. Based on the results of the analysis, information was obtained on the percentage of respondents who answered 'No' but still answered the follow-up questions for the three types of questions, respectively 81.13% (for questions that took a long time accessing the SIAP application), 32.6% (for communication questions with the PIC of the subject matter, and 75% (for communication questions with the SIAP administrator). The details can be seen in Table 2.
Taking a Long Time to Access the SIAP Application According to research conducted by several experts about computer-based applications, the average access needed to open applications is 4.7 seconds (Hastomo & Yuhana, 2013), 0.93 seconds (Shubhi, Yuliana, & Winarno, 2011), and 6.959 kB/s (Mulyana & Sholekan, 2010). However, each application used certainly has different access times. Software specifications used, when the situation opens access (morning, noon, or night) (Mulyana & Sholekan, 2010), availability of Wifi (Pangesti, 2017), as well as the software used are some of the factors that cause the speed of loading or data access.
Surely, it also applies to the SIAP application that is accessed by the writing participants the question that became the respondent of this study. Of the 295 respondents who participated, 189 respondents answered 'Yes' and 106 respondents answered 'No'. It means that most respondents need a long time to access SIAP. There were 86 people from all respondents who answered 'No' (45.50%) respond to the next questions. This number 86 consists of 42 female and 44 male respondents. It seems to support the results of the study by Parquette (1952) that men tend to be difficult to understand reading at once reading. Although only two points adrift, it can be said that men need time to understand reading compared to women. In fact, in the research conducted by Qian, Buchmann, and Zhang (2018), women tend to have good educational adaptability.
However, if they hold the opinions of respondents number 31 and 82 (both women of the same age), in the part of the question regarding the experience of accessing the SIAP application, they wrote that the problem they experienced when accessing was network. They will not have trouble as long as the signal when they open the application is in a strong position or from the side of the SIAP application itself, which is in maintenance status. Indeed, there are respondents (number 71, male) who say it takes two minutes at the beginning of the login, but it becomes smooth during the process of writing questions and saving and submitting. For this reason, he did not consider that it took a long time to access the SIAP application. Sometimes, their understanding of applications that are still relatively new or they are still unfamiliar with using applications makes them stutter technology (based on respondents' answers number 37, 59, and 154; all three are male and are in the same age group, namely 30 to 39 years old). It is undeniable that most people need a long time to understand or learn something new. The same is the case with the results of a research by Onwuegbuzie et al. (2004) conducted on undergraduate graduates in America.
Communication with the Person in Charge of the Subject Matter SIAP application provides an opportunity for the writer of the questions to communicate with the PIC of the material through the Chat feature. However, some participants did not use the facility. Of the 295 respondents, 200 people (68%) answered 'Yes', 92 people (31%) answered 'No', and three people (1%) did not answer. Of the 92 respondents who answered 'No', 30 respondents continued to answer questions. Of the total 92 respondents who answered 'No', 30 respondents still answered further questions. Male respondents numbered 13 people (43%) and female respondents numbered 17 people (57%). It indicates that more females are not willing to communicate with the PIC of the subject matter.
This condition is in contrary to consistency in continuing to answer questions, namely, in this case, women can be said to be inconsistent, which can be caused by psychological factors (Hornung, 1977), such as stress due to communicating with the PIC of the subject matter that must be socially elevated. In addition to stress, personal health conditions or family can also become the trigger. For example, the condition that occurred in respondent number 264 (female, aged 40 to 49 years old) said that she could not focus because she was caring for his mother who was hospitalized. The same is true for the answer of respondent number 164 (female, aged 50 years old or more) who said that she accessed the SIAP application when she was not ready because she was ill.
It is possible that female respondents did communicate with the PIC of the subject matter, but they received a rejection (it could be in the form of neglecting chat, having to make improvements to the questions submitted, even more extreme: the rejection of the question). It seems that in this case, the stimulus factors and responses (Hovland, 1948) are the determinants of communication that can be established well or not. The reason for fear of being rejected for the questions they wrote seems to be justified when reading the respondent's answers in the question section on the experience of accessing the SIAP application. Respondent number 238, female, postgraduate education, with five years teaching experience (respondents included in the sample who answered follow-up questions) wrote that she was happy and worried about the questions she wrote and submitted. She was afraid that the problem would be rejected so she was very aware of writing questions and submitting them. Errors in running an application can also cause unauthorized written questions. These errors include mistakes in placing stimulus, choice of answers, and key answers and reasons.
Subconsciously, those female respondents accidentally choose the option 'No'. It is likely the inability of females to suppress expressive language tendencies (Maynard, 1988) which is more dominant than male language behavior. She means to answer 'Yes', but accidentally clicks the choice of 'No'.

Communication with the SIAP Administrator
The results of the studies have provided information that technological advances in the field of communication cause various changes in society, such as changes in the speed of information exchange (Zamroni, 2009). Administrators as parties that facilitate the questions writers with various information, such as, the time of the assignment, indicators that must be done, the number of questions that must be written, must be able to take advantage of technological advances in communicating. To support this, in addition to providing communication facilities with the PIC of the material directly, the SIAP application also provides communication facilities between the question writer and the SIAP administrator.
Of the 295 respondents who responded, 266 people (90.16%) used this facility. The number of male respondents was 112 people and female respondents amounted to 154 people, while the number of respondents who answered 'No' was four people and those who did not answer are 25 people. Of the four respondents who answered 'No', only three people participated in answering further questions. The interesting thing is from the three questions that have follow-up questions, specifically the communication questions with the admin getting the response that did not answer the most, namely 25 people. There were 13 female respondents with undergraduate education and five male students; four females with post-graduate education and three males with post-graduate education.
The biggest possibility of respondents who did not provide answers regarding communication with the SIAP administrator was respondents who had less memorable experiences. For example, the answer of respondent number 285 (female, 50 years old or more, post-graduate education) on the question of the criticism and suggestions that she requested that the SIAP administrator be faster and more responsive in answering the questions submitted by the participants. They need immediate answers because the time given by the system for writing questions is not too long, especially for those who communicate via WhatsApp private lines (respondent number 225, woman, undergraduate).
Another reason identified was the respondents' unpreparedness when they got a call from the SIAP administrator who tells them about assigning questions. Often the ad-ministrator suddenly contacts the question writer to be assigned several indicators that must be written for a short period of time (respondent's answers number 66, 198, 202, and 290). Upset? Most people will feel annoyed if they have just gotten information but have been demanded to be able to immediately implement with deadlines that are not so long.
As an administrator, it is appropriate to provide the best service to service recipients. Everyone who serves as an administrator must have high commitment and motivation in carrying out their duties (Moenir, 2010). If the administrator can provide optimal service and meet customer demands, the organization can be regarded as the greatest form of success on the service side. On the contrary, if the service provided is not satisfactory, it is necessary to improve the system and implementation mechanism.

Conclusion
The web-based questionnaire or digital questionnaire still has many disadvantages. The presence of data outliers, for example, cannot be avoided. There are only factors that cause respondents to give unexpected data, such as psychological factors when responding to questionnaires. Actually, it can be anticipated if the digital questionnaire is accompanied by an interview to get in-depth information about the aspects being evaluated. It is okay to use a web-based questionnaire, but in its implementation, it will be more targeted if the respondent is accompanied when inputting data on the media used (such as devices, mobile phones, tablet computers, etc.). It provides an opportunity for respondents to ask if there are questions that are difficult to understand. Interviewers can also supervise carefully, especially taking preventive actions related to conditional questions (in this paper stated in terms of follow-up questions).
Data in the form of outliers do not have to be deleted or treated by means of statistical averaging, but can be used as a data source to conduct in-depth interviews with respondents who gave the response. It can be a new field for constructivist paradigm researchers. They can explain outliers from another point of view. Even for psychometrics, outliers can be used as new data to see the relationships between the variables of nonparametric scale data found in the respondents, for example, to see if there is a relationship between the length of teaching time, gender, and age with communication with the subject matter PIC, and SIAP administrator.
In addition, it is suggested that in the formulation of questionnaires (both paperbased and web-based), it is necessary to consider the respondent's experience of novelty and current, such as responding in a new form (digital questionnaire), layered questions (questions that require further questions), because not all of the respondents were in a focus position to answer, even though they were highly educated, of productive age, and had a longer experience.