THE IMPACT OF UNDERGRADUATE PROGRAM DEVELOPERS AND USER PARTICIPATION ON THE QUALITY OF SCHOOL INFORMATION SYSTEM

Developer's expertise and user participation are two critical factors of successful School Information System (SIS). Many researchers focus on user-initiated and professional SIS, while this study observes a campus-initiated SIS which involves undergraduate developers and volunteer users. We qualify the system using the System Usability Scale (SUS) and observation. The final prototype reaches an acceptable level of SUS and the outcomes among students whose GPA above 3.0 are not much different, but we recommend to find students with good grades in the supporting courses. The volunteer users come from 2 public and two private schools in Yogyakarta, Indonesia. We also examined user participation and found they were good and even excellent, although each developer may have a different standard of perceiving user participation. This study also reveals user inconsistency and interface issues were still become problematic, as changes cause the project to be overdue.


INTRODUCTION
The academic quality of students in a study program may be seen in diverse ways, including GPA (Grade Point Average), involvement in the recognized activities or competitions, and scientific experiences.Having an industrial or social experience becomes a new necessity and a new trend in the curriculum in the high-quality education so that students are not only adept at theory but are also flexible in solving problems [1].In the field of information technology, a student having work experience on Information Technology (IT) projects will have a valuable portfolio to work in the industry, as service-job category need more complex competencies [2].This is because the real experience will hone their soft skills and hard skills as they have interacted directly with users or IT project leaders [3].
Most of IT projects have been initiated by government, private, or higher education institutions.In fact, in the context of developing countries, there are many cases when the solutions to a problem or opportunities are initiated by these three parties [4].For example, it is the government who provided public facilities in the form of railway ticket reservations, but in its operation, it is the private sector who complemented it (ticket agents, ecommerce marketplace), and it is the educational institutions who widely studied its modelling, simulation and technology.Similarly, in the field of education, Dapodik as a national data collection system (initiated by the government) is assisted by School Information Systems (SIS) made by the private sector [5].
Although there have been many SISs developed by private parties, educational institutions still have opportunities to help schools manage data.Therefore, the researchers from our university took the initiative to develop SIS as one of the IT final projects for undergraduate students.The campus randomly selects any students to join the project to allow them with equal opportunity to interact directly with the community.
The variation in students' ability, the school's business processes complexity, and the diversity of user participation during the Rahayu et al., The Impact of Undergraduate Program Developers and User Participation on the Quality of School Information System development of SIS presents a challenge for this study.To get a comprehensive picture, it is necessary to assess students and school partners regarding the project.It is also necessary to evaluate students' experiences of interacting with school partners, to be compared to the objective assessment of school partners on the use of developed SIS.
Previous researches have shown that user participation and user involvement are positively correlated to system success, which is known much related to the quality of the system.In this study, a more comprehensive review of aspects that affect the quality of the system is conducted, both from the developers and users.The level of user participation and student competence (as developers) which is associated with the system's quality from usability aspects is analyzed.
The program developer experience regarding user participation has been the subject of many previous studies.Usability measurement standards may also employ several methods, such as usability scale systems (SUS) [6]- [8], ISO / IEC 9126 standards [9], usage-based metrics [7], and other specific standards for specific systems [10].The use of SUS for measuring usability in education domains has been largely undertaken, especially in e-learning systems that involve students as users [6], [8], [11] not in the framework of being a developer.Therefore, this study is expected to identify the pattern of user participation on the usability of the SIS products developed by undergraduate students.
This SIS has kinds of system users namely superadmin, headmaster, administration staff, curriculum staff, school counselor, students, student's supervisor, and prospective students.They can use the system for various activities, from the master data and personnel management, student admission, class distribution, scheduling, presence system, assessment, counseling guidance to extracurricular management.
This SIS was developed by six undergraduate students as their final projects/undergraduate thesis.Since the system is divided into six modules, each of the students built one module.As a program developer, they did the software development lifecycle using waterfall methodology from the analysis, design, implementation to testing phase in approximately six months.We collaborated with four schools located in Yogyakarta as clients to help the developers in each of the four phases.The result is an SIS prototype that already tested and accepted by the clients.
The Government of the Republic of Indonesia has provided various systems for the management of schools nationally, such as Dapodik as a data collection system of each educational unit and Sekolah Kita as a service provider of school profiles and comparison.More recently, the government has also released the e-report system as part of the school's administrative software.
School administration software is known abroad as the School Information System (SIS).SIS is defined as a system for managing students' data and other data related to students in the educational institution, both for formal and non-formal affairs.In general, there are two kinds of local SIS applications, namely (1) paid-SIS, e.g.SISKO and MySister, and (2) free SIS, e.g.JIBAS [12] and Sisfokol.
Many kinds of research have revealed that the quality of user participation determines the quality of the Information System, both in the designing stage [13] and implementation.In a recent study involving 86 Information System studies, it comes to light that user participation and involvement are positively correlated by 92% of system success, in which one factor is related to the quality of the system [14].This reinforces that the quality of information systems is influenced entirely by user participation.
To achieve user satisfaction, it is necessary to evaluate the usability of the information system in the education environment, both in the early and late stages of the system development cycle.SUS (System Usability Scale) is a measuring instrument of a questionnaire that has been widely used and validated by various studies [15]- [17].Table 1 shows related previous studies around the world.

METHOD
This study was conducted with four main stages.The first stage was school partner selection by examining the ranks of the schools based on the scores of National Exam for the academic year of 2014/2015.A total of 526 schools were on this list, and twelve schools met the criteria.Afterwards, a research proposal was sent to them, and during the development, there were still two public and two private schools who became volunteer users.The second stage was SIS development that consisted of six modules.The third stage was the SIS quality assessment used SUS.It was assessed twice, and developers revised the prototype before the second test.Details of each assessment were as follows: (a) Developer chose user tester from each school, (b) Users were required to operate each module, according to the scenarios and rate the SUS questionnaire, (c) The assessment generated performance measure (duration of work for each scenario, starting from instructing the work completion) and SUS score for each module.The last stage of the study was assessment of user participation in the system development by asking the students to fill out the questionnaire.
The participants involved in this study were two public and two private schools located in Yogyakarta province.User representatives consisted of 14 persons from four schools with various roles.Six undergraduate program undergraduate program developers were involved in implementing the SIS and assesing the user participation.In order to simplify the text, undergraduate program developer is shortened into "developer" term.
For measuring the quality of the system, a SUS questionnaire that has been widely used and validated by various studies was used [15], [16].SUS consists of 10 items to be assessed on a Likert scale (1 to 5) ranging from strongly disagree to agree strongly, and five positive and five negative statements.Table 1 summarizes some studies that measure usability in the education system using SUS.
SUS score is divided into 4 groups: <50 (not acceptable), 50 to 62.5 (marginal low), 62.5 to 70 (marginal high) and> 70 (acceptable).The calculation method of the SUS method is as follows: (1) each statement is given a scale of values from 1 to 5, ranging from strongly disagree to strongly agree, (2) the final value for the odd number is the value of the testers minus 1 point, (3) the final value for the even number is 5 points minus the given value scale, and (4) the SUS result is the total amount obtained from SUS and multiplied by 2.5 points.
For measuring user participation in the system development, the questionnaire compiled from several references related to an assessment from the developer side [17] was used.The limitations in this study are as follows: (1) system complexity through comprehensive metrics such as function point analysis, the line of codes, and cyclomatic complexity was not counted, and (2) the influence of the faculty's assistance during the final project because it will produce bias in assessment result.

RESULTS AND DISCUSSION
The 2017 SIS developer team consists of six students with various work experience and academic performance.Two of them had been a courses' assistant or a laboratory assistant, and one of them had been in the information systems project.In general, students' academic performance is reflected in the GPA.While in detail, the academic ability is projected in their grades of four courses: fundamental programming (Programming Algorithm I and Web Programming), Database and Information Systems.There is only one student (M5) who get a grade under 3.0 for four supporting courses.Figure 1 shows the grade distributions of those program developers.In the SUS assessment, the team encountered some difficulties to get software testers who represent each user actor.Because of the different completion time, each module is tested individually by the available user.The list of user testers for each module is in Table 2.It was a benefit since students did not wait for each other.Furthermore, various activities in school partners make it difficult for all user actors to join the testing session in a single time.On the other hand, students have limited time to complete all of the tasks.
Table 3 shows the result of SUS assessment in the first and second tests.It was concluded that the users spent less duration time in the second test.It should be noted that the developer improves the modules based on user review and critics from the first test.Three modules (DIST, BKEK, and PRES) got more than a 30% increase in terms of duration time.It is found that this increase is not related to academic performance as shown in Figure 1.
The average SUS score increased by 8.63 points to 71.08, so the prototype is categorized with the acceptable level (formerly: marginally low).The most significant contributor to this growth is PRES, BKEK and PEGW modules.Again, this data does not show any correlation between the grades of the supporting courses and GPA.Nevertheless, low SUS score for BKEK module is positively correlated with the corresponding GPA and courses' grade of the developer (M5).
In the user participation, two types of questions were asked to student developers: 14 closed questions with response of Strongly Disagree to Strongly Agree, and 3 open questions.First type assesses user attendance, user support, user feedback, and user approval for each stage of SIS development.The second one asks developers' opinions about the project's benefits and drawbacks and their suggestions related to user participation.User-developer meetings are divided into four phases of development: analysis, design, implementation and testing.User attendance was positively appreciated by the developer, as there is no "Disagree" answer.While all developers coincident responded "Agreed" for the design phase, a total of 83.3% of developers answered "Strongly Agree" in the testing phase.It implies that users participate in all phases, but user-developer meetings in the design phase were not optimal.
Developers appreciate user supports as they usually supply needed documents.Nevertheless, developers said that user inconsistency still exists in the design and implementation phase.The forms of the documents include student registration forms, ledger, report books, student attendance forms, print-outs from the Dapodik system, and schedules.
User feedback and their approval were perceived vary by developers, although none of them stated: "Disagree".The design phase of system input or output is full of user feedbacks and fully approved by users (⅓ states Strongly Agree, and the rest responds to Agree).The same result also applied in the design phase of system features.Furthermore, the three modules perform better, i.e.PPDB, JDWL and BKEK.Users provide adequate feedback to those modules, but surprisingly, all developers stated that users greatly appreciated the final features.It shows that developers were capable of recognizing the details and incorporating schools' business processes.
A special case occurred in the interface design phase since user feedback and approval varies as presented in Figure 2. As half-sized bar indicates "Agree" and the full one denotes "Strongly Agree", All possible combinations between feedback and approval are noticed.It is then concluded that the interface design phase is still complicated for both developers and users.It is understandable since interface integrate both system functionality and the aesthetic aspect.
In general, students appreciate user involvement as final project's benefits.Indeed, one student stated they are thankful as users can directly remark the work, and if additional data is urgently needed, school give the sample document immediately.Nevertheless, a total of 5 out of 6 students identified user consistency is a major drawback.As users often change business process after the design is implemented, it contributed to late system completion.Inconsistency usually occurs as the government change the regulation and users did not consider the details.The system's complexity began incorporating each school's process business.As the project was not initiated by users, some developers suggest building a long-term relationship with users to maintain its sustainability.
Table 4 summarizes the aspect of developers and users, related to system quality.The quality of developers assessed through student GPAs with a range of 3.00 to 3.25 are categorized sufficient, 3.25 to 3.5 are categorized high enough, 3.5 to 3.75 categorized as high and above 3.75 as very high.The level of user participation is measured through a questionnaire and calculated by averaging the value of user participation in the analysis, design, implementation and testing phases.The values are grouped into four class: 3.00 to 3.25 categorized moderate, 3.25 to 3.5 is categorized high enough, 3.5 to 3.75 categorized as high and above 3.75 as very high.System quality is measured by using the System Usability Scale questionnaire, where the calculated value is converted to adjective ratings [15].M1, M2, M3, M4 data show fairly consistent results, high user participation rates and high average developer competencies can produce system quality in both good and excellent range.However, in M5 and M6 data, there is an interesting phenomenon, as M5 rated that the user has a high participation rate, but the quality of the system in terms of usability is considered low by the user.Otherwise, M6 developers judge that user participation is at a lower level, but the system rated by the user has the best SUS value among the others.This may happen because the developers have a different standard when assessing the level of user participation or it may be occurred due to the developer competence factor that predominantly affects the quality of the system in both cases.
Based on the academic diversity in Figure 1, any students are randomly selected to join the project.The quality of SIS among the top five students is not far.However, a student who has the lowest score for GPA and four supporting courses also shows the lowest SUS score, both in the first and the second tests.Likewise, the assessment standard is quite varied among developers.Those findings indicated a positive correlation between their academic competence and system development.
Finding school partners that are in accordance with the criteria is not easy, as onethird of schools finally accepted our proposal.Nevertheless, developers still face problems with partners in attendance rate, regulations and user inconsistency.
As shown in Figure 2, all possible combinations between feedback and approval occurred in the interface design phase.Therefore this phase is still complicated for both developers and users.It is understandable since this phase integrate both system functionality and aesthetic aspects.

CONCLUSION
The diversity of academic achievement (in this case, courses' grades and GPA) does not significantly affect the quality of the prototype.It should be noted that all students have GPA of more than 3.0 on a 4.0 scale.Nevertheless, student M5 whose lowest academic achievement attained the lowest SUS score.In order to create the best team, it is recommended to find students with good grades in the supporting courses.The second test shows SUS average score gets increased and reach the acceptable level.Nevertheless, there are two modules which have SUS score below 70 points.Based on user participation data, interface improvement would make the modules better.Undergraduate students appreciate well for user attendance, user feedback and user approval, especially in the design phase of input, output and system features.Notwithstanding, user inconsistency is a crucial issue for project sustainability, as changes cause the project to be overdue.

Figure 1 .
Figure 1.GPA Distribution Data and Course Grades for Six Students (M1 to M6)

Figure 2 .
Figure 2. Perceived Interface Design Phase by Developers

Table 1 .
Previous Study Related to SUS in Educational Institutions

Table 2 .
List of User Testers for Each Module (*t=teacher, s=student, c=curriculum division, a=administrator)

Table 3 .
Results of the 1 st Test and 2 nd Test

Table 4 .
Impact of Developer and User Related to