Evaluation of the use of e-learning in the implementation of Stake’s countenance model in the era of the Covid 19 pandemic

The Master of Art in Arts Education Study Program encountered difficulties since it was impossible to attend offline meetings during the Covid-19 outbreak. The objective of this study is to apply the Stake's countenance model to assess the effectiveness of e-learning. The countenance model by Stake was applied in this study as a qualitative evaluation approach. The following three components of the curriculum implementation are evaluated: (1) antecedent (context, input), (2) transactions (processes), and (3) outcomes (results). The results of this study showed that the implementation of e-learning in the study program ran effectively. The meetings that are mostly effective account for 21.57% of the learning performance. The student learning results are 88.46% appropriate in terms of Graduate Learning Outcomes (GLO). According to the questionnaire's findings, 83.33% of the appropriate student learning outcomes (80–100%) were met. These findings describe the student learning outcomes that complied with the curriculum's minimal standards.


INTRODUCTION
For lecturers, carrying out learning that was in line with the objectives during the Covid-19 pandemic presented significant difficulties. According to Elen & Putri (2020), all levels of teaching and learning had to switch to online classes as a result of the Covid-19 pandemic that hit the planet in 2020. Due to this pandemic, the teaching and learning process must be done online, according to Muqorobin & Rais (2021). The use of information system technology in lectures has significant educational benefits. According to Miswanto (2022), both groups and individual students encounter a variety of challenges when using the internet and assignment-based systems.
Online learning can be carried out by utilizing technology. The use of technology for conducting lectures provides a solutive solution, with the help of technology lecturers can do many things from planning lectures to evaluating the learning process and student learning outcomes. Likewise in the Master of Art in Arts Education Study Program, Faculty of Languages, Arts, and Culture, Yogyakarta State University, the implementation of learning in limited face-to-face conditions required lecturers to be able to carry out learning online. When lecturers have to organize online everything must be well prepared starting from lesson planning, implementation to evaluation and results to be achieved as contained in the curriculum and other standard demands.
The Master of Art in Arts Education Study Program consists of four fields of arts education, each of which has its own characteristics, namely: music education, visual arts education, dance education and theater education whose lectures consist of theory and practice. The learning objectives for each subject are contained in the Semester Learning Plan (SLP) referring to the objectives contained in the curriculum and made by each course teaching lecturer. The problem that arises is whether the existence of the Covid-19 pandemic with the existing limitations of implementing online lectures can learning objectives be achieved?
The Semester Learning Plan (SLP), which references the objectives set forth in the curriculum and created by each course's lecturers, lists the learning objectives for each subject. The issue that arises is whether learning objectives can be met given the Covid-19 pandemic and the current restrictions on using online courses. Given the characteristics of the four fields and the nature of lectures, namely theory and practice. For this reason, an in-depth evaluation of the implementation of lecture learning in the Masters of Arts Education Study Program is needed. Considering the traits of the four disciplines and the format of lectures, i.e., theory and practice. This calls for a thorough analysis of how lecture learning has been implemented in the study program.
The purpose of evaluation research on learning implementation is to ascertain the suitability of the learning objectives that are present in the curriculum and its implementation, which includes program input readiness, the program implementation process itself, and the program's success in producing graduate products that meet predetermined standards. The countenance evaluation model by Stake was applied in this study as a qualitative evaluation approach. These three elements of the curriculum implementation were assessed: (1) antecedent (context, input), (2) transactions (processes), and (3) outcomes (results). The antecedent data for the first stage came from the documentation of the vision, mission, curriculum, lesson plans, and the lecturers' readiness, which was reinforced by their individual skills. The second stage of the transaction included questionnaires and interviews to assist lecturers in gathering information regarding the learning process as it related to lecture practice and theory, students' capacity for following along with and participation in the learning process, and the availability and use of facilities and infrastructure. The final phase are the student learning outcomes. To get an overview of the use of online learning in the study program, everything that became the focus in this stage was taken into account (judged) against current standards.
Globally, technology and information development is accelerating at a rapid rate, and this has implications for the field of education. Due to the Covid-19 pandemic, the learning process in tertiary institutions underwent a change in approach that initially used a face-to-face approach between lecturers and students, then switched to a face-to-face in class to an online approach. Students that use e-learning didn't just listen to lecturers describe the topic; they also actively participated by performing, performing, demonstrating, etc. To make teaching materials more engaging and dynamic and to inspire pupils to study, they could be virtualized in a variety of formats. Information and communication technology was used in e-learning to help students learn. Online implementation was one of the methods. Students could finish the learning process both inside and outside of the classroom with the aid of online learning, which removed time and space constraints.
It is highly recommended to conduct an in-depth evaluation in order to gather precise, understandable data and gain a general understanding of the limitations and issues that exist in the program being evaluated. This will allow for the formulation of appropriate recommendations for improvement and the facilitation of decision-making in the future. An adequate evaluation model is required in order to provide good evaluation findings. The Stake's coutenance model is one evaluation methodology that is appropriate for evaluating e-learning. When compared to other evaluation models, this paradigm has one benefit. This evaluation model can significantly influence assessments and is a robust enough idea for future advancements in evaluation. In this model, evaluation is carried out by contrasting a program with a reference program.
The justification for this research is that e-learning has been already widely employed in the field of education, particularly in the Master of Art in Arts Education Study Program during the Covid-19 epidemic, and it will continue to be used as a learning medium in the future. The problem formulated in this study is how to evaluate the utilization of online learning in the Master of Art in Arts in Education Study Program using the maintenance stake model. The objective of the study is to assess how well the Stake's countenance model was being used to implement learning in the study program.

METHOD
This study is an example of an assessment study since it looks at how learning is put into practice and how results are obtained by employing a countenance model method. The approach used in this evaluation is qualitative evaluation research using the Stake's model, which highlights the existence of two key points, namely: (1) descriptions and (2) judgments, and distinguishes the existence of three stages in program evaluation, namely: (1) antecedents (antecedent/context), (2) transactions (transactions/process), and (3) outputs (outcomes).
According to Pidarta, the research design is as follows based on the type of research mentioned above: (1) photographing the desired focus, (2) evaluating the focus after it has been discovered by comparing it to a predetermined good standard or concept, (3) offering solutions to focuses that have not been implemented correctly, and (4) summarizing the evaluation's findings. To provide a clear picture of the object under study, photographs of the research focus were taken in phases for each focus. Each focus that is photographed needs to have an evaluation attached to it that compares it to established standards or theories. In order to find a remedy, it is desired that current issues can be recognized as early as feasible. The stages listed below are used to conduct an evaluation using the Stake's evaluation model.

Data gathering
The evaluator gatheres information on the initial circumstances, transactions, and outcomes that the program developer desired. Both document studies and interviews are used to gather data. The evaluator initially establishes a frame of reference pertaining to antecedents, transactions, and results before beginning data collecting. This is done in order to make the evaluation's goal more clear as well as to check whether it is consistent with the transactions known as the antecedents and outcomes.

Data analysis
This step of the process involved both logical and empirical analysis. The provision of considerations regarding the interrelationships between initial prerequisites, transactions, and objective outcomes requires logical analysis. The evaluator must be able to determine whether the suggested transaction plan will result in the initial prerequisites stated by the program developer. Or perhaps another, more efficient transaction model exists. The connection between transactions and anticipated results is similar. Additionally, empirical analysis functions largely in the same way as logical analysis but uses empirical data.

Congruence analysis
This analysis compares what is stated in the objective (intended) with what really occurs throughout the action (observation). In this instance, the evaluator examines any discrepancies from what is planned in the objectives and the implementation in the field. The team of experts and program participants receives the congruence analysis from the evaluator once it has been finished. This team will evaluate the accuracy of the outcomes of the evaluator's analysis and offer their opinions on the crucial congruence elements.

Evaluation of outcomes
The evaluator's next responsibility is to offer consideration for the program under investigation. Evaluators require norms for that. The evaluator who renders conclusions about the program under evaluation is the general focus or the crucial element in this Stake's model. Stake claims that a description is distinct from a judgment or an assessment on the one hand. The antecedents (input), transactions (process), and outcomes (results) data in this model are compared not just with absolute standards for evaluating program benefits, but also to see if there are discrepancies in goals from the actual scenario.
The evaluator employed the Stake's Countenance Assessment Model and performed the description matrix and consideration matrix as well as examining the contingency and congruence of the collected data. Logical and empirical contingencies are both types of contingency. The evaluator's examination of the logical compatibility between the inputs, processes, and results yielded logic contingency. The evaluator's first decision-making step took this into account. On the basis of data gathered at the research site, the evaluator also offered empirical considerations (empirical contingency) between input, procedure, and results.
The assessor considered the congruence or disparities between what had been initially intended for a program or curriculum and what actually occurred in the research in addition to checking for contingency. Figure 1 presents a diagram of the assessment flowchart that utilizes Stake's Countenance Assessment Model. (https://www.semanticscholar.org/paper/Evaluation-of-the-2017-updated-secondary-school-of-Aksoy/e305fb7ed4c6624bb35ec78883b17759cac07806/figure/1)

Findings
Based on the research instrument answers that included a number of question indicators on the Google Form, 19 answers were gathered. The following is a description of each indicator's responses.

Availability of Semester Learning Plans (SLP)
The availability of SLP was determined to be 55 out of 31 courses in the Master of Art in Arts Education Study Program based on the findings of the questionnaire. As can be seen, SLP is present in all courses. The SLP offers 55 of the 31 courses, therefore there are 24 courses that it does not offer. One subject has many SLPs; this issue is brought on by the fact that some courses change lecturers every semester.

Completeness of SLP
A complete lesson plan must include a variety of learning tools. Information about how complete the semester lesson plans were acquired is shown in Table 1. With a percentage of 28.125%, online learning resources made up the majority of SLP equipment. Electronic learning materials make up the least LSP equipment. This demonstrates that the professors in the Master of Art in Arts in Education Study Program made the best use possible of the internet to ensure that the lesson plans are complete. E-learning tools were employed in the study program. During the CO-19 pandemic, these learning resources supported the educational process in the study program.

Implementation of lectures in accordance with the LSP
Based on the results of filling out the questionnaire, data on the conformity of the implementation of the lectures with the SLP was obtained, as shown in Table 2. The results of the answers show that 91.67% of the implementation of lectures with RPS was entirely appropriate. A total of 8.33% stated that the implementation of lectures was mostly appropriate. This illustrates that the implementation of lectures in the Master of Arts Education Study Program was in accordance with the RPS. The lecturers carried out lectures in accordance with the prepared lesson plans.
According to the responses, 91.67% of the implementation of lectures using LSP was perfect. 8.33% of respondents thought that the way lectures were implemented was generally appropriate. This demonstrates how the RPS was followed in the execution of lectures. Lessons were delivered by lecturers in accordance with the planned lesson strategies.

Effectiveness of Learning Process
Based on the responses to the questionnaire, the data regarding the effectiveness of the learning process is presented in Table 3. Based on the results of the questionnaire responses, it was obtained that the learning process' effectiveness in the category of all effective meetings was 78.43%. The category of meetings that were mostly productive made up 21.57% of the entire learning process. This information demonstrates that during the Covid-19 epidemic, classes in the study program continued as usual.

Constraints on the Learning Process
Based on the responses to the questionnaire, data about the limitations of the learning process is presented in Table 4. During the Covid-19 pandemic, there were substantial and technical challenges with the study program's learning process. These challenges included the restrictions of virtual classes (56.72%), the features of the learning materials (up to 13.43%), and the technical constraints of online learning (up to 29.85%). The data demonstrates that despite the study program's adoption of learning tools during the Covid-19 epidemic, there were problems that should be solved by the lecturers. The restrictions of virtual classes were the biggest barrier to learning. The studies of art involved emotions, which could not be best acquired in a virtual classroom. Additionally, due to the reliance on virtual classroom tools, the interaction between students and lecturers was constrained.

Virtual Face-to-face Media
Based on the responses to the questionnaire, data about the use of virtual face-to-face media is presented in Table 5. According to the data above, Google Meet had the highest percentage of users (35.96%) for faceto-face media. Use of media Zoom was 56%. Due to the ease of usage of both of these media, they were both widely employed. Discussions and learning could happen between lecturers and students at any time. The resources like media presentations and comment sections helped to foster the involvement that took place throughout learning.

LMS
Based on the responses to the questionnaire, data about the use of learning management system (LMS) media is presented in Table 6. According to the questionnaire's responses, 43% of respondents utilized Google Classroom, while 27.45% of respondents used Be-Smart. The use of LMS, such as Google Classroom and Be-Smart, was made easier for lecturers and students. The university provided support for both learning management systems (LMSs).

Teaching Methods
Based on the responses to the questionnaire, data about the use of teaching methods is presented in Table 7. According to the questionnaire's responses, it was discovered that individual assignment made up 21,07 % of the learning strategy used in the study program. Lecture came in second place, with a percentage of 20,31 %. Students were given individual assignments in order to practice autonomous study. The students had a positive attitude toward independent learning.

Conformity between Implementation of Teaching and Teaching Schedule
Based on the responses to the questionnaire, data about the match between schedule and implementation of is presented in Table 8. Based on the questionnaire's results, it was obtained that 86.11% of the lectures conformed with the teaching schedule (80-100%). This information demonstrates that the lecturers generally carried out lectures according to the teaching schedule. They could change the teaching schedule when they had to carry out other official duties.

Constraint of Student's Participation
Based on the responses to the questionnaire, the data regarding constrained of student's participation is presented in Table 9. Based on the questionnaire's responses, information gathered indicated that technological problems prevented students from engaging in online learning during the Covid-19 pandemic. Technical limits brought on by device and signal limitations. These two instruments aid in the efficient operation of online learning.

Lecturer's Satisfaction with the Implementation of Online Learning
Based on the responses to the questionnaire, the data regarding the lecturer's satisfaction with the implementation of learning is presented in Table 10. According to the questionnaire responses, 62.50% of the lectures were very satisfied with how the learning was implemented and 37.50% of lecturers were satisfied with how well learning is being implemented. The existence of the Covid-19 pandemic made online media helpful for the implementation of learning.

Conformity of Student Learning Achievement with GLO
Based on the responses to the questionnaire, the data regarding the conformity of student learning achievement with graduate learning outcomes (GLO) is presented in Table 11. According to the questionnaire's results, there was 88,46% of student learning achievement that conformed with the Graduate Learning Outcomes (GLO) (80-100%), while the remaining 11,54% of student learning achievement mostly conformed with the GLO. This information indicates that the lecturers generally conducted their lectures based on the Graduate Learning Outcomes (GLO).

Student Participation in Semester Final Exam (SME)
Based on the responses to the questionnaire, the data concerning student participation in Semester Final Exam (SME) is presented in Table 12. Based on the responses to the questionnaire, it was found that 88.89% of the students entirely completed the Semester Final Exam (SME). The remaining of 11.11% students mostly completed the Semester Final Exam (SME). This high level of student participation was due to the fact that students were required to take the SME for the final assessment of the courses.

Form of Semester Final Exam (SME)
Based on the responses to the questionnaire, the data regarding the form of Semester Final Exam (SME) is presented in Table 13  Based on the questionnaire's responses, it was found that the Semester Final Exam (SME) consisted of 40.00% assignments, 33.60% projects, and 26.40% written tests. Assignments were the most common form of SME assessment. The aim of lecturers was for students to master the learning content by conducting independent study.

Achievement of Student Learning Outcomes
Based on the responses to the questionnaire, the data regarding the achievement of student learning outcomes is presented in Table 14. Based on the results of the questionnaire, it was obtained that 16.67% of students got an average level of performance, compared to 83.33% of students who performed well (80-100%). These results demonstrate that student learning outcomes adequately satisfied the accomplishment standard.

Lecturer Satisfaction with Student Learning Achievement
Based on the responses to the questionnaire, the data regarding the lecturer satisfaction with student learning achievement is presented in Table 15. Based on the questionnaire's responses, it was found that while only 30.16% of the lectures expressed dissatisfaction with the student learning results, the remainder 69.84% did. These results indicate that, in general, the lecturers were satisfied with student learning outcomes.

Discussion
During the Covid-19 epidemic, art education had difficulties. The challenges that students and lecturers face are more inclined toward practical lectures, according to Yasa et al. (2022). Relocating art learning is one method of online education. According to Yanuartuti & Astutiningtyas (2020), the government and educators in Indonesia, including instructors and lecturers, believe that shifting art education online will be the most successful in boosting student creativity during the co-19 pandemic. According to the study's findings, digital technology can significantly boost creativity during the Covid-19 pandemic, especially if arts learning is planned using digital media. This is in keeping with the world's increasingly quickening development.
The Master of Art in Arts Education Study Program now offers online courses. According to this study, the Covid-19 epidemic, in general, had no effect on instruction in the Master of Arts Education Study Program. Meetings that went rather smoothly account for 21.57% of the learning process' overall smoothness. The information demonstrates that during the co-19 epidemic, classes in the study program continued as usual. The use of internet was largely responsible for the Semester lesson plan (SLP) comprehensiveness. This demonstrates that the lecturers in the study program made the best use possible of the internet to ensure that the lesson plans are comprehensive. E-learning tools were employed which supported the educational process in the study program. According to Wibowo & Nugraheni (2021), the institution's emergency curriculum syllabus, which is created through daily programs that are implemented through social media, summarizes the use of creative arts learning media.
Positive and negative aspects of learning during the Covid-19 epidemic can be seen in the Master of Art in Arts Education Study Program. The benefits of online learning can be carried out flexibly in terms of time and place, the positive is that learning activities could be reached by students because they did not have to travel to campus, the implementation and learning outcomes in this course could be used as an evaluation of learning achievement, and online learning focused students.
The drawbacks include the lack of variety in activities and interactions between students, lecturers, and fellow students; restrictions on Turnitin for students; technical restrictions on internet networks and limiting learning interactions; difficulties for lecturers in providing technical guidance and assistance in developing multimedia projects despite the existence of printed modules and video tutorials; and the inability of original multimedia applications to function.
The lack of variety in activities and interactions between students and lecturers as well as between students, the limitations placed on students' access to plagiarism checkers, the instability of the internet network, and the constraints placed on learning interactions and guidance are the negative aspects. Students should have access to field study facilities, laboratory research, reading and downloading literature books from the internet, and plagiarism checkers in order to overcome these challenges. Offline learning is still an option, though, should the need arise and the circumstances permit. There should be more digital learning tools available to students. The absence of data and internet packages, in accordance with Anhusadar (2020), are the obstacles that impede the online lecture process. Because interactions between lecturers and students may be conducted in the classroom and do not require an application, students prefer in-person lectures over online lectures.
Learning about character education amid the Covid-19 pandemic should be possible. According to Arrazaq & Aman (2020), character education is crucial for molding a country's psyche. According to Arrazaq (2019), historical legacy can be used to acquire the values of character education. The subject of study, historical legacy, contains artistic components that can be employed as teaching aids for art classes.

CONCLUSION
According to the research results, instruction in the Master of Art in Arts Education Study Program during the Covid-19 epidemic was, in general, successful. Meetings that went rather smoothly account for 21.57% of the learning process' overall smoothness. The information demonstrates that during the Covid-19 epidemic classes in the study program continued as usual. The internet was largely responsible for the Semester Learning Plans (SLPs) comprehensiveness. This demonstrates that the lecturers in the study program made the best use possible of the internet to ensure that the lesson plans were comprehensive. E-learning were employed in the study program.
The conformity of student learning outcomes with GLO was 88.46%. The conformity was appropriate because the lecturers developed their Semester Lesson Plans (SLPs) based on the GLO. According to the questionnaire's results, it was obtained that there was 88,46% of student learning achievement that conformed with the Graduate Learning Outcomes (GLO) (80-100%), while the remaining 11,54% of student learning achievement mostly conformed with the GLO. This information indicates that the lecturers generally conducted their lectures based on the Graduate Learning Outcomes (GLO).
Even though during the Covid-19 pandemic the learning process at the Master of Arts Education Study Program ran smoothly with all the limitations. This study used a qualitative evaluation method with the countenance evaluation model by Stake. The things that are evaluated include (1) Antecedent (context, input); (2) Transactions (processes) and (3) Outcomes (results) of curriculum implementation. In accordance with the problem and research objectives is to evaluate the use of E-Learning in the implementation of learning in the Masters of Arts Education Study Program with the Coutenance Stake model. In accordance with the Stake evaluation model, the implementation of learning is divided into three parts, namely preparation, process and results. Everything is generally smooth and student results are as expected.
Even with all the restrictions, the Master of Art in Arts Education Study Program's educational process continued to run efficiently during the Covid-19 pandemic. The countenance evaluation model by Stake was applied in this study as a qualitative evaluation approach. The implementation of the curriculum's antecedent (context, input), transactions (processes), and outcomes (results) are all evaluated. To evaluate the use of e-learning in the study program's implementation of learning using the maintenance stake model is in line with the problem and research objectives. The Stake's evaluation model divides the process of putting learning into practice into three parts: preparation, procedure, and results. Overall, everything went smoothly, and student test results were as anticipated.