Analysis of the Level of Satisfaction and Importance among Trainees towards Distance Training in the Covid-19 Pandemic Era

Abstrak: This study aims to analyze the level of satisfaction and importance among trainees towards distance training and identify the strengths and weaknesses of distance training implementation so that the organizers can maximize the satisfaction of the trainees towards the implementation of distance training. This research used quantitative and qualitative analysis within the framework of Level 1 (Reaction) of the Kirkpatrik Evaluation Model, as well as the Importance Performance Analysis (IPA) method. The results showed that: (1) the trainees were very satisfied with the implementation of distance training; (2) the implementation of distance training is considered very important to be implemented; (3) Some evaluation indicators have positive gap values, while others have negative gap values, meaning that some evaluation indicators have not met the expectations of stakeholders; and (4) most evaluation indicators are in quadrants A and B, meaning that they either must be prioritized to improve the performance, or maintained because they are already considered good. Therefore, the Financial Education and Training Agency through work units in the region, especially the Makassar branch, needs to improve the performance of distance training implementation, especially in the evaluation items/indicators in quadrant A to increase the satisfaction of participants in the next training sessions.


Introduction
In carrying out its duties and functions, namely managing education, training (learning), certification and knowledge management in the Covid-19 pandemic emergency, the Financial Education and Training Agency continues to implement health protocols, as well as innovate and adapt in the services they provide. This is in line with the direction of the Minister of Finance which states that the performance target in the perspective of learning and growth as the service area of the Financial Education and Training Agency must not change or decrease. Therefore, instead of halting services due to pandemic conditions, the Financial Education and Training Agency adapts or converts their services by optimizing digital platforms (Financial Education and Training Agency, 2020).
Optimization of digital services is not recently done by the agency. Since 2017, several policies have been established and given a foundation for the Financial Education and Training Agency to adapt during this pandemic, namely by encouraging innovative digital-based learning services, especially through the Ministry of Finance Learning Center platform called the KLC system. However, with the rampant pandemic requiring the human resources of the Ministry of Finance to work from home, there is a demand for learning through digital platforms to be more massively applied. This means that the Financial Education and Training Agency is expected to create and deliver new models of learning that address the current conditions (Financial Education and Training Agency, 2020).
As a form of adaptation to technological developments, flexibility of learning space, consideration of effectiveness and efficiency, and ease of the application of integrated learning, Financial Education and Training Agency has developed a learning program in the form of distance training as an alternative form of learning in the agency. Distance learning is a learning process held outside the training venue and puts the emphasis on independent learning that is managed systematically and not limited by distance or time by using various learning media (Iskandar & Subekan, 2020). Online learning or distance learning aims to meet the needs of human resource development and improvement within the Ministry of Finance through the use of technology and information despite the challenge of restrictions of the training implementation during the pandemic (Marlena et al., 2018;Nugroho & Lestyowati, 2020;Yang et al., 2012).
As mentioned above, distance learning or distance training at the Financial Education and Training Agency is carried out through a web-based learning center system (www.klc.kemenkeu.go.id) for non-face-to-face sessions, in addition to Zoom or other virtual meeting applications for face-to-face sessions, as well as Google Form and other applications for practice sessions, assignments, and exams (evaluations). This distance learning model is a combination of synchronous and asynchronous learning models. The synchronous model in distance learning is carried out through the Zoom virtual meeting application which allows trainees and instructors to interact simultaneously with each other (Narayana, 2016;Iskandar & Subekan, 2020;Darmawan, 2018;Nana & Surahman, 2019). Meanwhile, the asynchronous model in distance learning is carried out through the learning center system where learning is centered on trainees with an independent learning approach through the use of relevant online learning resources (Iskandar & Subekan, 2020;Elyas, 2018).
Given that distance learning is a form of adaptation or new innovation at the Financial Education and Training Agency during the pandemic and the new normal transition, it is important to evaluate the implementation of this learning model, especially from the level of satisfaction and importance among the training participants. Several previous studies have reviewed and evaluated the implementation of distance training. Septian (2020) evaluated the implementation of distance learning through elearning and the introduction of Edmodo, a social media application. This study found that distance learning can increase the ability of civil servant teachers in using Edmodo as a distance learning tool and that the application is found to be useful as a medium of learning in schools. Hayadin (2012) conducted a study to determine the context, input, process and outcomes of remote education and training held by the Ministry of Religion since 2009. This study used a survey method at several research locations in Indonesia and concluded that the implementation of long-distance training since 2009 at several training centers, has contributed to three things, namely: a) increasing the number of training participants; b) providing ICT competencies to trainers/educators and training center staff by training them to becomes online tutors as well as administrators for remote education and training; and c) providing a Virtual Learning Environment (VLE), and hardware and software applications in the form of internet websites with content that supports distance learning. Furthermore, Sari (2020) evaluated distance training for teachers in using the Google Form application as a distance learning medium. The results of this study indicate that the trainees gain additional skills and are able to understand the principles of making online quizzes using the Google Form application.
Although these studies have examined and evaluated distance training implementation, it remains rare to find research on the study and analysis of the satisfaction and importance level on the training implementation among training participants, especially one that is conducted in the special condition of a pandemic. In order to fill this gap, this study was aimed at analyzing the level of satisfaction and importance among trainees on distance learning, as well as to identify the strengths and weaknesses of the distance learning implementation so that the organizing committee can maximize the participants' satisfaction on the implementation of distance learning.
In evaluating the education and training implementation, including the distance learning approach, the Makassar branch uses Kirkpartrick's evaluation model. The model consists of four levels focusing on different aspects, in which each level affects the next one (Kirkpatrick, 2008). Level 1 (reactions) measures how participants react to learning, or in other words measuring customer (participant) satisfaction. Level 2 (learning) is an evaluation on the learning process, or the transfer learning based on the participants' learning results, including change in behavior, the development of knowledge, and/or the improvement of skills. Level 3 (behavior) aims to examine to what extent the participants' behaviors change after learning, and Level 4 (results) is an evaluation on the final results after learning is completed, including increased productivity and quality, reduced costs, decreased workplace accident rates, increased sales, decreased turnover of employees, and increased profits (Kirkpatrick & Kirkpatrick, 2006).
The scope of the study is limited to the study of Kirkpatrick's evaluation criteria (Kirkpatrick & Kirkpatrick, 2006) on Level 1 (Reactions). Evaluations on Level 2 (Learning), Level 3 (Behavior) and Level 4 (Results) were not included in the scope of the study because they require more time and can only be done in a period between six months and one year (short term) or even up to two or ten years (long term) (Iskandar, 2019). This study was conducted on the alumni of two distance training programs organized by the Makassar Financial Education and Training Center as one of the 13 technical implementing units in the region during the period of August to December 2020.
Method Based on the research objectives, this study can be classified as descriptive research, which aims to depict a certain situation or phenomenon in detail (Hamdi & Ismaryati, 2019). In terms of approach, this study employs the quantitative approach with a survey and questionnaire as the data collection method (John, 2013;Sarwono, 2006). There are two types of data based on the source, namely primary and secondary data. Primary data was obtained directly from the participants in the form of questionnaire responses, whereas the secondary data was obtained from a number or sources, including the training program terms of reference, outlines of learning programs, results of the training needs identification, and reports on the implementation of the evaluation and information section of the Makassar Financial Education and Training Center. The research variables used in this study can be seen in Table 1.  (Kirkpatrick & Kirkpatrick, 2006) Analyzing the questionnaire results to: 1. Evaluate the implementation 2. Evaluate teaching Likert scale is used for all questionnaire items and ranges from "very satisfied", "satisfied", "dissatisfied", to "very dissatisfied" for perceptions/facts, and "very important", "important", "not important", "very unimportant" for importance/expectation. The research population consists of 332 alumni of distance learning during the period of August -December 2020, with a total of 13 classes. This study applied the nonprobability sampling technique in the form of saturation sampling which uses all the population as the research sample (Sugiyono, 2017;Hamdi & Ismaryati, 2019). Therefore, the number of research samples was a total of 332 people.The collected questionnaire data for the Level 1 evaluation (Reaction) were then analyzed using the descriptive statistics technique in order to learn the distribution frequency of each answer to each questionnaire item. Data were tabulated and presented with the mean score as well as percentage for each item.
In addition, the Importance Performance Analysis (IPA) technique was also used to compare the satisfaction/performance (X) and the importance/expectation (Y) of the training participants.
The result of the comparison is interpreted based on the satisfaction and importance matrix, as presented below:  (2017) and Nugroho & Lestyowati (2020) The data were grouped in the Cartesian quadrant which shows the level of satisfaction and importance. The IPA method made it possible to observe which aspects/parts of the implementation of distance learning require more attention. To represent the results in numbers and support the analysis, the Statistical Package for the Social Sciences (SPSS) was used to facilitate the calculations. The IPA method was first introduced by Martilla & James (1977) with the aim of measuring the relation between consumer perceptions and priorities for improving product/service quality, also known as the quadrant analysis. In this study, it is used to map the relation between the importance and performance of each offered attribute and the gap between the performance and expectation of these attributes.
The interpretations of the quadrant are described below (Nugroho & Lestyowati, 2020). a. Main Priority (Concentrate Here) -This quadrant covers factors that are considered important or expected by the consumers, yet their performance is considered to be unsatisfactory. Therefore, the company must focus their attention and allocate their resources on these aspects in order to improve the performance. b. Keep Up the Good Work -This quadrant includes factors that are considered important and expected as factors that support customer satisfaction, which must be maintained by the company/organization to keep the achievement/good performance. c. Low Priority -This quadrant contains factors with a low perception level or low actual performance level and not very significant or lowly expected by customer, which means that there is no need for the company to pay extra attention or put these aspects in priority.

d. Possibly
Overkill -This quadrant consists of factors that are considered as insignificant or not expected by customer. Therefore, the company should allocate resources to other factors with a higher level of priority.

Result and Discussion
The learning evaluation was conducted based on the guidance in the Regulation of the Head of the Financial Education and Training Agency Number PER-5/PP/2017 on the Learning Evaluation Guidance in the Scope of the Ministry of Finance and Number KEP-82/PP/2020 on the Distance Learning Implementation Guidance in the Scope of the Financial Education and Training Agency. The evaluation aspects include the participants, the teaching staff, and the implementation.
The satisfaction level and the average satisfaction level of the distance training participants were analyzed based on the respondent assessment on the Likert scale, using the following equation (Nugroho & Lestyowati, 2020): in which RK : average satisfaction/importance JSK : total score of all respondents JK : number of respondents The satisfaction and importance scores were each grouped based on four levels, namely "very satisfied", "satisfied", "dissatisfied" and "very dissatisfied" and "very important", "important", "not important", and "very unimportant" (for importance/expectation). The interval was determined based on the following formula (Nugroho & Lestyowati, 2020): Based on the interval, the scores can be grouped as follows (perception/ importance): 1 -2.0 : very dissatisfied/very unimportant; 2.1 -3.0 : dissatisfied/not important; 3.1 -4.0 : satisfied/important; 4.1 -5.0 : very satisfied/very important.
The formula was used for calculating the mean score of satisfaction and importance toward the teaching staff and organizing committee among the training participants based on the questionnaire answers. The result can be seen below.  Table 1) Source: Primary Data (processed)

Figure 2. Satisfaction/Performance Evaluation Components
Note: Items 1 -8 are the evaluation indicators (See Table 1) Source: Primary Data (processed) Based on the relevant equation, the average level of satisfaction and importance towards the implementation of distance training among training participants based on the evaluation indicators can be seen below (Figure 1 and 2). 1. Satisfaction/Performance Level -Based on the calculation on the average satisfaction level using evaluation indicators, it was found that the training participants were very satisfied with the implementation of the distance training. All eight evaluation indicators scored above 4.1. The highest point was found in item or indicator number 1, namely the conformity between the learning materials and the expectations/needs of the participants, with a score of 4.70 (Very Satisfied). 2. Importance/Expectation Level -Based on the calculation on the average importance level using evaluation indicators, it was found that the implementation of distance training was considered very important to be implemented. All points scored above 4.1, meaning that they were considered very important. The highest points are found in items or indicators number 1 and 2, namely the conformity between the learning materials and the expectations/needs of participants and easy-to-understand teaching materials with a score of 4.62 (Very Important).
Based on the measurement of the level of satisfaction and the level of importance on all evaluation indicators, a formula is created to calculate the level of conformity between the level of satisfaction and the level of importance among the distance learning participants, as depicted below.
= 100% in which Tki : Conformity level of the respondents Xi : The average score of satisfaction/performance level Yi : The average score of importance/expectation The level of conformity is the result of the comparison of scores between the level of satisfaction and the level of importance. In this analysis, there are two variables represented by the letter X and the letter Y, in which the letter X is the level of satisfaction/performance, while the letter Y is the level of importance/expectation. The level of conformity between the level of satisfaction and the level of importance for each evaluation indicator is shown in the following table in detail. Based on Table 2, it can be seen that some of the evaluation indicators have positive gap values, while others have negative gap values. This means that some of the performance evaluation indicators have not met the expectations of stakeholders. Respondents assessed that satisfaction/performance was below the importance/expectation with varying gap values. The highest negative gap is -0.08 in the evaluation indicator number 4, namely sufficient time allocation for delivering the distance learning materials, followed by evaluation items number 7 and 3, namely accessible distance training facilities and the suitability of the learning method for the distance training materials. This means that respondents consider that the three indicators have not met their expectations, namely sufficient time allocation for delivering the distance training materials, accessible distance training facilities, and the suitability of learning method for the distance training materials. Meanwhile, the highest positive gap is in the evaluation item number 8, namely easy-to-use distance training facilities. This means that the use of distance training tools namely the KLC system, Zoom application, etc. is considered easy by respondents.
Furthermore, from the data in Figure 1 and 2 above, further analysis is needed to determine the priority scale in the effort to improve each of the existing evaluation indicators. The priority scale for improvement can be determined using the IPA method. The IPA method measures the levels of satisfaction and expectations of the training participants to be mapped in the importance performance matrix quadrants using a Cartesian diagram. Cartesian diagram is a chart that is divided into four parts and bounded by two lines that intersect perpendicular to the average point of satisfaction/perception/reality and the average level of importance/expectation. The horizontal axis (X) represents the satisfaction level scores, while the vertical axis (Y) represents the importance level scores.
Based on the data on the average X (Satisfaction/Performance) and Y (importance/expectation), all evaluation indicator items are mapped in Figure 3. distance learning method for the learning materials. This means that respondents consider that the following indicators have not met their expectations, i.e. sufficient time allocation for delivering the distance training materials, accessible distance training facilities, and the suitability of the learning method for distance training materials. Meanwhile, the highest positive gap is in the evaluation item number 8, namely easy-to-use distance training facilities. This means that the use of distance training tools, such as the KLC system, Zoom virtual meeting, etc. is considered easy by respondents. Moreover, the IPA analysis shows that most of the evaluation indicators are in quadrants A and B, which means that they either must be prioritized to improve their performance or must be maintained because they are already considered adequate, with three indicators in each quadrant. In addition, there are two indicators in quadrant C, which means that the evaluation indicators are not very important.
Based on the results of the analysis and conclusion, this study proposes the following suggestions. First, the Financial Education and Training Agency through the regional work units, especially the Makassar branch, needs to improve the performance and implementation of distance training, especially on evaluation items/indicators in quadrant A. In this study, the indicators are the suitability of the learning method for the distance training materials and sufficient time allocation for delivering the distance training materials, and accessible distance training facilities. This aims to increase participants' satisfaction in the next training programs. Second, data collection and identification of problems that occur during and after the training must be done to obtain the next strategy or technical step in the context of continuous improvement in the implementation of distance training, especially during the pandemic in the "new normal" world. Third, the Financial Education and Training Agency and all the regional work units recognize the use of distance training tools such as the KLC system and Zoom virtual meeting application as an advantage in delivering the learning service both for the synchronous and asynchronous models. Therefore, the Financial Education and Training Agency needs to pay great attention to the development and improvement of the quality of the ministry's KLC system in the future.
From the practical aspect, the implications of this research are expected to provide input for the Financial Education and Training Agency and all the regional work units as the training developers and implementers, especially on the important aspect of evaluation for improving training services for stakeholders. This research is limited to the scope of trainees at the Makassar branch, meaning that if this research was conducted in another work unit, it might obtain different results. In addition, the data used in this study was obtained from the recapitulation of questionnaires distributed to respondents. The researcher ruled out any other data that might exist and affect the results of the study. Researchers have not explored the level of satisfaction towards distance training participants using other frameworks/models. Therefore, further research is needed to determine the variety of evaluation results through different approaches.