ANALYSIS OF
FACTORS AFFECTING SYSTEM SUCCESS IN E-LEARNING SYSTEM OF XYZ UNIVERSITY JAKARTA
Frans
Setiawan1, Sfenrianto2
Bina Nusantara University, Indonesia
frans.setiawan@binus.ac.id1, sfenrianto@binus.edu2
Abstract
The e-learning system is currently a hot topic of
conversation considering the Covid-19 pandemic situation which has hampered the
teaching and learning process in various countries. XYZ University as one of
the best technology universities in Indonesia has used the e-learning system
long before the pandemic. Therefore, this research was conducted using
quantitative methods to determine the factors that influence system success in
the XYZ university e-learning system in hope that this research can be a
benchmark as a effective solution. This study uses
several literature studies including e-learning, and knowledge acquisition.
Data analysis was carried out using the SEM method using the SmartPLS 3 application. There were 6 independent variables,
namely information quality, system quality, service quality, system usage, user
satisfaction, and knowledge acquisition. The dependent variable is system
success. From the results of the questionnaire distribution, 385 respondents
were obtained from XYZ university undergraduate students who are still actively
studying out of a total of 418 respondents. A total of 224 respondents were
male and 194 respondents were female. This study proposes 12 hypotheses that
have 11 accepted hypotheses and 1 rejected final results. This research contain indicators per variable that used for the
questionnaire also several data analysis process was served. The results show
that the knowledge acquisition is positively
influenced by system quality but not significant and the information quality have
a significant positive effect on the knowledge acquisition. There are limitations in this study, where the selected respondents are
limited by the researcher cause of some access to respondents. Further research
can be conducted on students from various majors so that they can gain
different perspectives for more concrete evidence. Further research can be
carried out using other variables that support proving the factors that
influence the success of the system using the DeLone
and McLean models.
Keywords: E-Learning, Knowledge Acquisition, System Success, Online Learning, DeLone and McLean
Introduction
Along with the development of an increasingly modern era,
many changes accompany this development, one of which is technology. The
internet as one of the most massive technological developments is used as a
means of communication, a source of knowledge and information from all over the
world as well as for daily activities. In the COVID-19 pandemic situation as is
currently being experienced by almost all countries in the world, various
activities cannot be carried out normally, one of which is the teaching and learning
process which is required by the government to be carried out at home. The
internet as a means of communication, a source of knowledge and information has
once again shown its expertise in dealing with a pandemic situation like this (Handoko,
Churniawan, & Riyanta, 2022). XYZ University, which is often
known as one of the best technology universities in Indonesia, has various
systems created to make work easier and to facilitate the teaching and learning
process. One of the systems implemented long before the pandemic was the
E-learning system known as the LMS. During the COVID-19 pandemic, XYZ
university is again innovating to develop their E-learning system by adding a
video conferencing feature to replace face-to-face meetings in the classroom.
However, based on data obtained from XYZ University's quality management center
(QMC), in 2021 it was recorded that the satisfaction level of 2 students
regarding the use of the LMS was still below 80%, namely 75% in general from
various study programs, where the information systems study program obtained an
average of 78% academic satisfaction, one of the components of which is student
satisfaction. Therefore the author will discuss the
factors that influence the success of the e-learning system at XYZ Jakarta
university.
Method
This
study uses the ISS research model which was developed to identify the
effectiveness of the E-Learning or LMS system at XYZ university.
The
population in this study is focused on the use of the E-Learning system at XYZ
university. Because all XYZ university students already use the E-Learning
system, all students meet the requirements to be the population of this study.
Based on the data recorded by PDDikti, there were
6,359 active undergraduate students in Informatics Engineering and 3,522 active
undergraduate students in Information Systems, the population of this study was
9,881 people.
One
technique that can be used to calculate the number of samples is by using the Slovin method as follows:
Thus,
based on the above formula the required number of samples from a population of
9,881 students with an error rate of 5% (95% confidence level) is:
Results and Discussion
Based on the results of distributing the questionnaires, 418 respondents were obtained and 385 respondents were active undergraduate students using the E-Learning system. After further analysis, it was found that 224 respondents were male and 194 respondents were female. The data also shows that 83.8% of the platforms used are various (social media, websites, meeting applications), the rest answer using one platform with an average duration of more than 60 minutes.
By distributing questionnaires, information was obtained that 0.7% of respondents used e-learning for less than 30 minutes, followed by 14.7% of respondents used e-learning for 31-60 minutes, and 91.9% of respondents used e-learning for more than 60 minutes every time they use it.
Structural Model (Inner Model)
After the questionnaire was successfully distributed and 385 respondents were obtained who had been analyzed as needed, the next step was to describe the research model into SmartPLS 3 for processing.
Figure 3: Inner Model Result
Evaluation of the Measurement Model (Outer
Model)
Convergent
Validity Test
A validity test is conducted to measure the level of validity of a questionnaire so that it can be known whether a questionnaire can be used and is able to explain the purpose of a study. The level of validity can be seen from the Loading Factor (Outer Loadings) value of each statement. The minimum value used in this study for Loading Factor is 0.7 based on research conducted by (Hair Jr, Sarstedt, Hopkins, & Kuppelwieser, 2014; Leguina, 2015) it is said that the minimum criterion value for the Loading Factor is 0.7.
In the validity test, there is 1 indicator in 1 invalid variable so the indicator is deleted referring to the research conducted by (Hair Jr et al., 2014). Then retested and got the results as shown in Table 4 Validity Test Results.
Convergent validity test results were also obtained by looking at the value and value of Average Variance Extract (AVE) on each indicator of a variable. The minimum value used in this study is 0.5 for the loading factor and also the AVE can be said to be valid.
Table
4: Validity Test Results
Variable |
Code |
Loading Factor |
AVE |
Desc |
|
Knowledge Acquisition |
KA1 |
0.926 |
0.862 |
Valid |
|
KA2 |
0.931 |
Valid |
||
|
Information Quality |
QI1 |
0.842 |
0.710 |
Valid |
|
QI2 |
0.860 |
Valid |
||
|
QI2 |
0.827 |
Valid |
||
|
System Quality |
QS1 |
0.771 |
0.628 |
Valid |
|
QS2 |
0.777 |
Valid |
||
|
QS3 |
0.828 |
Valid |
||
|
Service Quality |
SQ2 |
0.855 |
0.898 |
Valid |
|
SQ3 |
0.894 |
Valid |
||
|
System Use |
SU1 |
0.861 |
0.710 |
Valid |
|
SU2 |
0.820 |
Valid |
||
|
SU3 |
0.846 |
Valid |
||
|
User Satisfaction |
US1 |
0.899 |
0.771 |
Valid |
|
US2 |
0.886 |
Valid |
||
|
US3 |
0.849 |
Valid |
||
|
System Success |
SS1 |
0.869 |
0.776 |
Valid |
|
SS2 |
0.879 |
Valid |
||
|
SS3 |
0.894 |
Valid |
Reliability Test
Table
5: Reliability Test Results
|
|
Cronbach's Alpha |
Composite Reliability |
|
Information Quality |
0.796 |
0.880 |
|
Knowledge Acquisition |
0.840 |
0.926 |
|
Service Quality |
0.886 |
0.946 |
|
System Quality |
0.704 |
0.835 |
|
System Success |
0.856 |
0.912 |
|
System Use |
0.796 |
0.880 |
|
User Satisfaction |
0.851 |
0.910 |
To test the reliability, it can be done by looking at the value of Cronbach's alpha and composite reliability, where the minimum criterion value is 0.6. And based on the results in Table 5 Reliability Test Results, it is known that all variables meet the criteria and it can be said that the variables can be trusted.
Evaluation of Structural Model
R2 Test
The function of the R-Squared is to find out how much influence an independent variable has on the dependent variable.
Table
6: R2 Result
|
|
R Square |
R Square Adjusted |
|
Knowledge
Acquisition |
0.536 |
0.532 |
|
System Success |
0.676 |
0.674 |
|
System Use |
0.455 |
0.451 |
|
User
Satisfaction |
0.493 |
0.489 |
The R-Squared results in Table 6 show that the Knowledge Acquisition (KA) variable is affected by 53.6%; System Success (SS) variable is influenced by 67.6%; System Use (SU) variable is affected by 45.5%; and the User Satisfaction (US) variable is influenced by 49.3%.
Path
Coefficients
Used to see the direction of the variable relationship whether it is positively or negatively related. In general, the path coefficient is -1 and 1, which means that if the value is close to 1, it has a positive effect and vice versa.
Table
7: Path Coefficients
|
|
QI |
KA |
SQ |
QS |
SS |
SU |
US |
|
QI |
|
0.433 |
|
|
|
0.418 |
0.316 |
|
KA |
|
|
|
|
0.208 |
|
|
|
SQ |
|
0.275 |
|
|
|
0.152 |
0.233 |
|
QS |
|
0.139 |
|
|
|
0.197 |
0.262 |
|
SS |
|
|
|
|
|
|
|
|
SU |
|
|
|
|
0.114 |
|
|
|
US |
|
|
|
|
0.600 |
|
|
All
hypotheses in this study have a positive relationship. The strongest
relationship is shown by the US variable which affects SS by 60.0%. Besides
that, there is also a weak relationship, namely SQ and QS to KA, SU and US then
there is also a moderate relationship, namely QI to KA by 43.3% followed by QI
to SU by 41.8% and QI to US by 31.6 %
Significant Test
To prove whether a hypothesis has a significant effect, analysis needs to be done by looking at the T-Statistic and P-Value through the bootstrap process with a significance level of 5%. Referring to the research conducted by (Leguina, 2015) if the T-Statistic > 1.96 and the P-Value value is smaller than the 0.05 significance level, the hypothesis has a significant effect.
Table
8: T-stat and P-value
|
|
T-stat |
P-value |
|
QI -> KA |
6.313 |
0.000 |
|
QI -> SU |
7.005 |
0.000 |
|
QI -> US |
4.958 |
0.000 |
|
KA -> SS |
4.855 |
0.000 |
|
SQ -> KA |
4.205 |
0.000 |
|
SQ -> SU |
2.568 |
0.011 |
|
SQ -> US |
4.202 |
0.000 |
|
QS -> KA |
1.869 |
0.062 |
|
QS -> SU |
2.644 |
0.008 |
|
QS -> US |
3.769 |
0.000 |
|
SU -> SS |
2.224 |
0.027 |
|
US -> SS |
11.962 |
0.000 |
Table
9: Result interpretation from table 8
|
Hypothesis |
Result |
|
|
H1 |
Knowledge Acquisition will have a positive impact on
System Success |
Accepted |
|
H2 |
User Satisfaction will have a positive impact on System
Success. |
Accepted |
|
H3 |
System Usage will have a positive impact on System
Success. |
Accepted |
|
H4 |
System Usage is positively affected by
Information Quality. |
Accepted |
|
H5 |
User Satisfaction will be positively influenced by
Information quality. |
Accepted |
|
H6 |
Knowledge Acquisition will be positively influenced
by Information Quality. |
Accepted |
|
H7 |
System Usage is positively affected by System
Quality. |
Accepted |
|
H8 |
User Satisfaction is positively influenced by System
Quality. |
Accepted |
|
H9 |
Knowledge Acquisition is positively influenced by System
Quality. |
Rejected |
|
H10 |
Service Quality will have a positive impact
on System Usage. |
Accepted |
|
H11 |
Service Quality will have a positive impact on User
Satisfaction. |
Accepted |
|
H12 |
Service Quality will have a positive impact on
Knowledge Acquisition. |
Accepted |
Interpretation Hypothesis
H1 : Knowledge Acquisition will have a
positive impact on System Success.
The first hypothesis proves that Knowledge Acquisition has a significant positive effect on System Success, this is evidenced by the results of hypothesis testing which shows T-statistics > 1.96, which is 4.855, and P-Values < 0 ,05 which is equal to 0.000. Based on these results, it is concluded that H1 is accepted. These results are in accordance with the results of research (Emelyanova & Voronina, 2014) which says that Knowledge Acquisition affects the success of an LMS.
This proves that E-Learning can be said to be successful if students can absorb the knowledge provided. Therefore, XYZ University Jakarta must continue to maintain and innovate in conducting online teaching techniques. This can be done in various ways, one of which is with interactive lesson content so that students can actively participate in the online teaching and learning process.
H2 : User Satisfaction will have a positive impact
on System Success.
The results of the second hypothesis test prove that User Satisfaction has a significant positive impact on System Success, the results of the hypothesis test show the results of T-statistics > 1.96, which is 11.962, and P-Values < 0.05, which is 0.000. These results are in accordance with the results of research conducted by (Fernando, Titan, Surjandy, & Meyliana, 2020; Freeze, Alshare, Lane, & Joseph Wen, 2010; Subaeki et al., 2020) which says that User Satisfaction affects the success of E-Learning.
Universities
as e-learning service providers must be able to provide a good system that
meets the needs of both students and teachers, such as providing easy access to
attractive features so as to increase user satisfaction.
H3 : System Usage will have a positive impact
on System Success.
The third hypothesis proves that Knowledge Acquisition has a significant positive effect on System Success, this is evidenced by the results of hypothesis testing which show T-statistics > 1.96, which is 2.224, and P-Values < 0.05, which is 0.027. Research conducted by (Jafari, Salem, Moaddab, & Salem, 2015) also shows results that say that System Use affects the success of an LMS.
These results prove that the more often students and teachers use the system, the more successful the system can be. To encourage the level of system usage, in various ways can be done, one of which is to maintain and increase Blended Learning activities and special online classes. This method can increase the use of the system and can increase the capacity of new student admissions.
H4 : System Usage is positively affected
by Information Quality.
The results of the hypothesis test show the results of T-statistics > 1.96, which is 7.005, and the P-Values <0.05, which is 0.000. Based on these results, it is concluded that Information Quality has a significant positive effect. These results are also proven by (Fernando et al., 2020; Freeze et al., 2010; Jafari et al., 2015) who say that Information Quality has a positive effect on the success of E-Learning.
Quality information needed by students and instructors must be provided periodically because the continuity of teaching and learning depends on the system so that users are bound and the use of the system can increase.
H5 : User Satisfaction will be positively
influenced by Information Quality.
The results of the hypothesis test show the T-statistics value > 1.96, which is 4.958, and the P-Values <0.05, which is 0.000. Based on these results, it is concluded that Information Quality has a significant positive effect on User Satisfaction. Testing was also carried out in previous research by (Freeze et al., 2010) which said that Information Quality had a significant positive effect on User Satisfaction.
The ability of the system to provide information needed by students and teaching is highly expected so that the workflow that must be carried out can be understood clearly. This can be done such as giving individual notifications so that the information provided can be right on target, not just general information.
H6 : Knowledge Acquisition will be positively
influenced by Information Quality.
The results of the hypothesis test that show the results of T-statistics > 1.96 are 6313 and the P-Values < 0.05 are 0.000. Based on these results, it is concluded that Information Quality has a significant positive effect on Knowledge Acquisition. These results are in accordance with the results of research by (Petter, DeLone, & McLean, 2008) which says that the desired quality of information from the system output is, management reports and web pages. Example: relevance, easy to understand, accurate, concise, and complete for easy acceptance.
The information provided must be clear and easy to understand so that users can easily absorb the information provided.
H7 : System Usage is positively affected
by System Quality.
The results of the hypothesis test show that the T-statistics value is >1.96, which is 2.644 and the P-Values <0.05, which is 0.008. Based on these results, it is concluded that System Quality has a significant positive effect on System Usage. These results are in accordance with the results of research by (Freeze et al., 2010) which says that System Quality affects System Usage.
To increase the use of a system, of course, quality needs to be considered and improved. The system must be able to provide up-to-date and minimal problems so that users can easily and comfortably use the system anytime, anywhere, especially during productive times
H8 : User Satisfaction is positively influenced
by System Quality.
The results of the hypothesis test show that the T-statistics value is >1.96, which is 3.769 and the P-Values <0.05, which is 0.000. Based on these results, it is concluded that System Quality has a significant positive effect on User Satisfaction. These results are in accordance with the results of research by (Fernando et al., 2020; Freeze et al., 2010; Subaeki et al., 2020) which says that System Quality affects User Satisfaction.
These results prove that the quality of a system can provide satisfaction for its users. One way to improve quality is to update the system and server to minimize bugs.
H9 : Knowledge Acquisition is positively influenced
by System Quality.
The results of the hypothesis test show that the T-statistics value is <1.96, which is 1.869 and the P-Values is >0.05, which is 0.062. Based on these results, it is concluded that Knowledge Acquisition has a positive but not significant effect on System Quality. These results are not in accordance with the results in research (Jain & Gupta, 2019) which says that System Quality improves performance, and innovation, and facilitates the teaching and learning process that affects the acceptance of knowledge.
A good quality system can increase the absorption of knowledge and information for users. This is a special concern so that users can use the system easily and smoothly. The system must also provide information as well as some important features for users.
H10: Service
Quality will have a positive impact on System Usage.
The results of the hypothesis test that show the results of T-statistics > 1.96 are 2.568 and the P-Values < 0.05 are 0.011. Based on these results, it is concluded that Service Quality has a significant positive effect on System Usage. These results are in accordance with the results of research by (Fernando et al., 2020) which says that the quality of the system affects users using the system.
Improving user services such as question and answer services can be a solution so that all user activities can be carried out through the system, including complaints from users. This can help users if they experience problems with the system so that users can be assisted by existing services.
H11: Service
Quality will have a positive impact on User Satisfaction.
The results of the hypothesis test showed the results of T-statistics > 1.96, which was 4.202, and the P-Values < 0.05, which was 0.000. Based on these results, it is concluded that Service Quality has a significant positive effect on User Satisfaction. These results are in accordance with the results of research by (Fernando et al., 2020; Subaeki et al., 2020) which say that User Satisfaction is directly affected by Service Quality.
The system should improve the Help Desk so that users can receive assistance and solutions when needed.
H12: Service
Quality will have a positive impact on Knowledge Acquisition.
The results of the hypothesis test that show the results of T-statistics > 1.96 are 4.205 and the P-Values < 0.05 are 0.000. Based on these results, it is concluded that Service Quality has a significant positive effect on Knowledge Acquisition. These results are in accordance with the results of research (Zaheer & Munir, 2020) which says that Service Quality is considered a form of service feature contained in the E-Learning system. A user-friendly makes it easy to create a smart classroom that can embrace lecturers and students to be involved in an interactive and constructive learning process.
The
materials and information provided must be accessible and even downloadable as
needed. The information provided is not only limited to learning content but
also administrative information.
Conclusion
The rapid development of the era
has made many changes that accompany these developments, one of which is technology.
The internet is used as a means of communication, sources of knowledge and
information from around the world and for daily activities. E-Learning as one
of the topics chosen in this study due to the Covid-19 pandemic situation that
hit various countries so that it hampered the offline teaching and learning
process.
The results of the research on the
factors that influence system success in the e-learning system at XYZ
University show that information quality and system quality have a significant positive
effect on knowledge acquisition. This makes the knowledge acquisition variable
have a positive effect on system success.
There are limitations in this
study, where the selected respondents are limited by the researcher. Further
research can be conducted on students from various majors so that they can gain
different perspectives for more concrete evidence. Further research can be
carried out using other variables that support proving the factors that
influence the success of the system using the DeLone
and McLean models.
BIBLIOGRAPHY
Emelyanova, Natalya, & Voronina,
Elena. (2014). Introducing a learning management system at a russian
university: Students’ and teachers’ perceptions. International Review of
Research in Open and Distance Learning, 15(1), 272–289.
https://doi.org/10.19173/irrodl.v15i1.1701
Fernando, Erick, Titan, Surjandy,
& Meyliana. (2020). Factors influence the success of e-learning systems for
distance learning at the University. Proceedings of 2020 International
Conference on Information Management and Technology, ICIMTech 2020, (August),
294–299. https://doi.org/10.1109/ICIMTech50083.2020.9211163
Freeze, Ronald D., Alshare, Khaled a,
Lane, Peggy L., & Joseph Wen, H. (2010). IS success model in e-learning
context based on students’ perceptions. Journal of Information Systems
Education, 21(2), 173.
Hair Jr, Joe F., Sarstedt, Marko,
Hopkins, Lucas, & Kuppelwieser, Volker G. (2014). Partial least squares
structural equation modeling (PLS-SEM): An emerging tool in business research. European
Business Review.
Handoko, Handoko, Churniawan, Erifendy,
& Riyanta, Wawan. (2022). Simulasi Penanganan Keadaan Darurat pada
Perlintasan Sebidang di JPL Kota Cilegon. Journal Locus Penelitian Dan Pengabdian,
1(8), 637–646. https://doi.org/10.58344/locus.v1i8.287
Jafari, Seyed Mohammadbagher, Salem,
Suha Fouad, Moaddab, Mohaddece Sadat, & Salem, Sharif Omar. (2015).
Learning Management System (LMS) success: An investigation among the university
students. 2015 IEEE Conference on E-Learning, e-Management and e-Services
(IC3e), 64–69. https://doi.org/10.1109/IC3e.2015.7403488
Jain, Namita, & Gupta, Vikas.
(2019). The impact of knowledge management system on student performance: A
case study of the University of Delhi. VINE Journal of Information and
Knowledge Management Systems, 49(1), 115–135. https://doi.org/10.1108/VJIKMS-07-2018-0065
Leguina, Adrian. (2015). A primer on
partial least squares structural equation modeling (PLS-SEM). In International
Journal of Research & Method in Education (Vol. 38).
https://doi.org/10.1080/1743727x.2015.1005806
Petter, Stacie, DeLone, William,
& McLean, Ephraim. (2008). Measuring information systems success: Models,
dimensions, measures, and interrelationships. European Journal of
Information Systems, 17(3), 236–263.
https://doi.org/10.1057/ejis.2008.15
Subaeki, Beki, Manaf, Khaerul,
Pitoyo, Djoko, Setiawan, Hayun, Sugiri, Tia, & Effendi, Asep. (2020). New
Normal Concept in E-Learning Context: An Application of the Delone and Mclean
Information System Model. 2020 14th International Conference on
Telecommunication Systems, Services, and Applications (TSSA, 1–6.
https://doi.org/10.1109/TSSA51342.2020.9310891
Zaheer, Muhammad, & Munir, Saba.
(2020). Research supervision in distance learning: issues and challenges. Asian
Association of Open Universities Journal, 15(1), 131–143.
https://doi.org/10.1108/aaouj-01-2020-0003
Cross, J. (2004). An informal history of eLearning. On
the Horizon, 12(3), 103–110. https://doi.org/10.1108/10748120410555340
Dağ, F., & Geçer, A.
(2009). Relations between online learning and learning styles. Procedia -
Social and Behavioral Sciences, 1(1), 862–871.
https://doi.org/10.1016/j.sbspro.2009.01.155
DeLone, W. H., & McLean, E. R. (1992). Information
systems success: The quest for the dependent variable. Information Systems
Research, 3(1), 60–95. https://doi.org/10.1287/isre.3.1.60
DeLone, W. H., & McLean, E. R. (2003). The DeLone and McLean model of information systems success: A
ten-year update. Journal of Management Information Systems, 19(4), 9–30.
https://doi.org/10.1080/07421222.2003.11045748