Exploring the factors associated with expert systems success

Explorando os fatores associados com o êxito de sistemas especialistas

Exploring the factors associated with expert systems success

Explorando os fatores associados com o êxito de sistemas especialistas

Tor Guimaraes^I; Youngohc Yoon^II; Quinton O’O’Neal^III

^IJ. E. Owen Chair of Excellence. Tennessee Technological University. Cookeville, TN 38505. [phone]. Fax: [phone]. E-Mail (BITNET): [e-mail]

^IIDepartment of Information Systems. Virginia Commonwealth University. Richmond, VA 23284

^IIIManufacturing Expert System Project Center. International Business Machines Corporation. San Jose, CA 95192

ABSTRACT

As the widespread use and company dependency on Expert Systems increase, so does the need to assess their value and to ensure implementation success. This study identified and empirically tested eight major variables proposed in the literature as determinants of ES success, in this case measured in terms of user satisfaction. IBM's Corporate Manufacturing Expert Systems Project Center collected information from 69 project managers to support the study. The results clearly support the hypothesized relationships and suggest the need for ES project managers to pay special attention to these determinants of ES implementation success. ES success is directly related to the quality of developers and of the ES shells used, end-user characteristics and degree of user involvement in ES development, as each has been defined in this study. For exploratory purposes, the component items for each of these major variables were correlated with the components of user satisfaction. Based on the results, several recommendations are proposed for ES project managers to enhance the likelihood of project success, including: adding problem difficulty as a criterion for ES application selection; increasing ES developer training to improve their people skills, ability to model and to use a systems approach in solving business problems; shaping end-user attitudes and expectations regarding ES; improving the selection of domain experts; more thoroughly understanding the ES impact on end-user jobs; restricting the acquisition of ES shells based on a proposed set of criteria; and ensuring a proper match of ES development techniques and tools to the business problem at hand.

Key-words: expert systems, ES, expert systems success, user satisfaction, determinants of success, ES development, ES implementation.

RESUMO

Com o crescimento das aplicações de Sistemas Especialistas (SE), surge a necessidade de avaliar seu valor e assegurar o sucesso da implantação. Este estudo identificou e testou empiricamente oito variáveis principais apresentadas na literatura como determinantes do êxito de um SE, neste caso medido em termos de satisfação do usuário. O Centro de Projeto de Sistemas Especialistas da Divisão de Manufatura da IBM, coletou dados de 69 gerentes de projeto para dar suporte ao estudo. Os resultados confirmam claramente as hipóteses e sugerem a necessidade dos gerentes de projetos de SE dedicarem especial atenção aos determinantes do êxito da implantação. O êxito do SE está diretamente relacionado à qualificação dos que o desenvolvem, à qualidade dos shells utilizados, às características da finalidade da aplicação e ao grau de envolvimento do usuário no desenvolvimento do SE, conforme definidos neste estudo. Para propósitos de investigação, os itens que compõem cada uma dessas variáveis foram correlacionados com os itens componentes da variável satisfação do usuário. Com base nesses resultados, diversas recomendações são propostas aos gerentes de projetos de SE visando o aumento da probabilidade de êxito do projeto: incluir o grau de dificuldade do problema como critério para seleção de aplicação de SE; aumento de treinamento das pessoas que desenvolvem o SE para melhorar suas qualificações; definir as atitudes e expectativas do usuário final em relação ao SE; melhorar a seleção dos especialistas; melhorar o entendimento sobre o impacto do SE no trabalho do usuário final; restringir a aquisição de shells baseando-se em um conjunto proposto de critérios; e assegurar a adequação das técnicas e ferramentas de desenvolvimento de SE ao problema abordado.

Palavras-chave: sistemas especialistas, SE, êxito de sistemas especialistas, satisfa­ção do usuário, determinantes de êxito, desenvolvimento de SE, implantação de SE.

1. Introduction

Expert Systems (ES) are being widely deployed throughout industry, government and other types of institutions. Many organizations have now developed hundreds of ES to assist in a wide variety of areas (SVIOKLA, 1990; LIEBOWITZ, 1990). These facts alone provide strong evidence that ES technology, in general, has been successfully implemented. Several case studies have reported successful implementations of specific ES such as Expertax and DEFT and the significant benefits from these systems (SHPILBERG, GRAHAM, & SCHATZ, 1986; O'NEAL & PALESE, 1988). As ES technology gained credibility and wide application in organizations, it has become a powerful business tool for personal use and for organizations to gain competitive advantage (FEIGENBAUM, MCCORDUCK, & NII, 1988; LIEBOWITZ, 1990). On the other hand, as the investment in the technology grows, so does the need to more carefully evaluate the payback and to better understand the factors related to success or failure when applying the technology. As with many other technologies, there is a substantial time gap among organizations in the adoption of ES technology and in the derived benefits from implementation (LU & GUIMARAES, 1989). Why are some organizations more successful than others with ES applications? As the number of organizations implementing the technology increases and applications within organizations proliferate, understanding the factors important to success becomes essential.

The literature has an abundance of studies assessing computer-based system success factors. Unfortunately, the vast majority of the studies deal with Transaction Processing Systems (TPS), Decision Support Systems (DSS), Executive Information Systems (EIS), or other computer-based information systems besides ES. The term ES in this case refers to systems which comprise at least a knowledge base, an inference engine, an explanation module, and a user interface in order to mimic expert decision making (JIH, 1990). While obviously there are many similarities, such ES are quite different from other systems in many ways. For example, the basis for ES is the capture and use of knowledge from high-level experts to assist the less proficient ES end-users. The other systems deal with the flow of data and information through the organization unit, and their development requires considerable end-user input to define the nature, functions and features of the system. Therefore, for TPS, EIS and DSS the end-users are the "domain experts."

ES are also dramatically different from other types of systems in other ways. TPS perform routine data processing and focus on business procedures. DSS support decisions and focus on computational and statistical models. On the other hand, while ES also make or support decisions, they focus on knowledge/expertise rather than on transaction processing or computational techniques (DUDA & SHORTLIFFE, 1983). ES are heuristics-based and are particularly viable for problems that cannot be solved algorithmically (WRITZEL & KERSCHBERG, 1989). Even a summary discussion of the many differences between ES and other systems would require several pages, but three references stand out and should be mentioned. JIH (1990) found differences between ES and TPS along seven areas: basic mission, knowledge structure definition, requirements for input certainty and completeness, system shareability, system human interfaces, data abstraction, and validation. TURBAN & WATKINS (1986) discussed the differences between DSS and ES along eleven attributes: objectives, who makes the decisions, major orientations, major query direction, nature of support, data manipulation method, characteristics of problem areas, repetitiveness of problems addressed, content of databases, reasoning capability, and explanation capability. Subsequently, TURBAN (1990) addressed the differences among TPS, MIS, DSS, EIS and ES in terms of application types, support focus, nature of the databases, decision making/support capabilities, numeric versus symbolic manipulation, types of information produced, highest organizational level served, and primary impetus for the system.

Given the dramatically different nature of ES, results obtained from previous studies are likely to provide at best a partial picture of the major issues surrounding ES development and implementation in practice. There are several reasons why studies dealing specifically with ES are important and must be undertaken: (1) Despite widespread use and increasing importance of ES technology, little effort has been made to systematically identify and empirically test the determinants of ES success. (2) Managers responsible for the development and implementation of ES cannot assume that the set of determinants for success with other system types are equally important for ES. Even if one believes that to be so, this assumption must be tested empirically. (3) Many important factors are unique to ES, i.e., their "expert mimicking" nature, the domain oriented problems addressed, the characteristics of ES shells, the required activities and characteristics of domain experts and knowledge engineers, and their unique relationship with end-users. (4) The reports about ES (BARSANTI, 1990; IGNIZIO, 1991; KEYES, 1989; O'NEAL, 1990; PRERAU, 1990; SMITH, 1988) are mostly based on the opinion and personal experience of one individual and have not been empirically tested. Empirical studies conducted by BYRD (1992) and TYRAN and GEORGE (1993) reported on general issues and factors surrounding ES implementation, but much more research is needed to synthesize previous findings, formulate and empirically test hypotheses regarding the likely determinants of ES success, and to build a theoretical foundation in this important area. The purpose of this study is to empirically test the various factors proposed in the literature as important to ES implementation success. The next section describes the theoretical underpinnings for the study.

2. The Theoretical Framework

Much of the research on computer-based system implementation has been focused on identifying factors conducive to success or failure, including user involvement (BARKI & HARTWICK, 1989; BARONAS & LOUIS, 1988), management support (LEE, 1986; LEITHEISER & WETHERBE, 1986), end‑user's expectation and attitude (GINZBERG, 1981; MAISH, 1979; ROBEY, 1979), politics (MARKUS, 1983), communications between developers and end‑users (De BRABANDER & THIERS, 1984), task structure (SANDERS & COURTNEY, 1985; GUIMARAES, IGBARIA & LU, 1992), and end‑users' training and experience (FUERST & CHENEY, 1982; NELSON & CHENEY, 1987).

Similarly, prior studies have identified several factors related to ES successful implementation. BARSANTI (1990) reported on five ES success factors: top-down corporate support, expert selection, support group, tool selection, and project selection. Four major variables encompassing factors regarding the organization, the implementation process, technical aspects, and ES ease of use were included in an experimental study of ES implementation success conducted by TYRAN and GEORGE (1993). In another study by BYRD (1992), the factor analysis revealed three factors related to ES success: knowledge engineering, job impact, and knowledge acquisition.

Some of the success factors related to other system types may also be applicable as determinants of ES success. However, as mentioned earlier, one cannot assume that without empirical evidence. Therefore, while the focus is on ES, a comprehensive list of variables proposed in the literature as important determinants of system success was developed. Eight major variables were selected as independent variables for this study. Their selection is based on the existence of literature supporting their relevance as likely determinants of ES implementation success and/or on the opinion of eight experts in the ES area who assisted with questionnaire development. User satisfaction, the dependent variable and surrogate measure for ES success, is discussed next. That is followed by a discussion of each of the major independent variables in Table 1 and the rationale for their respective hypothesis.

User Satisfaction: Prior research has employed various measures of system success, including user satisfaction (GALLETTA & LEDERER, 1989; IVES, OLSON, & BAROUDI, 1983; KENDALL, BUFFINGTON, & KENDALL, 1987; MAHMOOD & SNIEZEK, 1989), level of system usage (FUERST & CHENEY, 1982; MYKYTYN, 1988), perceived benefits of systems (DAVID, 1989; MONEY, TROMP, & WEGNER, 1988), improved decision quality and performance (ALDAG & POWER, 1986; CATS‑BARIL & HUBER, 1987; KOTTEMANN & REMUS, 1989), and business profitability (BENBASAT & DEXTER, 1982; ECKEL, 1983; SHARDA, BARR, & McDONNELL, 1988). The choice of "best" measure for system success depends on the study objectives. All the measures mentioned above, including user satisfaction, have advantages and disadvantages. On the other hand, user satisfaction has been proposed by some as "a substitute for objective determinants of information system effectiveness" (IVES, et al., 1983), as the most useful surrogate measure of system success (GUIMARAES & GUPTA, 1988), and as "the most useful assessment of system effectiveness" (HAMILTON & CHERVANY, 1981). Further, a comprehensive discussion of system success by DeLONE & McLEAN (1992) proposes that compared to other factors, user satisfaction has been widely used, making it important as an enabler for inter-study comparison. For all these reasons we used it in this study.

Problem Characteristics (Difficulty): The relationships between task characteristics and the success of computer‑based information systems have been studied by several authors (CHENEY, MANN, & AMOROSO, 1986; GUIMARAES, et al., 1992; SANDERS & COURTNEY, 1985; PIEPTEA & ANDERSON, 1987). For ES development, some authors have stated that selection of an improper domain is likely to result in ES implementation failure (BARSANTI, 1990; BECKMAN, 1991; KEYES, 1989; LIEBOWITZ, 1989; SLAGLE & WICK, 1988; WATERMAN, 1986). PRERAU (1990) stressed that the selection of an appropriate ES domain is more closely related to the ultimate success of the project than any other factor. SLAGLE and WICK (1988) echoed PRERAU's statement and proposed an evaluation method for choosing appropriate applications. Of particular importance in the case of ES is the stability of task knowledge since constantly modifying a knowledge base to reflect changes in the business task is arduous work (BECKMAN, 1991). Other desirable business task characteristics for successful ES implementation are simplicity, ease of understanding, and manageable size (SMITH, 1988; WATERMAN,1986). CASEY (1989) proposed that ES tasks with a narrow, well‑defined focus would be more likely to be successfully implemented. Task uncertainty is thought to reduce the accuracy of system results, decreasing the likelihood of ES success. However, most of these statements are based on personal opinion and/or single case observation. Despite widespread belief in the importance of the many variables underlying problem difficulty to ES success, there is practically no empirical research testing their relationships. As alluded to earlier, ES are quite different from MIS and DSS. In the case of MIS and DSS, the end-users are the "domain experts;" thus, developers are required to have considerable interaction with end-users to define the nature, functions and features of the system. In the case of ES, the more advanced knowledge of domain experts is used to assist the system end-users in solving problems. Assuming the experts can properly address the problem, increased problem difficulty to end-users represents a major opportunity for the ES to be of greater service to its end-users. Since, in this study, user satisfaction with the ES is the measure for success, we expect a direct relationship between problem difficulty and ES success, and propose H1: Problem difficulty is positively related to ES success.

Developer(s) Skill: A number of developer characteristics were found to affect system implementation success. One is the developers' abilities to minimize conflicts with end‑users (GREEN, 1989; KAISER & BOSTROM, 1982; LUCAS, 1975). WHITE and LEIFER (1986) explored the impact of developers' skills on MIS success and reported that a range of skills, including both technical and process skills, are important for system success. Due to the special nature of ES and its development process, the importance of skillful ES developers has been emphasized by several authors (PAYNE & AWAD, 1990; SHACKLETT, 1990). Different from most other systems types, the construction of an ES requires developers to elicit the decision rules employed by domain experts. In order to elicit the decision rules, developers must ask relevant questions and quickly comprehend the decision procedures reasoned by domain experts while problem solving. The knowledge elicitation procedure-‑the lengthy process of interviews--is widely known to be a bottleneck in ES development. It is obviously desirable for a developer to possess excellent communication skills in order to alleviate the difficult knowledge elicitation process. A developer with poor communication skills may not be able to properly perform the critical knowledge acquisition task, causing project implementation failure (PRERAU, 1990). Strong knowledge of various functional areas in an organization improves developers' communication with end-users, as well as domain experts, and also helps save everyone's time and effort. Once knowledge is elicited from domain experts, it is represented and stored in a knowledge base using a programming language or an ES shell. For this purpose, a developer should be familiar with various knowledge representation paradigms and ES building tools, which are important requirements for technical competence. Thus, H2: Developer(s) skill is positively related to ES success.

Domain Expert(s) Characteristics: A unique feature of ES development is the involvement of domain experts in the development process. Domain experts' problem‑solving expertise in a specific domain is the basis of an ES, and the domain knowledge embodied in a knowledge base to assist in solving problems is considered to be the essence of an expert system (WATERMAN, 1986). The importance of domain experts to ES success cannot be overemphasized since often the only source of domain knowledge for the ES is one or more domain experts. Depending upon the quality of the domain experts, ES implementation may or may not be successful (IGNIZIO, 1991). BARSANTI (1990) also stressed that when developing an ES, the issues regarding domain experts must be carefully considered. Domain experts must be articulate and willing to participate in the project (PRERAU, 1990). Thus, H3: Domain expert characteristics are positively related to ES success.

End-User(s) Characteristics: In the ES literature, the dominant end‑user characteristics affecting ES success include user attitude, user expectation and user knowledge of computer and ES technology (SMITH, 1988). User attitude has been considered as an important factor to ES success since end‑users with a negative attitude toward an ES will not utilize the system, completely wasting development costs. As discussed below, in the case of ES, users often have great fears about the ES affecting their job security, thus developing strong negative attitudes and challenging the system implementation (LU & GUIMARAES, 1988). This problem is more apparent with ES which by their very nature are more likely to significantly change the nature and requirements of the job and replace human tasks with artificial systems, i.e. the effect of XCON (SVIOKLA, 1990). Another end‑user characteristic mentioned in previous studies is the user knowledge of computer and ES techniques. Unlike domain experts who are expected to know a great deal of ES techniques, end‑users do not need to have much knowledge about AI and expert systems, thus creating a mysterious aura about such systems. These important statements should be empirically tested, thus H4: End-user characteristics are positively related to ES success.

ES Impact on End-User(s) Job: Due to the unique nature of ES described earlier, knowledge engineers have recognized ES impact on the end‑users' jobs as an important factor to successful ES implementation (BYRD, 1992). Changes in a work environment frequently increase end-users' fears about their jobs and, in turn, may generate resistance against a new ES (SLOANE, 1991). The psychological impact and the organizational changes brought on by ES technology can be far more marked than those created by any other type of system. Indeed, this has been considered as a particularly important factor. SVIOKLA (1990) analyzed the reason for the successful implementation and use of an ES (XCON at DEC) by examining its impact on end‑users' jobs. Previous studies showed that the end‑users suffered from stress and loss of control when AI technologies were introduced to their working environments (ARGOTE & GOODMAN, 1986; ARGOTE, GOODMAN, & SCHKADE, 1983). The fear of job loss can result in end‑users resisting the ES implementation and making its success very difficult. The ES impact on end‑users' jobs seems to be closely related to ES success, thus H5: ES desirable impact on the end-user(s) job is positively related to ES success.

Shell Characteristics: ES shells represent a unique feature for ES development which save developers a substantial amount of time and effort. The commercial availability of ES shells with highly sophisticated features in today's market has significantly raised shell usage and introduced another potentially very important factor to ES success. An ES shell can be generally characterized by its features, including the knowledge representation methods, inference engine methods, developers interface, end‑users interface, and others (HARMON, MAUS, & MORRISEY, 1988; KIM & YOON, 1992; VEDDER, 1989; VEDDER, FORTIN, LEMMERMANN & JOHNSON, 1989). The shell used to develop an ES can be an important determinant to its quality (BARSANTI, 1990), therefore, employing an ES shell appropriate to the ES tasks is vitally important to ES success. The desirable features of an ES shell will vary significantly depending on the tasks to be performed by the ES. The ES reasoning process is required to match that of human experts. If an ES cannot be constructed to reason in the same fashion as human experts due to the shell's inflexibility in controlling methods, the system will not behave "naturally" to end‑users, discouraging ES use.

For many applications, shells must enable the ES to be easily integrated with existing computer-based systems. However, many ES are capable of only limited interface with existing systems due to the ES shell used. This impediment forces the ES to acquire data through an interactive process with end‑users even though the raw data are already stored in existing data bases. Needless to say, entering data in an interactive manner is tedious and inconvenient, discouraging users from utilizing the ES and decreasing its usefulness. Similarly, a shell providing a friendly user interface encourages end‑users to more frequently use the system. Thus, H6: Shell characteristics are positively related to ES success.

User Involvement: In contrast with most DSS whose development are heavily dependent on user involvement (GUIMARAES, et al, 1992) for information requirements definition, in ES development domain experts are in most cases the primary source of knowledge and inference about the problem. However, high levels of user involvement are still considered by some to be important to ES success (SMITH, 1988). Users who initiated an ES project and were involved in establishing its goals/objectives are more likely to be satisfied with the system. KEYES (1989) argued that if the end‑users were excluded up front, they would exclude themselves at the end and not use the ES. TURBAN (1990) claimed that in the verification and validation phases, user involvement becomes important. In general, for ES development, user requirements are unknown at the early stages. Instead, they evolve during iterative development processes in which a working model is used as a vehicle to help communications between users and developers. Therefore, the high level of user involvement is strongly encouraged during the validation and verification process in order to ensure the development of ES which would satisfy the user information need. To increase user satisfaction with the ES, heavier user involvement is also recommended to design the user interface and to determine the system availability and access according to their needs. Thus, H7: User involvement is positively related to ES success.

Management Support: The importance of management support to ES success has been emphasized by several authors (DEPREE, 1988; KEYES, 1989; SMITH, 1988). KEYES (1989) reported that lack of management support was a critical barrier to ES success, and BARSANTI (1990) said that a key predictor of ES success in an organization is the existence of top‑down corporate support. The importance of management support for the successfully implemented ES (XCON) at Digital Equipment Corporation was illustrated by LEONARD‑BARTON (1987) and LEONARD‑BARTON & DESCHAMPS (1988). The survey conducted by BYRD (1992) also revealed management support as an important factor to ES success. Several reasons for the importance of management support can be described. First, management support is essential to receive personnel and monetary resources necessary to the development. Without their support, a system development cost will not be funded, resulting in a system failure. Second, the adoption of a new technology by an organization always results in some change in the manner in which decisions are made, business tasks are performed, and power is allocated. Changes in a work environment frequently increase end‑users' fears about their jobs and, in turn, may generate resistance against a new ES (SLOANE, 1991). The psychological impact and the organizational changes brought on by ES technology can be far more marked than those created by any other type of system. In such cases, management support is crucial to mitigate end‑users' negative attitudes toward ES, and to overcome the user resistance. Third, a desirable application area for an ES project is where expertise is scarce or expensive or where the experts are overworked (LU & GUIMARAES, 1988). In such cases, the direct supervisors of domain experts may find it difficult to share their time for ES development. However, higher‑level management may take a longer view and be more amenable to investing substantial amounts of time over the period necessary for effective ES development (PRERAU, 1990). Management can also play a vital role in dealing with uncooperative domain experts by assuring them about their jobs and convincing them of the potential ES benefits to the organization. Thus, H8: Management support is positively related to ES success.

3. Study Methodology

This section addresses the study setting, the reliability and validity of the measures used, and the sampling procedure. Due to the serious lack of empirical evidence about the determinants of ES success in practice, instead of a multi-company survey, a case study approach to data collection has been used. This approach is expected to reduce the possibility of confounding results due to inter-organization differences such as ES development sophistication, budget, methodologies used, development resources available, policies, conflicts between user departments and the company's ES development group, etc. The company in this study is International Business Machines (IBM), selected because of its sophisticated ES development environment with many experienced ES developers and an impressive collection of ES applications known to have provided substantial benefits to the end-user community (O'NEAL & PALESE, 1988). IBM created its Corporate Manufacturing Expert Systems Project Center (CMESPC) in San Jose, California, in 1988 to develop, implement and measure the results from Knowledge Based Systems on internal manufacturing operations. Since the Center's inception, close to 460 ES manufacturing applications have been developed and implemented internationally throughout the organization in areas such as process control, integrated testing and analysis, manufacturing repository, multimedia systems, and the "factory of the future" in Fujisawa-Japan.

3.1 Variable Measurement

The specific items composing each major independent variable are shown in the body of Table 1. The components for user satisfaction are at the bottom of the table.

User Satisfaction: The psychometric qualities of the measure used in this study is relatively well established. Starting with the user satisfaction measure originally developed by BAILEY & PEARSON (1983), IVES, et al. (1983) produced a shorter form by excluding 26 items from the original 39 item instrument. RAYMOND (1985) also adapted the instrument and developed a 20‑item questionnaire. By factor analysis these items have been grouped into four factors: output quality, user‑system relationship, support, and user relationship with EDP staff (RAYMOND, 1985). Instead of measuring user satisfaction in all four areas, this study used only the 9 items related to the first two areas. The first six assess user satisfaction with the quality of an expert system's output. The other three assess user satisfaction with the ES ease of use, ease of learning, and the usefulness of the available documentation. This scale excluded items such as management support, user relationship with EDP staff, and vendor support which are not directly related to the ES. This approach is deemed more appropriate in the case of ES and is similar to the one used by LUCAS (1978) to measure the quality of a model in terms of its output, problem solution, and system interfaces with the users.

Problem Difficulty: Problem difficulty has been defined by VAN DE VEN & FERRY (1980) as the degree to which an individual problem solver can follow a formal and well defined search procedure to solve a problem. Many variables were employed to assess problem difficulty in previous studies. In order to measure ES task difficulty, eight variables were operationalized according to BECKMAN (1991), GUIMARAES, et al. (1992), and WATERMAN (1986).

Developer(s) Skill: There are some unique skills required from ES developers. For ES development, DEBENHAM (1990) listed four essential skills of knowledge engineers: the ability to extract accurate and complete knowledge from human experts; the ability to represent and implement knowledge; the ability to design an ES for maintenance; and the ability to design an ES that exploits existing investments in information processing. A comprehensive list of knowledge and skills necessary for ES developers has been developed by PAYNE & AWAD (1990). It includes knowledge of computer technology, knowledge of general fact‑finding techniques, knowledge of prototype methods, knowledge of human factors, knowledge of functional areas, communication skills, project planning skills, human relations skills, organizational skills, and personal attributes. Although the range of system developers' skills and abilities varies slightly between studies, they were classified into six general categories, according to NUNAMAKER, COUGER, & DAVIS (1982), which are used in this study.

Domain Expert(s) Characteristics: Prior studies have employed various domain expert characteristics thought to influence the likelihood of ES implementation success (IGNIZIO, 1991; PRERAU, 1990; SLAGLE & WICK, 1988). The quality of an ES depends on the skills and abilities (expertise) of its domain experts. The domain experts' ability to articulate the reasoning process and to communicate with knowledge engineers becomes an important factor. Another crucial attribute is the experts' willingness to cooperate. Lack of domain experts' cooperation may come from apprehension that an ES may threaten their jobs. Experts may consider participation in an ES development project to be an inconvenient or an unimportant break from their jobs. Sometimes, domain experts are assigned by management to an ES project against their will. Experts who are uninterested or resentful of participating in an ES development project may not devote themselves enough or may withhold their knowledge. This is particularly serious for large and complex problems, when the ES development costs are substantial. In such cases, it may take many weeks or months to elicit domain knowledge and complete the knowledge acquisition process. Therefore, uncooperative domain experts are likely to be a serious barrier to ES development and successful implementation. Last, without understanding ES technology, domain experts may have too high expectations from ES developers and/or the development process; or they may underestimate ES capabilities and potential usefulness. Therefore, based on PRERAU (1990), a domain expert's quality was measured in terms of four items.

End-user Characteristics: As proposed by SMITH (1988), three items were used which address user attitude, expectation, and knowledge of computers and AI technology.

ES Desirable Impact on End-User(s) Job: The factor analysis conducted by BYRD (1992) revealed ES impact on end‑user(s) jobs as consisting of two items: fear of loss of control and fear of job loss. SVIOKLA (1990) analyzed the impact of XCON on end‑users' jobs by examining the changes in input and output, the increase in task accuracy and amount of work completed, the shifts in the end‑users' role and responsibilities, and job satisfaction. Based on BYRD (1992) and SVIOKLA (1990), this study employed 11 variables to measure the desirable impact of ES on end-users' jobs.

Shell Characteristics: The desirable features of a shell listed in the literature include flexibility of knowledge representation, flexibility of the inference engine, ability of monitoring session activities, knowledge-base editing facility, knowledge acquisition facility, debugging aids, managing data uncertainty, end‑user interface, explanation facility, integration with external databases and other systems, integration with other programming languages, ease to use, ease to learn, portability, response time, real time support, documentation, and vendor support (BRODY, 1989; HARMON, et al., 1988; KIM & YOON, 1992; VEDDER, 1989; VEDDER, et al., 1989). Obviously, some features are applicable only to specific expert systems; i.e., the capability of real‑time support is applied exclusively to a real‑time ES application. Based on HARMON, et al. (1988), KIM & YOON (1992), and WATERMAN (1986), 10 features deemed applicable to a wide range of ES applications were used.

User Involvement: We assessed user involvement in nine different activities. The first eight items are the measure of user involvement by DOLL & TORKZADEH (1989), the last represents LUCAS' concept of user control, determining system access and availability (1978).

Management support: Based on the propositions by SLOANE (1991), PRERAU (1990), and BYRD (1992) which were discussed earlier, we used four items to assess management support.

3.2 Validity of the Measures

Despite the exploratory nature of this study, several precautions were taken to ensure the validity of the measures used. Many of the recommendations by CARMINES & ZELLER (1979) were followed. To ensure content validity, a thorough survey of the relevant literature was undertaken to understand the important aspects of each major variable and its components, and not to neglect any important dimension of any variable.

As discussed earlier, the theoretical underpinnings of this study are relatively well established, with most of the constructs and the proposed relationships having been addressed before by several authors. To further reduce the possibility of any non-random error, the main source of invalidity, a group of four academics and three practitioners, experts in the area of ES development management, reviewed the questionnaire for validity (measuring the phenomena intended), completeness (including all relevant items), and readability (making it unlikely that subjects will misinterpret a particular question). A few questions were reworded to improve readability; otherwise, the items composing each major variable remained as derived from the literature.

3.3 Reliability of the Measures

Many of the measures used were chosen because they have been previously used and their psychometric properties are relatively well known. Factor analysis showed that the items for each scale loaded unambiguously (> .50 into one factor and < .30 into the others) thus indicating construct unidimensionality, a requirement for computing the Cronbach's Alpha. As shown in Table 1, the internal consistency reliability coefficients (alpha) for the constructs in this study are all well above the level of .50 acceptable for exploratory studies (NUNALLY, 1967). One exception is the alpha coefficient for domain experts characteristics which was .23 when education level and work experience were part of the scale. These two items were dropped from the scale raising its alpha to .63 and were considered separately in the sub-items correlation analysis.

3.4 Sampling Process

The 170 project managers who are in charge of the development, as well as the maintenance, of expert systems throughout IBM installations worldwide were invited to participate in this study via telephone and e-mail. Subsequently, questionnaires were distributed to project managers via e-mail. They were explicitly asked to provide information on an ES whose development they had managed and which was in operation for at least one year. Project managers should be considered the most appropriate sources of information for this study for three main reasons: (1) They are closely involved with the business processes. (2) They are the only people familiar with the overall ES implementation process: the domain experts, the developers, the development tools, and the end-users who participated in the development process of specific ES. Furthermore, through the company's performance evaluation and reward system, project managers are induced to assume an end-user perspective and a long term view of project success. In other words, the project managers' performance is judged on the ability of the ES to accomplish specific end-users objectives. An unduly favorable project manager view of an ES is meaningless in this case. Overall end-user satisfaction with the system in the long-run, regardless of this study, is the only performance measure that matters. (3) To have ES developers or domain experts rate their own performance would unduly raise the risk of response bias. Nevertheless, readers are cautioned that most of the information collected for this study represent the opinions of project managers.

For the measures of user satisfaction and desirable ES impact on users' jobs, it was deemed important that end-users provide their own opinion of the ES. The primary end-user for the particular ES (the one who uses it most often) was chosen by one of the authors, a manager in the host company, to provide his/her response directly to him via e-mail or hard copy.

Of the 170 questionnaires which were sent out, 69 were filled out in time to be processed for this report (a response rate of 40 percent). Table 2 shows selected demographic information on project managers. Based on the wide representation of project managers in terms of country of origin, education, and job experience, no sampling bias is apparent. Further, one of the authors (a host company manager) found no reason to suspect sample bias in any way. Some of the respondents did not provide some demographic information, thus the unknown category in Table 2. For the other variables, missing items were negligible.

3.5 Data Analysis Procedures

In order to examine whether the hypothesized relationships were significantly different according to the type of ES applications or the type of ES building tools being used, non-parametric Mann-Whitney (M-W) tests were performed. Given the relatively small sample size and the possibility that the relevant variables are not normally distributed, non-parametric tests were considered more appropriate to analyze the statistical differences between groups of different ES applications along major study variables. Similar to the question about potential ES types bias, the ES shell used for development may affect the study's results. If only one shell is used, one may question whether the results are applicable to ES developed with other shells. Again, M-W tests were conducted to analyze the difference between ES developed with different shells. The hypothesized relationships were tested by computing correlation coefficients between the major study variables. Last, the correlation coefficients among major variables' sub-items were also computed.

4. Discussion of Results

4.1 Sample Description

The 69 ES reported on by each project manager are rule-based expert systems which have been in operation for at least one year. A description of the participating ES project managers was presented earlier. The end-users of these ES can be classified as: operational level (41), business professional (10), programmer (6), management (10) and unknown (2). Table 3 shows the demographic information on end-users and domain experts. On the average, the domain experts are more experienced and educated than end-users. Therein lies a basic reason for developing ES--experts assisting the less advanced end-users in solving problems.

The 69 applications represent a wide variety of areas grouped as (not mutually exclusive): systems configuration (5), training/education (10), problem diagnosis (31), process control (9), procedural directives (12), planning (12), alternative selection (3), forecasting (5) and others (11). For the M-W test, the 69 ES were divided into "for diagnostic" (n=31) and "for others" (n=38). The test revealed no significant differences between diagnostic and non-diagnostic ES along the major study variables, with the lowest p-value at .15 (user involvement) and the others ranging from .31 to .91.

The ES in this study were developed using a wide variety of shells. Many (27) were developed with ESE, the rest are based on sixteen different shells such as KEE (5), TIRS (5), KNOWLEDGE TOOL (4), KNOWLEDGE DIRECTOR (3), ART-IM (2), one ES each developed with DAD, OCTOPSY, REXX, and others. Results from an M-W test comparing the ESE shell against the others as a group along all the major variables in this study revealed no statistically significant differences except for problem difficulty. Apparently, the ESE applications, on the average, tend to be associated with simpler business problems.

4.2 Hypotheses Testing Results

The means, standard deviations, and the matrix of intercorrelations among the nine major study variables are presented in Table 4. User satisfaction correlated positively and significantly with all eight independent variables. The following hypotheses are corroborated at the 0.01 significance level or better:

H2: Developer(s) skill is positively related to ES success.

H4: End-user characteristics is positively related to ES success.

H5: ES desirable impact on end-user(s) is positively related to ES success.

H6: Shell characteristics is positively related to ES success.

H7: User involvement is positively related to ES success.

The hypotheses below are corroborated at the 0.05 or better significance level:

H1: Problem difficulty is positively related to ES success.

H3: Domain expert quality is positively related to ES success.

H8: Management support is positively related to ES success.

While not proposed for empirical testing, other significant relationships in Table 4 are worth interpreting for exploratory purposes. As should be expected, ES with greater desirable impact on end-users' jobs tend to address relatively more difficult problems, to use better qualified domain experts and better quality shells, and to have greater user involvement in the ES development process. To the credit of project managers, the more skillful developers are using higher quality shells and are being assigned to the development of ES which are addressing more difficult business problems. The latter, in turn, enables ES to have greater beneficial impact on end-users' jobs. For a difficult problem, knowledge acquisition and representation become more intractable requiring excellent communication and technical skills. Difficult problems enable ES to have a greater impact on users' jobs because the more advanced knowledge of top domain experts is leveraged to assist the system end-users, and without the ES, users would encounter difficulties addressing the problem.

4.3 Sub-item Intercorrelations

Given the lack of theoretical support for many of the sub-item intercorrelations, this analysis was conducted strictly for exploratory purposes and readers are cautioned in interpreting these results. The intercorrelations between each sub-item of the dependent variable and those of the eight independent variables are presented in Table 5. In the case of ES output value (US1), ES assistance is found to be better appreciated in solving more difficult problems than simpler ones. This corroborates the opinion of other authors (PRERAU, 1990; WATERMAN, 1985; IGNIZIO, 1990) that the domain task selected should not be too trivial or the ES may not be valuable. Obviously, if an ES solves a problem that is too difficult for end-users to deal with, they would greatly appreciate the ES assistance. Three items under developers' skills (people, model, and systems) are directly related with the value end-users assign to their ES output and their satisfaction with its timeliness. Developers' effective communication and interpersonal skills enable them to more thoroughly understand end-users, leading to higher satisfaction along these two variables. Similarly, when developers can skillfully model business problems, the ES output tends to be more useful to end-users.

The result shows that end-users' positive attitude clearly influences their satisfaction with the value of ES output. The importance of the end-users' jobs, job appeal, feedback on job performance, opportunity for advancement, and job satisfaction are also related to the value of ES output. One may assume that when end-users benefit from an ES in performing their jobs, they tend to assign more value to its output. The shell's appropriateness for the problem is shown to be an important factor in determining the quality of an ES. All shells are not equally applicable to a problem. A shell inappropriate for a given problem is not likely to allow developers to properly address user business requirements. That, in turn, reduces the value of the ES output.

For ES output timeliness (US2), users tend to view ES output as more timely when the system is addressing problems which are large, complex and require high expertise. Experts' cooperation is another important factor affecting user satisfaction with the timeliness of ES output. Many authors have emphasized the importance of domain expert cooperation in developing an ES (PRERAU, 1990; TURBAN, 1990). Since a domain expert is a primary source of high-level expertise for an ES, their cooperation has often determined the success or failure of an ES implementation. Even after successful ES deployment, the knowledge base of many ES need to be continuously updated to reflect the change in domain knowledge. The knowledge base is often maintained by end-users with domain experts' assistance. The latter's cooperation becomes vitally important for prompt modification. Four shell characteristics are directly related to user satisfaction with ES output timeliness: ease to use, ease to learn, response time capability, and shell's appropriateness for the problem. The meaning of these findings is obvious and emphasizes the importance of carefully selecting a shell with appropriate features, properly matching the business problem being addressed.

Users seem to be more impressed with ES output reliability (US3) when it tackles labor intensive, complex problems which require high expertise. It is obviously more difficult to impress users about ES reliability when addressing simple problems. The relationship between ES reliability and the user interface can be attributed to ES explanation facilities, an integral part of the user interface. The facility provides information regarding how and why an ES reached a particular conclusion. A weak explanation facility could lead users to believe that the ES outcome is unreliable.

The response/turnaround time of a selected shell is an importance factor influencing end-users' satisfaction with the ES response time (US4). For example, an ES could be inherently slow due to an inefficient inference engine in the shell used. The results also show that the user interface of a shell affects user satisfaction with the ES response time. Depending upon the information content and form displayed by end-users' interface, users may have different impressions about ES response time. For example, displaying the stepwise execution of the inference engine may increase user satisfaction with the system's perceived response time. The results support the notion that carefully selecting a shell appropriate to the problem is likely to minimize design problems whose symptoms (slow response time) become apparent later when the ES is operational. User involvement in determining ways to meet their requirements is directly related to their satisfaction with the ES output response time, likely because developers have the opportunity to learn first hand about user response time requirements.

For obvious reasons, ES output accuracy (US5) is directly related to the modeling skills of developers and domain experts' cooperation. In cases where the ES decreases the amount of work required, users tend to be more satisfied with the accuracy of the ES output. Very likely this is due to the difference in ES versus user accuracy for the tasks involved. The shell's user interface is also related to user satisfaction with the ES output accuracy. Similar to user perceptions about response time, this is likely to be dependent on the output information content and form. For example, the interface may provide alternative output forms for cross-validation. User satisfaction with the accuracy of ES output is also directly related to user involvement in determining user requirements, determining ways to meet requirements, identifying sources of data and information, outlining information flow, and developing input forms/screens. One likely interpretation is that users have greater confidence in ES output accuracy when they participated in these ES development activities.

User satisfaction with the completeness of ES output (US6) is directly related to domain expert cooperation and the modeling skills of developers (their ability to provide complete representations of the business problems involved). For example, uncooperative experts may withhold some of their knowledge, causing an incomplete knowledge base. The results also show that the shell's user interface is important here. Providing complete explanations for the why and how of ES behavior is often a very difficult task. Many ES render only partial explanations, possibly causing users to think that the ES output itself is incomplete. Last, the shell's appropriateness for the problem is also an important factor for user satisfaction with ES output completeness since an improper shell is likely to hinder developers in representing all the important features of a chosen domain.

User satisfaction with ES ease of use (US7) and ease of learning (US8) are directly related to users' positive attitude about ES, the shell appropriateness for the problem, job security, and job satisfaction. While the first two have obvious interpretations, the last two are difficult to interpret. Possibly, ES which enhance job security and satisfaction receive more attention from users who perceive the efforts for learning how to use them and for using them as worthwhile. It is interesting to note that shell ease of use and of learning are no insurance that the ES itself will have the same characteristics. Management support expressed in different ways (i.e., providing resources, training, moral support) is apparently one of the most important factors to ensure ES ease of use and of learning. The unimportance of developer skills in this case is counter-intuitive. Apparently, unless the shell is properly matched to the business problem involved, developer skill is not enough to deliver a "user friendly" ES.

Regarding user satisfaction with the usefulness of the ES documentation (US9), for obvious reasons, the results indicate that user expectations and attitude toward ES are important. As ES require users to increase their skill level to perform their jobs, the available documentation becomes inadequate. On the other hand, as the ES increase job satisfaction and freedom in how the user can perform the job, users are more satisfied with the ES documentation. Last, shell appropriateness for the problem seems to simplify the requirements for ES documentation, thus increasing user satisfaction.

It is apparent that, while all hypothesized relationships among major variables were corroborated (Table 4), many results regarding sub-items contradict our literature-based expectations (Table 5). For example, degree of problem structure and developer skills regarding computers, organization and society, show no significant relationships with overall user satisfaction with the ES or any of its components. In some cases (i.e., work experience and availability of domain experts), the absence of relationships can be explained by the fact that in the host company most domain experts are fairly experienced and available. Nevertheless, these contradictions of the literature should be investigated further before the variables are considered unimportant to ES success.

Among the many independent variables sub-items, four are especially important for having significant relationships with five or more user satisfaction sub-items at the significance level of 0.01 or better: model skills of developers, positive user attitude on ES, increase in job satisfaction, shell's appropriateness for the problem. The model skills of developers are significantly correlated with five user satisfaction sub-items at 0.01 level or better, indicating the developer's ability to formulate and solve analytic models is an important determinant of user satisfaction with the ES. The positive user attitude about ES shows significant relationships with all nine user satisfaction sub-items: seven items at 0.01 level or better, and two items at 0.05 level or better, making it an important determinant of user satisfaction with ES. The ES impact on job satisfaction is also a very important determinant, affecting all nine user satisfaction sub-items: eight items at the significant level of 0.01 or better, and one item at the level of 0.05 or better. Finally, the shell's appropriateness for the problem is another important factor having significant relationships with eight user satisfaction sub-items at 0.01 level or better. ES developer selection of an appropriate shell for addressing the business problem at hand is apparently very important to increase the level of user satisfaction with the system.

5. Managerial Recommendations

The main objective of the study was to empirically test the determinants of ES implementation success proposed in the literature. The results indicate that ES implementation success, as measured by user satisfaction, is related to several major factors: business problem difficulty, developer(s) skill, ES shell characteristics, domain expert quality, end-user characteristics, the ES impact on the end-user's job, user involvement in the ES development process, and management support. While some of these factors cannot be directly controlled in the short run, ES development managers can be more aware of potential ES implementation difficulties, attempt to preempt the likely problems and develop plans to facilitate the development of more successful ES applications. Based on the results several recommendations can be advanced for managerial consideration. These recommendations fall into four major areas: ES application selection, project planning, ES development process, and training and company environment.

5.1 ES Application Selection

Several sub-items under business problem difficulty are directly related to ES output value, as perceived by end-users. Assuming experts can be found to tackle a problem and build an ES, difficult business problems may provide great opportunities for leveraging the organization's knowledge resources. In other words, the more difficult business problems addressable by ES will make ES technology more useful to the end-user community and the company as a whole. Companies should consider including problem difficulty among the other criteria previously proposed in the literature for selecting ES applications (LU & GUIMARAES, 1988) which include two other items addressed in this study: need for high expertise and job labor intensity.

The impact of the ES on the end-users' jobs is an important factor for its success. Again, the results in this case provide useful insights for selecting ES applications. The selection criteria should take into consideration how the ES would affect the end-user's job in a variety of ways: how the ES will affect the importance of the job within the organization; whether it will increase or decrease the amount of work necessary to do the job; its impact on job appeal; whether or not it would improve feedback on job performance; its effect on opportunities for career advancement, job security, relationship with fellow employees; and overall end-user job satisfaction. The data indicates that the ES impact on the amount of work to do the job, increased opportunity for advancement, and on overall user satisfaction with the job, are the two most important determinants of user satisfaction with the ES. To increase user satisfaction, ES project managers developing applications which may have negative implications for the end-users' jobs should consider job re-engineering before starting ES development.

5.2 Project Planning

Before embarking on a major ES development effort, project managers should ensure that it will enjoy appropriate levels of management support, user involvement, and domain expert collaboration. As discussed later, these are important factors to ES success and project managers should not proceed in their absence.

The results clearly indicate the importance of ES shell quality in general to successful ES development and implementation. The quality of the shell's user interface facility and the shell's appropriateness to address the business problem at hand seem to be the two most important factors. Based on that, management should consider restricting ES development groups to acquiring only shells with the desirable characteristics described earlier. Further, as the collection of ES development techniques and commercially available shells increase, it becomes important for ES developers and project managers to ensure a proper match of development techniques and tools to the business problem at hand (YOON & GUIMARAES, 1993; YOON, GUIMARAES, & SWALES, 1993).

5.3 ES Development Process

Domain expert cooperation is an important factor to ES success. Project managers are likely to benefit from efforts to establish good relations with user department managers, as well as domain experts, to ensure such cooperation. Such cooperation must be secured not only in principle but also operationally (i.e. time scheduling) throughout the development process, whenever necessary.

User involvement has been confirmed as an important factor for user satisfaction with ES. ES developers should strive to give end-users a chance to feel ownership over the particular ES being developed. There are several things that ES project managers can do to increase user involvement (WELLS & GUIMARAES, 1992). It is interesting to note that at a time when end-users are independently developing their own systems, relatively few end-users develop ES without knowledge engineers. However, as the user interface for more advanced ES shells become commercially available, end-users are more likely to independently develop ES. Meanwhile, user involvement should be cultivated by ES project managers to benefit from the psychology of ownership and a host of other reasons: to gradually introduce the ES application under development to the end-users' world, to de-sensitize end-users fearful of ES technology and/or business process change, and to collect feedback on how they feel about the overall system and its features. The results show that user involvement in the many ES development activities addressed in this study are directly related to the accuracy and the value of ES output to end-users.

The importance of management support is addressed in the next section as a company environment requirement for ES success in the long run. However, for specific projects, it is important for project managers to ensure that management support does not waiver when it is needed the most, i. e. to ensure system quality and user participation.

5.4 ES Training and Company Environment

There is need for project managers to cultivate on-going management support for ES applications in general, perhaps by ensuring a few successful ES implementations with obvious benefits to the company, and temporarily avoiding large expensive projects whose benefits are expected in the long run. The results corroborated the direct relationship of management and organization support with the successful implementation of ES. However, the indirect importance of management support may be even greater for two major reasons: First, it is widely known that computer technology in general, and AI technology in particular, has been quite threatening to people. Not only because the technology is incredibly accurate, fast, and tireless, but, in the case of ES, it also can mimic expert behavior in problem solving (an activity reserved in the past for humans). In cases where the ES project has created high user anxiety, user involvement with the project is not likely without strong direct management support. While Table 4 shows that the relationship between these two major variables is not statistically significant, their sub-items intercorrelations revealed some interesting clues. Management understanding of potential ES benefits is directly related to user involvement in determining ES availability and access. Management encouragement to use the ES is related to user involvement in determining ways to meet requirements, developing input forms/screens, developing output forms/screens, and determining system availability and access. Also, having the necessary help/resources is related to user involvement in developing input forms/screens and determining system availability/access.

Second, one may surmise that with high levels of management support for implementing ES technology, the ES development team is more likely to receive sufficient funds for the acquisition of the necessary tools, methodologies, training, etc. Indeed, the data shows direct relationships between that and user satisfaction with ES ease to learn and ease to use. On the other hand, intuitively developers and managers of specific ES projects should realize that management support will be more or less necessary depending on project characteristics. It is likely to be more important in cases where user involvement is critical (i.e., because the business problem requires heavy user interaction with the ES) but difficult (i.e., because the ES is threatening to users). The same is likely to be true in cases of expensive experimental projects with high risk of failure, where users, lower level managers, or the ES development group, would be concerned about wasting their time, resources, and/or reputation. In this category we have high visibility projects necessary to prove the technology's capability in a critical application area, or projects where the ES will have a profound impact on end-user jobs, user departments or the whole organization.

The need for training developers and end-users is also clear from the results. Developers must be trained to develop people skills, formulate models of business problems, and be able to use a systems approach to problems. ES developer analytic modeling skills seem to be particularly important given their direct relation to ES output value, information timeliness, reliability, accuracy, and completeness.

While one may think that because of the presence of the domain expert, the role of end-users in ES development is relatively less critical than for other system types, i.e., DSS, the results indicate that end-user expectations and attitude toward the ES are important factors for successful ES implementation. As suggested by PRERAU (1990), to improve user expectations and attitudes, companies should establish short seminars to explain the potential and limitations of ES technology, to interpret ES conclusions and output, to incorporate the system into users jobs, and to effectively interact with specific ES. Managing end-user attitudes and expectations from a specific system should be an important item for ES project managers to include in meeting agendas. Improvement in this area may call for substantial changes from what is going on in industry today since training for ES developers and end users has been found lacking in most organizations (WELLS & GUIMARAES, 1992).

The results show that, in general, the caliber of the domain experts involved in the development of an ES is indeed an important factor to its success. The caliber of the domain expert is even more important for difficult problems where an organization can derive greater benefits from the deployment of ES and leverage expert knowledge. Timely access to the best possible domain experts call for organizations to develop directories of internal/external experts in the various domains important to the organization. There is much that organizations can do to improve internal domain expert willingness to cooperate in ES development (WELLS & GUIMARAES, 1992). BYRD (1992) has also discussed domain experts' fear of job loss, which make it important for managers to establish policies and mechanisms to induce experts to fully cooperate in ES development.

6. Study Limitations And Conclusions

Given the dearth of empirically validated knowledge about ES implementation success and the large collection of variables which are likely to be organization specific, we chose to collect information on ES within a single organization to reduce the likelihood of confounding results. We chose our target organization for its sophistication and rich collection of ES applications in the manufacturing area. The reader now has to carefully interpret the results and question their generalizability over other areas beyond manufacturing, and in less sophisticated ES development environments.

A major limitation of this study stems from the relatively small sample size. It is large enough to ensure good representation but too small to enable multivariate data analysis for exploring moderating relationships between independent variables as they affect ES success. Several new studies are needed to further understand the determinants of ES implementation success. With larger sample sizes one can explore the relationships between shell quality and developer skills, and between developer skills and problem difficulty, as they affect ES success. We also need to explore other measures for ES success: ES usage, return on investment, end-user benefits, etc.

While this study strived to identify and use the most valid and reliable measures possible, there is substantial room for improvement. For example, there is need for measuring the major constructs in different ways to test their discriminant and convergent validity. There is also need to re-test the proposed construct relationships with new data sets to strengthen the theoretical basis for ES success studies over time. An on-going systematic re-testing process addressing ES success factors is likely to be necessary as the technology and its use change over time. For example, the importance of domain experts is likely to decrease as new ES development tools and methods such as a case-based approach are used. Similarly, the important features of ES shells and end-user characteristics are likely to change as these new methods are used to develop new types of ES applications. Some important questions remain unanswered and deserve more attention in future studies. For example, are these results generalizable to other industry sectors besides manufacturing? To smaller, less sophisticated ES development organizations? To different types of application areas such as monitoring/opportunity, configuration/design, scheduling, planning, etc.? While the differences between ES for diagnostic purposes versus non-diagnostic were not statistically significant, a larger sample size will allow researchers to more thoroughly investigate possibly important differences.

Despite these limitations, this study represents one of the first attempts to identify and empirically test the determinants of ES success, as measured through end-user satisfaction. The results provided significant evidence that ES development managers can improve the likelihood of ES project success by carefully managing problem difficulty, developer(s) skill, ES shell characteristics, domain expert quality, end-user characteristics, the impact of the ES on the end-user's job, user involvement in the ES development process, and management support, according to the recommendations proposed above.

• ALDAG, R.J. & POWER, D.J.: "An Empirical Assessment of Computer‑Assisted Decision Analysis," Decision Sciences, (17:4), 1986, pp. 572‑588.
• ARGOTE, L. & GOODMAN, P.S.: "The Organizational Implications of Robotics," Managing Technological Innovation, Davis, D. D. (ed). San Francisco: Jossey‑Bass, 1986, pp. 127‑153.
• ARGOTE, L.; GOODMAN, P.S.; SCHKADE, D.: "The Human Side of Robotics: How Workers React to a Robot," Sloan Management Review, (24), 1983, pp. 31‑41.
• BAILEY, J.E. & PEARSON, S.W.: "Development of a Tool for Measuring and Analyzing Computer User Satisfaction," Management Science, (29:5), May 1983, pp. 530‑545.
• BARKI, H. & HARTWICK, J.: "Rethinking the Concept of User Involvement," MIS Quarterly, (13:1), March 1989, pp. 53‑63.
• BARONAS, A.M.K. & LOUIS, M.R.: "Restoring a Sense of Control During Implementation: How User Involvement Leads to System Acceptance," MIS Quarterly, (12:1), March 1988, pp. 111‑124.
• BARSANTI, J.B.: "Expert Systems: Critical Success Factors for Their Implementation," Information Executive, (3:1), 1990, pp. 30‑34.
• BECKMAN, T.J.: "Selecting Expert‑System Applications," AI Expert, (6:2), February 1991, pp. 42‑48.
• BENBASAT, I. & DEXTER, A.S.: "Individual Differences in the Use of Decision Support Aids," Journal of Accounting Research, (20), Spring 1982, pp. 1‑11.
• BRODY, A.: "The Experts," INFOWORLD, June 19, 1989, pp. 59‑75.
• BYRD, T.A.: "Implementation and Use of Expert Systems in Organizations: Perceptions of Knowledge Engineers," Journal of Management Information Systems, Spring 1992, (8:4), pp. 97‑116.
• CARMINES, E.G. & ZELLER, R.A.:Reliability and Validity Assessment, Sage University Paper, 1979.
• CASEY, J.: "Picking the Right Expert System Application," AI Expert, (14:9), September 1989, pp. 44‑47.
• CATS‑BARIL, W.L. & HUBER, G.P.: "Decision Support Systems for Ill‑Structured Problems: An Empirical Study," Decision Sciences, (18:3), 1987, pp. 350‑372.
• CHENEY, P.H.; MANN, R.I.; AMOROSO, D.L.: "Organizational Factors Affecting the Success of End‑User Computing," Journal of Management Information System, (3:1), 1986, pp. 65‑80.
• DAVID, F.D.: "Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology," MIS Quarterly, (13:3), September, 1989, pp. 319‑339.
• DEBENHAM, J.K.: "Knowledge Engineering: The Essential Skills," Expert Systems for Management and Engineering, Balagurusamy, E. and Howe, J. (Eds). New York, NY: Ellis Horwood, 1990, pp. 36‑66.
• De BRABANDER, B. & THIERS, G.: "Successful Information System Development in Relation to Situational Factors Which Affect Effective Communication Between MIS‑Users and EDP‑Specialist," Management Science, (30:2), 1984, pp. 137‑155.
• DeLONE, W. & McLEAN, E.: "Information Systems Success: The Quest for the Dependent Variable," Information Systems Research, (3:1), March 1992, pp. 60-95.
• DePREE, R.: "Implementing Expert Systems," Micro User's Guide, Summer 1988.
• DOLL, W.J. & TORKZADEH, G.: "Discrepancy Model of End-User Computing Involvement," Management Science, 1989, (35:10), pp. 1151-1171.
• DUDA, R.O. & SHORTLIFFE, E.H.: "Expert System Research," Science, (220:4594), (April) 1983, pp. 261-268.
• ECKEL, N.L.: "The Impact of Probabilistic Information on Decision Behavior and Performance in an Experimental Game," Decision Sciences, (14), Fall 1983, pp. 483‑502.
• FEIGENBAUM, E.; McCORDUCK, P.; NII, P.:The Rise of the Expert Company, Alexandria, VA: Time Life, 1988.
• FUERST, W.L. & CHENEY, P.H.: "Factors Affecting the Perceived Utilization of Computer‑based Decision Support Systems in the Oil Industry," Decision Sciences, (13:4), 1982, pp. 554‑569.
• GALLETTA, D.F. & LEDERER, A.L.: "Some Cautions on the Measurement of User Information Satisfaction," Decision Sciences, (20:3), Summer 1989, pp. 419‑438.
• GINZBERG, M.J.: "Early Diagnosis of MIS Implementation Failure: Promising Results and Unanswered Questions," Management Science, (27:4), 1981, pp. 459‑478.
• GREEN, G.I.: "Perceived Importance of System Analysts' Job Skills, Roles, and Non‑Salary Incentives," MIS Quarterly, (13:2), June 1989, pp. 115‑133.
• GUIMARAES, T. & GUPTA, Y.: "Measuring Top Management Satisfaction With The MIS Department," OMEGA, (16:1), 1988, pp. 17-24.
• GUIMARAES, T.; IGBARIA, M.; LU, M.: "Determinants of DSS Success: An Integrated Model," Decision Sciences, (23:2), March/April 1992, pp. 409-430.
• HAMILTON, S. & CHERVANY, N.L.: "Evaluating Information System Effectiveness," MIS Quarterly, (5:1), March 1981, pp. 76‑88.
• HARMON, P.; MAUS, R.; MORRISEY, W.:Expert Systems Tools and Applications, New York, NY: John Wiley & Sons, Inc., 1988.
• IGNIZIO, J.P.: Introduction to Expert Systems, New York, NY: McGraw‑Hill, Inc. 1991.
• IVES, B.; OLSON, M.H.; BAROUDI, J.J.: "The Measurement of User Information Satisfaction," Communications of the ACM, (26:10), October 1983, pp. 785‑793.
• JIH, W.J.K.: "Comparing Knowledge-Based and Transaction Processing Systems Development," Journal of Systems Management, May 1990, pp. 23-28.
• KAISER, K.M. & BOSTROM, R.P.: "Personality Characteristics of MIS Project Teams: An Empirical Study and Action‑Research Design," MIS Quarterly, (6:4), December 1982, pp. 43‑60.
• KENDALL, K.E.; BUFFINGTON, J.R.; KENDALL, J.E.: "The Relationship of Organizational Subcultures to DSS User Satisfaction," Human Systems Management, (7), 1987, pp. 31‑39.
• KEYES, J.: "Why Expert Systems Fail," AI Expert, (4:11), November 1989, pp. 50‑53.
• KIM, C. & YOON, Y.: "Selection of A Good Expert System Shell for Instructional Purposes in Business," Information Management, (23), 90-92, pp. 249-262.
• KOTTEMANN, J.E. & REMUS, W.E.: "A Study of the Relationship Between Decision Model Naturalness and Performance," MIS Quarterly, (13:2), June 1989, pp. 171‑181.
• LEE, D.S.: "Usage Patterns and Sources of Assistance to Personal Computer Users," MIS Quarterly, (10:4), December 1986, pp. 313‑325.
• LEITHEISER, R.L. & WETHERBE, J.C.: "Service Support Levels: An Organized Approach to End‑User Computing," MIS Quarterly, (10:4), December 1986, pp. 337‑349.
• LEONARD‑BARTON, D.: "The Case for Integrative Innovation: An Expert System at Digital," Sloan Management Review, (29:1), October 1987, pp. 7‑19.
• LEONARD‑BARTON, D. & DESCHAMPS, I.: "Managerial Influence in the Implementation of New Technology," Management Science, (34:10), October 1988, pp. 1252‑1265.
• LIEBOWITZ, J.: "Problem Selection for Expert Systems Development," Structuring Expert Systems, Liebowitz, J. and De Salvo, D.A. (Eds), Englewood Cliffs, NJ: Prentice Hall, 1989, pp. 1‑24.
• LIEBOWITZ, J.:Expert Systems for Business & Management, Englewood Cliffs, NJ: Yourdon Press, 1990.
• LU, M. & GUIMARAES, T.: "A Guide to Selecting Expert Systems Applications," Systems Development Management, (32-03-20), December 1988, pp. 1-11. Reprinted in Journal of Information Systems Management, Spring 1989, pp. 8-15. Reprinted in Expert Systems, Summer 1989.
• LUCAS, H.C.:Why Information Systems Fail, Columbia University Press, New York, NY. 1975.
• LUCAS, H.C.: "Empirical Evidence For a Descriptive Model of Implementation", MIS Quarterly, (2:2), June 1978, pp. 27‑42.
• MAHMOOD, M.A. & SNIEZEK, J.A.: "Defining Decision Support Systems: An Empirical Assessment of End‑User Satisfaction," Information Systems & Operational Research (INFOR), (27:3), August 1989, pp. 253‑271.
• MAISH, A.M.: "A User's Behavior Toward His MIS," MIS Quarterly, (3:1), March 1979, pp. 39‑52.
• MARKUS, M.L.: "Power, Politics, and MIS Implementation," Communications of the ACM, (26:6), 1983, pp. 430‑444.
• MONEY, A.; TROMP, D.; WEGNER, T.: "The Quantification of Decision Support Benefits Within the Context of Value Analysis," MIS Quarterly, (12:2), June 1988, pp. 223‑236.
• MYKYTYN, P.P.: "End‑User Perceptions of DSS Training and DSS Usage," Journal of System Management, (39:6), June 1988, pp. 32‑35.
• NELSON, R. & CHENEY, P.: "Training End‑Users: An Exploratory Study," MIS Quarterly, (11:4), December 1987, pp. 547‑559.
• NUNALLY, J.C.:Psychometric Theory, 2nd Edition, McGraw-Hill, New York, NY, 1978.
• NUNAMAKER, J.; COUGER, J.D.; DAVIS, G.B.: "Information Systems Curriculum Recommendations for the 80s: Undergraduate and Graduate Programs," Communications of the ACM, (25:11), November 1982, pp. 781‑794.
• O'NEAL, Q.:Planning and Managing Successful KBS Applications, presented at IAKE 90.
• O'NEAL, Q. & PALESE, D.: "Developing and Implementing Diagnostic Expert Systems," IBM Manufacturing Technology Digest, (6:1), 1988, pp. 43-48.
• PAYNE, S.C. & AWAD, E.M.: "The Systems Analyst as A Knowledge Engineer: Can the Transition Be Successfully Made?" Proceeding of the 1990 ACM SIGBDP Conference on Trends and Directions in Expert Systems, October 1990, pp. 115‑169.
• PIEPTEA, D.R. & ANDERSON, E.: "Price and Value of Decision Support Systems," MIS Quarterly, (11:4), December 1987, pp. 515‑527.
• PRERAU, D.S.: Developing and Managing Expert Systems, Reading, MA: Addison‑Wesley Pub. 1990.
• RAYMOND, L.: "Organizational Characteristics and MIS Success in the Context of Small Business," MIS Quarterly, (9:1), March 1985, pp. 37‑53.
• ROBEY, D.: "User Attitudes and MIS Use," Academy of Management Journal, (22:3), 1979, pp. 527‑538.
• SANDERS, G.I. & COURTNEY, J.F.: "A Field Study of Organizational Factors Influencing DSS Success," MIS Quarterly, (9:1), March 1985, pp. 77‑93.
• SHACKLETT, M.E.: "In Search of the Knowledge Engineer," UNISPHERE, August 1990, pp. 16‑17.
• SHARDA, R.; BARR, S.H.; McDONNELl, J.C.: "Decision Support System Effectiveness: A Review and an Empirical Test," Management Science, (34:2), February 1988, pp. 139‑159.
• SHPILBERG, D.; GRAHAM, L.E.; SCHATZ, H.: "Expertax: An Expert System for Corporate Tax Planning," Expert Systems, (3:3), 1986, pp. 136‑150.
• SLAGLE, J.R. & WICK, M.R.: "A Method for Evaluating Candidate Expert System Applications," AI Magazine, (9:4), 1988, pp. 44‑53.
• SLOANE, S.B.: "The Use of Artificial Intelligence by the United States Navy: Case Study of A Failure," AI Magazine, (12:1), Spring 1991, pp. 80‑92.
• SMITH, D.L.: "Implementing Real World Expert Systems," AI Expert, December 1988, (3:2), pp. 51‑57.
• SVIOKLA, J.: "The Examination of the Impact of Expert Systems on the Firm: The Case of XCON," MIS Quarterly, (14:2), June 1990, pp. 126‑140.
• TURBAN, E.:Decision Support and Expert Systems, New York, NY: MacMillan Publishing Co., 1990.
• TURBAN, E. & WATKINS, P.R.: "Integrating Experts System and Decision Support Systems," MIS Quarterly, (10:2), June 1986, pp. 121-136.
• TYRAN, C.K. & GEORGE, J.F.: "The Implementation of Expert Systems: A Survey of Successful Implementation" DataBase, Winter 1993, pp. 5-15.
• VAN DE VEN, A.H. & FERRY, D.L.:Measuring and Assessing Organizations, Wiley‑Interscience Publication, New York, 1980.
• VEDDER, R.G.: "PC‑based Expert System Shells: Some Desirable and Less Desirable Characteristics," Expert Systems, (6:1), February 1989, pp. 28‑42.
• VEDDER, R.G.; FORTIN, M.G.; LEMMERMANN, S.A.; JOHNSON, R.N.: "Five PC‑based Expert Systems for Business Reference: An Evaluation," Information Technology and Libraries, March 1989, pp. 42‑54.
• WATERMAN, D.A.: A Guide to Expert System, Reading, MA: Addison‑Wesley Publishing Co., 1986.
• WELLS, F.S. & GUIMARAES, T.: "End-User Development of Expert Systems," Proceedings of 1992 IRMA Conference, May 1992, pp. 133-140.
• WHITE, K.B. & LEIFER, R.: "Information Systems Development Success: Perspectives from Project Team Participants," MIS Quarterly, (10:3), September 1986, pp. 215‑223.
• WRITZEL, J.R. & KERSCHBERG, L.: "Developing Knowledge-based Systems: Reorganizing the System Development Life Cycle," Communications of the ACM, (32:4), April 1989, pp. 482-488.
• YOON, Y. & GUIMARAES, T.: "Selecting Expert System Development Techniques," Information & Management, (24:4), April 1993, pp. 209-223.
• YOON, Y.; GUIMARAES, T.; SWALES, G.: "Integrating Artificial Neural Networks with Rule-Based Expert Systems," DSS Special Issue on Artificial Neural Network, Spring, 1993.

Publication Dates

• Publication in this collection
  31 May 2010
• Date of issue
  Apr 1997