Journal of Management Information Systems

Volume 16 Number 4 2000 pp. 3-10

Special Issue: Impacts of Information Technology Investment on Organizational Performance

Mahmood, M Adam and Mann, Gary J

THIS SPECIAL ISSUE ADDRESSES A SUBJECT OF MUCH DEBATE among interested researchers: the relationships between organizational investment in information technology (IT) and organizational performance and productivity. Despite the enormous investment in IT during recent years, demonstrating the effects of such investment on organizational performance has proven extremely difficult. The present Special Issue is intended to enhance this area of research by encouraging the development of new perspectives of the IT investment-performance relationships, thereby expanding the portfolio of methods used in investigating these relationships.

Some recent studies of relationships between investment in IT and organizational performance and productivity [1, 2, 4, 10, 12, 14] have reported positive and significant effects of such investment. Some researchers question these results on the grounds that the studies involved examination of primarily cross-sectional data. This criticism stems at least in part from the premise that the benefits of IT investment can be realized only over longer periods of time. However, it is possible, indeed likely, that in many instances IT has the potential to provide important benefits within the same year the investment is made. In any event, research reflecting relationships between IT investment and organizational performance and productivity might be more convincing if it were based on IT investment in both current and earlier periods. Although some authors [3, 9, 13] have speculated on the lagged effects of IT investment, there have so far been no empirical studies, with the exception of one reported in a conference [11], that have demonstrated a relationship between such investment and organizational performance and productivity in subsequent periods. This Special Issue contains a paper by Sarv Devaraj and Rajiv Kohli that reports on their longitudinal research design in identifying lag effects of IT investment.

Another reason for disagreement among researchers involves the argument that correlations reflecting relationships between IT investment and organizational performance and productivity do not necessarily imply causation, particularly if the correlations are based on data from the same year only [12]. It has also been emphasized that causality cannot be established by using conventional statistical techniques. The call for papers for this Special Issue encouraged researchers to apply multivariate (e.g., canonical correlation analysis) and nonparametric (e.g., data envelopment analysis) methods, as opposed to more commonly used methods such as correlation and regression analyses, to enable them to infer causality, if present, between IT investment and organizational performance and productivity. Accordingly, Sumit Sircar, Joe L. Turnbow, and Bijoy Bordoloi, as well as Byungtae Lee and Nirup M. Menon in their papers in this Special Issue report on their use of multivariate and nonparametric methods in analyzing study data.

Some researchers have flatly rejected the conclusions of studies that found positive relationships between investment in IT and organizational performance and productivity. One went so far as to call such conclusions "the big lie of the information age" [15]. Perhaps a more valid method of determining whether IT is living up to expectations is through analyses involving both cross-sectional and longitudinal databases involving hundreds or even thousands of data points from various industries. Large databases would serve to average out extremes and provide a clearer picture of the underlying relationship between IT investment and organizational performance. A report on findings from a study using multiyear, cross-sectional data is provided here by Sumit Sircar, Joe L. Turnbow, and Bijoy Bordoloi, and by Byungtae Lee and Nirup M. Menon.

Based on the review of a number of research studies that were presented at the Workshop on Information Systems and Economics (WISE), Brynjolfsson and Hitt [5, p. 50] state that, "[w]hile the average returns to IT investment are solidly positive, there are huge variations across organizations, some have spent vast sums on IT with little benefit, while others have spent similar amounts with tremendous success." They hold that the greatest benefits of IT appear to be realized by organizations when IT investment is coupled with other complementary investments such as organizational reengineering, restructuring, and redesign. Also, it is quite possible that the nature of some businesses may be such that they may not be able to obtain as great a return from IT as other organizations might. Another purpose of this Special Issue was to encourage researchers to go beyond the sole effects of IT investment and explore these complementary issues. We are pleased to report that Sarv Devaraj and Rajiv Kohli examine the combined effects of IT and business process reengineering on organizational performance, while Byungtae Lee and Nirup M. Menon examine the relationship between IT investment levels and technical and allocative efficiencies in a firm. Michael J. Davern and Robert J. Kauffman also contribute in this area by presenting a perspective for understanding a firm's potential IT value and for relating this to realized value.

"Despite the fact [that] the productivity paradox of IT is an international phenomenon," according to Dewan and Kraemer [6, p. 57], most of the existing studies in this area involve firm-level analysis mainly conducted in the United States using American firms. With the exception of the Dewan and Kraemer research, no international-level studies have previously been conducted. We believe it is important that researchers add an international dimension to the matter of IT investment-performance relationships, extending beyond the United States to encompass the experience of organizations in other developed countries. In this Special Issue, Paul P. Tallon, Kenneth L. Kraemer, and Vijay Gurbaxani, as well as Arik Ragowsky, Dennis A. Adams, and Myles Stern report on their respective uses of international data.

It has been argued that traditional IT investment-performance analyses have not been very successful in the past because of their overreliance on financial data. Some researchers (e.g., [4]) have called for additional research to identify "hidden costs and benefits" that are typically not included in a traditional analysis of IT investment relationships with organizational performance and productivity. In their paper in this Special Issue, Sherry D. Ryan and David A. Harrision provide insights into what they believe to be one of the sources of such hidden costs-namely, those associated with human resources.

The divide between the several groups of researchers investigating the effects of IT investment, discussed above, is rather common knowledge. One group, emphasizing the need for use of qualitative analysis, believes that quantitative measures have received preferential consideration in the research performed to date. The quantitative group is somewhat vocal about what it considers the superiority derived from the rigors of its approach. Some members of this group even go so far as to say that they do not believe in qualitative measures of IT productivity. They argue that qualitative measures can be used only if they first concur with the quantitative measures of IT payoff. Yolande E. Chan does a fine job in bringing forward and explaining this controversy. This Special Issue also includes a number of research studies that use qualitative data (e.g., by Paul P. Tallon, Kenneth L. Kraemer, and Vijay Gurbaxani; by Akemi Takeoka Chatfield and Philip W. Yetton; and by Sherry D. Ryan and David A. Harrison) to emphasize that there is room for both camps in IT productivity research. Ideally, the work of each group should complement that of the other.

In summary, we believe this Special Issue makes a significant contribution to the ongoing debate over the value of IT investment by shedding new light on organizational experiences with IT investment. We feel the research presented here, by providing new insights, moves the area of IT investment research to a new level of inquiry. It is clear that, while progress is being made in the area, the various approaches used in measuring IT payoffs are not universally accepted and the conclusions are arguable. The papers presented here focus on critical, yet insufficiently explored, dimensions of the IT payoff question. These papers, selected from a large number of manuscripts submitted by authors within and outside the United States, collectively exemplify the study of this critical question in both national and global environments. We now introduce to you the papers included in this Special Issue.

As described in the first paper, Sherry D. Ryan and David A. Harrison interviewed fifty IT decision-makers in various industries in conducting an investigation of social subsystem costs related to IT decisions. Ryan and Harrison's study supported the observation that IT decisions are traditionally focused on financial or technological issues. The results further indicated that human-related costs often are not considered, or are minimized or ignored until after IT is implemented, resulting in less than optimal investment decisions and thereby reducing the potential benefits of IT investments. The paper extends existing theory by describing systematic patterns of inclusion and exclusion of related costs and benefits. In addition, a decision aid is provided to help IT executives begin to think about which social subsystem costs and benefits should be incorporated into various decisions.

Sarv Devaraj and Rajiv Kohli examined monthly data from eight hospitals over a three-year period to evaluate possible relationships between IT investment measures and performance measures of profitability and quality. Holding that the difficulty in identifying the impacts of technology in past research has been the isolation of such benefits from other factors that may also contribute to organizational performance, they sought to determine whether another organizational factor-namely, business process reengineering (BPR)-had a positive impact on measures of organizational profitability and quality. Using a longitudinal research design, the study found positive lag effects of IT investment on organizational profitability. Also, IT investment was found to have a positive effect on organizational quality initiatives. BPR alone was not found to lead to improvement in profitability; however, evidence was found that IT capital investment combined with BPR had a positive impact on profitability.

Sumit Sircar, Joe L. Turnbow, and Bijoy Bordoloi reviewed earlier studies of the impact of information technology (IT) on firm performance, noting that previous research provided conflicting findings. Extending this earlier work, they analyzed a large database by use of canonical correlation, finding strong empirical validity for the premise that a relationship exists between sets of IT investment measures (as opposed to individual measures) and firm performance. Further investigation revealed that, overall, IT investment (computer capital, MIS staff and staff training, and other MIS), and non-IT investment (labor and noncomputer capital) were directly related to firm performance measures.

Byungtae Lee and Nirup M. Menon extended previous research of the contributions of IT to the enhancement of output. Using 19 years of data from hospitals, they applied both parametric and nonparametric approaches to evaluating the effects of IT. In addition to productivity measures, they estimated error terms representing various inefficiencies in business processes. Efficiency, measured through post-hoc analyses, provided information on how well the mix of inputs impacted production, as well as on how well costs and the productivity impacts of inputs were factored into decision making during resource allocation. In addition, the authors used efficiency results to distinguish between over- and underinvestment in IT and to infer the relationship between level of investment and process reengineering.

Michael J. Davern and Robert J. Kauffman present a method of measuring IT value that emphasizes the importance of understanding where potential value lies and how best to relate it contextually to the measurement of a firm's realized value. Using concepts such as locus of value and value conversion contingencies, the authors developed the idea that complementary assets (particularly business process design and human capital) have some bearing on the firm's realization of value. In contrast to earlier process models of IT value that begin with IT expenditures, their analysis emphasizes the initial assessment of potential value for an IT investment. They suggest that the critical level of analysis for understanding how potential value is transformed into realized value may be different from the level of analysis at which the business case for investment, and thus potential value, was initially established.

Paul P. Tallon, Kenneth L. Kraemer, and Vijay Gurbaxani used another approach to assessing the impacts of IT, applying a process-oriented model to estimate possible impacts on critical business activities within the value chain. The model incorporates four types of corporate goals for IT and management practices as key determinants of realized IT payoffs. To evaluate the model, and in recognition of the increasing role that executives play in IT investment decisions, the authors based their assessment of IT payoffs on a worldwide survey of executives' perceptions of realized IT impacts. The survey confirmed that the goals were useful indicators of IT payoffs. It was also found that the adoption of certain management practices enables firms to realize higher levels of IT business value.

The research reported by Arik Ragowsky, Myles Stern, and Dennis A. Adams had two objectives. The first was to find support for the premise that benefits derived through the use of information systems (IS) depend on an organization's operating characteristics (e.g., number of suppliers and lead time for purchase orders). The second was to show that the relationship between the benefits of an IS and an organization's operating characteristics was stronger for a specific IS application than for the entire IS applications portfolio considered as a whole. Using data from over 500 manufacturing firms in Israel and the United States, the authors found no beneficial relationship between the overall organizational IS and the operating characteristics of organizations for either country. However, for identical specific IS applications in both countries, a beneficial relationship with operational characteristics was noted.

Focusing on a more narrow aspect of information technology applications, Akemi Takeoka Chatfield and Philip W. Yetton report on the conduct of cross-case analyses of three organizations that initiated sophisticated electronic data interchange (EDI) networks but that experienced different levels of strategic payoffs. Applying sociological theories of embeddedness, they postulate that the relationship between EDI use and strategic benefits is moderated by EDI embeddedness, which they define as a measure of how central or peripheral a specific EDI network is to managing firm interdependence.

Finally, Yolande E. Chan observes that, to provide evidence that would be considered by an executive audience, many past studies of the relationships between IT investment and organizational performance have focused exclusively on quantitative performance measures. She calls for renewed recognition of the important contributions made by "soft" IT value measures and individual-level measures and suggests that researchers need to recognize the important contributions that can be made by qualitative measures, and by individual and group-level analyses in IT value research.

Acknowledgments: We wish to recognize the contributors to this Special Issue for being highly responsive to reviewers' and guest editors' suggestions in preparing revisions of their manuscripts. If there is one single factor underlying the success of this Special Issue, it can be attributed to the authors' participation and cooperation. They have our sincere thanks for their outstanding individual efforts.

We also recognize the contributions of the scholars who diligently reviewed and critiqued the original proposals, and the initial and revised manuscripts. All accepted manuscripts were reviewed by at least two reviewers and went through at least three revisions. Each reviewer spent a large amount of time carefully evaluating the assigned manuscript and its subsequent revisions until the manuscript was finally accepted or rejected. Their professionalism in moving this project to completion is truly appreciated.

We express our special thanks to Jo Willems at the College of Business, University of Texas at El Paso, who was instrumental in keeping the Special Issue project on track. Her efforts and dedication to the project are truly appreciated.

The reviewers for the Special Issue were:

William Acar, Kent State University

Dennis A. Adams, University of Houston

Niv Ahituv, Tel Aviv University

Irma Becerra-Fernandez, Florida International University

James R. Burkman, Indiana University

Jeff Butterfield, Western Kentucky University

James I. Cash, Jr., Harvard University

Yolande E. Chan, Queen's University

Akemi Takeoka Chatfield, University of New South Wales

Debabroto Chatterjee, Washington State University

Michael J. Davern, New York University

Sarv Devaraj, University of Notre Dame

Kelley Donalds, University of Connecticut

Kevin J. Dooley, Arizona State University

T. Selwyn Ellis, Ouachita Baptist University

Gerald E. Evans, University of Montana

Edmond P. Fitzgerald, University of Southern Queensland

Dale Goodhue, University of Georgia

Sanjay Gosain, University of Southern California

Tom G. Gough, University of Leeds

Barbara Haley, University of Virginia

Diana Hoogeveen, Erasmus University

Murray E. Jennex, Claremont Graduate University

Kailash Joshi, University of Missouri

Robert Kauffman, University of Minnesota

C. Bruce Kavan, University of North Florida

Betty A. Kleen, Nichols State University

Kevin Kobelsky, University of California

Stefan Koch, Vienna University of Economics & Business Administration

Rajiv Kohli, Holy Cross Health System

Kenneth L. Kraemer, University of California

Tor J. Larsen, Norwegian School of Management

Byungtae Lee, University of Illinois at Chicago

In Lee, Western Illinois University

Jack T. Marchewka, Northern Illinois University

Fred Niederman, University of Baltimore

Prashant Palvia, University of Memphis

Arik Ragowsky, Wayne State University

Manash Ray, Lehigh University

Pekka Reijonen, University of Turku

Marcus A. Rothenberger, Arizona State University

Sherry D. Ryan, University of North Texas

Benjamin B. M. Shao, State University of New York at Buffalo

Susan A. Sherer, Lehigh University

Namchul Shin, Rowan University

Wesley Shu, Santa Clara University

Sumit Sircar, University of Texas at Arlington

Paul P. Tallon, University of California

Asher Tishler, University of Iowa

Marvin D. Troutt, Kent State University

Stijn Viaene, Katholieke Universiteit Leuven

John Wallace, Marshall University

Hugh J. Watson, University of Georgia

Susan Rebstock Williams, Georgia Southern University

REFERENCES

1. Alpar, P., and Kim, M.A. A microeconomic approach to the measurement of information technology value. Journal of Management Information Systems, 7, 2 (1990), 55-69.

2. Barua, A.; Kriebel, C.H.; and Mukhopadhyay, T. Information technologies and business value: an analytic and empirical investigation. Information Systems Research, 6, 1 (March 1995), 3-23.

3. Brynjolfsson, E. The productivity paradox of information technology. Communications of the ACM, 36, 12 (1993), 67-77.

4. Brynjolfsson, E., and Hitt, L. Paradox lost? Firm-level evidence on the returns to information systems. Management Science, 42, 4 (1996), 541-558.

5. Brynjolfsson, E., and Hitt, L. Beyond the productivity paradox. Communications of the ACM, 41, 8 (1998), 49-55.

6. Dewan, S., and Kraemer, K.L. International dimensions of the productivity paradox. Communications of the ACM, 41, 8 (1998), 56-62.

7. Hitt, L.M., and Brynjolfsson, E. Productivity, business profitability, and consumer surplus: three different measures of information technology value. MIS Quarterly, 20, 2 (1996), 121-141.

8. Holden, T., and Wilhelmij, P. Improved decision making through better integration of human resource and business process factors in a hospital situation. Journal of Management Information Systems, 12, 3 (1995-96), 21-41.

9. Loveman, G.W. An assessment of the productivity impact of information technologies. In T.J. Allen and M. Scott Morton (eds.), Information Technology and the Corporation of the 1990s. Oxford: Oxford University Press, 1994, pp. 81-117.

10. Mahmood, M.A., and Mann, G.J. Measuring the organizational impact of information technology investment: an exploratory study. Journal of Management Information Systems, 10, 1 (1993), 97-122.

11. Mahmood, M.A.; Mann, G.J.; Dubrow, M.; and Skidmore, J. Information technology investment and organizational performance: a lagged data analysis. Proceedings of the Information Resources Management Association, Boston, 1998, pp. 19-22.

12. Mitra, S., and Chaya, A.K. Analyzing cost-effectiveness of organizations: the impact of information technology spending. Journal of Management Information Systems, 13, 2 (1996), 29-57.

13. Osterman, P. The impact of computers on employment of clerks and managers. Industrial and Labor Relations Review, 39, 2 (1986), 175-186.

14. Rai, A.; Patnayakuni, R.; and Patnayakuni, N. Technology investment and business performance. Communications of the ACM, 40, 7 (1997), 89-97.

15. Schrage, M. The real problem with computers. Harvard Business Review, 75, 5 (1997), 178-183.

16. Semich, J.W. Here is how to quantify IT investment benefits. Datamation, 40, 1 (1994), 45-48.

From the Editor-in-Chief

WE ARE DELIGHTED TO WELCOME to the Editorial Board Alan Dennis of the University of Georgia and Varun Grover of the University of South Caroina.

Wishing all of you a good millennium,

Vladimir Zwass