This paper describes an initial exploration of how organizations are using system development methodologies. A survey of over one hundred organizations indicates that nearly sixty­five percent of these organizations developed their methodology in­house. While most report some increase in productivity and quality, they are not universally satisfied with their methodologies. Over eighty­five percent of the organizations report that their methodology is adapted on a project by project basis, and over half do not believe that it was important to use the methodology on all projects. Interviews conducted with IS managers and system developers at four organizations supported the survey data. Three of the four organizations use methodologies developed in­house. Although the types of methodologies differ, in each of the organizations the methodology is viewed as a general framework of phases or activities. Varying degrees of adherence to the methodology were reported, with decisions regarding which particular tools or techniques to use, or which activities to perform, typically made at the project­team level.

INTRODUCTION

System development methodologies are promoted as a means of improving the management and control of the software development process, structuring and simplifying the process, and standardizing the development process and product by specifying activities to be done and techniques to be used. It is often tacitly assumed that the use of a system development methodology will improve system development productivity and quality. However, there is little empirical evidence to support this assumption.

There is a growing body of literature which questions the efficacy of formal system development methodologies (e.g., Baskerville, Travis & Truex, 1992; Fitzgerald, 1994; Keyes, 1992; Lyytinen, 1989). In particular, existing methodologies may not effectively support the changing nature of both the process and product of system development.

Most research to date has focused on the development of new methodologies and frameworks for the selection and comparison of methodologies, rather than on their evaluation or use in practice. Although the number of methodologies has proliferated, they are largely untested. It is not known if or how they are used, how effectively they are used, or whether they are useful. Much IS development research implicitly assumes that methodologies are used, and that they are useful and effective.

The purpose of this research is to discover how development methodologies are being used in organizations. Specific issues addressed are what types of methodologies are being used, what level of adherence is there to the particulars of the methodology, and how satisfied are organizations with their methodologies.

Many questions remain unanswered about system development methodologies (Wynekoop & RUSSO, 1993). This study specifically addresses two of these issues: the extent of methodology use and the nature of methodology adaptation within organizations.

Are methodologies used? This appears to be a simple question. However, the question might better be phrased, to what extent are methodologies used? It should not be assumed that because an organization has a methodology that all projects follow it at all or follow it completely (Pressman, 1989; Thayer, Pyster & Wood, 1981). Limited research indicates that conditions surrounding projects may prevent ideal development practices from being used (Curtis, Krasner & Iscoe, 1988; Sauer & Lau, 1994). Therefore, this study will examine organization's adherence to methodology standards to determine if organizations are using the adopted methodology on all development projects or on some selected subset of projects. A related issue to address is that if the methodology is used, is it used in its entirety, or are bits and pieces of the methodology selected for use in specific development projects.

Methodologies may be purchased and adapted because it is cheaper than developing a new methodology in­house, and methodologies must be continually refined to meet changing needs (Van De Velde, 1992). The adaptation of methodologies to fit a particular situation appears to be common. A UK study indicates that SSADM users have adapted the methodology by adding or omitting techniques and steps (Edwards, Thompson & Smith, 1989a, 1989b, 1989c). In addition to adapting a single methodology to a specific situation, different methodologies may be used on different projects (Edwards et al., 1989a) or methodologies may be adapted on a project by project basis (Smolander, Tahvanainen, & Lyytinen, 1987). However, it is not known how such decisions are made or how such adaptation is done, how frequently it is done, whether there are any controls over the changes, and how well the adapted methodologies work.

Research regarding methodology use has for the most part been limited to studies comparing a small number of methodologies (e.g., Dekleva, 1992; Necco, Gordon, & Tsai, 1987). However, although hundreds of different methodologies exist, the differences between them are slight and often no more than semantic in nature (cf. Fitzgerald, 1994). For that reason, this study will not address specific methodologies, but will instead examine methodology use in general.

RESEARCH METHOD

To address the focus of this research ­­ the nature of methodology use ­­ two approaches were used. A questionnaire survey was used to gain a broad picture of the state of methodology use in organizations throughout the US. Although surveys have been the dominant research method used to examine the use and adaptation of methodologies, surveys at best provide a snapshot of methodology use. To understand the motivations for using or adapting particular methodologies in particular contexts, a more in­depth approach is needed. To this end, interviews were conducted with four organizations.

The questionnaire was distributed to IS managers at seven hundred North American organizations. The questionnaire included questions regarding the type of development methodology used, the most useful features of the methodology, and the degree of methodology utilization. Usable responses were received from 132 organizations (19%) The size of the organizations ranged from application development staffs of less than 10 to over 400, and IS operating budgets ranged from several hundred thousand to nearly a billion dollars.

While the survey provides a broad overview of the use of development methodologies in today's organizations, additional information is needed to understand how organizations are using and adapting their methodologies. Interviews were conducted with IS managers and developers at four organizations. These individuals were asked to describe their development methodologies and how they were used. Individuals at various levels of management were interviewed to try to distinguish formal development procedures (i.e., how things should be done) and actual development procedures (i.e., how things are really done). This research is a first step in understanding the nature of methodologies in use and the extent to which they are used. This sample may not be representative of IS organizations in general, nor may their experiences. But it is only through a database of such experiences that we will understand how methodologies are in use today and what their shortcomings might be.

METHODOLOGY USE

Survey Results

The questionnaire was administered to IS managers of 700 organizations from a wide range of industries across the United States. These organizations were identified as those using a formal methodology for application development from a larger random sample of 1000 organizations. Because the purpose of this study is to describe the use of methodologies in organizations today, those organizations not using a methodology were dropped from the sample. Of the remaining 700 organizations, 132 returned usable questionnaires, resulting in a response rate of 19% The responding organizations appeared to be representative of the target sample in terms of industry, size, and location.

The first part of the survey addressed the characteristics of the methodologies being used. Since the focus of the study was not to identify individual methodologies, the questions addressed the general type of methodology and the characteristics that the organizations identified as important. In Table 1, the percentages of organizations reporting their methodologies to be of the various general types are shown. By far the majority of the organizations are using some type of structured methodology.

Table 1. Types of Methodologies Used

Methodology Type	Percentage of Organizations
Structured approach (SDLC) Prototyping/iterative approach Rapid application development (RAD) Object oriented Other	76.5% 7.6% 5.3% 0% 10.6%

Organizations were asked how they had acquired their methodology. Approximately 35 % of the organizations reported that they had purchased their methodology. The remaining 65 % developed their methodology in­house.

In an attempt to understand the forces behind organizations decisions to adopt methodologies, a list of ten activities related to methodology use was developed. This list was compiled based on a survey of the practitioner literature and discussions with practicing system developers.

From this list, the IS managers participating in the survey were asked to identify the three most important characteristics used in selecting their development methodologies and the three most important characteristics describing the use of the methodology. The three most important features both for selecting and using the methodologies were structured development techniques, well­defined corporate policies/procedures, and sharing of information between developers. The complete lists are provided in Table 2.

It was encouraging to see the similarity between the responses to the selection characteristics and the use characteristics. It would appear that the methodologies are actually being used to perform the tasks for which they were selected. However, later in the study it becomes clear that although these organizations are able to use the methodologies and their related tools to perform the types of activities they see as important (such as using structured techniques), they may not be using the entire methodology to do this. Selection of the particular pieces of the methodology that fit the particular development project appears to be common. The only differences between the two rankings were slight. Whereas the use of CASE tools for design was ranked as fourth in the selection criteria, it moved to fifth place in the use ranking, which appears to indicate that although the organizations expected to and do use the CASE tools for design, in practice the use of the methodologies' design documents and tools in JAD has been found to be particularly useful.

Table 2. Most Important Characteristics of Methodologies

Methodology Characteristic	Selection Rank	Use Rank
Use structured development techniques Enforce well­defined corporate policies/procedures Share information between developers Use of front­end CASE tools for design Use joint application development (JAD) Support of prototyping Use of back­end CASE tools for code generation Object­oriented methods	1 2 3 4 5 6 7 8	1 2 3 5 4 6 7 8

The remainder of the questionnaire addressed the nature of the use of the methodologies in the organizations. These result are summarized in Table 3.

Table 3. Methodology Use

System Development Methodology (SDM) Use	Agree	Unsure	Disagree
One or more SDMs are used in this firm The SDM should be used on all development A single SDM is used for all projects The SDM is adapted on a project­by­project basis Productivity is increased by using the SDM System quality is better due to the use of the SDM Using the SDM improves communication System developers are satisfied with the SDM IS management is satisfied with the SDM	84.2% 73.5% 48.5% 88.6% 72.0% 83.4% 84.1% 43.2% 45.5%	8.3% 8.3% 15.1% 2.3% 22.7% 15.1% 15.9% 39.4% 34.8%	7.5% 18.2% 36.4% 9.1% 5.3% 1.5% 0% 17.4% 19.7%

The first item shown in Table 3 was used to confirm that the responding organizations were, in fact, using a system development methodology. Ten of the organizations (7.5%) said that they were not using a system development methodology in applications development. This is particularly interesting because the sample organizations were specifically selected because they had responded affirmatively on an earlier survey to a question that asked if their organization had a system development methodology. This highlights the difference between having a methodology and using a methodology. Eleven of the IS managers responding were unsure if their organizations used a system development methodology. This type of response, while disconcerting, is not unique. An earlier study of system development methods (Dekleva, 1992), also found that many respondents did not know what development methods were used in their organizations.

The survey results provide some information on how extensively methodologies are used in these organizations. Although the majority of the IS managers responding believe that methodologies should be used on all development projects, there is a great deal of disagreement as to whether or not a single methodology is appropriate for all projects. So, whereas methodologies are viewed as beneficial, no one methodology is ideal for all projects. This is idea is reinforced by the responses to the next item, which indicates that in nearly 90% of the organizations the methodology is adapted on a project­by­project basis. This supports findings of previous case studies which described this type of adaption (Edwards et al., 1989a, 1989b, 1989c; Smolander et al., 1987).

To assess the "success" of the methodologies, respondents were asked to comment on the impact of the methodology on productivity, quality, and communication, and on general satisfaction with the methodology, from the view of both the IS managers themselves and the system developers. Seventy­two percent of the organizations reported increases in development productivity due to using their methodology, and an even higher percentage (83.3%) reported that system quality improved with the use of the methodology. Over 80% of the IS managers felt that communications between developers was improved through the use of the methodology. Based on these positive responses, one would also expect to see highly positive responses on the overall satisfaction issues. However, this was not the case. In particular, high number of IS managers were unsure whether the developers or even they themselves were satisfied with the methodologies. So, even though the methodologies are bringing improvements to productivity, quality, and communication, they are not uniformly accepted.

Interview Results

Interviews were conducted with companies from a cross section of industries: two were oil and gas companies, one a paper products manufacturer, and one a credit processing firm. Methodology use in each of the companies is described below.

Company A: Natural Gas Company

The current methodology was developed in­house by the development center from 1989­1991, and first used in 1991. Although the company had been using a commercial methodology, they found it inflexible, and increasing competition in the industry and user expectations motivated the development of a methodology that would provide shorter development times and higher quality products. The goal was to combine waterfall and prototyping models and base the methodology on structured methods.

The current methodology contains a series of checklists of activities. Tools and techniques for activities are suggested; guidelines for choosing appropriate ones are included. Project leaders choose which activities, tool, and techniques should be used on a given system ­­ choices are usually made based on their experience. Methodology enforcement varies among the systems groups.

Company B: Oil Company

The current methodology has been in use for a little over two years. The methodology, and the CASE tool it includes, have been used in their entirety on only a few projects. The methodology, developed in­house by a methodology group, is based on information engineering and LBMS. The methodology calls for various activities to be performed and specifies tools, techniques, deliverables, and people for each.

The methodology essentially contains five different waterfall­based lifecycles: standard waterfall lifecycle, package lifecycle, release lifecycle, maintenance lifecycle, and client/server lifecycle. According to one lead designer, the methodology is not always followed, and, according to another, use of the methodology has not been enforced.

For each project, the "core team" makes decisions regarding the appropriate lifecycle, and which tasks, tools, techniques, and deliverables to use. Major deviations, such as not using the CASE tool or changing the methodology, may need some type of approval, but the method by which this is done varies by team.

Company C: Paper Product Manufacturer

This organization has no single, formal methodology. Instead, they have several "methodology streams" in place. Consultants working on a large project brought in their own methodology (Method 1/Foundation), and some of the paper companies own developers have been using that methodology or parts of it. The organization has purchased another commercial methodology (from Computer Partners) which is a typical waterfall or SDLC­type methodology which provides very precise instructions as to what is to be done in each phase of the development process, with specified deliverables. Several project teams have used this methodology. However, there is no standard approach, and no enforcement of methodology use at this time.

The organization would like to develop a "framework" to guide method choice. This framework would outline the minimum required steps for any project, with anything else left up to the individual. They would like to identify a set of two to three acceptable formats for requirements specifications and design documents, and allow the individual developer or project team determine which is best for a particular project. They believe that narrowing the number of available documentation methods would make it easier to communicate with users and to share work and communicate between developers.

Company D: Credit Processing Firm

The systems development group in this organization is using what they call a "homegrown" methodology, which was developed in 1993. It is described as a formal process with eight phases. It provides a series of steps to follow and deliverables to produce. It was developed by a group at the corporate office. The group reviewed past experiences and existing methods, including SDM70, STRADIS, Foundation and Method 1, and came up with a set of activities that seemed most appropriate for their development environment.

Use of the methodology is not mandatory, but is encouraged. Acceptance is considered marginal. It is estimated that 80% of the developers are using the methodology "in spirit", but only 50% are using it effectively. Usage varies from team to team. The method is adapted on a project­to­project basis. No details or guidelines are provided in the methodology itself for how to do this adaptation. The driving force is tied to specific applications.

The data from these organizations indicates that the methodologies tend to be viewed as a general framework, providing a series of steps or phases. Within that framework, numerous tools and techniques are included. In each of these organizations, the decision of whether or not to follow the methodology and which tools and techniques to use was made at the project team level.

Summary of Results

The information from the survey and interviews indicates that system development methodologies, in their intended forms, are not truly meeting the needs of organizations today. Organizations are developing their own methodologies or adapting commercial methodologies. Even with these "home­grown methodologies", adaptation is common. Formal methodologies do not apply to every development project. They are modified on a project­by­project basis, typically with little or no guidance on how this adaptation should take place provided by the methodology itself or by organizational guidelines. Although benefits of using methodologies, particularly in the areas of productivity, quality, and communication, are recognized, organizations nevertheless are not entirely satisfied with their methodologies.

Whereas organizations appear to want the benefits that a methodology can provide, it is not clear that existing methodologies are meeting these needs. Further research is needed to determine whether refinements of current methodologies, such as guidelines for determining how to adapt the methodology to a particular development project, will be sufficient, or whether a totally new form of methodology is needed.

This study has provided initial information regarding the use and adaptation of methodologies. Whereas many organizations, in policy, have a formal system development methodology, in practice adherence to the methodology does not seem to be enforced, and the extent to which the methodology is followed varies by project or system. In most cases formal procedures or guidelines for adapting or modifying the methodology do not exist, and when they do, they are not always followed.

Additional issues remain to be addressed. Should organizations rely on a single development methodology, or is a generic tool­kit of tools, techniques, and methods, with procedures for their selection, more appropriate? Are new forms of methodologies needed to support new forms of organizations? Are formal methodologies in general detrimental to the creativity of the development process?

The results of this study should be viewed as an initial step in understanding the use of methodologies in today's organizations. The survey provided a snapshot of a limited set of information at one period of time. The organizations studied in the interviews may not be representative of the general IS community. However, these results, when combined with the results of additional studies, can provide the basis for the development and use of future methodologies.

REFERENCES

Avison, D.E. and Fitzgerald, G. "Information Systems Development: Current Themes and Future Directions." Information and Software Technology, Volume 30, 1988, pp. 458­466.

Baskerville, R.; Travis, J.; and Truex, D. "Systems Without Method: The Impact of New Technologies on Information Systems Development Projects." The Impact of Computer Supported Technologies on Information Systems Development, K.E. Kendall, K. Lyytinen, J.I. DeGross (Eds). Amsterdam: North­Holland, 1992, pp. 241 ­269.

Curtis, B.; Krasner, H.; and Iscoe, N. "A Field Study of the Software Design Process for Large Systems." Communications of the ACM, Volume 31, 1988, 1268­1287.

Dekleva, S.M. "The Influence of the Information Systems Development Approach on Maintenance." MIS Ouarterly. Volume 16. 1992, pp. 355­372.

Edwards, H.M., Thompson, J.B., and Smith, P. "Results of Survey of Use of SSADM in Commercial and Government Sectors in the United Kingdom," Information and Software Technology, 1989a, Vol. 31, No. 1, pp. 21­28.

Edwards, H.M., Thompson, J.B., and Smith, P. "Experiences in Use of SSADM: Series of Case Studies. Part 1: First Time Users," Information and Software Technology, 1989b, Vol. 31, No. 8, pp. 411­419.

Edwards, H.M., Thompson, J.B., and Smith, P. "Experiences in Use of SSADM: Series of Case Studies. Part 2: Experienced Users," Information and Software Technology, 1989c, Vol. 31, No. 8, pp. 420­428.

Fitzgerald, B. "The Systems Development Dilemma: Whether to Adopt Formalised Systems Development Methodologies or Not?" In Baits, W. (Ed.), Proceedings of Second European Conference on Information Systems, Nijenrode University Press, Breukelen, 1994, pp. 691 ­706.

Keyes, J. "How Software is Developed Undergoing Basic Changes," Software Magazine, January 1992, pp. 38­55.

Lyytinen, K. "Expectation Failure Concept and Systems Analysts' View of Information System Failures: Results of an Exploratory Study." Information & Management, Volume 14, 1988, pp. 45­56.

Lyytinen, K. "New Challenges of Systems Development: A Vision of the 90's." Data Base, Volume 20, 1989, pp. 1­12.

Necco, C.R.; Gordon, C.L.; and Tsai, N.W. "Systems Analysis and Design: Current Practices." MIS Ouarterly, Volume 11, 1987, pp. 461­476.

Pressman. R.S. Software Engineering: A Practitioner's Approach. New York: McGraw­Hill, 1992.

Sauer, C. and Lau, C. "The Adoption of Information Systems Methodologies ­ An Analytical Framework and a Case Study," Proceedings of the 4th Australasian Conference on Information Systems, 1994.

Smolander, K.; TaLvanainen, V.; and Lyytinen, K. "How to Combine Tools and Methods in Practice ­­ A Field Study." In B. Steinholtz, A. Solvberg, and L. Bergman (eds.), Information Systems Engineering. Berlin: Springer­Verlag, 1987, pp. 195­285.

Thayer, R.H.; Pyster, A.B.; and Wood, R.C. "Major Issues in Software Engineering Project Management." IEEE Transactions in Software Project Management, Volume SE­7, 1981, pp. 333­342.

Van De Velde, R. "CASE Adoption at Imperial Oil," Journal of Systems Management, Vol. 43, 1992, pp. 24­38.

Wynekoop, J.L. and RUSSO, N.L. "System Development Methodologies: Unanswered Questions and the Research­Practice Gap," in J.I. DeGross, R.P. Bostrom, and D. Robey (eds.), Proceedings of the Fourteenth International Conference on Information Systems, 1993, pp. 181 ­ 190.