Dante Carrizo1 Cristián Ortiz1 Luis Aguirre1
1Departamento Informática y Ciencias de la Computación. Universidad de Atacama. Avda. Copayapu 485. Copiapó, Chile. E-mail: dante.carrizo@uda.cl; luis.aguirre@alumnos.uda.cl; cristian.ortiz@alumnos.uda.cl
ABSTRACT
Researchers often claim that the most appropriate elicitation techniques should be chosen to capture software requirements, but: what does it mean for them? This work carried out a systematic mapping on comparisons among elicitation techniques for knowing the constructs utilized to represent their appropriateness. This study identified 43 works that utilized 58 ways of measuring the goodness of the techniques. These metrics were classified, mainly, by types of constructs such as quality, adequacy, effectiveness, quantity and efficiency. The results show the large dispersion occurring between researchers on how to select the most appropriate technique for each elicitation session and, therefore, more convergent proposals are required.
Keywords: Elicitation requirements, techniques appropriateness, systematic review, elicitation techniques requirements, systematic mapping.
RESUMEN
Para capturar los requisitos de software los investigadores frecuentemente pregonan que se debe seleccionar la técnica de educción más adecuada, pero ¿qué significa para ellos esto? En este trabajo se realiza un mapeo sistemático de investigaciones acerca de comparaciones de técnicas para captura de requisitos con el fin de conocer los constructos que se utilizan para representar su adecuación. El estudio identificó 43 trabajos que utilizaron 58 formas de medir la bondad de las técnicas. Estas métricas se clasificaron principalmente en tipos de constructos de calidad, adecuación, efectividad, cantidad y eficiencia. El estudio demuestra la gran dispersión que existe entre los investigadores respecto de cómo seleccionar la técnica más apropiada para cada sesión de educción y que, por tanto, se requiere más propuestas convergentes.
Palabras clave: Educción de requisitos, adecuación de técnicas, revisión sistemática, técnicas de educción de requisitos, mapeo sistemático.
INTRODUCTION
The software requirements process, regardless the theoretical model considered (eg: [1-3]), has as first phase, the capture of relevant information from the problem domain and the stakeholders' needs to specify the requirements that the software
product must meet. To do this, there are a myriad of techniques, some of them from other disciplines such as psychology or social sciences [4].
Because of this diversity of origins, and the intrinsic nature of each technique, its performance in capturing information from stakeholders is obviously different.
Therefore, practitioners need to have methods and tools to select the most appropriate technique to be used at any time of the software development project [5].
Most of these methods [6-7] considered influential aspects of the context of the process that, in opinion of their authors, modulate the outcome of each elicitation session. However, in order to compare and select the most appropriate elicitation technique you need to use a metric system that faithfully represents this construct [8].
In order to move in this direction, this study aims to gather how researchers represent the appropriateness of the techniques. For this, we have conducted a systematic mapping of the publications on elicitation techniques, used in various domain areas. If the researchers visualize the elicitation techniques in a convergent and valid way then such vision may guide future experimental work on comparisons of techniques and thus form a body of knowledge about their performance. Conversely, if there is divergence of views, there will be the need for more research proposals to determine the construct of appropriateness. Once the researchers' view is established, it is intended by the authors to continue capturing the practitioners' view proceeding to compare them and then to induce a model that represents this construct.
For the presentation of this work, the paper is structured as follows: in next section backgrounds on systematic mapping and related work are discussed, then, the research methodology is described and the results of this work are presented, finally, the conclusions and future work are shown.
BACKGROUND AND RELATED WORK
A systematic mapping study is a methodology used in other areas such as medical research, but it has been suitable for being used in areas of software engineering [9]. It requires less effort than a systematic review of literature as it provides a broader view in order to identify more and less treated areas of research. A systematic review considers obtaining more detailed information than can be processed to form a body of knowledge based on empirical evidence with certain degrees of reliability.
The main goal of the systematic mapping performed in this study is to obtain an overall quantitative and qualitative overview of existing research on the performance evaluation of elicitation techniques.
This review has noted a significant number of studies on the use of the techniques, however, there are not any researches about the techniques with a focus on establishing a unique way to measure their performance or that show the great existing diversity. Only one work, Dieste and Juristo [10], deals with the dependent variables that represent the effectiveness of elicitation techniques in empirical studies. However, there is no further discussion of this diversity and these variables are only treated in order to generalize and obtain empirical evidence.
METHODOLOGY
Research question
The systematic mapping begins with the specification of the research questions to be answered. In this case, as it is an exploratory study, a main and three secondary questions are considered.
The main question is: What do researchers mean by appropriateness of elicitation techniques? This question relates directly to the constructs that represent the performance of a technique in a given contextual situation.
The other questions are:
• What techniques are used in studies about adequateness constructs?
• In what application domains certain constructs for appropriateness of the techniques are more commonly used?
• What kinds of studies use certain constructs for appropriateness of techniques?
Selection of studies
The identification stage of primary studies was performed by searching in the following databases: IEEEXPLORE, ACM DL, SCIENCE DIRECT. Eventual Internet searches were also conducted between the references of the selected articles and other papers already identified. The searching period included from the year 1984 to November 2013.
The searching string was:
(Requirements) AND (elicitation OR gathering OR capture OR acquisition) AND (techniques OR methods)
This search string was adjusted to the formats of each database and focused on the publications abstracts. The following inclusion/exclusion criteria were considered for selecting studies:
• Scientific publications related to elicitation techniques and requirements related books were considered.
• Studies may focus on one or more techniques. Thus, candidates could be publications on the application of a technique to a case study, descriptive and prescriptive techniques comparisons, and empirical evaluations of techniques, among others.
• No restriction on the application area of the literature was performed, as long as elicitation techniques that are used in software engineering are used.
To identify primary studies the following filters were performed:
First Filter (1F)
• Title: Each author reviewed the titles of publications contained on each database. One of them also made the opportunistic searching.
• Abstract: The abstract of those publications that were selected by their title, were reviewed by each assigned author.
Second Filter (2F)
• Full Text: Finally, publications that passed the previous filter were subjected to a revision of their contents. Those articles, in which an assigned author had doubts, were reviewed by the three authors assigned.
Classification scheme
In order to answer the research questions, the literature was reviewed focused on the following aspects of the classification scheme: Adequateness metric, elicitation techniques used, application domain, type of study, degree of reliability.
The adequateness metric refers to the definition and / or formula showing how the authors of the publication state the performance of the technique. The techniques used are those considered in each publication. The application domain is the area in which one or several elicitation techniques are used. The type of study refers to the main method that supports the study. The degree of reliability is a validity assessment of the authors' proposal. In this last case, the following values were considered:
• Low: expert opinions of books or articles, or empirical studies without statistical validation results
• Media: empirical studies with results without statistical validation
• High: empirical studies with conclusive results and statistical validation
ANALYSIS OF THE RESULTS
The results of the literature searching are summarized in Figure 1 by DCM format. After applying the filters presented above, 43 primary articles were identified. The eventual searching was the most productive, since much of the related literature belongs to conferences and in specialized journals on software requirements but which are not indexed in the selected databases.
Figure 1. Graphic of the searching.
Of these primary studies [6, 11-52], 58 constructs or ways of evaluating the performance of the techniques were identified. This means that some of the reviewed studies proposed more than one independent metric. For reasons of space it is not possible to show the detail of the information obtained from the review of the identified bibliography. However, it can be reviewed at https://docs.google.com/file/d/0BzV_sIkoJTGAXy1oSzVvOU9XZ0k/edit. A summary of the primary papers found, with information about their origin, is shown in Table 1.
Table 1. Identified works.
This information was analyzed considering four related aspects with the research questions: constructs or metric by which the appropriateness of the techniques is measured, the types of studies and their relationship with these constructs, types of application domains and their relationship with the constructs, and finally, the techniques used in these studies. Each of these results is reviewed in the following sections.
Constructs of appropriateness
When the identified papers were analyzed, 58 different ways to measure the appropriateness of the techniques used were obtained. Many of these measures are quantitative and have metric shape, such as: Number of generated concepts per time unit, Number of identified use cases, Numbers of goals elicited, Number of requirements, among others. Many others are qualitative, such as: Different levels and types of knowledge and information, Types of requirements, etc. And finally others are quite subjective and do not have defined metrics, such as: Fit with problem types (Analysis problems, Synthesis problems, Combination problems), Fit properties of requirements elicitation techniques with contextual facets, Suitability of elicitation techniques according to stakeholders' preferences, among others. These representations were grouped according to nature in 13 main constructs, which are shown in Figure 2 together with the number of times that are used by researchers to define the adequacy of elicitation techniques.
Figure 2. Constructs found in primary studies.
As can be clearly seen, Quality of information elicited is the most used construct to represent the concept of appropriateness, followed by Adequacy as such, Quantity of Elicited Information and Effectiveness.
Due to the constructs Quality level of understanding and Quality of elicited information belong to the same concept, were joined in one construct called Quality, the same was done with the constructs Quantity of information elicited y Quantity of knowledge elicited forming the construct Quantity. On the other hand, the constructs with low incidence: Productivity, Utility, Performance, Specification and Usability, were named Others because their low use. Figure 3 shows those 6 types of constructs as final result of the characterization table and the percentage in which they are used to define the adequacy. These types will be taken into account in the rest of this article as a basis for various analyses.
Figure 3. Percentage of main types of constructs.
Type of study
The identified papers use the constructs, in its proposals, in form of expert opinions or empirical studies. Figure 4 shows the distribution of the 43 items according to the type of study presented, being classified in 4 categories: Experimental, Theoretical, Case Study and Survey.
Figure 4. Percentage of articles according to study type.
Figure 5 illustrates the detail of the distribution showing at the same time that nearly half of the constructs are utilized in experimental articles. Theoretical proposals also have a significant proportion (33%). In this case, a preference to use the Adequacy construct by researchers is observed, that is to say, researchers claim that this or that technique is the most appropriate based in many cases on their experience and in others in the literature. They also use the Quality construct, but in a descriptive or qualitative form.
Figure 5. Distribution of type of study with respect to the main constructs.
The case of the research conducted by experiments is different. In them, researchers use mostly constructs Quality and Quantity to define which technique is most appropriate based on empirical evidence from controlled experiments. In these studies, an increase in the presence of other "quantitative" constructs like Effectiveness and Efficiency is also observed, and opposite to the case of the Theoretical studies, the Adequacy construct is the least used.
The articles that deal with a Case Study show a similar trend of the Experiment ones, but with very few articles. Following this approach, conclusive information cannot be extracted. Survey based studies presented the same situation, with a few of the articles. These latter types of studies mainly focus on constructs of Quantity and Quality of elicited information.
Application Domains
Among the selected articles six application domains were identified: Information Systems, Knowledge engineering, Software Engineering, Marketing Analysis, Discrete Cell Control Systems (Manufacturing), and Product design. The last three joined a group called Others, because they were not found in more than one article. This distribution is presented by Figure 6.
Figure 6. Distribution of articles according to the application domain.
As expected, most of the selected studies apply the elicitation techniques in Software Engineering domain. A significant amount of articles uses techniques in Knowledge Engineering area, since, as it is known, many elicitation techniques were first used in this discipline.
Figure 7 shows the distribution of the application domain related with the constructs. In Software Engineering domain, the most considered construct is Adequacy followed by the Quality construct.
Figure 7. Distribution of the application domain with respect to the main constructs.
Instead, the Quantity and Efficiency constructs are mainly used in Knowledge Engineering domain. In the area of Information Systems, with much less number of studies; focus on measuring the Quality and Quantity of information elicited.
It is noteworthy also that six studies the elicitation techniques were focused in areas of marketing, design and manufacturing.
engineers. In particular, in branch of research such as Evidence-based Software Engineering, where aggregated evidence. The greater the diversity of constructs is used the greater the generalization is required, which means a significant loss of prescriptive information.
Techniques considered
Finally, Figure 8 shows a graph of word cloud with all of the elicitation techniques studied by researchers in the 43 primary articles. On the map, the most named techniques by researchers are highlighted in larger font size. A convergence of most articles for studying the best-known techniques such as the various forms of Interviews, Brainstorming, Questionnaire, Prototyping, JAD, amongst others, is appreciated.
Figure 8. Cloud of elicitation techniques used.
This studied aspect also highlights the fact that there are more than fifty elicitation techniques that have been studied and applied in areas such as software engineering, knowledge engineering and information systems areas.
CONCLUSIONS AND FUTURE WORK
Systematic mapping performed in this work showed that, despite there is a significant amount of research related to the use of elicitation techniques, there is no common way to assess their performance yet. Moreover, a great variability among metrics and / or constructs proposed in the primary studies was found. Only few cases less significant, whose authors have worked together in prior projects, use similar definitions.
This evidence is relevant because it has implications on the results of empirical research. Mainly, it is relevant for the purpose of shaping a body of knowledge to generate guidelines for requirements engineers. In particular, in branch of research such as Evidence-based Software Engineering, where aggregated evidence. The greater the diversity of constructs is used the greater the generalization is required, which means a significant loss of prescriptive information.
For this reason, it is necessary to carry out more proposals to help converging towards a unique way to measure the performance of requirement elicitation techniques.
In this direction, our planned future work aims to contrast the views of practitioners and to raise a proposal on how to find a faithful representation of the performance of the techniques.
ACKNOWLEDGMENTS
The proofreading of this work was done by Marcia Poblete Ríos of the Language Department, Universidad de Atacama.
REFERENCES
[1] Swebok. "Software Engineering Body of Knowledge". 2005. URL: http://www.swebok.org
[2] A. Davis. "Software Requirements: Objects, Functions and States". Englewood Cliffs, N.J. Prentice-Hall. 1993.
[3] P. Loucopoulos and V. Karakostas, "System Requirements Engineering". McGraw-Hill. 1995.
[4] S. Robertson. "Requirements trawling: Techniques for discovering requirements". International Journal of Human-Computer Studies. Vol. 55, pp. 405-421. 2001.
[5] D. Carrizo, O. Dieste and N. Juristo. "Study of elicitation techniques adequacy". Proceedings XI Workshop on Requirements Engineering. WER 2008, pp. 104-114. 2008.
[6] L. Jiang and A. Eberlein. "Selecting Requirements Engineering Techniques based on Project Attributes - A Case Study". Proc. of the 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems. 2007.
[7] E. Kheirkhah and A. Deraman. "Important factors in selecting Requirements Engineering Techniques". International Symposium on Information Technology. 2008.
[8] D. Zowghi and C. Coulin. "Elicitation: A Survey of Techniques, Approaches, and Tools". A. Aurum, and C. Wohlin (Ed). Engineering and Managing software requirements. Springer-Verlag, pp. 19-46. New York, USA. 2005.
[9] B. Kitcheham, T. Dyba and M. Jorgensen. "Evidence-based software engineering". Proceeding of the 26th Int. Conf. on Software Engineerig. IEEE Computer Society, pp. 373378. 2006.
[10] O. Dieste and N. Juristo. "Systematic review and aggregation of empirical studies on elicitation techniques". IEEE Transactions on Software Engineering. Vol. 37 N° 2, pp. 283-304. 2011.
[11] L. S. Al-Salem and A. Abu Samaha. "Eliciting Web application requirements - an industrial case study". Journal of Systems and Software. Vol. 80, Issue 3, pp. 294-313. 2007.
[12] C.-H. Chen, L. P. Khoo and W. Yan. "A strategy for acquiring customer requirement patterns using laddering technique and ART2 neural network". Advanced Engineering Informatics. Vol. 16, Issue 3, pp. 229-240. 2002.
[13] A. R. Hevner and H. D. Mills. "Box-structured requirements determination methods". Decision Support Systems. Vol. 13 N° 3-4, pp. 223-239. 1995.
[14] J. Jin, M. Loftus and I. Franks. "A method for the acquisition of users requirements in discrete manufacturing cell systems". Computer Integrated Manufacturing Systems. Vol. 11 N° 3, pp. 229-242. 1998.
[15] M. Laguna, J. Marqués and F. García. "DocFlow: workflow based requirements elicitation". Information and Software Technology. Vol. 45 N° 6, pp. 357-369. 2003.
[16] V. Laporti, M. Borges and V. Braganholo. "Athena: A collaborative approach to requirements elicitation". Computers in Industry. Vol. 60, N°6, pp. 367-380. 2009.
[17] H. Saiedian and R. Dale. "Requirements engineering: making the connection between the software developer and customer". Information and Software Technology. Vol. 42 N° 6, pp. 419-428. 2000.
[18] J. Moore and F. Shipman. "A comparison of questionnaire-based and GUI-based mean by "the right requirements elicitation techniques"? requirements". Proceedings of the Fifteenth IEEE International Conference on Automated Software Engineering, pp. 35-43. 2000.
[19] A. Sutcliffe. "A technique combination approach to requirements engineering". Proceedings of the Third IEEE International Symposium on Requirements Engineering, pp. 65-74. 1997.
[20] M. Sadiq, S. Ghafir and M. Shahid. "An Approach for Eliciting Software Requirements and its Prioritization Using Analytic Hierarchy Process". International Conference on Advances in Recent Technologies in Communication and Computing. ARTCom '09, pp. 790-795. 2009.
[21] M. Serna. "Analysis and selection to requirements elicitation techniques". 7th Colombian Computing Congress, pp. 1-7. 2012.
[22] S. Kausar, S. Tariq, S.Riaz and A.Khanum. "Guidelines for the selection of elicitation techniques". 6th International Conference on Emerging Technologies, pp. 265-269. 2010.
[23] A. Sen and K. Hemachandran. "Elicitation of Goals in Requirements Engineering Using Agile Methods". IEEE 34th Annual Computer Software and Applications Conference Workshops (COMPSACW), pp. 263-268. 2010.
[24] N. Boulila, A. Hoffmann and A Herrmann. "Using Storytelling to record requirements: Elements for an effective requirements elicitation approach". Fourth International Workshop on Multimedia and Enjoyable Requirements Engineering - Beyond Mere Descriptions and with More Fun and Games (MERE), pp. 9-16. 2011.
[25] S. Vieira, D. Viana, R.do Nascimento, T. Conte. "Evaluating a technique for requirements extraction from business process diagrams through empirical studies". XXXVIII Conferencia Latinoamericana en Informática (CLEI), pp. 1-10. 2012.
[26] A.M. Hickey and A.M. Davis. "Requirements Elicitation and Elicitation Technique Selection: A Model for Two Knowledge-Intensive Software Development Processes". Proceeding of the 36th Annual Hawaii International Conference on System Sciences (HICSS'03) - Track 3 - Vol. 3, pp 96.1. 2003.
[27] B.A Mustafa. "An Experimental Comparison of Use Case Models Understanding by Novice and High Knowledge Users". Proceedings of the 2010 conference on New Trends in Software Methodologies, Tools and Techniques, pp. 182-199. 2010.
[28] G. Browne and M. Rogich. "An Empirical Investigation of User Requirements Elicitation: Comparing the Effectiveness of Prompting Techniques". Journal of Management Information Systems. Vol. 17, pp. 223-249. Spring, 2001.
[29] E. Duggan and C. Thachenkary. "Higher Quality Requirements: Supporting Joint Application Development with the Nominal Group Technique". Information Technology and Management. Vol. 4, pp. 391-408. 2003.
[30] W.J. Lloyd, M. Rosson and J. Arthur. "Effectiveness of elicitation techniques in distributed requirements engineering". Proceedings IEEE Joint International Conference on Requirements Engineering, pp. 311-318. 2002.
[31] R. Agarwal and M. Tanniru. "Knowledge Acquisition Using Structured Interviewing: An Empirical Investigation". Journal of Management of Information Systems. Vol. 7, N°1, pp. 123-141. 1990.
[32] E. Batista and A. Carvalho. "Uma Taxonomía Facetada para Técnicas de Elicitação de Requisitos". Anais do Workshop em Engenharia de Requisitos WER03, pp. 48-62. 2003.
[33] T. Tsumaki and T. Tamai. "Framework for matching requirements elicitation techniques to project characteristics". Software Process Improvement and Practice. Vol. 11 N° 5, pp. 505-519. 2006.
[34] N. Maiden and G. Rugg: "ACRE: selecting methods for requirements acquisition". Software Engineering Journal. Vol. 11 N° 3, pp. 183-192. 1996.
[35] A. Massey and W. Wallace. "Focus groups as a knowledge elicitation technique: an exploratory study". IEEE Transactions on Knowledge and Data Engineering. Vol. 3, Issue 2, pp. 193-200. 1991.
[36] Z. Zhang. "Effective Requirements Development - A Comparison of Requirements Elicitation Techniques". E. Berki, J. Nummenmaa, I. Sunley, M. Ross, G. Staples, (Ed.). Software Quality Management XV: Software Quality in the Knowledge Society. British Computer Society, pp. 225240. 2007.
[37] T. Byrd, K. Cossick and R. Zmud. "A Synthesis of Research on Requirements Analysis and Knowledge Acquisition Techniques". MIS Quarterly. Vol. 16, pp. 117-138. 1992.
[38] G. Aranda, A. Vizcaino, A. Cechich and M. Piattini. "Choosing groupware tools and elicitation techniques according to stakeholders' features". Proceedings of the Seventh International Conference on Enterprise Information Systems ICEIS 2005. Vol. 3, pp. 68-75. 2005.
[39] B. Corbridge, G. Rugg; N. Major, N. Shadbolt and A. Burton. "Laddering - technique and tool use in knowledge acquisition". Knowledge Acquisition. Vol. 6, pp. 315-341. 1994.
[40] C. Chao and G. Salvendy: "Impact of cognitive abilities of experts on the effectiveness of elicited knowledge". Behaviour and Information Technology. Vol. 14 N° 3, pp. 174-182. 1995.
[41] S. Lauesen. "Software requirements: Styles and techniques". Addison-Wesley. 2002.
[42] J. Hua. "Study on knowledge acquisition techniques". Proceedings 2nd International Symposium on Intelligent Information Technology Application, pp. 181-185. 2008.
[43] F. Scapolo and I. Miles. "Eliciting experts' knowledge: A comparison of two methods". Original Research Article Technological Forecasting and Social Change. Vol. 73, Issue 6, pp. 679-704. 2006.
[44] B. McCloskey, J. Geiwitz and J. Kornell. "Empirical comparisons of knowledge acquisition techniques". Proceedings of the Human Factors Society, pp. 268-272. 1991.
[45] W. Wagner, Q. Chung and M. Najdawi. "The impact of problem domains and knowledge acquisition techniques: A content analysis of P/OM expert system case studies". Expert Systems with Applications. Vol. 24 N° 1, pp. 79-86. 2003.
[46] C. Holsapple, V. Raj and W. Wagner. "An experimental investigation of the impact of domain complexity on knowledge acquisition (KA) methods". Expert Systems with Applications. Vol. 35, Issue 3, pp. 1084-1094. 2008.
[47] M. Keil and E. Carmel. "Customer-developer links". Communications of the ACM, Vol. 38 N° 5, pp. 33-44. 1995.
[48] G. Rugg, C. Corbridge, N.Major, A. Burton and N. Shadbolt. "A comparison of sorting techniques in knowledge acquisition". Journal of Knowledge Acquisition. Vol. 4, pp. 279-291. 1992.
[49] J. Sauer, S. Schramme and B. Ruttinger. "Knowledge acquisition in ecological product design: the effects of computer-mediated communication and elicitation method". Behaviour and Information Technology. Vol. 19 N° 5, pp. 315-27. 2000.
[50] J. Fowlkes, E. Salas and D.Baker. "The utility of event-based knowledge elicitation". Human Factors. Vol. 42, pp. 24-35. 2000.
[51] D. Vásquez-Bravo, M Sánchez-Segura, F. Medina-Domínguez and A. Amescua. "Knowledge management acquisition improvement by using software engineering elicitation techniques". Computers in Human Behavior. 2013.
[52] J.S. Dhaliwal and I. Benbazat. "A framework for the comparative evaluation of knowledge acquisition tools and techniques". Knowledge Acquisition. Vol. 2 N° 2, pp.145-166. 1990.
Received: October 6, 2014 Accepted: July 22, 2015
