Analysis and comparison of deep learning models for user stories identification

Abstract

Nowadays, most software companies have adopted agile development methodologies, which suggest the capture of requirements through user stories. Issues Management Systems allow development teams to manage user stories and other issues, such as errors, change requests, and others. Although these systems provide features for categorizing or labeling issue types, the user often needs to include or specify this information correctly. A poor issue categorization causes many user stories to end up buried in a large volume of data, making it difficult to identify them. This article presents and compares three neural network models to classify issues as User Stories. As the ultimate goal of this research is to improve the quality of the software development project documentation, the comparison is practical to select a model to be embedded in an IMS tool for automatically categorizing issues. The compared models are a BRNN-LSTM model, an Elmo-based model, and a BERT-based model. It applied the CRISP-MD methodology to train, validate, and test the three proposed neural network models. Then, a comparison was performed regarding their accuracy and performance. As a result, the article shows that the BERT-based model is the one that best fits the problem posed, managing to classify the issues as user stories with an accuracy of approximately 97%. This model can analyze the text syntactically and semantically with the best accuracy and performance.