Validating requirements reviews by introducing fault-type level granularity

A machine learning approach

Maninder Singh, Vaibhav Anu, Gursimran S. Walia, Anurag Goswami

Research output: Chapter in Book/Report/Conference proceedingConference contributionResearchpeer-review

Abstract

Inspections are a proven approach for improving software requirements quality. Owing to the fact that inspectors report both faults and non-faults (i.e., false-positives) in their inspection reports, a major chunk of work falls on the person who is responsible for consolidating the reports received from multiple inspectors. We aim at automation of fault-consolidation step by using supervised machine learning algorithms that can effectively isolate faults from non-faults. Three different inspection studies were conducted in controlled environments to obtain real inspection data from inspectors belonging to both industry and from academic backgrounds. Next, we devised a methodology to separate faults from non-faults by first using ten individual classifiers from five different classification families to categorize different fault-types (e.g., omission, incorrectness, and inconsistencies). Based on the individual performance of classifiers for each fault-type, we created targeted ensembles that are suitable for identification of each fault-type. Our analysis showed that our selected ensemble classifiers were able to separate faults from non-faults with very high accuracy (as high as 85-89% for some fault-types), with a notable result being that in some cases, individual classifiers performed better than ensembles. In general, our approach can significantly reduce effort required to isolate faults from false-positives during the fault consolidation step of requirements inspections. Our approach also discusses the percentage possibility of correctly classifying each fault-type.

Original languageEnglish
Title of host publicationiSOFT - Proceedings of the 11th Innovations in Software Engineering Conference, ISEC 2018
PublisherAssociation for Computing Machinery
ISBN (Electronic)9781450363983
DOIs
StatePublished - 9 Feb 2018
Event11th Innovations in Software Engineering Conference, ISEC 2017 - Hyderabad, India
Duration: 9 Feb 201811 Feb 2018

Publication series

NameACM International Conference Proceeding Series

Conference

Conference11th Innovations in Software Engineering Conference, ISEC 2017
CountryIndia
CityHyderabad
Period9/02/1811/02/18

Fingerprint

Learning systems
Inspection
Classifiers
Consolidation
Learning algorithms
Automation
Industry

Keywords

  • Ensemble
  • Fault types
  • Inspection reviews
  • Machine learning
  • Supervised learning

Cite this

Singh, M., Anu, V., Walia, G. S., & Goswami, A. (2018). Validating requirements reviews by introducing fault-type level granularity: A machine learning approach. In iSOFT - Proceedings of the 11th Innovations in Software Engineering Conference, ISEC 2018 [a10] (ACM International Conference Proceeding Series). Association for Computing Machinery. https://doi.org/10.1145/3172871.3172880
Singh, Maninder ; Anu, Vaibhav ; Walia, Gursimran S. ; Goswami, Anurag. / Validating requirements reviews by introducing fault-type level granularity : A machine learning approach. iSOFT - Proceedings of the 11th Innovations in Software Engineering Conference, ISEC 2018. Association for Computing Machinery, 2018. (ACM International Conference Proceeding Series).
@inproceedings{01fe6cd3e1a648e186d7b4ef57318583,
title = "Validating requirements reviews by introducing fault-type level granularity: A machine learning approach",
abstract = "Inspections are a proven approach for improving software requirements quality. Owing to the fact that inspectors report both faults and non-faults (i.e., false-positives) in their inspection reports, a major chunk of work falls on the person who is responsible for consolidating the reports received from multiple inspectors. We aim at automation of fault-consolidation step by using supervised machine learning algorithms that can effectively isolate faults from non-faults. Three different inspection studies were conducted in controlled environments to obtain real inspection data from inspectors belonging to both industry and from academic backgrounds. Next, we devised a methodology to separate faults from non-faults by first using ten individual classifiers from five different classification families to categorize different fault-types (e.g., omission, incorrectness, and inconsistencies). Based on the individual performance of classifiers for each fault-type, we created targeted ensembles that are suitable for identification of each fault-type. Our analysis showed that our selected ensemble classifiers were able to separate faults from non-faults with very high accuracy (as high as 85-89{\%} for some fault-types), with a notable result being that in some cases, individual classifiers performed better than ensembles. In general, our approach can significantly reduce effort required to isolate faults from false-positives during the fault consolidation step of requirements inspections. Our approach also discusses the percentage possibility of correctly classifying each fault-type.",
keywords = "Ensemble, Fault types, Inspection reviews, Machine learning, Supervised learning",
author = "Maninder Singh and Vaibhav Anu and Walia, {Gursimran S.} and Anurag Goswami",
year = "2018",
month = "2",
day = "9",
doi = "10.1145/3172871.3172880",
language = "English",
series = "ACM International Conference Proceeding Series",
publisher = "Association for Computing Machinery",
booktitle = "iSOFT - Proceedings of the 11th Innovations in Software Engineering Conference, ISEC 2018",

}

Singh, M, Anu, V, Walia, GS & Goswami, A 2018, Validating requirements reviews by introducing fault-type level granularity: A machine learning approach. in iSOFT - Proceedings of the 11th Innovations in Software Engineering Conference, ISEC 2018., a10, ACM International Conference Proceeding Series, Association for Computing Machinery, 11th Innovations in Software Engineering Conference, ISEC 2017, Hyderabad, India, 9/02/18. https://doi.org/10.1145/3172871.3172880

Validating requirements reviews by introducing fault-type level granularity : A machine learning approach. / Singh, Maninder; Anu, Vaibhav; Walia, Gursimran S.; Goswami, Anurag.

iSOFT - Proceedings of the 11th Innovations in Software Engineering Conference, ISEC 2018. Association for Computing Machinery, 2018. a10 (ACM International Conference Proceeding Series).

Research output: Chapter in Book/Report/Conference proceedingConference contributionResearchpeer-review

TY - GEN

T1 - Validating requirements reviews by introducing fault-type level granularity

T2 - A machine learning approach

AU - Singh, Maninder

AU - Anu, Vaibhav

AU - Walia, Gursimran S.

AU - Goswami, Anurag

PY - 2018/2/9

Y1 - 2018/2/9

N2 - Inspections are a proven approach for improving software requirements quality. Owing to the fact that inspectors report both faults and non-faults (i.e., false-positives) in their inspection reports, a major chunk of work falls on the person who is responsible for consolidating the reports received from multiple inspectors. We aim at automation of fault-consolidation step by using supervised machine learning algorithms that can effectively isolate faults from non-faults. Three different inspection studies were conducted in controlled environments to obtain real inspection data from inspectors belonging to both industry and from academic backgrounds. Next, we devised a methodology to separate faults from non-faults by first using ten individual classifiers from five different classification families to categorize different fault-types (e.g., omission, incorrectness, and inconsistencies). Based on the individual performance of classifiers for each fault-type, we created targeted ensembles that are suitable for identification of each fault-type. Our analysis showed that our selected ensemble classifiers were able to separate faults from non-faults with very high accuracy (as high as 85-89% for some fault-types), with a notable result being that in some cases, individual classifiers performed better than ensembles. In general, our approach can significantly reduce effort required to isolate faults from false-positives during the fault consolidation step of requirements inspections. Our approach also discusses the percentage possibility of correctly classifying each fault-type.

AB - Inspections are a proven approach for improving software requirements quality. Owing to the fact that inspectors report both faults and non-faults (i.e., false-positives) in their inspection reports, a major chunk of work falls on the person who is responsible for consolidating the reports received from multiple inspectors. We aim at automation of fault-consolidation step by using supervised machine learning algorithms that can effectively isolate faults from non-faults. Three different inspection studies were conducted in controlled environments to obtain real inspection data from inspectors belonging to both industry and from academic backgrounds. Next, we devised a methodology to separate faults from non-faults by first using ten individual classifiers from five different classification families to categorize different fault-types (e.g., omission, incorrectness, and inconsistencies). Based on the individual performance of classifiers for each fault-type, we created targeted ensembles that are suitable for identification of each fault-type. Our analysis showed that our selected ensemble classifiers were able to separate faults from non-faults with very high accuracy (as high as 85-89% for some fault-types), with a notable result being that in some cases, individual classifiers performed better than ensembles. In general, our approach can significantly reduce effort required to isolate faults from false-positives during the fault consolidation step of requirements inspections. Our approach also discusses the percentage possibility of correctly classifying each fault-type.

KW - Ensemble

KW - Fault types

KW - Inspection reviews

KW - Machine learning

KW - Supervised learning

UR - http://www.scopus.com/inward/record.url?scp=85044398564&partnerID=8YFLogxK

U2 - 10.1145/3172871.3172880

DO - 10.1145/3172871.3172880

M3 - Conference contribution

T3 - ACM International Conference Proceeding Series

BT - iSOFT - Proceedings of the 11th Innovations in Software Engineering Conference, ISEC 2018

PB - Association for Computing Machinery

ER -

Singh M, Anu V, Walia GS, Goswami A. Validating requirements reviews by introducing fault-type level granularity: A machine learning approach. In iSOFT - Proceedings of the 11th Innovations in Software Engineering Conference, ISEC 2018. Association for Computing Machinery. 2018. a10. (ACM International Conference Proceeding Series). https://doi.org/10.1145/3172871.3172880