A Comparison of Nano-Patterns vs. Software Metrics in Vulnerability Prediction

Kazi Zakia Sultana, Byron J. Williams, Amiangshu Bosu

Research output: Chapter in Book/Report/Conference proceedingConference contributionResearchpeer-review

Abstract

Context: Software security is an imperative aspect of software quality. Early detection of vulnerable code during development can better ensure the security of the codebase and minimize testing efforts. Although traditional software metrics are used for early detection of vulnerabilities, they do not clearly address the granularity level of the issue to precisely pinpoint vulnerabilities. The goal of this study is to employ method-level traceable patterns (nano-patterns) in vulnerability prediction and empirically compare their performance with traditional software metrics. The concept of nano-patterns is similar to design patterns, but these constructs can be automatically recognized and extracted from source code. If nano-patterns can better predict vulnerable methods compared to software metrics, they can be used in developing vulnerability prediction models with better accuracy. Aims: This study explores the performance of method-level patterns in vulnerability prediction. We also compare them with method-level software metrics. Method: We studied vulnerabilities reported for two major releases of Apache Tomcat (6 and 7), Apache CXF, and two stand-alone Java web applications. We used three machine learning techniques to predict vulnerabilities using nano-patterns as features. We applied the same techniques using method-level software metrics as features and compared their performance with nano-patterns. Results: We found that nano-patterns show lower false negative rates for classifying vulnerable methods (for Tomcat 6, 21% vs 34.7%) and therefore, have higher recall in predicting vulnerable code than the software metrics used. On the other hand, software metrics show higher precision than nano-patterns (79.4% vs 76.6%). Conclusion: In summary, we suggest developers use nano-patterns as features for vulnerability prediction to augment existing approaches as these code constructs outperform standard metrics in terms of prediction recall.

Original languageEnglish
Title of host publicationProceedings - 25th Asia-Pacific Software Engineering Conference, APSEC 2018
PublisherIEEE Computer Society
Pages355-364
Number of pages10
ISBN (Electronic)9781728119700
DOIs
StatePublished - 2 Jul 2018
Event25th Asia-Pacific Software Engineering Conference, APSEC 2018 - Nara, Japan
Duration: 4 Dec 20187 Dec 2018

Publication series

NameProceedings - Asia-Pacific Software Engineering Conference, APSEC
Volume2018-December
ISSN (Print)1530-1362

Conference

Conference25th Asia-Pacific Software Engineering Conference, APSEC 2018
CountryJapan
CityNara
Period4/12/187/12/18

Fingerprint

Learning systems
Testing

Keywords

  • Vulnerability
  • nano-pattern
  • software metrics
  • software security

Cite this

Sultana, K. Z., Williams, B. J., & Bosu, A. (2018). A Comparison of Nano-Patterns vs. Software Metrics in Vulnerability Prediction. In Proceedings - 25th Asia-Pacific Software Engineering Conference, APSEC 2018 (pp. 355-364). [8719530] (Proceedings - Asia-Pacific Software Engineering Conference, APSEC; Vol. 2018-December). IEEE Computer Society. https://doi.org/10.1109/APSEC.2018.00050
Sultana, Kazi Zakia ; Williams, Byron J. ; Bosu, Amiangshu. / A Comparison of Nano-Patterns vs. Software Metrics in Vulnerability Prediction. Proceedings - 25th Asia-Pacific Software Engineering Conference, APSEC 2018. IEEE Computer Society, 2018. pp. 355-364 (Proceedings - Asia-Pacific Software Engineering Conference, APSEC).
@inproceedings{5e4e195aee5c456f82b6ec45f37dcabf,
title = "A Comparison of Nano-Patterns vs. Software Metrics in Vulnerability Prediction",
abstract = "Context: Software security is an imperative aspect of software quality. Early detection of vulnerable code during development can better ensure the security of the codebase and minimize testing efforts. Although traditional software metrics are used for early detection of vulnerabilities, they do not clearly address the granularity level of the issue to precisely pinpoint vulnerabilities. The goal of this study is to employ method-level traceable patterns (nano-patterns) in vulnerability prediction and empirically compare their performance with traditional software metrics. The concept of nano-patterns is similar to design patterns, but these constructs can be automatically recognized and extracted from source code. If nano-patterns can better predict vulnerable methods compared to software metrics, they can be used in developing vulnerability prediction models with better accuracy. Aims: This study explores the performance of method-level patterns in vulnerability prediction. We also compare them with method-level software metrics. Method: We studied vulnerabilities reported for two major releases of Apache Tomcat (6 and 7), Apache CXF, and two stand-alone Java web applications. We used three machine learning techniques to predict vulnerabilities using nano-patterns as features. We applied the same techniques using method-level software metrics as features and compared their performance with nano-patterns. Results: We found that nano-patterns show lower false negative rates for classifying vulnerable methods (for Tomcat 6, 21{\%} vs 34.7{\%}) and therefore, have higher recall in predicting vulnerable code than the software metrics used. On the other hand, software metrics show higher precision than nano-patterns (79.4{\%} vs 76.6{\%}). Conclusion: In summary, we suggest developers use nano-patterns as features for vulnerability prediction to augment existing approaches as these code constructs outperform standard metrics in terms of prediction recall.",
keywords = "Vulnerability, nano-pattern, software metrics, software security",
author = "Sultana, {Kazi Zakia} and Williams, {Byron J.} and Amiangshu Bosu",
year = "2018",
month = "7",
day = "2",
doi = "10.1109/APSEC.2018.00050",
language = "English",
series = "Proceedings - Asia-Pacific Software Engineering Conference, APSEC",
publisher = "IEEE Computer Society",
pages = "355--364",
booktitle = "Proceedings - 25th Asia-Pacific Software Engineering Conference, APSEC 2018",

}

Sultana, KZ, Williams, BJ & Bosu, A 2018, A Comparison of Nano-Patterns vs. Software Metrics in Vulnerability Prediction. in Proceedings - 25th Asia-Pacific Software Engineering Conference, APSEC 2018., 8719530, Proceedings - Asia-Pacific Software Engineering Conference, APSEC, vol. 2018-December, IEEE Computer Society, pp. 355-364, 25th Asia-Pacific Software Engineering Conference, APSEC 2018, Nara, Japan, 4/12/18. https://doi.org/10.1109/APSEC.2018.00050

A Comparison of Nano-Patterns vs. Software Metrics in Vulnerability Prediction. / Sultana, Kazi Zakia; Williams, Byron J.; Bosu, Amiangshu.

Proceedings - 25th Asia-Pacific Software Engineering Conference, APSEC 2018. IEEE Computer Society, 2018. p. 355-364 8719530 (Proceedings - Asia-Pacific Software Engineering Conference, APSEC; Vol. 2018-December).

Research output: Chapter in Book/Report/Conference proceedingConference contributionResearchpeer-review

TY - GEN

T1 - A Comparison of Nano-Patterns vs. Software Metrics in Vulnerability Prediction

AU - Sultana, Kazi Zakia

AU - Williams, Byron J.

AU - Bosu, Amiangshu

PY - 2018/7/2

Y1 - 2018/7/2

N2 - Context: Software security is an imperative aspect of software quality. Early detection of vulnerable code during development can better ensure the security of the codebase and minimize testing efforts. Although traditional software metrics are used for early detection of vulnerabilities, they do not clearly address the granularity level of the issue to precisely pinpoint vulnerabilities. The goal of this study is to employ method-level traceable patterns (nano-patterns) in vulnerability prediction and empirically compare their performance with traditional software metrics. The concept of nano-patterns is similar to design patterns, but these constructs can be automatically recognized and extracted from source code. If nano-patterns can better predict vulnerable methods compared to software metrics, they can be used in developing vulnerability prediction models with better accuracy. Aims: This study explores the performance of method-level patterns in vulnerability prediction. We also compare them with method-level software metrics. Method: We studied vulnerabilities reported for two major releases of Apache Tomcat (6 and 7), Apache CXF, and two stand-alone Java web applications. We used three machine learning techniques to predict vulnerabilities using nano-patterns as features. We applied the same techniques using method-level software metrics as features and compared their performance with nano-patterns. Results: We found that nano-patterns show lower false negative rates for classifying vulnerable methods (for Tomcat 6, 21% vs 34.7%) and therefore, have higher recall in predicting vulnerable code than the software metrics used. On the other hand, software metrics show higher precision than nano-patterns (79.4% vs 76.6%). Conclusion: In summary, we suggest developers use nano-patterns as features for vulnerability prediction to augment existing approaches as these code constructs outperform standard metrics in terms of prediction recall.

AB - Context: Software security is an imperative aspect of software quality. Early detection of vulnerable code during development can better ensure the security of the codebase and minimize testing efforts. Although traditional software metrics are used for early detection of vulnerabilities, they do not clearly address the granularity level of the issue to precisely pinpoint vulnerabilities. The goal of this study is to employ method-level traceable patterns (nano-patterns) in vulnerability prediction and empirically compare their performance with traditional software metrics. The concept of nano-patterns is similar to design patterns, but these constructs can be automatically recognized and extracted from source code. If nano-patterns can better predict vulnerable methods compared to software metrics, they can be used in developing vulnerability prediction models with better accuracy. Aims: This study explores the performance of method-level patterns in vulnerability prediction. We also compare them with method-level software metrics. Method: We studied vulnerabilities reported for two major releases of Apache Tomcat (6 and 7), Apache CXF, and two stand-alone Java web applications. We used three machine learning techniques to predict vulnerabilities using nano-patterns as features. We applied the same techniques using method-level software metrics as features and compared their performance with nano-patterns. Results: We found that nano-patterns show lower false negative rates for classifying vulnerable methods (for Tomcat 6, 21% vs 34.7%) and therefore, have higher recall in predicting vulnerable code than the software metrics used. On the other hand, software metrics show higher precision than nano-patterns (79.4% vs 76.6%). Conclusion: In summary, we suggest developers use nano-patterns as features for vulnerability prediction to augment existing approaches as these code constructs outperform standard metrics in terms of prediction recall.

KW - Vulnerability

KW - nano-pattern

KW - software metrics

KW - software security

UR - http://www.scopus.com/inward/record.url?scp=85066783110&partnerID=8YFLogxK

U2 - 10.1109/APSEC.2018.00050

DO - 10.1109/APSEC.2018.00050

M3 - Conference contribution

T3 - Proceedings - Asia-Pacific Software Engineering Conference, APSEC

SP - 355

EP - 364

BT - Proceedings - 25th Asia-Pacific Software Engineering Conference, APSEC 2018

PB - IEEE Computer Society

ER -

Sultana KZ, Williams BJ, Bosu A. A Comparison of Nano-Patterns vs. Software Metrics in Vulnerability Prediction. In Proceedings - 25th Asia-Pacific Software Engineering Conference, APSEC 2018. IEEE Computer Society. 2018. p. 355-364. 8719530. (Proceedings - Asia-Pacific Software Engineering Conference, APSEC). https://doi.org/10.1109/APSEC.2018.00050