TY - GEN
T1 - A Comparative Analysis between AI Generated Code and Human Written Code
T2 - 2024 IEEE International Conference on Big Data, BigData 2024
AU - Patel, Abhi
AU - Sultana, Kazi Zakia
AU - Samanthula, Bharath K.
N1 - Publisher Copyright:
© 2024 IEEE.
PY - 2024
Y1 - 2024
N2 - In today's world where generative artificial intelligence has almost become an integral part of the coding, new challenges must be faced. Therefore, evaluating software bugs in both human written and AI generated code can be useful for the developers. A comparative analysis of these two coding practices is not only helpful for the decisions taken by the developers, it will also assist to give a direction on how to improve the quality of AI driven code. Currently, researchers have leveraged the role of software metrics to compare human written code and AI generated code as these metrics have long been utilized for software bug and vulnerability prediction by the researchers. They also analyzed the secure coding practices in terms of the number of bugs found in both AI and human written code. Our study is an extension of the current works as this study focuses on a set of metrics and a set of bugs as identified by some static analyzer tools. Investigating these two coding practices from different angles can help to reveal unknown relationships and factors that further can be analyzed to improve code quality of recent AI tools. Therefore, the main objective of our work is to identify the relationships between software metrics and bugs in AI generated code and human written code to compare and contrast the coding profiles of the two approaches. This will offer developers critical knowledge to enhance their strategies in mitigating potential bug risks across different coding methodologies. We have utilized top-rated Java solutions to 90 LeetCode problems, generated corresponding AI Java solutions to them, and utilized various static analysis tools to collect metrics and bugs. In this study, we found that two software metrics CountLineCodeDecl and CountLineCodeExe are positively correlated with the bug DLS-DEAD-LOCAL-STORE and the metric AvgCyclomatic is related to the bug AvoidLiteralsInIfCondition in both human written and AI generated code. These findings provide developers with critical insights into potential bug risks, enabling more effective mitigation strategies across different coding methodologies.
AB - In today's world where generative artificial intelligence has almost become an integral part of the coding, new challenges must be faced. Therefore, evaluating software bugs in both human written and AI generated code can be useful for the developers. A comparative analysis of these two coding practices is not only helpful for the decisions taken by the developers, it will also assist to give a direction on how to improve the quality of AI driven code. Currently, researchers have leveraged the role of software metrics to compare human written code and AI generated code as these metrics have long been utilized for software bug and vulnerability prediction by the researchers. They also analyzed the secure coding practices in terms of the number of bugs found in both AI and human written code. Our study is an extension of the current works as this study focuses on a set of metrics and a set of bugs as identified by some static analyzer tools. Investigating these two coding practices from different angles can help to reveal unknown relationships and factors that further can be analyzed to improve code quality of recent AI tools. Therefore, the main objective of our work is to identify the relationships between software metrics and bugs in AI generated code and human written code to compare and contrast the coding profiles of the two approaches. This will offer developers critical knowledge to enhance their strategies in mitigating potential bug risks across different coding methodologies. We have utilized top-rated Java solutions to 90 LeetCode problems, generated corresponding AI Java solutions to them, and utilized various static analysis tools to collect metrics and bugs. In this study, we found that two software metrics CountLineCodeDecl and CountLineCodeExe are positively correlated with the bug DLS-DEAD-LOCAL-STORE and the metric AvgCyclomatic is related to the bug AvoidLiteralsInIfCondition in both human written and AI generated code. These findings provide developers with critical insights into potential bug risks, enabling more effective mitigation strategies across different coding methodologies.
KW - Artificial Intelligence
KW - Bugs
KW - Software Metrics
KW - Static Code Analysis
UR - http://www.scopus.com/inward/record.url?scp=85218042052&partnerID=8YFLogxK
U2 - 10.1109/BigData62323.2024.10825958
DO - 10.1109/BigData62323.2024.10825958
M3 - Conference contribution
AN - SCOPUS:85218042052
T3 - Proceedings - 2024 IEEE International Conference on Big Data, BigData 2024
SP - 7521
EP - 7529
BT - Proceedings - 2024 IEEE International Conference on Big Data, BigData 2024
A2 - Ding, Wei
A2 - Lu, Chang-Tien
A2 - Wang, Fusheng
A2 - Di, Liping
A2 - Wu, Kesheng
A2 - Huan, Jun
A2 - Nambiar, Raghu
A2 - Li, Jundong
A2 - Ilievski, Filip
A2 - Baeza-Yates, Ricardo
A2 - Hu, Xiaohua
PB - Institute of Electrical and Electronics Engineers Inc.
Y2 - 15 December 2024 through 18 December 2024
ER -