Comparing general purpose pre-trained Word and Sentence embeddings for Requirements Classification

Federico Cruciani; Samuel Moore; CD Nugent

Comparing general purpose pre-trained Word and Sentence embeddings for Requirements Classification

Federico Cruciani, Samuel Moore, CD Nugent

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

173 Downloads (Pure)

Abstract

The recent evolution of NLP has enriched the set of DL-based approaches to include a number of general purpose Large Language Models (LLMs). Whereas new models have been proven useful for generic text handling, their applicability to domain-specific NLP tasks still remains doubtful, particularly because of the limited amount of dataset available in certain domains, such as Requirements Engineering. In this study, different pre-trained embeddings were tested in three requirements classification tasks, in search of a tradeoff between accuracy and computational complexity. The best F1-score results were obtained with BERT (90.36% and 84.23%), with DistilBERT identified as optimal tradeoff (90.28% and 82.61%).

Original language	English
Title of host publication	Joint Proceedings of REFSQ-2023 Workshops, Doctoral Symposium, Posters & Tools Track and Journal Early Feedback (REFSQ-JP 2023)
Subtitle of host publication	REFSQ Co-Located Events 2023
Publisher	CEUR-WS
Volume	3378
Publication status	Published online - 20 Apr 2023
Event	6th Workshop on Natural Language Processing for Requirements Engineering: REFSQ Co-Located Events 2023 - Barcelona, Spain Duration: 17 Apr 2023 → …

Workshop

Workshop	6th Workshop on Natural Language Processing for Requirements Engineering
Abbreviated title	NLP4RE
Country/Territory	Spain
City	Barcelona
Period	17/04/23 → …

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

NLP4RE-paper4_publishedFinal published version, 1.69 MBLicence: CC BY

https://ceur-ws.org/Vol-3378/NLP4RE-paper4.pdfLicence: CC BY

Improving reliability in the internet of things through anomaly detection
Moore, S. J. (Author), Zhang, S. (Supervisor), Nugent, C. (Supervisor) & Cleland, I. (Supervisor), Sept 2022
Student thesis: Doctoral Thesis

File

Cite this

Cruciani, F., Moore, S., & Nugent, CD. (2023). Comparing general purpose pre-trained Word and Sentence embeddings for Requirements Classification. In Joint Proceedings of REFSQ-2023 Workshops, Doctoral Symposium, Posters & Tools Track and Journal Early Feedback (REFSQ-JP 2023): REFSQ Co-Located Events 2023 (Vol. 3378). CEUR-WS. Advance online publication. https://ceur-ws.org/Vol-3378/NLP4RE-paper4.pdf

@inproceedings{9685eb3eb2174b0588750ac45138c681,

title = "Comparing general purpose pre-trained Word and Sentence embeddings for Requirements Classification",

abstract = "The recent evolution of NLP has enriched the set of DL-based approaches to include a number of general purpose Large Language Models (LLMs). Whereas new models have been proven useful for generic text handling, their applicability to domain-specific NLP tasks still remains doubtful, particularly because of the limited amount of dataset available in certain domains, such as Requirements Engineering. In this study, different pre-trained embeddings were tested in three requirements classification tasks, in search of a tradeoff between accuracy and computational complexity. The best F1-score results were obtained with BERT (90.36\% and 84.23\%), with DistilBERT identified as optimal tradeoff (90.28\% and 82.61\%).",

author = "Federico Cruciani and Samuel Moore and CD Nugent",

year = "2023",

month = apr,

day = "20",

language = "English",

volume = "3378",

booktitle = "Joint Proceedings of REFSQ-2023 Workshops, Doctoral Symposium, Posters \& Tools Track and Journal Early Feedback (REFSQ-JP 2023)",

publisher = "CEUR-WS",

note = "6th Workshop on Natural Language Processing for Requirements Engineering : REFSQ Co-Located Events 2023, NLP4RE ; Conference date: 17-04-2023",

}

Cruciani, F , Moore, S & Nugent, CD 2023, Comparing general purpose pre-trained Word and Sentence embeddings for Requirements Classification. in Joint Proceedings of REFSQ-2023 Workshops, Doctoral Symposium, Posters & Tools Track and Journal Early Feedback (REFSQ-JP 2023): REFSQ Co-Located Events 2023. vol. 3378, CEUR-WS, 6th Workshop on Natural Language Processing for Requirements Engineering, Barcelona, Spain, 17/04/23. <https://ceur-ws.org/Vol-3378/NLP4RE-paper4.pdf>

Comparing general purpose pre-trained Word and Sentence embeddings for Requirements Classification. / Cruciani, Federico ; Moore, Samuel ; Nugent, CD.
Joint Proceedings of REFSQ-2023 Workshops, Doctoral Symposium, Posters & Tools Track and Journal Early Feedback (REFSQ-JP 2023): REFSQ Co-Located Events 2023. Vol. 3378 CEUR-WS, 2023.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Comparing general purpose pre-trained Word and Sentence embeddings for Requirements Classification

AU - Cruciani, Federico

AU - Moore, Samuel

AU - Nugent, CD

PY - 2023/4/20

Y1 - 2023/4/20

N2 - The recent evolution of NLP has enriched the set of DL-based approaches to include a number of general purpose Large Language Models (LLMs). Whereas new models have been proven useful for generic text handling, their applicability to domain-specific NLP tasks still remains doubtful, particularly because of the limited amount of dataset available in certain domains, such as Requirements Engineering. In this study, different pre-trained embeddings were tested in three requirements classification tasks, in search of a tradeoff between accuracy and computational complexity. The best F1-score results were obtained with BERT (90.36% and 84.23%), with DistilBERT identified as optimal tradeoff (90.28% and 82.61%).

AB - The recent evolution of NLP has enriched the set of DL-based approaches to include a number of general purpose Large Language Models (LLMs). Whereas new models have been proven useful for generic text handling, their applicability to domain-specific NLP tasks still remains doubtful, particularly because of the limited amount of dataset available in certain domains, such as Requirements Engineering. In this study, different pre-trained embeddings were tested in three requirements classification tasks, in search of a tradeoff between accuracy and computational complexity. The best F1-score results were obtained with BERT (90.36% and 84.23%), with DistilBERT identified as optimal tradeoff (90.28% and 82.61%).

M3 - Conference contribution

VL - 3378

BT - Joint Proceedings of REFSQ-2023 Workshops, Doctoral Symposium, Posters & Tools Track and Journal Early Feedback (REFSQ-JP 2023)

PB - CEUR-WS

T2 - 6th Workshop on Natural Language Processing for Requirements Engineering

Y2 - 17 April 2023

ER -

Comparing general purpose pre-trained Word and Sentence embeddings for Requirements Classification

Abstract

Workshop

UN SDGs

Access to Document

Fingerprint

Student theses

Improving reliability in the internet of things through anomaly detection

Cite this