Date Awarded
2024
Document Type
Thesis
Degree Name
Master of Science (M.Sc.)
Department
Computer Science
Advisor
Denys Poshyvanyk
Committee Member
Adwait Nadkarni
Committee Member
Oscar Chaparro
Abstract
A growing interest for Large Language Models (LLMs) is how increasing their size might result in changes to their behavior not predictable from relatively smaller-scaled models. Analyzing these emergent capabilities is therefore crucial to understanding and developing LLMs. Yet, whether LLMs exhibit emergence, or possess emergent capabilities, is a contested question. Furthermore, most research into LLM emergence has focused on natural language processing tasks and models suited for them. We focus on investigating emergence in the context of software engineering, and recontextualize the discussion of emergence in the context of prior research. We propose a multifaceted pipeline for evaluating and reasoning about emergent capabilities of LLMs in any context and instantiate this pipeline to analyze the emergent capabilities of the CodeGen1-multi model across four scales ranging from 350M parameters to 16.1B parameters. We examine the model's performance on the software engineering tasks of automatic bug fixing, code translation, and commit message generation. We find no evidence of emergent growth at this scale on these tasks and consequently discuss the future investigation of emergent capabilities.
DOI
https://dx.doi.org/10.21220/s2-889y-nd72
Rights
© The Author
Recommended Citation
O'Brien, Conor, "Emergent Capabilities Of Llms For Software Engineering" (2024). Dissertations, Theses, and Masters Projects. William & Mary. Paper 1727787986.
https://dx.doi.org/10.21220/s2-889y-nd72
