Abstract:
The rapid adoption of Large Language Models (LLM) and Generative AI (GenAI) tools in
programming education raises critical questions about whether natural language can directly
replace traditional programming language knowledge. This study investigated ChatGPT’s (4o,
4o-mini and 3.5) ability to interpret natural language instructions and convert them into executable
code. This experimental research employed 15 programming tasks with varying difficulty,
each tested with clear and ad-hoc prompts across three modes of ChatGPT (loggedin,
logged-out, incognito), resulting in 90 instruction-context combinations. Each interaction
passed through a three-phase framework: converting human instruction to a conceptual model,
converting the conceptual model into program code, and converting program code to executable
machine code. Data included ChatGPT responses (code snippets, explanations, sample outputs)
and metadata (GPT model version and programming language generated). All result program
code outputs were executed in Visual Studio Code to determine correctness and functionality.
Analysis combined quantitative and qualitative approaches. The quantitative analysis used chisquare
tests to examine relationships between variables such as the presence of comments, explanations,
sample outputs, programming language used, against independent factors including
GPT version, task type, instruction clarity and user context. The qualitative analysis focused on
task-based evaluations of frequency of instruction attempts, code correctness, logic implementation,
input handling, explanation clarity, and conversational tone. Results showed that the user
context influenced the model version used and the explanation provided. Clear instructions produced
cleaner code, while unclear instructions generated more detailed explanations. ChatGPT
primarily provided Python programs, but occasionally switched to Java when the instructions
used terms like “length” to describe size or counting. Barrier testing confirmed ChatGPT cannot
directly compile code into machine code but effectively simulates reasoning, explains logic,
and provides clear manual execution guidance, bridging the first two cognitive phases. Chat-
GPT performed well in creating functional code with both instruction types. Clear instructions
led to organised output; ad-hoc instructions resulted in richer explanations and interactive conversations.
These findings suggest natural language could instruct LLM and GenAI tools to
generate usable program codes. This could shift programming education dynamics by challenging
the need to learn coding skills in particular programming languages, but rather learning
to express instructions into AI tools for successful results. Future programming education may
focus more on computational thinking, logical thinking and communication than excelling in
programming language syntax and semantics.
