Programming with GenAI: How ChatGPT responds to different instructions of a novice programmer

Abstract

The rapid adoption of Large Language Models (LLM) and Generative AI (GenAI) tools in programming education raises critical questions about whether natural language can directly replace traditional programming language knowledge. This study investigated ChatGPT’s (4o, 4o-mini and 3.5) ability to interpret natural language instructions and convert them into executable code. This experimental research employed 15 programming tasks with varying difficulty, each tested with clear and ad-hoc prompts across three modes of ChatGPT (loggedin, logged-out, incognito), resulting in 90 instruction-context combinations. Each interaction passed through a three-phase framework: converting human instruction to a conceptual model, converting the conceptual model into program code, and converting program code to executable machine code. Data included ChatGPT responses (code snippets, explanations, sample outputs) and metadata (GPT model version and programming language generated). All result program code outputs were executed in Visual Studio Code to determine correctness and functionality. Analysis combined quantitative and qualitative approaches. The quantitative analysis used chisquare tests to examine relationships between variables such as the presence of comments, explanations, sample outputs, programming language used, against independent factors including GPT version, task type, instruction clarity and user context. The qualitative analysis focused on task-based evaluations of frequency of instruction attempts, code correctness, logic implementation, input handling, explanation clarity, and conversational tone. Results showed that the user context influenced the model version used and the explanation provided. Clear instructions produced cleaner code, while unclear instructions generated more detailed explanations. ChatGPT primarily provided Python programs, but occasionally switched to Java when the instructions used terms like “length” to describe size or counting. Barrier testing confirmed ChatGPT cannot directly compile code into machine code but effectively simulates reasoning, explains logic, and provides clear manual execution guidance, bridging the first two cognitive phases. Chat- GPT performed well in creating functional code with both instruction types. Clear instructions led to organised output; ad-hoc instructions resulted in richer explanations and interactive conversations. These findings suggest natural language could instruct LLM and GenAI tools to generate usable program codes. This could shift programming education dynamics by challenging the need to learn coding skills in particular programming languages, but rather learning to express instructions into AI tools for successful results. Future programming education may focus more on computational thinking, logical thinking and communication than excelling in programming language syntax and semantics.