I tried coding with AI, and became its micro-manager instead

As AI tools continue to gain traction in the world of software engineering, claims that AI can replace junior (and even mid-level) developers have become increasingly common. GitHub CoPilot, an AI-powered coding assistant, is one of the most widely discussed tools in this space, readily available for free to Github users. With the promise of speeding up development by suggesting code and automating tasks, I decided to dive into an experiment of my own. 

For context: I have a broad background in infrastructure, cloud, software delivery, and DevOps, but I don't consider myself a software engineer per se (even if I've written a lot of code). Surely, with the help of Github CoPilot, I should be able to build something simple? 

In the spirit of "AI replacing the developer" I was going to get this done through Vibe Coding.

What is Vibe Coding?

"Vibe Coding" refers to a programming approach where the developer relies heavily on AI tools for code generation, iterating with minimal hands-on intervention. The term was coined by Andrej Karpathy, who introduced the concept as a way to describe coding with AI tools in a more relaxed, experimental way — letting the AI "vibe" with the problem and produce code suggestions. While it offers the potential for rapid prototyping or quick solutions, some say it results in code that requires substantial oversight to ensure its quality. 

(You can read more about the origin and definition of Vibe Coding on Wikipedia.)

In my case, it was a fun experiment to see how well I could rely on CoPilot to produce something usable without diving too deep into TypeScript specifics, or get a PhD in using AI tools first. Here's how it went.

The Setup: A Simple Echo Service

The goal was simple: build a web service with an /echo endpoint in TypeScript that would return a 200 OK response for GET requests, and echo back any data sent in POST requests. In addition, I wanted CoPilot to handle creating a Docker image, adding tests, and configuring the devcontainer.json to develop the service in a DevContainer.

CoPilot generated a bunch of files in a matter of seconds. Awesome, right? It even provided instructions on how to build and run the TypeScript code. But my first hurdle appeared when I attempted to get everything running inside a DevContainer.

The First Hurdle: DevContainer Setup Fails

While CoPilot quickly generated a devcontainer.json file for me, it didn’t work right out of the gate. The base image it suggested didn’t exist, and it had syntax issues due to outdated JSON structures that led to build failures. I tried a few times to let CoPilot fix the issues, but it struggled even with something as simple as merging two JSON blocks together. Eventually, I took over and fixed the errors myself.

A few moments later, I had a working DevContainer, and the echo service was up and running. This was a win — but my excitement was short-lived.

The Service Runs, But… It Doesn’t Quite Work

The Echo Service was running, and I could access the /echo endpoint, but there was a problem: POST requests didn’t return the data as expected. CoPilot “fixed” the code several times, each time adding more lines, but the net result was still the same — no working POST request handling. After a few attempts to get it working, I realized that CoPilot had implemented the fixes in the wrong order, meaning the necessary code wouldn’t actually be triggered. This left me micro-managing it more than I expected.

Ultimately, I had to manually tweak the code and guide CoPilot through what was supposed to be simple fixes. I ended up with a working echo service, but it took far longer than I expected.

I also encountered an issue I didn’t anticipate: my echo service didn’t handle SIGINT (the signal that lets you stop the process with Ctrl+C). I didn’t realize that a Node.js service doesn't handle it by default, so I didn’t think to ask for it in my prompt. However, despite me making clear the Echo Service was meant to be run in a container (thus needing proper SIGINT handling for graceful shutdown), CoPilot never even suggested adding it. 

Testing: Mostly Right, but Still a Lot of Work

Once I had the service running, I wanted to add tests. Again, CoPilot came through quickly, generating a basic test structure in just a few seconds. It got about 80% of it right, but the remaining 20% required me to learn a bit about TypeScript testing libraries and manually adjust the test assertions. 

As before, CoPilot responded well to micro-management — more so than a typical junior developer would. It didn't get frustrated or defensive; it just kept suggesting fixes as I guided it through what needed to be done.

But here's the problem: while the AI is good at generating quick drafts, it’s seemingly unable to fully understand the intricacies of your specific problem (as we’ve seen with the SIGINT handling). CoPilot doesn’t have the same level of reasoning, problem-solving, or adaptability that a human developer brings to the table. It can give you the skeleton of a solution, but to get everything working properly, you still need to do a significant amount of work.

What I Learned: AI Can Help, But It Can’t Replace Developers

At the end of the day, I had a working Echo Service with tests, running in a DevContainer. But I still don’t know if the code quality is good. It might be terrible. And I definitely wouldn’t trust it in a production environment without significant further review and refactoring.

So, does this mean AI tools like CoPilot can replace junior developers? My experience says no, at least not yet. AI is certainly helpful, but it’s far from perfect. While CoPilot helped me speed up certain tasks, I still needed to constantly intervene to fix mistakes, adjust code, and ensure things were working as expected. It might be faster than a junior developer, but that seems to be offset by the fact that the typical junior developer will learn from mistakes they make. AI, much less so. It will, confidently, keep making the same mistakes.

This brings me to an important realization: AI can augment the work of skilled engineers, but it can’t replace them. If you want to use AI tools effectively, you still need someone with a solid understanding of software engineering to guide it. These tools are best used by experienced developers who can ask the right questions and recognize when the AI is going off track. And, as we all know, skilled engineers are still in high demand. It’s not enough to just rely on AI to generate “good” code — we still need skilled humans to teach the AI, write the training data, and manage the process.

Pure 'Vibe Coding': A Tool for Prototyping, Not Production

While Vibe Coding can be a fun and useful way to prototype quickly or experiment with new ideas, it’s best left to skilled engineers who can ensure that the results aren’t disastrous. It can dramatically reduce the time it takes to get something running, but don’t expect the generated code to be production-ready, secure, or scalable. At this stage, AI-generated code is best suited for non-production codebases or proof-of-concept applications where speed is more important than quality, and I would happily use it for that purpose.

Final Thoughts

In the future, as AI continues to improve, we may see it take on more of the heavy lifting in software development. But for now, skilled human developers are still essential to ensure the code is functional, maintainable, and of high quality. The reality is that we need to keep investing in developing and mentoring junior engineers, so they can learn to use these AI tools responsibly and effectively. And that’s something no AI can replace just yet.

Finally, the AI landscape is very rapidly evolving. I chose to be a little naive in my approach to Vibe Coding, and as you’ve read the results leave a little to be desired. Using different tools, different models, or simply being better at structuring prompts can get you better results. That means that in itself, using AI requires skills that need to be developed.

Watch this space for more upcoming blogposts on this subject. If you want to work with us, feel free to reach out!