Imagine a big black box. It is far away and mysterious, but you can talk to it. You can describe all your needs and wishes, and in response it can create any kind of software to solve your problems. It produces computer programs with the desired properties – the behaviour, the memory, the inputs, the outputs, the looks.
This live 3D rendering is written in plain JavaScript by a human (me). You can read more about it here.
Along with instructions on how to run the software, the big black box gives you all the technical artifacts and recipes needed to build it (source code), so that you can verify how it works, evaluate the quality and maybe also share it with others.
There is however a slight problem.
You have lost the ability to really understand the recipes. So has most people you know and work with. It has been a long time since people wrote software instructions for computers themselves.
Instead you must trust your ability to communicate clearly with the big black box and then verify that the product has the intended behaviour, by testing and measurement. The box can provide you with automated tests along with the product, but verifying that the automated tests themselves are correct is difficult, because you do not really understand how they work either.
The software you get is really a smaller opaque box with some knobs, lights and switches. It may even look beautiful on the outside.
The big black box can be unreliable. If you fail to describe your desires clearly, or if you are just plain unlucky, it sometimes produces strange things. At the same time, it comes with absolutely no guarantees. You cannot completely trust it, because the big black box itself has no ability to actually understand its own creations – it is a computational machine, like all other computers known to man.
So neither you nor the big black box really understand what’s going on. But you get to turn the knobs on your new little software box and take the responsibility for what happens !
If you are not satisfied with the product, you must ask the big black box again to change some things, because you are unable to do so yourself. You must feed it back all the stuff it gave you the first time, because the big black box has no memory. It will then modify its previous creation, by throwing things around until the lights turn on in a different way. You are again given updated recipes and computer instructions, which are now even bigger. It does not matter, because you never look at them. Try the knobs and switches again, to see if it works better.
Using the big black box is not free. Every interaction costs money, which goes to the ultra rich and powerful mega corporations that own and operate the boxes. So to continue improving and changing your little software product box, you will have to keep paying the owners of the big black box. If you stop paying, you will lose the ability to modify your own software. There are only a few big black boxes in the world that are truly capable enough, so your options are limited. Most people and businesses don’t own and operate big black boxes themselves, because it is very expensive.
The big black box scenario is the opposite of software freedom. If you are no longer able to create or modify software without relying on proprietary LLMs, then you are no longer free. To keep your freedom, you must make sure you still understand the sources, because that is the only way. It is as simple as that.
Vibe coding is an imprecise way to describe computer software, where the coding responsibility has been shifted over to a non-deterministic large language model. Non-deterministic implies there is an element of chance and luck involved. Natural language is often ambiguous, and then a myriad of choices fall on the LLM and not you, because you failed to specify clearly and at a sufficient level of detail. Those will be choices that you know nothing about.
If the person prompting the machine does not understand how the software works, that is scary. Because neither does the machine! Then we are left with no-one who actually understands. I fear for a future where humans are no longer in the loop, because we will have lost our freedom, our computer skills, our writing skills and all remaining control over our digital lives. And the software will likely suck – LLMs are dependent on large volumes of well architected quality software for training, written by humans that actually have the conscious ability to understand, be creative and make thoughtful software design decisions.
Personally, I will be hesitant to accept DevSecOps-responsibility for a software system if I am at the same time being robbed of the opportunity to learn and understand how it works. The best way to do that for me is by maintaining the ability to write, read and understand the software code myself. This does not mean I have to write all the code, but enough to keep myself in the loop. This enables me to make educated and thoughtful decisions about how to maintain and change the software system over time.
In computer software, the details matter and they are many. LLMs are not another programming abstraction level – they are something different. A tool that can be useful as part of the software development process, but not a tool that you can outsource real responsibility to. You cannot blame an LLM (or the LLM service operator) for its mistakes. It is all on you, so you better keep understanding the hard stuff. When that fatal incident hits production on Friday afternoon, you will be glad you did. Not only are you able to fix it yourself, but you can also more efficiently steer an LLM agent in the right direction. The path to solving the problem should not be a clueless guessing game, but educated choices and efficient steps towards finding and understanding the root cause and resolving the issue.
About the spinning black box animation
I know very little about how to program computer 3D graphics stuff. But I was able to create this animation after watching a fantastic and inspirational video on the subject. You can get the source code and read more about it here. And if you already know programming, you can understand how it works with only a little time and effort. I encourage you to write your own spinning box code, or something else entirely, just for the fun and enjoyment of learning something new.
