Lack of Theoretical Foundation Leaves Human-Built Systems Without Insight
| Source: Mastodon | Original article
Human-built and AI-generated systems risk losing core understanding. Code survives, but context is lost.
Recent developments in AI and human-built systems have highlighted a concerning trend: the loss of understanding and context in code development. As we reported on May 24, the distinction between fine-tuning and using Retrieval-Augmented Generation (RAG) can be unclear, leading to confusion among developers. This issue is further complicated by the fact that human-built systems often lose their underlying rationale and context when their creators leave, while AI-generated systems may never develop this understanding in the first place.
This phenomenon violates Peter Naur's view of programming, which emphasizes the importance of a mental model and context in code development. The implications are significant, as code without a underlying theory or understanding can lead to suboptimal decision-making and outcomes. This is evident in the contradictions between prospect theory and the theory of expected utility, which can result in choices that do not maximize utility.
As researchers and developers move forward, it will be essential to prioritize the integration of corrigibility and human oversight into AI decision-making processes. This may involve exploring new approaches, such as Functional Decision Theory, to ensure that AI systems are aligned with human values and goals. By addressing the issue of code without theory, we can work towards creating more transparent, accountable, and effective AI systems.
Sources
Back to AIPULSEN