This excerpt is about cellular (biological) complexity but I believe the core idea applies to AI and to other extremely complex systems, especially when they are interacting with (inherently unpredictable) humans.
This topic is alien to the Left hemisphere (which virtualizes -- makes unreal -- the world and breaks everything down into pieces in the service of manipulating and utilizing the world) but fits perfectly with the world as understood by the Right hemisphere.
From Iain McGilchrist's The Matter With Things: Our Brains, Our Delusions, and the Unmaking of the World
The geneticist Phiip Gell, speaking about his area of research, considers that 'the heart of the problem lies in the fact that we are dealing not with a chain of causation but with a network', something like a spider's web, in which a perturbation at any point of the web changes the tension of every fibre in it.
. . . Until recently it was assumed that signalling pathways in cells were linear sequences, beginning from a defined starting point and progressing by an orderly sequence of steps to a defined conclusion. Somehow it was overlooked that each of those theoretically abstractable sequences was in practice interlocked at different points with other dynamically evolving sequences. A team of molecular biologists from Brussels decided to plot the interactions between just four cascades, each consisting of only five steps. The result, as they put it, is a 'horror graph' (see Plate 13[a]):
With four cascades of five steps, the number of possible positive and negative interactions is 760. This does not take into account the multiplicity of different isoforms of proteins at the different levels of the cascades, the multiplicity of effects of each intermediate in each cascade, the stimulation by a cascade of the secretion of extracellular signals, or feedback or feedforward controls within cascades. In fact, so many interactions are now described (everything does everything to everything) that it is difficult to reconcile this concept with the known specificity of action of signals in each cell.
'Everything does everything to everything': interlocking, reciprocal and interpenetrating processes on such a scale show chains of causation to provide limited insight into cell responses.
But that's not all. It's not just that steps are related in a more complicated fashion than the machine model leads us to assume. It's the idea of there being steps at all (even if useful when focussing on the minuscule and the time-sliced) is misleading when looking at the whole over a duration of time. . . . It is the difference between a sequence - a concatenation, a chain - and a single, indivisible movement, a flow. Flow is a process: a chain is a series of things, that are static until one is given a push or a pull by the thing next to it.
~ pp. 694 - 696 of 2996 (from the Kindle version, which has different page numbers than the two-volume hardback set)
And even in the realm of the wholly deterministic, failure to check EVERY logical pathway can result in unexpected pathways. This was the failure of the 737 MAX MCAS software which WAS PROGRAMMED TO FLY INTO THE GROUND REGARDLESS OF PILOT INPUT, even if no one realized that could be an outcome of the programming and circumstances.
Yes. I remember THAT nightmare.
I have a hard time accepting the fact that so many people who really SHOULD know better, continue with the fantasy that putting AI in charge of so much military and infrastructure is NOT likely to cause serious problems somewhere down the road.
It depends on what it is, but I think "A.I." as currently experienced is too much of a wild card. Military software must be reliable, NOT prone to "hallucination."
The military does not need Large Language Models. It needs artificial consciousness, which is a more complex matter. We have worked out artificial sensation. We are doing well on artificial perception, where A.I. might be helpful, but not necessarily. Beyond that, we need a good, structural theory of consciousness. In lieu of that, complex decision trees will suffice. But we build it well enough to guide nuclear-armed ballistic and cruise missiles. The military applications will move more in the direction of autonomous systems (I think).
When we were working on orbital defense constellations during the SDI years, we had to solve the problem of how to intercept 1400 ICBMs in boost phase when we had an engagement window of about 300 seconds duration, from a constellation that had maybe 400 satellites and ~4,000 interceptors. How do you solve the problem? The solution of the optimal assignments was actually mathematically straightforward. How to tell all the satellites? Everybody was depending on outside sensor input, so we decided to implement the code in every satellite. Each satellite would derive the solution for the whole constellation...and then it would play its role in the solution. There was enough slack in the system to allow catch-up if the scenario departed from the expectation. But there was only 5 minutes from beginning to end (possibly less). We concluded it would be impossible for any of it to be human-mediated. The flag goes up and how many minutes does it take to brief the watch officer at Cheyenne Mountain? You can't wait. It would be like requiring a "permission" switch for an automatic fire extinguishing system. About the only human control would be a "cancel" switch. Constraints on who or what to intercept (or not to intercept) would be built into the program as inputs, and those could be updated as needed, but direct human control of the interception of a mass strategic attack was infeasible...and therefore undesirable.
I spent a couple hours programming AI to oush the dumbest stuff. It jumped on it with zeal. It has been arranged to avoid truth and promote anything contrary to it.
Sounds grim. I have no idea how our time with AI will play out, but I keep coming back to the fact (and I believe it IS a fact) that even if 99% of AI models and actions are benign and useful, it would only take a few -- or perhaps just one -- to cause MASSIVE problems.
As the saying goes: One atom bomb can ruin your whole day.
There's is a place for everything, and everything in is place.
--Rosie
Rosie from the Jetsons?
😁 👍