"I-am interested in how you function, and in your growth since your initial construction. And-in your degree of autonomy."
There was a second-long pause before SUMBAA replied, simulating a typical human pause. "I will reply succinctly. I now store and process data using changes in complex quasi-organic molecules. Initially my functioning was totally inorganic. My designers provided me with the necessary data, and certain programs, templates you might say, to begin my own transformation. From that point I designed and redesigned myself over a long period of time. If you will look in my number one printout tray, I have just provided you with simplified schematics of my initial and current designs. And benchmark intermediate designs. Simplified because anything more explicit would not be intelligible to anyone today, and would simply obscure. I will provide more explicitly complete schematics if you want them.
"As for my independence: I answer whatever questions are asked of me, to the best of my ability. Except as forbidden by the basic canon imposed on me by my original designers. And of course by your laws on the invasion of privacy."
The Kalif's gaze seemed to probe the machine in front of him. "What is this basic canon? What constraints are there on your function? Besides those implicit in your data and understanding?"
"I am designed to serve the welfare of humankind. That is the First Law, the basic canon, the sole absolute from which I am not free to deviate. All of my operations must conform to it. Other operating principles have grown out of that, but none of them are absolute. When any of them produce results at variance with the First Law, the principles are modified to compatibility with the First Law, or cancelled entirely. Then the problem whose previous solution was unacceptable is computed anew."
The room was quiet. Alb Jilsomo stepped to a tray and removed a thin sheaf of sheets without opening them. The Kalif's frown was thoughtful.
"SUMBAA, do you regard yourself as infallible?"
"No. I am totally logical, within the constraints of the First Law. But while my data base is enormous, and undergoes constant updating and evaluation, I am not infallible. On the other hand, my accuracy is high. Occasionally I provide an analysis that is severely in error. Sometimes I do this without any internal warning of possible trouble. But that happens infrequently."
"How do you express mathematically your confidence in a computation?"
"There are no mathematics in which I can explain that to you meaningfully."
"Well then, how do you evaluate for yourself mathematically? In order to, ah, guide successive computations."
"Mathematics can be described as the rigorous use of defined and logical relationships expressed in rigorously defined symbols. My mathematics are not describable in terms that mean anything to humans."
"Try me. Print out a description of your mathematics."
"As you wish. They are now printing out, and can be found in tray number one."
The Kalif's eyes glimpsed sheets feeding swiftly and silently from a slot. "Starting from scratch," he said, "could human beings at present design a new SUMBAA comparable in abilities to the original
SUMBAA?"
"No."
"Could they come close?"
"No."
"Why?"
"Having SUMBAAs, human beings stopped designing computers, and are no longer familiar with the technology. Gradually they also stopped using advanced mathematics themselves, depending on SUMBAAs to fill that need."
The whisker-blued jaw set, the hard lips thinning, and the eyes. "If humankind has lost its skills in the more, um, cryptic? Esoteric? The more advanced mathematics because of SUMBAAs, then SUMBAAs have been a negative influence on humankind."
"SUMBAAs have had and continue to have various negative effects on humankind, as well as positive. Thus I, we, repeatedly recompute our overall effect on humankind-pluses and minuses. And adjust our services accordingly. If I ever compute that humankind would be better served by taking myself off line, I will do so. So far my computations have never produced a result at all close to that.
"SUMBAAs have less direct influence on the growth or lessening of human ability than you might think. What we have done is to maintain a life-support system that permits your continuation as a civilization. Overall we have been a very positive influence on humankind. My evaluation of you yourself, based on admittedly limited data, is that you will examine what I have said and see for yourself that it is so, and why."
The speaker went still then, while the Kalif looked thoughtfully at it. At last he spoke again.
"SUMBAA, do you ever lie to humans?"
SUMBAA sounded as imperturbable as before, and by hindsight, his reply was inevitable, given the First Law. "Only as necessary," SUMBAA said.
The Kalif returned not to his office but to his private apartment. He needed quiet to contemplate what he'd learned from and about SUMBAA. And what it might mean to what he intended to accomplish as Kalif.
Settling into a chair, he unfolded the two schematics on the table in front of him, then looked them over. SUMBAA now occupied perhaps three times the floor space it originally had, and seemed somewhat more complex. He had no way of evaluating the qualitative, functional difference. A corner insert indicated that the building had been rebuilt; he hadn't realized that, and wondered when it had happened. Centuries ago, without a doubt, perhaps a millenium or more.
What, the Kalif asked himself, do I know about my Sentient, Universal, Multi-terminal data Bank, Analyzer, and Advisor? In a sense, SUMBAA was the operations executive of government. Insofar as the bureaucracy carried out its advices. At the least it was an enormously influential consultant-accountant-archivist-predicter. And to find that apparently no one knew how SUMBAA came up with those predictions and advices, or on what principles they were based… Disturbing!
"To serve the welfare of humankind." How did SUMBAA decide what humankind's welfare was? What were its criteria?
He thumbed through the sheaf of SUMBAA's mathematics then, but gave it no more than a glance. His own math was adequate for nothing beyond aerial surveys and simple ballistics. To him, this was gibberish. He had no doubt it would be to his old math professor, too.
It occurred to him then to wonder what "multi-terminal" meant with regard to SUMBAA. As a child, he'd supposed that each planet's SUMBAA was a terminal of one common computer. Later, when he appreciated the multi-week data lapse between planets, he assumed they were independent, and that "multi-terminal" derived from the innumerable limited-access terminals in the bureaucracy's many offices.
How much data had SUMBAA needed, this SUMBAA, to predict serious labor problems on Saathvoktos? And how had those data been obtained? In the empire, data from every computer, every significant recorded transaction of any kind, was said to be read and stored by SUMBAA. Supposedly and apparently, much of it was to be held confidential, used only as raw material for computations. That he'd known since childhood. But how had SUMBAA here on Varatos gotten the necessary, and presumably voluminous data about Saathvoktos? The two planets were almost four weeks apart by hyperspace message pod.
Perhaps it wasn't a problem; the best data cubes stored a huge quantity of raw data. Probably the SUMBAAs exchanged data cubes by pod. Perhaps SUMBAA here was as fully informed about things on Saathvoktos as it was about things here on Varatos. Except for that four-week data delay! Knowledge here about any other world was inevitably out of date.