The stages of a scientific discovery shown in Figure 3.3 are seldom completed in sequence at the first attempt. There is usually repeated backtracking before each stage is completed — or rather, solved, for each stage may present a problem whose solution itself requires all five stages of a subsidiary problem-solving process. This applies even to stage 1, for the initiating problem itself is not immutable. If we cannot think of good candidate solutions we may return to stage 1 and try to reformulate the problem, or even choose a different problem. Indeed, apparent insolubility is only one of many reasons why we often find it desirable to modify problems we are solving. Some variants of a problem are inevitably more interesting, or more relevant to other problems; some are better formulated; some seem to be potentially more fruitful, or more urgent — or whatever. In many cases the issue of what precisely the problem is, and what the attributes of a ‘good’ explanation would be, receive as much criticism and conjecture as do trial solutions.
Similarly, if our criticisms at stage 3 fail to distinguish between rival theories, we try to invent new methods of criticism. If that does not seem to work we may backtrack to stage 2 and try to sharpen our proposed solutions (and existing theories) so as to get more explanations and predictions out of them and make it easier to find fault with them. Or we may again backtrack to stage 1 and try to find better criteria for the explanations to meet. And so on.
Not only is there constant backtracking, but the many sub-problems all remain simultaneously active and are addressed opportunistically. It is only when the discovery is complete that a fairly sequential argument, in a pattern something like Figure 3.3, can be presented. It can begin with the latest and best version of the problem; then it can show how some of the rejected theories fail criticism; then it can set out the winning theory, and say why it survives criticism; then it can explain how one copes without the superseded theory; and finally it can point out some of the new problems that this discovery creates or allows for.
While a problem is still in the process of being solved we are dealing with a large, heterogeneous set of ideas, theories, and criteria, with many variants of each, all competing for survival. There is a continual turnover of theories as they are altered or replaced by new ones. So all the theories are being subjected to variation and selection, according to criteria which are themselves subject to variation and selection. The whole process resembles biological evolution. A problem is like an ecological niche, and a theory is like a gene or a species which is being tested for viability in that niche. Variants of theories, like genetic mutations, are continually being created, and less successful variants become extinct when more successful variants take over. ‘Success’ is the ability to survive repeatedly under the selective pressures — criticism — brought to bear in that niche, and the criteria for that criticism depend partly on the physical characteristics of the niche and partly on the attributes of other genes and species (i.e. other ideas) that are already present there. The new world-view that may be implicit in a theory that solves a problem, and the distinctive features of a new species that takes over a niche, are emergent properties of the problem or niche. In other words, obtaining solutions is inherently complex. There is no simple way of discovering the true nature of planets, given (say) a critique of the celestial-sphere theory and some additional observations, just as there is no simple way of designing the DNA of a koala bear, given the properties of eucalyptus trees. Evolution, or trial and error — especially the focused, purposeful form of trial and error called scientific discovery — are the only ways.
For this reason, Popper has called his theory that knowledge can grow only by conjecture and refutation, in the manner of Figure 3.3, an evolutionary epistemology. This is an important unifying insight, and we shall see that there are other connections between these two strands. But I do not want to overstate the similarities between scientific discovery and biological evolution, for there are important differences too. One difference is that in biology variations (mutations) are random, blind and purposeless, while in human problem-solving the creation of new conjectures is itself a complex, knowledge-laden process driven by the intentions of the people concerned. Perhaps an even more important difference is that there is no biological equivalent of argument. All conjectures have to be tested experimentally, which is one reason why biological evolution is slower and less efficient by an astronomically large factor. Nevertheless, the link between the two sorts of process is far more than mere analogy: they are two of my four intimately related ‘main strands’ of explanation of the fabric of reality.
Both in science and in biological evolution, evolutionary success depends on the creation and survival of objective knowledge, which in biology is called adaptation. That is, the ability of a theory or gene to survive in a niche is not a haphazard function of its structure but depends on whether enough true and useful information about the niche is implicitly or explicitly encoded there. I shall say more about this in Chapter 8.
We can now begin to see what justifies the inferences that we draw from observations. We never draw inferences from observations alone, but observations can become significant in the course of an argument when they reveal deficiencies in some of the contending explanations. We choose a scientific theory because arguments, only a few of which depend on observations, have satisfied us (for the moment) that the explanations offered by all known rival theories are less true, less broad or less deep.
Take a moment to compare Figures 3.1 and 3.3. Look how different these two conceptions of the scientific process are. Inductivism is observation- and prediction-based, whereas in reality science is problem- and explanation-based. Inductivism supposes that theories are somehow extracted or distilled from observations, or are justified by them, whereas in fact theories begin as unjustified conjectures in someone’s mind, which typically precede the observations that rule out rival theories. Inductivism seeks to justify predictions as likely to hold in the future. Problem-solving justifies an explanation as being better than other explanations available in the present. Inductivism is a dangerous and recurring source of many sorts of error, because it is superficially so plausible. But it is not true.
When we succeed in solving a problem, scientific or otherwise, we end up with a set of theories which, though they are not problem-free, we find preferable to the theories we started with. What new attributes the new theories will have therefore depends on what we saw as the deficiencies in our original theories — that is, on what the problem was. Science is characterized by its problems as well as by its method. Astrologers who solve the problem of how to cast more intriguing horoscopes without risking being proved wrong are unlikely to have created much that deserves to be called scientific knowledge, even if they have used genuine scientific methods (such as market research) and are themselves quite satisfied with the solution. The problem in genuine science is always to understand some aspect of the fabric of reality, by finding explanations that are as broad and deep, and as true and specific, as possible.