Tom Gold, a scientist at Cornell University, believes that bacteria are living under our feet in staggering quantities–not in our shoes, although they are certainly there too, but deep in the crust of the earth. They are known to thrive in acid, on hot coals, and in deep-sea vents at 480 degrees Fahrenheit, so he figures that wherever water can seep into microscopic cracks in the rock, there you will find some hardy bacterium making do with just a few chemicals and the heat emanating from the center of the earth. There must be so many of these creatures, the professor has reckoned, that if we dug them up and spread them out they’d form a mat five feet thick covering every square inch of land on the planet.

Forget the fate of dogs, small children, and those who happen to be sitting down during this experiment: who will now serve as spokesman–or spokes-zoon, to be more biologically correct–for this bacterial mat? It is none other than redoubtable Stephen Jay Gould, the Harvard paleontologist and prolific author of popular science. He pops his head out of the muck, shakes himself off, and immediately begins his lecture to the surviving bipeds: “See how many of these little devils there really are? By any objective standard–weight, numbers, variety, persistence–they are the true masters of life on earth. There are more bacteria swimming inside each one of your bellies than all the man-footed beasts who have ever been born. This is the Age of Bacteria: so it was 3.8 billion years ago, and so it shall ever be. We flit into a single instant of this immense span of time and then assume, like so many late arrivals with bad manners, that the whole party has been waiting just for us. People! The idea that life moves inevitably onward and upward is nothing more than a secretion of your swollen mammalo-centric head. Evolution goes here, it goes there, but in the end it goes nowhere. Do you doubt me? Look on its works, ye Mighty, and despair!”

I paraphrase, but that’s the gist of Gould’s new book, Full House: The Spread of Excellence From Plato to Darwin. He doesn’t intend it as a counsel of despair. Anyone who knows Gould’s work knows how much he delights in evolutionary chance, both for the endless stream of curiosities it creates and for the “freedom and consequent moral responsibility” it bestows on its most curious creation of all. In Full House Gould has grasped the biggest nettle he can find–“the problem of progress in the history of life”–and makes his strongest claim yet for how haphazard evolution really is. The book has a philosophical tinge, but its author is no sage meditating on an ageless puzzle, turning it this way and that. He’s a scientist who wants answers, and with the passion that is the great virtue of his style he attacks the idea of progress with all arms.

Evolution, Gould tells us, is not about creating something better; it simply fits species to particular circumstances, a task that is utterly neutral toward anything we might call progress. The tapeworm, it must be admitted, is as well suited to his business as is the book reviewer to his. Survival is the only standard nature offers, but this is too crude for soi-disant Homo sapiens. It is an insult to his pride. So he surveys the aeons and picks out a few groups that excite his admiration–the trilobites, fishes, amphibians, reptiles, and mammals we learned about in school–and lines them up in a great parade of progress with his own estimable self in the vanguard.

The image is gratifying, but Gould reports that it’s as false as the creation stories that, in some minds, it has come to replace. In reality there is no vanguard, and no parade either; it’s more like a mob of endlessly variable forms, varying endlessly. They leave descendants, they stagnate, and eventually they go extinct (except those damn bacteria!), all with complete disregard for what might seem “better” to the vanity of the future. It’s true that as species filled up the available places a few organisms became more efficient or complex or intelligent or whatever other vague and self-serving adjective we may choose, but that’s only because life began, as it had to, at the simplest level. (“You cannot begin by precipitating a lion out of the primeval soup.”) Any subsequent change, though perfectly random, had nowhere to go but up; or more exactly, nowhere but away from simplicity. And that’s all, according to Gould, that can be said for so-called progress.

Why, you may wonder, does an eminent scientist write a book to prove that his own kind is so much statistical noise in the evolution of life? What moves him to hail the “long and illustrious history” of bacteria while he derides that of man as a mere “dribble”? He isn’t joking, that’s for sure. Boosting bacteria is serious business for Gould, who’s made debunking scientific truths into his own special vocation. His purpose is not merely to reveal error but to expose its roots in personal and cultural bias–the only soil, he is always careful to add, that can yield any intuition in science, right or wrong. He’s probably best known for showing how a rank prejudice against the mental capacity of certain groups lay behind the false claims made for IQ tests, claims that eventually acquired a life of their own. The prejudices Gould wants to expose in Full House are much deeper than that: the pride and insecurity that go with being human, and the complacent faith that progress has given us this, the best of all possible worlds.

When we were made in the image of God we knew where we stood; then we got demoted by Charles Darwin and found ourselves just another animal, forced to fight it out with all kinds of brutes over unimaginable stretches of time. To repair the damage, Gould suggests, people invented evolutionary progress: if we haven’t dominated all history, then let us at least claim surrogates in the leading citizens of each age, the ones who worked hard, had big families, and so eventually transformed themselves into us. The idea of progress, as Gould presents it, is a kind of substitute religion: it discovers a noble essence in what appears to be base, and it makes sense out of life by showing that history has a purpose. And, like any religion, the belief that man is the goal of evolution is easily defeated by science on purely rational grounds.

This rather undemanding task is the main object of Full House. Once Gould defines the enemy as “our yearning for general progress–that is, the predictable and sensible evolution to dominance of a creature like us, endowed with consciousness,” the outcome is hardly in doubt. For what he’s describing is nothing but outright finalism, the idea that the end explains the beginning and the middle. Karl Marx, the greatest theologian of finality in modern times, made the case for end causes in biology like this: “Human anatomy contains a key to the anatomy of the ape.” This is indeed a strange way to look at evolution, but I wonder how many people today still think that the future can explain the past. They would have to believe, like the coauthor of The Communist Manifesto, that progress is intrinsic to history and that its results are as predictable as the motions of the planets. We can foretell where Jupiter is going to be at any moment millions of years from now, but is there anyone left who cares to predict the growth of our brainpans or the withering away of our little toes–not to mention the coming of the workers’ paradise?

There is a real scientific issue here, of course, a version of the old contest between chance and necessity: how much of the history of life is directed in some way, and how much is random? In Full House this question is presented as a stark choice between finalism on one side and absolute randomness on the other. Natural selection, which Darwin proposed as the directing element in evolution, seems to avoid these extremes by molding new types out of random mutations. But to Gould, selection is just “a negative force that can make nothing by itself,” a janitor that empties the trash and sweeps away the unfit after the real work–the creation of new varieties by chance–has been done.

This low view of natural selection is nothing new for Gould. In the 1970s he first made a name for himself in paleontology as cooriginator of the theory of punctuated equilibrium, which says that species don’t really change much except during isolated instants when new forms branch off abruptly. (The theory was also known, for reasons having nothing to do with the boldness of the theorists, as “evolution by jerks.”) Biology is irresistible as an arena for scoring political points by analogy, so Gould’s punctuations were seized upon, for better or worse, as proof of a more radical view of evolution; but in fact the instant of change is geological, big enough for both insurgents anxious to man the barricades and traditionalists content to work within the Darwinian system. What is more telling about the theory are the long governing periods of stability: during all this time, many millions of years, natural selection is relegated to pushing its humble broom.

While this fight was going on inside paleontology, Gould was also taking part in a more general assault on adaptation, the very citadel of Darwinism. Pushed to the limit, adaptation seems to imply that every part of every living thing exists for a reason; if the reason is obscure, it isn’t nature’s ingenuity that’s lacking but our own. To Gould and his allies this was at best naive optimism and at worst, in human sociobiology, a covert defense of the status quo as natural, inevitable, and correct. During the sociobiologists’ heyday in the mid-70s they reported adaptive genes for a grab bag of old favorites like aggression in men, submission in women, racial prejudice, conformity, spite, and fear of snakes. By a curious logic, even homosexuality received the blessing of natural selection. But while these discoveries were being warmly applauded by the public, a Boston-area collective called Science for the People (with comrade “Steve Gould” in its Sociobiology Study Group) denounced them in a series of polemics bitter even by academic standards.

It wasn’t only politics that raised the temperature of these debates. The basic issue, as Gould put it then, was biological potentiality versus biological determinism, and that issue hasn’t changed in 20 years. From the early days of punctuated equilibrium right through his recent attacks on The Bell Curve, Gould has been fighting determinism in one form or another. Full House is another strike against this enemy, but now the assault is global. What’s at stake is larger than the import of natural selection or the genetic basis of behavior or the nature of intelligence; Gould’s subject is really life itself in the largest possible sense–I mean the largest scientific sense. As a whole, he is saying, life is not bound in any direction. Its deepest essence is random, so it has no story to tell, no theme. Its history signifies nothing. Above all, life does not incline toward “the good,” whatever that may be.

This might seem like a bitter pill to swallow, but Gould also has his spoonful of sugar–“the spread of excellence.” If we can just stop worrying about which beasts have more brains, he says, we’ll discover that “it is, indeed, a wonderful life within the full house of our planet’s history of organic diversity.” Variety, not superiority, is what really counts, and not just in evolution. Variation is Gould’s master principle, “the primary expression of natural reality.” Here is the very stuff of the world, in other words, but we can’t see it because we’re so boggled by “the best.” To get this point across Gould spends about a quarter of his book explaining why nobody hits .400 in the major leagues anymore. The best batters haven’t gotten worse, he says; in fact the general level of play in baseball is better than ever, as in every sport. But because the rules are adjusted to maintain a balance between pitching and hitting (roughly a .260 average overall), more raw skill with the bat is now required to hit at any given percentage. A .400 season was always an exceptional feat; today it lies so close to “the wall,” the absolute limit of human performance, that Ty Cobb himself probably couldn’t do it. Thus the decline at the top of the sport is only apparent: it’s really a sign of the spread of excellence within the whole.

This solution is ingenious and convincing, but I don’t see what it has to do with Gould’s argument against progress in evolution. He can’t be saying that while the advent of mankind is meaningless, excellence has somehow been spreading through the whole batting order of biology. That would just let progress in through the back door. Gould acknowledges only one real trend in evolution: over time, more niches have been filled by a greater variety of organisms. They’re not necessarily better organisms, but they are more diverse. So here we come to a peculiar twist of logic: what was “the spread of excellence” in baseball becomes in evolution something more like the excellence of spread.

Diversity isn’t just a remarkable feature of life to Gould, it’s the only standard he allows for evolutionary success. The genus Homo is a lonely twig, but “when groups are truly successful…their tree contains numerous branches, all prospering at once.” This is not only a lot friendlier than the old nature red in tooth and claw, it makes rodents the most successful of all mammals. Among animals as a whole, Gould is delighted to report, insects rule. And if you want to look at the big picture, you’re back staring at all those bacteria. I’m not sure what Gould proves by totting up the numbers of different species to declare his winners, but he has few doubts himself. It is “absurd,” he says, a “ludicrous case of the tail wagging the dog,” to see progress in evolution just because a few creatures have lifted themselves up and become more complex and intelligent than the mass of germs, bugs, and rats.

Obviously a contest like this can be rigged any way you like, so who (or what) takes first prize will have less to do with evolution than with the logic and taste of the judges. Certainly those in the audience who hold diversity as a supreme value will be heartened to read that “the Full House model does teach us to treasure variety for its own sake.” Others may find that Gould is merely lending the weight of science to a shift in the meaning of “diversity” that is already well established in the American language at large. It’s like what happened to “ecology” a generation ago, when a neutral term for the interplay of organic and inorganic nature became a rallying point for political action–something you could be for, if not quite against. “Diversity” seems to have achieved its apple-pie status even faster. Full House can be read as an intricate and learned defense of this retooled “diversity,” now imported back into science, where it began.

Gould is aware of this reading and makes a point of disclaiming it at the end: “I hate to think that an intellectual position, hopefully well worked out in the pages of this book, might end up as a shill for one of the great fuzzinesses of our age–so-called ‘political correctness.'” In that case he ought to have told his publishers, who announce on the jacket of Full House that it offers “a new paradigm of progress in which variety–not complexity–is the true measure of excellence.” But they can hardly be blamed, since at the end of his book the author himself boils it down to the following, stated as a general truth: “Excellence is a range of differences, not a spot.” (I’m sure it’s just a coincidence, but this pronouncement is remarkably close to a phrase that surfaced briefly a few years ago at the University of California, Berkeley, until recently one of the major proving grounds of our affirmative action program out west. With a subtle nod to Orwell, they affirmed actively that “Diversity is Excellence.”)

“I am a strong advocate of the general argument that ‘truth’ as preached by scientists often turns out to be no more than prejudice inspired by prevailing social and political beliefs.” So Gould wrote 20 years ago, and in the present book he provides a good example of what he had in mind: Charles Darwin, who wrote in his great work that “as natural selection works solely by and for the good of each being, all corporeal and mental endowments will tend to progress towards perfection.” How, Gould wonders, could this most rational and far-seeing man fall into such a trap? His answer is that while Darwin was a scientist, he was also “a patrician Englishman, at the very apex of his nation’s thundering success.” In Victorian England, with an economic engine humming at home and a huge and beneficent empire abroad, the air itself must have smelled like progress; for a well-to-do country squire like Darwin there was nothing more natural than to crystallize his view of the world around a kernel of this optimism. Gould believes that one part of the man (“the intellectual radical”) understood that his theory implied the utter randomness of evolution, but another part (“the cultural conservative”) would simply not allow him to follow through. “Darwin,” he says, “could not bear to fail his own world.”

Everyone is a product of his time, true enough, but it’s unbecoming for Gould to present Darwin’s argument for progress as though he had merely “concocted a tenuous resolution” of some inner conflict. In a nutshell, here is what Darwin said: natural selection develops ever finer differences between species, and as they diversify in this way, plants and animals get better at filling the slots available in what he called “the general economy of any land.” They are able to find new places not yet occupied, and because as a group they’re “more perfectly diversified in structure,” they also beat out those holding the current positions. (Darwin’s “diversification” describes concretely how progress happens; Gould’s more abstract “diversity” argues that it doesn’t because each form is simply suited to its own time and place.)

This is why Darwin believed that the mammals of Europe and North America were superior to those of Australia. He saw the marsupials’ coarser division of labor as the mark of “an early and incomplete stage of development.” Obviously a match between these two groups must remain imaginary, much like the contests fought routinely in bars between, say, the 1996 Yankees and those of 1927. But I think I can guess where Gould would come down on that one, and for reasons not very different from Darwin’s.

All we can say for sure about scientific ideas is that they reflect, in a general way, the different worlds that create them. Gould is certainly right to describe Darwin as a thinker with a deep affinity for his society. He’s also right to ignore the usual crude analogies between the “struggle for existence” and the ruthlessness of Victorian capitalism, focusing instead on Darwin’s Victorian optimism. It’s impossible to read Darwin and not be struck by his belief–nowhere stated directly, but everywhere coloring the whole–that with enough time and fidelity to certain principles things will slowly, but surely, improve. For his part Gould is never anything but optimistic on the printed page, if not downright ebullient. Even so, it’s hard not to feel some despair when one of the best known scientists in the country starts telling us with all his considerable passion that there is no such thing as progress in evolution, nor could there ever be.

Full House: The Spread of Excellence From Plato to Darwin by Steven Jay Gould, Harmony Books, $25.

Art accompanying story in printed newspaper (not available in this archive): illustration by Russ Ando.