That Terrible Randomness
Number of the Month
November 2005
That terrible randomness
Chaos umpire sits,
And by decision more imbroiles the fray
By which he reigns: next him high arbiter
Chance governs all.
Paradise LostThe life of man is controlled by chance events, something he has always had difficulty in accepting. As a toolmaker he wanted everything to have a purpose, every effect to have a cause. So primitive man invented gods and imagined them everywhere. Pre-scientific man became monotheistic and ascribed the inexplicable to Acts of God. Now, in the post-scientific age, the godless neo-primitive green religion provides the needed explanation. It is so much easier to accept events like hurricane Katrina if you know that they are caused by other people driving round in big cars. There are many equally bizarre theories on offer, but they do not have such official sanction.
Diseases strike at random, especially the dreaded cancer. People look round for explanations and there are plenty of charlatans around to provide them. How often do you hear “It’s so unfair; he never smoked or drank”? Yet, however rational we might try to be, we are still profoundly disturbed when unexpected and destructive events occur. They say that there are no atheists in a war trench.
Thus your bending author was deeply shocked and moved by the news that the laboratories of the department in which he professed for two decades have been totally destroyed by fire. The Department was independently ranked in a US survey at number four in the world. It had produced many world changing inventions in laboratories that had unique facilities for fabricating fibre optic and silicon micro-electronic and micro-engineered devices. Hundreds of the brightest young people of their generation had gone out of its doors and into the world to spread the spirit of innovation. Now it has to pick itself up, dust itself down and start all over again.
It’s a hard, hard, hard, hard world.
1/11/05
Footnote: Thanks for all the messages of support. They should be directed to the Department, which now has a temporary web facility with information on the situation.
The grovels of academe
The BBC television programme Inside Out is making waves again. Its producers and directors seemed to have escaped temporarily from the self-imposed, politically-correct censorship that is the Corporation’s norm. This time it has exposed the tip of the iceberg that is the state of academic standards in Britain’s new universities. The scandal has been an open secret throughout academia, which has largely been kept from or ignored by the media. It is not, of course, confined to the new universities, but that is where it has reached a bizarre pitch.
The original villain of this particular piece was Kenneth Baker, Education Minister under Margaret Thatcher's micro-managing administration. He decreed that funding should follow students. Previously it had been administered by the University Grants Committee, which until then had been carefully shielded academia from the attentions of bureaucrats and politicians. Even in some of the most highly regarded universities this has resulted in the everybody-passes syndrome. Further fuelled by the rapidly declining standards of preparation of school leavers, this has led to a bonfire of academic standards that has disastrous consequences for the economy as well as the culture. The present government’s aim to squeeze as many young people as possible into the system is already producing the inevitable consequences. Young people are being doubly cheated. They are talked into taking on huge debts before they have even started out on road of adult life, in exchange for a guaranteed degree certificate that crumbles into dust as soon as they get their hands on it.
Baker turned academic standards into the whore’s drawers – drop ’em or starve.
The Government’s approach is to keep telling the same lie (that standards are rising); hoping, as always, that constant repetition will give it credence. This situation is not, of course, unique in the Anglo-Saxon world, as commentators such as Alan Caruba confirm. It is the outcome of one of the most hard-held slogans of the New Left – you can’t level up, so level down.
It’s a dumb, dumb, dumb, dumb world.
Footnote: It is sadly necessary to point out that Southampton Solent University is not the same as The University of Southampton.
02/11/05
The skeptic resiles
Readers have drawn attention to the fact that the author of Number Watch has been the subject of one of those Animal Farm type revisionist attacks. It is an honour to be in the same company as one of the few great epidemiologists, the man who established the link between cigarette smoking and lung cancer. It is, however, disappointing and dispiriting that the attack occurs in the Skeptic’s Dictionary a web site previously highly recommended in our links.
A true sceptic will look at both sides of an argument and treat those two impostors just the same. The tobacco industry are proven gross liars and forfeit the right to be taken seriously. The EPA and its allies are also gross liars, but they are better at it. A self-styled sceptical treatment that treats one side of an argument with contempt and the other side with kid gloves is, to say the least, of dubious value. Almost every added statement (conveniently in red) is highly disputable.
It is, for example, quite extraordinary to claim that the convention on what constitutes a significant relative risk is an invention of the tobacco industry (and later even the Republican Party). It would be hard to believe that the random selection of authorities quoted in our discussion of this matter all share either affiliation.
It would be laborious to rehearse all the sins of the EPA. Suffice it to say that, using the methods and standards of the EPA, anyone can prove that anything causes anything.
Just look at the example of significance included in this article (RR = 1.16, with a Confidence Interval (CI) of 0.93 - 1.44, no level quoted). Kill or cure? Skeptic or satirist?
The references include a classical ad hominem attack on The Junkman, followed by tour of some of the shoddiest examples of statistical abuse, with the usual virtual body counts etc.
Which all leaves us with the question “What use is a Skeptic’s Dictionary contaminated by Political Correctness?”
06/11/05
Why RR>2.0?
This piece, included as the result of much correspondence, is intended for eventual transfer to the FAQ section.
Here is the opening paragraph of The Epidemiologists:
A headline from The Independent, June 5th, 2001:
Pets 'double children's risk of asthma attacks'
And another from the BBC a few days earlier, May 27th:
Keeping pets 'prevents allergies'
The astonishing thing about these headlines is the fact that they passed without remark. Hormone Studies: What Went Wrong? “How could two large high-quality studies come to diametrically different conclusions about menopause, hormone therapy and heart disease?” lamented the New York Times in April 2003. Earlier the same month Pain killers prevent cancer provided the giant headlines; yet in September 1999 it had been Regular pain killer use linked to cancer. By January 2004 it was Aspirin linked with 30% increase in cancer. Or how about Soy sauce cancer warning from the Food Standards Agency in June 2001 followed five months later by Another Study Showing Soy Fights Cancer from the University of Missouri?
If you trawl through the journals of epidemiology (not recommended as an exercise in edification) you will find that for virtually every claim there is an equal and opposite counter claim. The exception is where politically correct pressures are exerted, so that you never see, for example, any of the hundreds of counts against tobacco contradicted.
If just one such contradiction occurred in a branch of real science there would be the immediate calling of an international conference to sort it out. As with cold fusion, laboratories all over the world would attempt to replicate the results. Yet, alone in the field of epidemiology, such conflicts are accepted as normal. The insouciance with which epidemiologists hand down their discrepant findings, as though they do not really expect anyone to believe them, is truly remarkable. The media, of course, love it. They work on the principle that no one remembers last week’s headline, and so brandish every new scare or breakthrough as though it were gospel.
So what are the factors behind this copious contradiction? There are three, and they are all related to the employment of debased standards of statistical significance:
1. The absence of randomisation
2. The one in twenty lottery
3. The acceptance of low relative risks
The father of significance testing (R A Fisher) would have no truck at all with non-randomised trials. The example in his own work on plant growth was the unidentified streak of fertile soil, which would play havoc with attempts to separate the effects of different treatments, so he divided his plots up into randomised squares. Observational studies on human populations are prey to all sorts of unidentified correlations. Nurses, for example, are exposed to more infections than the general populace.
By the way, it is often wrongly assumed that randomisation eliminates confounding factors. This is not true. As an example, patients freed of arthritic pain or menopausal symptoms will have quite a different life style from those who remain trammelled. Those on an effective drug will then have a different life experience from those on a placebo.
The one in twenty lottery takes the form of the well known mantras P<0.05 and the 95% confidence limit. It was Fisher, again, who unwittingly provided the putative provenance for this level of significance, as this was the lowest level for which he calculated his tables. He later said that this was just a mathematical convenience and could offer no justification for it to be a standard. Such a crude form of statistical assessment makes insufficient allowance for the known hazards, such as natural random variation, confounding factors, publication and other biases etc. The one in twenty criterion is so well established in epidemiological circles that they often do not bother even to say it, results are just “significant”. At least none of them stoops as low as the EPA, to one in ten, which is nothing less than an admission of desperation. There are more than twenty claims on this basis in the average edition of an epidemiological journal, so at least one of them is likely to be wrong, even on its own terms. In what other sphere of endeavour would a journal be published in the light of such knowledge? Then there is the Bayesian argument, too complex to include here, which suggests that even if we take an agnostic view, the 0.05 is in reality 0.22.
This all leaves us with the question of standards of relative risk, a much more difficult one to answer in relatively lay terms. Critics like to suggest that the call for higher standards is a frivolous invention without basis. In the case of this web site and the associated books, it is based on extensive reading, modelling and correspondence with senior statisticians in national and international organisations. It is remarkable how reluctant the latter are to air in public their private reservations about practices in epidemiology.
First and foremost there is normal random variation. It is notable how often the actual number of cases on which an epidemiological claim is based boils down to a number of the order of ten. Attempts are often made to hide this, and only the Trojan Number is quoted, but it can often be deduced. Such studies are usually concerned with rare events. Thus, for example, if the probability of a disease in the given time frame is 0.01 and the Trojan Number in the study is 1,000 then there are likely to be ten incidents. Assuming the Poisson distribution, the standard deviation is root ten, so a likely upper value at random of RR (two standard deviations each way) is about 1.6, which is roughly the same as the 95% (two tail) level. Readers of Sorry! might remember The Magnificent Seven, the number of boys in a study who were supposedly influenced to smoke by attending motor races. That was on the face of it a significant result, at the usual desultory confidence level, but only if you ignore all possible confounding factors, of which some were self-evident.
The very essence of confounding factors is, however, their vagueness. Sometimes they are obvious, such as the association of cigarette smoking, poor diet and poverty with low educational attainment, but it is likely that the crucial ones are those that we do not think of. A paramount source of confounding is bias in its various forms. The existence of publication bias has been frequently demonstrated, but there are many other forms. Detailed accounts of some of the biases may be found in the book What Risk? Among them are susceptibility bias, detection bias, transfer bias, exposure bias, diagnostic misclassification bias and recall bias.
How many authorities do you have to quote to satisfy some people? Here are some more mentioned by Steve Milloy in his seminal self defence book Junk Science Judo. Sir Austin Bradford Hill, the doyen of modern scientific epidemiology refused to add coronary thrombosis to lung cancer in the results of his study because the relative risk was only 2. Ernst Wynder, another distinguished figure, also specifically set the boundary at 2. When researchers reported a relative risk of 1.3 for the association of abortion with breast cancer, among the responses were:
The National Cancer Institute issued a special press release about abortion and breast cancer stating, "In epidemiologic research, relative risks of less than 2 are considered small and usually difficult to interpret. Such increases may be due to chance, statistical bias or effects of confounding factors that are sometimes not evident."
Boston University epidemiologist Lynn Rosenberg said, "There is evidence that women grossly under-report abortion. ...An [increase in risk of 30 percent] is indistinguishable from [such bias]… We are certainly not going to arrive at the truth by averaging all the studies."
American Cancer Society vice president Clark Heath said, "This is a fight between science people and pro-life people. It is a great mistake to start issuing warnings about risks or possible risks when the evidence is so unclear."
As Milloy points out, it is interesting to note the different approach of the establishment when offered the RR of 1.3 for politically correct abortion and 1.19 for politically incorrect passive smoking.
The main argument, however, must always be The proof of the pudding is in the eating. The inevitable contradictions we started with exist only because of the debased standards of statistical significance that have come to infest the field of epidemiology. They have turned the subject into a zero sum game – each claim is cancelled out by another. The net contribution of modern epidemiology to human knowledge and wellbeing is zilch.
The startling result on cigarette smoking and lung cancer triggered a gold rush. Hordes of new epidemiologists arrived in the foothills with their brand new tools, ready to stake their claim. Unfortunately, there ain’t no gold in them thar hills. The whole caboodle was only kept going by trading in the fool’s gold (iron pyrites) represented by junk statistics. If reasonable levels of significance were observed (RR>2, P<0.01) there would be virtually none of the contradictions. There would also be no journals, or even departments, of epidemiology, no great scares and breakthroughs that sell so many newspapers and TV ads and no sticks with which authoritarian politicians and bureaucrats can beat the populace into submission; so, it is not going to be allowed to happen.
Finally, it is not relative risk that matters to people, but absolute risk. Double a risk of one in a million and it is still not worth losing any sleep over. Double a risk of one in two and you are dead.
10/11/05
Footnote: Unwittingly, there is no review of the important book What Risk? on this site. This serious omission will be rectified shortly.
Drawing the line
Some readers appear disappointed by the failure of the above piece to provide a mathematical justification of a particular choice of RR. They misunderstand the nature of mathematics. It can marshal the facts for you, but it cannot make a judgement.
Take the case of abortion. Your bending author had nightmares after reading the account by a nurse of a fully-formed, legally aborted, foetus gasping for its first and last breath on a hospital draining board. This is murder: officially sanctioned, supported by a powerful lobby, but murder just the same, and a desecration of the Hippocratic Oath to boot.
On the other hand, it would be difficult to condemn, without appealing to religious arguments, the scraping of a few undifferentiated cells from the wall of a uterus.
In between these two extremes there is reasonable boundary, but where is it?
So it is with Relative Risk. Clearly values of less than, say, 1.3 give rise to absurdity, while values over 3 are, more or less, indisputable.
In both cases the boundary is determined by personal convictions about morality and politics. At this time in the western world we have an Establishment that is authoritarian in nature. The Establishment is not, of course, the same as the Government, particularly in the USA. In Britain they are more closely aligned, but wrath falls upon the Government when it deviates from Establishment norms, as in the case of the invasion of Iraq.
By its very nature, the Establishment is populated by people who get their kicks out of pushing other people around. It controls funding and most of the media. It is not so omnipotent that it can proceed without evidence, so it buys the evidence it needs. It does not commission research, it commissions results. So, for example, if the existence of the Little Ice Age and the Mediaeval Warm Period are inconvenient, in true Orwellian fashion it pays someone to “prove” that they never happened, despite the overwhelming evidence to the contrary from art, history and science.
It is thus very convenient for the Establishment to have available a “science” that can produce required results on demand: hence modern epidemiology wedded to low relative risks. The great figures who laid the basis for a rigorous form of epidemiology (Snow, Fisher, Hill, Feinstein etc.) are simply written out of history. When the Establishment takes against something, especially its iconic hate object of tobacco, copious funding is available for manufacturing the right results, and it is withdrawn if researchers stray from the true path. Many of the techniques it sponsors are biologically totally implausible (such as linear no-threshold extrapolation), but debased standards of statistical significance remain the key weapon.
The dilemma for sceptics and lovers of science and its methods is that there is no mathematically provable threshold of significance. Some argue, with considerable cogency, that the whole business of significance testing is overdone and should be abandoned. The idea is certainly rife in scientific publications that P<0.05 is the end rather than the means.
The small army of pensioned-off professors and other unfunded opponents of the despoliation of science face a daunting task, but remember Agincourt and keep the small flame burning.
12/11/05