Minutus

by Kate Chatfield (University of Central Lancashire) and Clare Relton.
ScHARR, University of Sheffield, posted with permission.
_______________________________________
Are the clinical effects of homeopathy placebo effects? - A full
critique of the article by Sahng et al (Lancet 2005;366 (9487):726-733)
The article by Shang et al (2005) can be critiqued on number of
different levels and this response has therefore been broken down into
sections for ease of referral. Each of the issues raised should have
been considered by the authors of the paper during production and
should also have been raised during the Lancet’s peer review process
prior to acceptance for publication. The fact that these issues have
not been fully explored by the authors or publishers could suggest
either a lack of understanding of research in homeopathy or a
deliberate attempt to mislead.
METHODOLOGICAL ISSUES

Matching
In their report Shang et al (2005) purport to match 110 homeopathy
trials with 110 allopathic trials
Their description of how these trials were matched according to similar
conditions and similar outcomes leaves many questions unanswered. What
exactly do they mean by similar? How similar are the conditions that
are matched? Variation between patient cohorts and severity of
complaint do not appear to have been taken into consideration.
Of the total number of homeopathy trials 16% involve individualised
(classical) prescribing. This involves a lengthy case-taking and hence
greater chance of the therapeutic relationship influencing the effects
of treatment. Hence the trials involving classical homeopathy, a
complex intervention, should have been matched with corresponding
trials of complex interventions rather than with simple interventions
such as pharmaceutical drugs.
Selection of the final 8 homeopathy and 6 allopathy trials
Of the total 220 trials the authors identify 21 homeopathy trials and 8
allopathy trials that are of higher standard according to their own set
criteria. Yet they present no comparative analysis of this subset of
trials. They then proceed to further select the small subset of
purportedly larger and higher methodological quality trials (8
homeopathy trials and 6 conventional medicine trials) from which the
paper's main conclusion is drawn. The authors fail to describe the
weighting of the two attributes size and methodological quality. The
authors do not provide an explanation as to how they chose the
particular cut off point that they used to select the final 14 trials.

The choice of the two parameters of size and methodological quality at
first appears to be reasonable but is problematic for homeopathy trials
for the following reasons:
1. High methodological quality in meta-analyses is defined as high
internal validity but ignores whether the study has any external
validity – its clinical relevance and its general real world
applicability of the study findings. In placebo controlled RCTs, high
internal validity is gained at the expense of low external validity
such that those of apparent high quality often bear little relation to
real world practice or relevance.
There are difficulties in attempting to research any complex health
care intervention such as individualised homeopathy. Careful modelling
of complex interventions is essential to ensure that interventions fit
with and reflect the complexities of the settings within which
interventions will be applied, and to ensure that the outcomes chosen
are those most appropriate to demonstrate any benefits or risks.
2. The use of trial size as a parameter introduces potential bias for
the homeopathy trials that is not addressed in the authors’ discussion.
The largest trials are ones funded by pharmaceutical companies.
Pharmaceutical companies fund trials of isopathy and complex
homeopathy, usually in the area of prevention (because that will
involve more people potentially buying the product). Isopathy and
complex homeopathy are the crudest and least clinically effective forms
of homeopathy. The largest trials are subject to the bias of
pharmaceutical profit funding, where the decision to fund a study will
be based on potential pharmaceutical profit rather than the clinical
relevance.

The authors fail to identify specifically the particular trials that
are used in the final analysis that would enable a fair critique of the
validity of the included trials.
Elucidation of the rationale for choosing this small number of studies
as
well as a list of which studies this seemingly comprehensive
interpretation was drawn from is essential.

Statistical analysis

It would be normal practice when comparing two different treatment
options to measure the statistical difference between the two options
for a fair comparison. The authors fail to do this, presumably as this
calculation shows clearly that there is no statistical difference
between the effects of the two interventions. That is, the statistics
tell us that there is no basis for saying that allopathic intervention
is any better than homeopathic intervention.

The authors use particular criteria for selection of trials that they
define as being of higher quality. Of the 220 trials only 21 homeopathy
trials and 8 allopathy trials meet these criteria for high quality.
With such a small number of trials meeting the criteria and with there
being a significantly larger number of homeopathy trials, the authors
should have stopped at this point and concluded that:
a. there are not enough high quality trials on which to carry out the
analysis, and
b. that the difference between the quality of the homeopathy and the
allopathy trials invalidates the matching.

The authors went to the great trouble of selecting 110 homeopathy
trials that met their inclusion criteria, matching them with 110
allopathy trials and then ignored all but 8 trials of homeopathy and 6
of allopathy in their final statistical analysis. Moreover the original
stated intention to compare trials of similar condition and outcome has
been ignored in the final analysis. The final small subset of trials is
not matched at all suggesting that different kinds of trials are being
compared, apples are being compared with oranges – a common failing in
meta-analyses.

INTERPRETATION BIAS

The authors generalise from an extremely small pool of data to draw
their apparently broad and negative conclusions. Their statistical
analysis and interpretation of results can be challenged in many
respects and their main conclusion that their “findings are compatible
with the notion that the clinical effects of homeopathy are placebo
effects” is unfounded.

While sample size is small in many individualised (classical)
homeopathy studies because of limited funding and the early
state-of-the-science need for preliminary studies to define the proper
design parameters (as in any proper research effort), the Shang et al.
paper does not support the conclusion that homeopathic remedies or
conventional drugs cause only placebo effects. It does support the
conclusion that homeopathic as well as conventional drug treatments in
clinical trials can produce placebo effects. This is no surprise and
does not logically support the conclusion that we should discard either
homeopathy or conventional drugs. (Bell,2005)
The authors display their own bias in interpretation when they dismiss
out of hand the substantially beneficial pooled effect from 8 trials of
homeopathic remedies in upper respiratory tract infections. This in
spite of
the fact that the trials perform well in the authors own set test of
funnel plot asymmetry which demonstrated that there was no significant
difference between effects in the higher quality trial and the lower
quality
trials. Here the authors speak of biases prevalent in these
publications to
excuse the effect without specifying in any way how they relate to these
trials and indicate that conclusions from these trials cannot be
trusted.

META-ANALYSES

Any meta-analysis of homeopathy will inevitably be considering only a
small number of trials as there are relatively few for any one
condition.
This leads to the pooling of heterogeneous data, a fact so often
completely ignored.

Meta-Analyses as a tool are akin to public surveys to glean opinions
about a politician. They are of vague value but do not reflect
individual truths which may carry much more meaning. That is especially
true of a meta-analysis of homeopathy, a science in which
individualization of prescription and interpretation of response
is fundamental to success. Meta-analyses flatten diversity and minimize
nuggets of truth by imposing criteria exclusive to allopathic
protocols. (Gray 2005)

In this report Shang et al (2005) different types of prescribing method
for homeopathy trials are all considered equally. Of the 100 trials
selected only 18 (16%) deemed classical homeopathy i.e. the type of
individualised prescribing that occurs in real life and there is no
indication of how many of the selected 8 trials involved this type of
prescribing.

RANDOMISED CONTROLLED TRIALS AND HOMEOPATHY

The authors’ conclusions are premised upon the supposition that the
placebo controlled randomised trial represents the gold standard against
which all research should be judged. It is becomingly increasingly
understood that there are considerable problems in using this method to
test
complex interventions such as homeopathy.

It is always going to be difficult to adopt the reductionist research
method of the placebo controlled RCT to measure effects of complex
intervention such as homeopathy.
Two primary concerns for homeopaths are that the treatment is holistic
and
that it is individualised. Treatment cannot be standardised and patient
response is unpredictable. RCTs are looking for specific effects
whereas homeopathy is attempting to produce general health effects as
well as specific effects – homeopathy treats the whole person.

Patients may choose homeopathy or acupuncture or other forms of
complementary and alternative medicine because of preferences and
perhaps even the potential for responding. Randomized controlled trials
do not necessarily recruit such patients – they take patients with a
conventional diagnosis, but do not screen for patients who typically
may gravitate to homeopathy or other types of complementary and
alternative medicine. The scientific evidence in this regard is
emerging in some studies and requires open-minded, albeit sceptical,
consideration toward understanding the true nature and potential
benefits of these treatments for some, but not necessarily all,
patients with a given conventional (Bell 2005)

“Actually there is no threat to anyone. Each system has its strengths
and limitations. If we work hand in hand, and not against each other,
it will really benefit our patients. In order to do this it is
important to understand the other system, and evaluate it according to
its principles and not see it from the view point of the other
system.”. (Sankaran 2005)
Are the clinical effects of homeopathy placebo effects? - A full
critique of the article by Sahng et al (Lancet 2005;366 (9487):726-733)
The article by Shang et al (2005) can be critiqued on number of
different levels and this response has therefore been broken down into
sections for ease of referral. Each of the issues raised should have
been considered by the authors of the paper during production and
should also have been raised during the Lancet’s peer review process
prior to acceptance for publication. The fact that these issues have
not been fully explored by the authors or publishers could suggest
either a lack of understanding of research in homeopathy or a
deliberate attempt to mislead.
METHODOLOGICAL ISSUES

Matching
In their report Shang et al (2005) purport to match 110 homeopathy
trials with 110 allopathic trials
Their description of how these trials were matched according to similar
conditions and similar outcomes leaves many questions unanswered. What
exactly do they mean by similar? How similar are the conditions that
are matched? Variation between patient cohorts and severity of
complaint do not appear to have been taken into consideration.
Of the total number of homeopathy trials 16% involve individualised
(classical) prescribing. This involves a lengthy case-taking and hence
greater chance of the therapeutic relationship influencing the effects
of treatment. Hence the trials involving classical homeopathy, a
complex intervention, should have been matched with corresponding
trials of complex interventions rather than with simple interventions
such as pharmaceutical drugs.
Selection of the final 8 homeopathy and 6 allopathy trials
Of the total 220 trials the authors identify 21 homeopathy trials and 8
allopathy trials that are of higher standard according to their own set
criteria. Yet they present no comparative analysis of this subset of
trials. They then proceed to further select the small subset of
purportedly larger and higher methodological quality trials (8
homeopathy trials and 6 conventional medicine trials) from which the
paper's main conclusion is drawn. The authors fail to describe the
weighting of the two attributes size and methodological quality. The
authors do not provide an explanation as to how they chose the
particular cut off point that they used to select the final 14 trials.

The choice of the two parameters of size and methodological quality at
first appears to be reasonable but is problematic for homeopathy trials
for the following reasons:
1. High methodological quality in meta-analyses is defined as high
internal validity but ignores whether the study has any external
validity – its clinical relevance and its general real world
applicability of the study findings. In placebo controlled RCTs, high
internal validity is gained at the expense of low external validity
such that those of apparent high quality often bear little relation to
real world practice or relevance.
There are difficulties in attempting to research any complex health
care intervention such as individualised homeopathy. Careful modelling
of complex interventions is essential to ensure that interventions fit
with and reflect the complexities of the settings within which
interventions will be applied, and to ensure that the outcomes chosen
are those most appropriate to demonstrate any benefits or risks.
2. The use of trial size as a parameter introduces potential bias for
the homeopathy trials that is not addressed in the authors’ discussion.
The largest trials are ones funded by pharmaceutical companies.
Pharmaceutical companies fund trials of isopathy and complex
homeopathy, usually in the area of prevention (because that will
involve more people potentially buying the product). Isopathy and
complex homeopathy are the crudest and least clinically effective forms
of homeopathy. The largest trials are subject to the bias of
pharmaceutical profit funding, where the decision to fund a study will
be based on potential pharmaceutical profit rather than the clinical
relevance.

The authors fail to identify specifically the particular trials that
are used in the final analysis that would enable a fair critique of the
validity of the included trials.
Elucidation of the rationale for choosing this small number of studies
as
well as a list of which studies this seemingly comprehensive
interpretation was drawn from is essential.

Statistical analysis

It would be normal practice when comparing two different treatment
options to measure the statistical difference between the two options
for a fair comparison. The authors fail to do this, presumably as this
calculation shows clearly that there is no statistical difference
between the effects of the two interventions. That is, the statistics
tell us that there is no basis for saying that allopathic intervention
is any better than homeopathic intervention.

The authors use particular criteria for selection of trials that they
define as being of higher quality. Of the 220 trials only 21 homeopathy
trials and 8 allopathy trials meet these criteria for high quality.
With such a small number of trials meeting the criteria and with there
being a significantly larger number of homeopathy trials, the authors
should have stopped at this point and concluded that:
a. there are not enough high quality trials on which to carry out the
analysis, and
b. that the difference between the quality of the homeopathy and the
allopathy trials invalidates the matching.

The authors went to the great trouble of selecting 110 homeopathy
trials that met their inclusion criteria, matching them with 110
allopathy trials and then ignored all but 8 trials of homeopathy and 6
of allopathy in their final statistical analysis. Moreover the original
stated intention to compare trials of similar condition and outcome has
been ignored in the final analysis. The final small subset of trials is
not matched at all suggesting that different kinds of trials are being
compared, apples are being compared with oranges – a common failing in
meta-analyses.

INTERPRETATION BIAS

The authors generalise from an extremely small pool of data to draw
their apparently broad and negative conclusions. Their statistical
analysis and interpretation of results can be challenged in many
respects and their main conclusion that their “findings are compatible
with the notion that the clinical effects of homeopathy are placebo
effects” is unfounded.

While sample size is small in many individualised (classical)
homeopathy studies because of limited funding and the early
state-of-the-science need for preliminary studies to define the proper
design parameters (as in any proper research effort), the Shang et al.
paper does not support the conclusion that homeopathic remedies or
conventional drugs cause only placebo effects. It does support the
conclusion that homeopathic as well as conventional drug treatments in
clinical trials can produce placebo effects. This is no surprise and
does not logically support the conclusion that we should discard either
homeopathy or conventional drugs. (Bell,2005)
The authors display their own bias in interpretation when they dismiss
out of hand the substantially beneficial pooled effect from 8 trials of
homeopathic remedies in upper respiratory tract infections. This in
spite of
the fact that the trials perform well in the authors own set test of
funnel plot asymmetry which demonstrated that there was no significant
difference between effects in the higher quality trial and the lower
quality
trials. Here the authors speak of biases prevalent in these
publications to
excuse the effect without specifying in any way how they relate to these
trials and indicate that conclusions from these trials cannot be
trusted.

META-ANALYSES

Any meta-analysis of homeopathy will inevitably be considering only a
small number of trials as there are relatively few for any one
condition.
This leads to the pooling of heterogeneous data, a fact so often
completely ignored.

Meta-Analyses as a tool are akin to public surveys to glean opinions
about a politician. They are of vague value but do not reflect
individual truths which may carry much more meaning. That is especially
true of a meta-analysis of homeopathy, a science in which
individualization of prescription and interpretation of response
is fundamental to success. Meta-analyses flatten diversity and minimize
nuggets of truth by imposing criteria exclusive to allopathic
protocols. (Gray 2005)

In this report Shang et al (2005) different types of prescribing method
for homeopathy trials are all considered equally. Of the 100 trials
selected only 18 (16%) deemed classical homeopathy i.e. the type of
individualised prescribing that occurs in real life and there is no
indication of how many of the selected 8 trials involved this type of
prescribing.

RANDOMISED CONTROLLED TRIALS AND HOMEOPATHY

The authors’ conclusions are premised upon the supposition that the
placebo controlled randomised trial represents the gold standard against
which all research should be judged. It is becomingly increasingly
understood that there are considerable problems in using this method to
test
complex interventions such as homeopathy.

It is always going to be difficult to adopt the reductionist research
method of the placebo controlled RCT to measure effects of complex
intervention such as homeopathy.
Two primary concerns for homeopaths are that the treatment is holistic
and
that it is individualised. Treatment cannot be standardised and patient
response is unpredictable. RCTs are looking for specific effects
whereas homeopathy is attempting to produce general health effects as
well as specific effects – homeopathy treats the whole person.

Patients may choose homeopathy or acupuncture or other forms of
complementary and alternative medicine because of preferences and
perhaps even the potential for responding. Randomized controlled trials
do not necessarily recruit such patients – they take patients with a
conventional diagnosis, but do not screen for patients who typically
may gravitate to homeopathy or other types of complementary and
alternative medicine. The scientific evidence in this regard is
emerging in some studies and requires open-minded, albeit sceptical,
consideration toward understanding the true nature and potential
benefits of these treatments for some, but not necessarily all,
patients with a given conventional (Bell 2005)

“Actually there is no threat to anyone. Each system has its strengths
and limitations. If we work hand in hand, and not against each other,
it will really benefit our patients. In order to do this it is
important to understand the other system, and evaluate it according to
its principles and not see it from the view point of the other
system.”. (Sankaran 2005)
[Non-text portions of this message have been removed]