Reading scientific reports - a few tips

[Irene de Villiers]

Robert & Shannon Nelson wrote:

Irene de Villiers wrote: [...]

Hi Shannon,
I used to take my ability to do this for granted, assuming everyone
else also did what I do - reading all studies in original and with a
very critical eye.... but then realized it was a learned skill and was
drummed into us with *daily* practise at the university where I studied
for my B.Sc degree (Rhodes University in SA). It was part of ALL the
undergraduate courses in any science - I guess the philosophy was that
if you wanted to be any kind of a scientist you'd better be able to
fully understand what was already researched so as to know where
something new might be needed - or where something was badly performed
before.
I hope to write a course on this soon as I see now that distilling
many different exercises on doing this, down to a single course with
lots of appropriate practise examples, might be very useful.
It's seldom that the statistics of a study are the main thing to be
worried about as those can be checked mathematically quite easily and
also the numbers involved should usually be clearly appropriate to the
task just by common sense.

The first thing I look at is the credentials of the person providing
the information, and I expect there to be RELEVANT credentials. I do
this when assessing website info as well. Too many times you see a bunch
of supposedly good science - but in the fine print you look for the
credentials of the author (or there is no author!) and they have a
theology degree or some such.
(That's an actual example!) So they are listed as "Doctor" but the
credibility of a doctor of theology on the subject of nutrition or cell
physiology is hardly convincing to me.
That is also where you will find who belongs to what corporation -
and usually they have minimal academic credentials if any in the
*relevant* field.

A lot of scientists will present good information in their own area
of specialized expertise - and then go on to spout in a spin-off area
where they have no specific credentials - and so I'll sift that out and
take only the relevant part.
For example Dr Peter D'Adamo does serious research on how foods
affect people of different blood types - and that info is really good.
But then he branches out into a book on "diabetes for Blood type O" or
whatever. What he knows about diabetes is nothing IMO other than the
nonsense dished out liberally by the American Diabetes Association (and
if you follow that, you are guaranteed to make grain companies and drug
companies rich as your diabetes gets steadily worse.)
My point is that d'Adamo's expertise STOPS at what foods do which
to a specific blood type as that is his area of direct research.
Applying it (such as in diabetes) is not his field. So use an expert in
diabetes to find out the diabetes part - and combine the two - but do
not take a really good expert's advice in an area where they are not
expert:-)

So one principle in checking out an information source is to check the
credentials of the author - thoroughly - make sure they are not a ten
dollar internet "doctor" from "Flybynight University" - or that they are
not spouting off in an area they did not *personally* study or research.
I use only what they *personally* studied.
How to find what that is:
Type their mane at PubMed, and see how many scientific papers come up
and on what subject/s:-)
Repeat at Google to see what books they wrote, etc and look up any
biographical blurbs.
One I checked recently, providing nutrition advice, was a geologist by
profession. Ummmmm.

The veterinarian whose homeopathy courses are the most popular in USA is
another example. What are his homeopathy credentials?
He lists DVM. Is that relevant?
If you check out "Academy of Veterinary Homeopathy (run by another DVM)
- it's an organisation which "certifies homeopathic veterinarians",
claims to be the "only one to so so in USA" and it's ONLY course
accepted is the one by the person just mentioned with DVM as his only
credential.
Do you see the point?
Where's the validity?
If you were to run a school in health care of any kind - would you do so
without a single credential in the relevant field?

There are several principles I use.
Next perhaps on the list is that the write-up needs to make sense to ME.
Never assume that it is hard to follow because YOU lack something. The
study you are reading was written up with plenty of care and time in the
phraseology and there is no excuse for it being in a mess.
If the article is not making sense - do not use it unless/until it does
make clear sense to *you*.
If it feels the least bit hazy or illogical anywhere, either work to get
clarity by looking up terms or whatever - or discard it, or write tot he
author for an explanation (Their address is invariably at PubMed with
the article). If it fails to communicate well - it has no value. If it
is the only article on the issue - write to the author for clarity.

Scientists might be good researchers but terrible report writers - but
if so, it is THEIR job to get a "ghost writer" to put the study into
proper understandable language - with the scientist signing off on it as
being an accurate representation of what was done and what was found -
so that it makes sense to ANY reader. So another principle is that the
study MUST make sense - to YOU.

This could get very long - and I do rather think that a course would be
an appropriate approach. I really think we all need to put scientists on
notice that we CAN and will make sense of what they are doing and can
see through shady "science" or biased work and will not accept it.
Statistics are not as relevant as other things with this proviso:
If you look at the number of individuals in the study, your common sense
will give you a pretty good idea of statistical relevance.
So for example if 7000 children are healthier on homeoprophylaxis than
7000 using vaccines - that seems a significant number just by common
sense. But:
For example there is a study on FIP in Japan involving 12 cats of
which 4 survived longer than three months - the 4 oldest ones. Eight
died within that time.
They mention what drugs were used (prednisone and interferon) and they
claim a "33% success rate with FIP by their treatment method".
Do you buy that?

Common sense should be asking:
What good is a 3 month "survival" - WHY the older cats. Fact: FIP
affects mainly kittens, it's quite hard to find an older cat with it.
So even if it did help older cats 33% of the time to survive for 3
months - is that relevant in the big picture?
No doubt in the older cats as they were less compromised by the
multiple annual vaccines now in use and so they made it for longer.
I consider it fraud that they write (in many places) in feline health
publications that "We get a 33% success rate in FIP cases with our method".
I see it as closer to zero percent.
If you look for somewhere another person uses that method - you see
death - it's not repeatable - there is not even one other case reported
positively. Nor will they say what happened to the 3 cats after 3 months.
So there is a need for a bit of follow-up and it is lack of logic -
or a flag like older cats doing better than young ones - that triggers
something being "off key" as a gut feel. Follow those up.
And watch out for "blanket" statements like that they get 33%
success.... Always ask "33% of what?" In this case it is 33% of a very
oddball small group, for a very limited time.
Be strict about interpretations. If theirs and yours look different (and
you SHOULD make an independent conclusion before reading theirs - to
test against theirs) then follow up to debunk/confirm - or discard the
research as not credible.
You can usually get the original abstract on PubMEd for free - just read
it well and do not assume the conclusion listed has value - make your
own...... If the article is of enough relevance to you then it is worth
buying the original in full - but it IS pricey - like $30 on average -
unless you get a subscription - also pricey - or join with others who
want to read it - or find another source with corroborating info - or
write directly to the author for more info, with a question.
Sometimes you can get a relatively good price on a compendium of
papers that are presented at a conference.... worth it if the entire
conference is of interest. Might even find it on ebay:-)

Phew - this is getting very long - there is lots more, but these are
starter ideas?

Namaste,
Irene
-
Irene de Villiers, B.Sc AASCA MCSSA D.I.Hom/D.Vet.Hom.
P.O. Box 4703 Spokane WA 99220.
www.angelfire.com/fl/furryboots/clickhere.html (Veterinary Homeopath.)
"Man who say it cannot be done should not interrupt one doing it."

[Leilanae]

snip.............
Hi Irene,

Thanks for taking the time to share your information.

Question: If a large study (lots of participants) was conducted where
10 participants died during the study, can they omit this fact by
saying "of the participants completing the study", the
results are?

Thanks,

Leilanae