Reading research is a skill. Just like any other skill, it takes time, patience, and practice. More and more content is being put out today with references tagged at the end which seems to increase the legitimacy of the writer or content. In reality, that means nothing. Anyone can site or reference papers and sources yet have no idea what they are talking about. I see it all too often. As readers, we NEED to be equipped with the proper skills to read critically which will help sift through and battle the BS. Here are the top 5 mistakes I see being made constantly when people read and interpret nutrition research or ANY research for that matter:
1. Peer-reviewed doesn’t necessarily mean to live or die by the findings.
Yes, peer-reviewed is considered a gold standard and what you should be looking for, but know that just because a study is published does not mean that it is a quality study. There are a lot of flaws in the peer review process which leads to inconsistency, bias, and abuse. Also, not all peer-reviewed journals are equal in quality, nor are the papers published in them. Unfortunately, that is just the name of the game for now so it is up to the reader to critically assess the components of the paper to determine its quality.
2. Skipping the methods and completely misunderstanding and misinterpreting the statistics.
When is the last time you had to take a stats class? College? High school? Maybe you never took one at all? Regardless, properly interpreting and understanding the statistics portion of a research paper is critical in determining the quality of the results and conclusions. Begin by understanding the meaning of words used to define the statistical findings such as significant, important, meaningful, worthwhile. Significant does not mean important! Go through the following questions when reading through the methods and stats portions.
(a)What is the error of measurements? What is the difference between the measured quantities and the true value?
(b) What is the reliability and validity of instrumentation used?
(c) Is the protocol used detailed extensively so that the project may be repeatable?
(d) Was a baseline established?
(e) Were repeated measures used?
(f) Was the study longitudinal or acute – meaning, did the study last months or was the study a day or week long and they concluded some sort of “significant” findings?
By answering these you can begin to determine the QUALITY of the study.
3. Were limitations exposed to these findings?
A good researcher will reveal in the conclusion the limitations of their study. Meaning, they list the problems or possible confounding factors that occurred during their study. This will lead to suggestions for future research. If the lead author does not expose the limitations of their study, then there is a problem – either the author is trying to mislead the readers or they are trying to make their conclusions seem like the end all be all.
4. Exhibiting confirmation bias
If you are truly not trying to be biased when reading the research, you should start by reading the method section and then go to results and conclusion. Confirmation bias is the tendency to search for, interpret, favor, and recall information in a way that confirms preexisting beliefs. The title and introduction prime you for what you are about to read and can steer you toward or away from a paper, based on your preexisting beliefs. Another mistake people make which leads to bias is failing to critically assess, compare, and contrast previous findings.
5. Inferring beyond the sample into a population or beyond a population
Look at the subjects used in the methods section of the study. Do they match up with the population referred to in the conclusion section? For example, you find a research paper that has the term “elite athlete” or “high-level athlete” in the title but in the methods section, they used very-well trained subjects with more than 2 years of resistance training experience. These two populations do not match up. Look at the number of subjects used, the age range, sex (male/female), pre-existing conditions, training age/history, and anthropometric measurements. The more detail provided in the study the better the findings can be applied to the specific population. You cannot take information found from X population and apply it to the Y population. The same principle applies to studies where animals are used. Yes, they can be useful but the findings should not be so black and white when applying them to humans.
Keep reading. Continue in your pursuit of knowledge. Good luck out there! And remember – don’t be a part of the problem. Be a part of the solution!
