Cherry-picked Datasets are breaking the Scientific Method: 3 useful tips
Sadly, here in the 3rd decade of the 3rd millennium AD, we’re regressing in our ability to communicate because few of us trust the same sources.
Recently a friend told me, “You’re an engineer for heaven’s sake, why can’t you see the statistical significance of your error?!” It never entered his mind, not once, that his chosen dataset was entirely different than mine, and that I had good reasons to mistrust his dataset more than my own. So it’s no wonder why he’s outraged. Self-induced myopia generally is.
Self-customization of news sources (and social media circles) have only made people become “experts” in what they already believed, closing them off to any external enlightenment, especially with regard to where they might be wrong. News gathering has become little more than self-imposed brainwashing, turning people more into mindless robots who just regurgitate what their favorite pundits say. Critical thinking is dead. For most people the “information highway” has become the bias-confirmation highway.
The most cursed effect of this has been the establishment of highly groomed / cherry-picked datasets and data sources. What are the latest pandemic numbers? It depends on the dataset. What are the employment numbers? Again, it depends on the dataset. Global temperatures? Again… dataset. That’s always been a problem, but it has never been as problematic as it is today. Opposing datasets are now carefully cherry-picked through the investment of billions of dollars to support whatever narrative people believe in, so that anyone can claim absolute certainty no matter how far fetched it may be. They have the data to prove it.
All cherry-picked datasets really do is break the scientific method. And they’re everywhere. Even in “comprehensive” reviews done by medical researchers and published in peer reviewed journals.
The result: People forever arguing while never actually learning anything new, and it’s an ever worsening phenomenon. As a society we’ve become self-validation experts, but self-evaluation morons.
So, what to do? Here’s 3 tips to help you sort out what’s real with regard to what’s been going on lately …
1) Stop Taking things at Face Value
What to do about it? Stop taking things at face value. When you hear about a study … research it from another perspective. Believe nothing until you’ve properly vetted it by fully investigating claims from different perspectives. In most cases people don’t have time to do that, and there’s nothing wrong with withholding an opinion until you can. Remember one study is just one study, and most serious issues involve the careful (not cherry-picked) analysis of many dozens of studies. Don’t let politics or other unrelated matters affect your analysis. Be okay with saying “I don’t know”.
2) Know the difference between proven, unproven, and proven false: 3 *different* things
When you hear something isn’t proven, realize that’s different than something proven to be wrong. More often than not people incorrectly interpret unproven as meaning proven false, which lack of insight has been problematic recently given the recent pandemic. For medical matters read what the NIH position is and know what they actually mean (proven means proven, unproven means unproven, proven false means proven false).
3) Note: The NIH is more truthful than the FDA
Again, back to how this relates to medicine, since it’s been a thing lately: if the FDA doesn’t recommend something then don’t read into that too much … they have to err on the side of caution. Remember early 2020 when they said masks don’t work for the general public but magically they do for doctors? Yeah. There was that.
The FDA is a regulatory agency … that’s what they focus on, and as such are pressured far more to push certain narratives. Look to the NIH for truth when it comes to scientific truth, and to the FDA for regulatory insights (and everything that might influence that). The NIH and FDA differ on opinions of things like treatments that work for a recent pandemic … and it is telling (see below).
Those 3 tips will dramatically improve your ability to navigate in this age of misinformation. Cherry-picked datasets? Yep, but with care you can know which things to trust.
—
PS — some recent cases where there have been disagreements between the two agencies:
- convalescent plasma: https://www.wsj.com/articles/a-little-debate-on-plasma-is-healthy-for-science-11600790067
- Masks: https://www.youtube.com/watch?v=PW9HyEEmyyo
- treatment options: https://www.clarkcountytoday.com/news/battle-over-whom-to-believe-about-ivermectin/, https://www.coloradopolitics.com/coronavirus/hydroxychloroquine-ivermectin-remdesevir-and-more-the-evidence-on-proposed-covid-19-treatments/article_ef433972-4d0f-11eb-a481-5b626171b7a8.html
