Coast

Someone asked me whether studying cognitive science changed my view of humanity.1

I had to think about it for a while, and my answer — which was only mildly surprising to me — was that instead of changing my views about humanity, it mostly changed my views about science and research. Specifically, that it made me a lot more skeptical whenever anyone claims that “science”, “research”, or studies have shown anything, say, about humanity.

I left academia for two reasons: first is that academia2 is extremely competitive; the jobs are few, the applicants are many3, and I didn’t want it as much as I saw other people did. Second is that soon after I started graduate school I started to become very disillusioned with the field, its practices, and its incentives. Eventually I became so cynical about a career that I hadn’t even started yet that I knew I should never try to start it in the first place.

Academia has a serious problem wherein it runs on a system of incentives that rewards bad scientists and pushes good scientists out. At risk of weirding you out by commenting on my own writing, that is a remarkable statement. I just told you that academia rewards bad scientists.

Most of this is not news. Every once in a while the BBC or The Economist will talk about the replication crisis and bad incentives in science, or the file-drawer problem, or p-hacking. But I think the enormity of the problem escapes the majority of people.

Here’s what I think you should do. Find a slow weekend morning or afternoon, make yourself a pot of coffee, and spend an hour or two reading Retraction Watch and Andrew Gelman’s site. I used to subscribe to Retraction Watch’s RSS feed but ended up unsubscribing because it was too prolific and I couldn’t keep up. Things are so busy over there that they publish a weekly weekend reads post that you couldn’t possibly finish reading in a weekend unless you had absolutely no other plans.

Here is the life cycle of an academic. There is very little variation in this cycle:

Undergraduate degree (do a thesis or something)
➝ Grad school, do a Master’s
➝ More grad school, do a PhD
➝ Almost definitely a Post Doctoral fellowship4

You remain in the Post Doc holding pattern until you find a job in a university or college. The dream is to land a tenure track position in a research-intensive institution. The reality is people are increasingly taking lesser and lesser positions because the demand for those dream tenure track jobs far outpaces the supply.

Landing a job, especially a good one (the tenure track and get-to-do-some-research-and-not-just-teach kind) is now determined by one factor and one factor only: publications.5 Publish or perish is not a joke, it is the law of the land. Most important is your past publications count and the likelihood that you will be as productive if not more so in the future. Those are also second and third most important. Fourth most important is the prestige of the journals where you get published. Are you publishing in Psych Science or in Frontiers? Makes some difference. I don’t know if anyone actually reads your papers, or whether the quality of your writing, methodology, or, you know, science, factors heavily.

Here is a list of things that, again, with possibly very few exceptions, mean absolutely nothing for your prospects at getting a job:

  1. You champion open science.
  2. You write blog posts about experiments that haven’t worked out, or interesting statistical issues or practices.
  3. You contribute to open source projects, or statistical or data visualization packages.
  4. You are an active mentor and are generous with your time with students or peers.

If there isn’t a publication coming out of it, it doesn’t matter.

It’s not hard to see what a system like that does to the quality of the scientific process. Science is misunderstood by many; it’s not certain, experiments require care, results are not guaranteed, and theories are sooner or later wrong. It takes time to do something right, and you can still end up with a null result; and no amount of hard work could have made a positive finding more likely. But you want a job, so you will do everything you can to end up with a positive result anyway. You will choose topics that are in fashion, you will try to choose easy experiments that can be published regardless of the result6, and yes, you will operate under the constant pressure to make your t-test or ANOVA give you a p-value less than .05, and you might justify bending the rules of statistics to get it.

The better researchers, the ones who prioritize good theory and well-designed experiments and analyses are at a disadvantage. They will not try to publish shoddy studies and will not squeeze a result out of data where none exists. They will want to run the experiments that will help the field choose between competing theories and make progress instead of running the experiment that will produce the 35th uninformative but curious interaction and publish that instead. And for those ideals, they will pay.

I am not a very social person and I never created a big network of academics when I was a student, and yet I personally know several incredible researchers who ejected out of academia as they saw their academic career prospects shrink and ended up in industry; where they are valued and get paid way more for their skills than in the field where they would prefer to be. What a tragedy.

In case you decide to not visit Andrew Gelman’s site – your loss, really – I’ve plucked out an example for you.

The absolute minimum background information you need to know is that psychology, especially social psychology, has been going through a replication crisis in which many popular and thought-to-be-bulletproof findings like ego depletion and power pose do not replicate.

The news and internet have not been kind to psychology during this tumultuous time, nor should they have been.

Susan Fiske, a social psychologist and past president of the Association for Psychological Science, wrote an article titled Mob Rule or Wisdom of Crowds? (PDF download) in which she writes one of the most ill-advised opinion pieces I’ve ever seen by an academic. In it, she – sigh, there is no other way to say this – rants and rails against “online vigilantes” and “self-appointed data police” “volunteering critiques of such personal ferocity and relentless frequency that they resemble a denial-of-service attack that crashed a website by sheer volume of traffic”.

I don’t know what the “website” is supposed to be here, but the idea that academics are suffering a denial of service attack from online critics is laughable. I know of few other institutions that live in their own protected bubble as the psychology department. The whole article is a shame and I am embarrassed on her behalf.

Even though this article was invited by the APS’s Observer, it seems the reaction was so negative that it was never published, and you might find it tricky to find a copy online. The link above gives you the PDF hosted on my own site, plus alternate link 1 and alternate link 2.

The Observer posted an unsurprisingly spineless comment on the issue, including this amazing final paragraph:

Those wishing to share their opinions on this particular matter are invited to submit comments in the space below. Alternatively, letters can be sent to apsobserver@psychologicalscience.org. We ask that your comments include your full name and affiliation.

Yes, please include your affiliation, lest we forget for a moment to judge the value and validity of your comment according to the authority of whether you’re a professor or just a normie.

In one of Andrew Gelman’s comments on the issue, aptly titled “What has happened down here is the winds have changed”, he writes:

In her article that was my excuse to write this long post, Fiske expresses concerns for the careers of her friends, careers that may have been damaged by public airing of their research mistakes. Just remember that, for each of these people, there may well be three other young researchers who were doing careful, serious work but then didn’t get picked for a plum job or promotion because it was too hard to compete with other candidates who did sloppy but flashy work that got published in Psych Science or PPNAS. It goes both ways.

I couldn’t have said it better.

Back to the original question (which I will rephrase to maintain the flow of the story I’m telling here): how has studying cognitive psychology changed my view on things?

For one, I find myself in the uncomfortable position of seeming like a contrarian to two opposing groups: authoritarian conspiracy theorists who assume all scientists are malicious liars with an agenda7, and the intellectual, educated but non-scientific class that has confused the platonic concept of “science” with the practice of scientific inquiry, and therefor defends anything with the smell of science as unquestionable and unassailable. It’s much easier, in fact, to deal with the first group. It’s the second one that depresses me.

Call them what you like, the intellectuals, the elite, the educated class, whatever, I’m not necessarily a fan of any of those terms. They are relatively affluent, internet-savvy people, who, in their attempt to fight back against anti-reason trends in the West (often socially conservative right-wing groups, although by no means all), have grossly overcorrected and now defend any output of human research as sacrosanct truth beyond reproach.

I won’t mince words, this is worship of authority. People confuse the day to day practice with the scientific method itself, and treat the results of the practice as though it was perfect and the output guaranteed by the sanctity of the lab coat and the scatterplot.

In another response to the Susan Fiske article titled “Weapons of math destruction”, NeuroAnaTody writes:

Science is moving forward so quickly that I don’t even think it’s necessary to point out ways in which the article is wrong. I will instead list a some elements of the scientific revolution that trouble me, even though I consider myself a proud (if quite junior) member of the data police.

  1. Belief in published results. I have so little of it left.
  2. Belief in the role of empirical research. Getting to otherwise hidden truths was our thing, the critical point of departure from philosophy.
  3. Belief in the scientific method. I was taught there is such a thing. Now it seems every subfield would have been better off developing its own methods, fitted to its own questions and data.
  4. Belief in statistics. I was taught this is the way to impartial truths. Now I’m a p-value skeptic.
  5. Belief in the academic system. It incentivizes competition in prolifically creating polished narratives out of messy data.

Emphasis mine, because point 1 is the headline for me. Belief in published results, I have so little of it left. That is how studying cognitive psychology has changed my view on things. Whenever I hear of a study that “showed something”, and especially if it’s in the field of psychology, my assumption is that it’s spurious.

So what am I telling you? Science is permanently broken and we are left rudderless in a sea of claims and counter-claims?

No, that would be confusing the practice of science with the scientific process as it should be, the same mistake I think the study-worshipper makes. The scientific method is still the best way we have to approach the truth about the world, we just need to set up the incentives to encourage following it better.

In the meantime, I think you should be extremely skeptical of everything you hear, which is is an uncomfortable position but is not an option at this point. The “study” goes through many stages on its way from having touched the truth and transformed into data, to your eyes. It has gone through an experimental design (done by a human), data collection (done by a different human), analysis (possibly done by a third different human), write-up (one or more humans), review (I will stop mentioning that things are done by humans), and interpretation by a journalist or reader.

In all of these stages, a human makes a judgement call to the best of their abilities, and as all other humans, they operate under pressures and incentives. Speaking of incentives, I am also telling you that academia is not the world of enlightened philosopher kings and queens operating outside the realm of dirty wants and desires the rest of us live in. Academics operate within a terrible, broken system of incentives, and you must keep that in mind whenever you’re consuming their research.

The other message I want to leave you with is that academia is broken and I don’t see it being fixed any time soon. It won’t be fixed until academics are evaluated based on more than their number of publications. It won’t be fixed until hiring committees stop looking at how many papers you’ve published and start looking at the quality of your contribution to knowledge. Yes, it’s much harder to decide whether you created knowledge and contributed to theory than it is to look at your impact factor, but that’s what has to happen. That’s it.

  1. Someone else pointed out that it would be difficult for me to answer that question because I didn’t know an alternative life where I didn’t study cognitive science and what my view of humanity would be in that world. But the question is still valid because I can compare to before I started studying cognitive science, or reflect on how my views changed during the study. ↩︎

  2. My personal academic experience is in psychology, but the points I make in this post generalize to all disciplines as far as I know. ↩︎

  3. Many. The ratio is quite bad. You shouldn’t necessarily trust the numbers from any of those articles, but you can conclude that the picture is bleak for anyone who is in the crème de la crème of their field, and hopeless for everyone else. ↩︎

  4. A Post Doc is not a student. They are an employee who is often an independent researcher in a lab and get paid way too little. ↩︎

  5. There might be exceptions to this, but they are exceptional exceptions. ↩︎

  6. I have personally received this piece of advice, explicitly, more times than I can count. ↩︎

  7. I think scientists do have an agenda that we ought to acknowledge more. They are, after all, people. ↩︎