Tag Archives: research

Bad Science: Thomson-Reuters Publishes Flawed Ranking of Hottest Research

Thomson-Reuters has reportedly published their yearly analysis of the hottest trends in science research. Increasingly, governments and funding organisations use such documents to identify strategic priorities. So it’s profoundly disturbing that their conclusions are based on shoddy methodology and bad science!

The researchers first split recent papers into 10 broad areas, of which Physics was one. And then the problems began. According to the official document

Research fronts assigned to each of the
10 areas were ranked by total citations and the top 10 percent of the fronts in each area were extracted.

Already the authors have fallen into two fallacies. First, they have failed to normalise for the size of the field. Many fields (like Higgs phenomenology) will necessarily generate large quantities of citations due to their high visibility and current funding. Of course, this doesn’t mean that we’ve cracked naturalness all of a sudden!

Second their analysis is far too coarse-grained. Physics contains many disciplines, with vastly different publication rates and average numbers of citations. Phenomenologists publish swiftly and regularly, while theorists have longer papers with slower turnover. Experimentalists often fall somewhere in the middle. Clearly the Thomson-Reuters methodology favours phenomenology over all else.

But wait, the next paragraph seems to address these concerns. To some extent they “cherry pick” the hottest research fronts to account for these issues. According to the report

Due to the different characteristics and citation behaviors in various disciplines, some fronts are much smaller than others in terms of number of core and citing papers.

Excellent, I hear you say – tell me more! But here comes more bad news. It seems there’s no information on how this cherry picking was done! There’s no mention of experts consulted in each field. No mathematical detail about how vastly different disciplines were fairly compared. Thomson-Reuters have decided that all the reader deserves is a vague placatory paragraph.

And it gets worse. It turns out that the scientific analysis wasn’t performed by a balanced international committee. It was handed off to a single country – China. Who knows, perhaps they were the lowest bidder? Of course, I couldn’t possibly comment. But it seems strange to me to pick a country famed for its grossly flawed approach to scientific funding.

Governments need to fund science based on quality and promise, not merely quantity. Thomson-Reuters simplistic analysis is bad science at its very worst. It seems to offer intelligent information but  in fact is misleading, flawed and ultimately dangerous.

Why does feedback hurt sometimes?

Research is hard. And not for the reasons you might expect! Sure, my daily life involves equations which look impenetrable to the layman. But by the time you’ve spent years studying them, they aren’t so terrifying!

The real difficulty in research is psychological. The natural state for a scientist is failure – most ideas simply do not succeed! Developing the resilience, maturity and sheer bloody mindedness to just keep on plugging away is a vital but tough skill.

This letter, written by an experienced academic to her PhD student is a wonderfully candid account of the minefield of academic criticism, both professional and personal. What’s more, it lays bare some important coping strategies – I certainly wish I’d read it before embarking on my PhD.

Above all, this letter is an admission of humanity. As researchers, we face huge challenges in our careers. But the very personal process of responding to them is precisely what makes us better scientists, and perhaps even improves us as people.

The Thesis Whisperer

This letter was written by an experienced academic at ANU to her PhD student, who had just presented his research to a review panel and was still licking her wounds.

The student sent it to me and I thought it was a great response I asked the academic in question, and the student who received it, if I could publish it. I wish all of us could have such nuanced and thoughtfu feedback during the PhD. I hope you enjoy it.

Screen Shot 2014-02-05 at 7.27.05 PMA letter to…My PhD student after her upgradeWell you did it. You got your upgrade. But from the look on your face I could tell you thought it was a hollow victory. The professors did their job and put the boot in. I remember seeing that look in the mirror after my own viva. Why does a win in academia always have the sting of defeat?

Yeah, it’s a…

View original post 1,120 more words

Asking the Obvious Question

I’m now about to finish my first year as a PhD student. Along the way I’ve done a lot of physics! Some of the concepts are very hard. I’ve sure spent my fair share of hours battling with abstract maths! But I’ve learnt something much tougher and infinitely more valuable in the past 12 months – how to do research.

The blessing and curse of research is it’s very hard to teach. You need just the right combination of perserverance, creativity and inspiration. Unlike most forms of employment, science is wonderfully, frustratingly unpredictable!

There’s one principle that stands out through every success and failure this year. Ask the obvious question! Whether in a seminar, a conversation with colleagues or in front of your desk, never be afraid to say something stupid. Often it’s the most basic idea which leads to the richest consequences.

At the end of the day, research is something of a confidence game. It’s a bit similar to my limited experience on a snooker table. If I think I’m going to win, I usually do. But when those doubts creep in, it’s much harder to keep the break going!

That’s why it’s so important that scientists communicate. Sadly the human brain doesn’t seem to be wired up to think deeply and laterally simultaneously. Regular breaks for discussion, evaluation and presentation of your work are vital!

I’ve had my clearest thoughts on walks to the tube, after chatting over coffee or writing a blog post. Although the life of a scientist might appear relaxed, ours is not a job where you can just clock in and out!

Asking the obvious question is not just important for researchers. Students, journalists, politicians, civil servants, lawyers, managers, even executives pose simple questions every day. In fact, it’s when public figures disguise their questions and answers with complex language that we struggle to relate.

A stupid, obvious question can do no harm. And more often than not, it’s exactly what you need to say.

Physics Through the Looking Glass

I recently took part in the popular Three Minute Thesis competition. Each contestant gets just 3 minutes to explain their research to a panel of laymen. Although I didn’t make it to the national finals, it was nevertheless great fun.

Here’s an audio recording of my speech, taken live at the QMUL finals. For the experts among you, I’m giving an account of my attempts to use twistor techniques to investigate subleading soft theorems in gauge theory and gravity!