We need metrics(!), so let’s make them as good as possible

This is a post that is forthcoming at the London School of Economics Impact blog, I only post it here so I can receive feedback and link to it somewhere else. 🙂

 

—————————————————————————————

 

Recently at the LSE Impact blog several individuals have argued against metrics, at least journal level metrics. Jane Tinkler put it well when, in a recent post here, she said,  ‘One of the most common concerns that colleagues discussed with us is that impact metrics focus on what is measurable at the expense of what is important.’ One ill-informed individual went so far as to say that any attempt to measure science is ill fated.

With such crazy comments, I feel obliged to stress three small things, the first of which is:

Metrics are good (but can be bad).

While it is true that ‘systems based on counting can be gamed,’ science has always been about taking the complexity of the world and making it measurable and predictable. Our evaluation of science should be no different. ‘What good science is?’, is a complex and many faceted problem, but is it any more complicated than the well-being or happiness of an individual? I would suggest not. It is true that metrics often miss some nuance, but the goal of the metric is to maximally explain the cases, on average, rather than as an individual. It is psychologically interesting that scientists are ready to apply metrics to everything in the world, except how good or bad their work is.

More than ‘metrics’ simply being the way science works, their value and necessity is evidenced most clearly by their increasing usage and popularity. Who does good science is an important variable to predict within a number of contexts, and if we spent the time to read each of the articles a potential hiree has put out (even just top 5 or 1), there would be little time for anything else.

Even while metrics are useful and necessary, they are still flawed. People are still people, and they will search for ways to increase their score on any widely accepted metric. This is why we clean up before we have someone over to our house and why students study for (and cheat on) exams. There may be fundamental problems with metrics, but they are more due to the humans who use them, rather than the metrics themselves.

It is because metrics are so necessary and widely spread that bibliometrics is such an essential study (unlike those who suggest it is a fake study at the fringe of science). People make real decisions, about real people’s lives, utilizing these tools, and it is important to make sure that they are as excellent as possible.

Creating metrics (!) that matter

Rather than expecting people to stop utilizing metrics all together (which is unreasonable given the value they offer), we should focus on making sure the metrics are effective and accurate. Blind use of bibliometrics is as inadvisable as blind use of any other data source. Importantly, this does not mean the data are bad. The metrics we utilize to measure our own worth will never problem free, but we can work to make them as useful and problem free as possible; and this is exactly what we discussed at the recent workshop on Quantifying and Analyzing Scholarly Communication on the Web at Websci15.

One of the large topics at this meeting, similar Whalen et al discussed the potential to better understand impact by looking at how  ’topically distant’ the target article is being cited in. My response focused more on the potential utility of social media discussions between scientists, utilizing either their keywords, or the sentiment in their discussion to learn more about the target article.

This desire for more metrics is similar to several calls at the LSE blog, which suggest looking across metrics to better understand the facets of research impact, rather than striving for any single definition or measure of quality. Such a goal is easily achievable by integrating APIs into one hub by which to collect and analyze data about scientists (help us build it! 😀 ). Such a data source would be helpful not only in understanding research impact, but also ‘more fundamental’ questions (e.g., what team or personal characteristics leads to optimal knowledge exchange?).

Having many metrics and understanding the facets of research will allow users to form a fuller understanding of a particular researcher’s work. Such a system might also make it harder to cheat, as it becomes more difficult to manipulate all metrics (assuming they are measuring different things) than a single one alone.

More than simply a plethora of metrics, we need metrics that will help, rather than hurt, the scientific endeavor. This implies not only an ongoing effort to monitor the effects of metrics on the research enterprise, but also:

The inclusion of theory and the scientist in metric creation

It is possible to build many metrics, but the best metrics will probably utilize the accumulated scientific knowledge into their development. The metrics we utilize now are mostly simple counts of things, but there is much to be gained by utilizing the keywords, review rating, and other data to their full advantage.

Scientists are humans (even if we pretend not to be) and this implies a large number of predictable behavioral patterns and biases. Many fields (e.g., Psychology, Philosophy of Science, Sociology, Marketing, Computer Science, Communication) can be utilized fruitfully to better understand scientific communication and impact.

Some instances of things science knows about include how biases: affect the types of experiments scientists run, how they conceptualize their experiments, who they cite, how they search for information, what happens when something unexpected happens, how the group responds to controversy, how we can conceptualize conflict in science, and many other questions. These hypotheses can probably even be tested and confirmed among scientists, if people saw that it would be worth the time and effort.

More than just using science to make the most informative metrics, we can utilize it to try and understand the long term consequences of these particular metrics. It is generally accepted within psychology that living things (scientists probably included) look to maximize their reward while minimizing their input. Keeping this in mind in a small way while developing and implementing novel metrics can go a long way toward avoiding later problems for the field.

Ultimately, it is an empirical question how science works most efficiently, and I most definitely think we should use science to improve science, not just to build better metrics, but to build a better functioning science and society, generally speaking.

In summary

Metrics are a necessary and valuable aspect of the scientific endeavor, and thus are good in general, even if they sometimes miss nuance or are harmful in small ways.

It is because metrics are so necessary and widely spread that bibliometrics is such an essential study (unlike those who suggest it is a fake study at the fringe of science). People make real decisions, about real people’s lives, utilizing these tools, and it is important to make sure that they are as excellent as possible.

The study of bibliometrics can also benefit from bringing in the understanding that the rest of science has built up about: humans, groups, systems, knowledge exchange, knowledge creation, and literally over one  million other topics of study.

Most generally, we should be using science to improve science, not just to build better metrics, but to build a better functioning science, generally speaking.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s