Do engineering researchers care about truth?

So ICSE, the top software engineering conference, rejected our paper, again. The reviewers were actually quite positive: high-quality work, little or no flaws, interesting. One of the reviewers found the paper’s results surprising, asked for more details, and suggested new research directions. The final conclusion of both reviews, however, was the same: The work has no merit because it only explains the world, it does not improve it.

Our paper provides a high-quality model of a key aspect of programming behavior in open source, basically the modeling behind this earlier empirical paper. As such, it is a descriptive empirical paper. It takes a large amount of data and provides an analytically closed model of the data so that we can explain or predict the future (better). That’s pretty standard operating procedure in most of natural and social sciences.

Suggest to a physicist that a paper that explains how a particle spins can only be published if in the same paper the physicist shows how to make the particle spin backwards. Or contribute to human well-being. He or she will give you the finger. Suggest to a sociologist that it is not enough to understand medical practices in a hospital, that he or she in the same paper also has to provide empirical validation on how to improve these practices. You’ll be out the door before you can say I’m sorry.

Only in engineering research do we seem to see no value in seeking knowledge and truth for its own sake. Only in engineering research is merit defined by how some work improves the world. Maybe that’s why many don’t consider it a science. It seems obvious to me that before improvement comes understanding, but in repeated experiences with ICSE and elsewhere, reviewers consistently require that both of it has to be packaged into the same 10 page paper. I think this is ludicrous.

I can accept an argument that some particular knowledge is of little merit because it is uninteresting. I cannot accept the requirement that deep and interesting insight only gains publishable merit by being combined with its application in some context.

Happy new year and a successful 2012 to everyone!

Dirk

PS: Some explanation for those not working in computer science / software engineering. ICSE is the top conference of the field, by most measures, and ranks higher than the journals. (Which is another weird thing about computer science, but not of concern here.) Computer scientists don’t like to read (a lot) and hate discursive papers (try publishing Grounded Theory work) so many publishing outlets have stringent page requirements, typically 10 pages or 10000 words.

Posted on

2012-01-05

1. Research, 1.1 Research (General)

Subscribe!

Comments

Leave a ReplyCancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Ewan Tempero

2012-01-10

Hi Dirk, You do have to wonder what some reviewers want in an acceptable paper sometimes! But let me pick up on your comment “(Which [the CS/SE view of conferences vs journals] is another weird thing about computer science, but not of concern here.)”
I disagree. I think that this exactly what is of concern here. This kind of “it must be novel, change the world, and really cute before I’ll recommend acceptance”-attitude is, I believe, a consequence of the fact that conference publications are held in such great esteem (deserved or otherwise). A paper accepted at a conference must fit within the page limitations (and all the flaws that introduces) and its fate determined by a snap decision by someone who has 5 (or 25) other papers on the pile. The other disciplines publish in journals, where it’s more likely to be 4 weeks to review 1 paper. I wonder which system is likely to yield better decisions?!
We do have good journals (TSE, ESE, TOSEM, JSS, etc), but unfortunately, the conference game is the one that we get judged on so that’s the one we play.

Reply
1. Dirk Riehle
  
  2012-01-10
  
  Hi Ewan,
  conferences vs journals is an interesting discussion. More mature (traditional?) disciplines all use conferences for community and journals for archiving. Which makes sense to me.
  CS does it largely the other way round. Let’s think it through strategically to answer the question: Where should you publish?
  Case 1. You are invested in a conference. So you tell your students to publish there. You defend high selectiveness in the PC meeting. You keep quality and supposedly reputation (and “impact factor” if measured) up. The cycle keeps repeating in your realm.
  Case 2. You are not invested in a conference, maybe because you never published there before or maybe because you are changing research subjects. If you are senior, you can travel there without having a paper. If you know what you are doing, you can make yourself known without a paper there. So you publish in journals that are largely equivalent. I, for one, don’t mind the “impact factor” difference between ICSE and TSE. And you still go to the community gathering known as conference. Thus, you are pushing journals up, conferences down.
  Case 3. You don’t care, because you are tenured. I tell my PhD students that for every good journal paper they can go to an (quality equivalent) conference. So I’m driving them to journals. In general, I think journals are slightly underrated in CS and conferences are overrated. It can only change, IMO, and move toward the experiences of the other disciplines.
  Thus, IMO, the smart money is on journals, not on conferences. It is a drawn-out fight because of existing investments, but conference-proponents can only lose.
  Cheers,
  Dirk
  
  Reply
Brian Burg

2012-01-07

I’m saddened to hear that, but not surprised at all.
In fact, I would second Andy’s notion that you should publish elsewhere. Not only is ICSE conservative, reviewers are biased towards whimsical novelty instead of empirical results. (Don’t believe me? Try to find some convincing evidence that anyone in industry cares about code clones, for example.) Now, “empirical” SE conference communities have their own biases and issues, but at least they are not in denial about what happens in the real world.
If your goal is to impact the professional software engineering field, you may have the best success by submitting a blog post to Hacker News or by taking out paid Facebook advertisements to lure software developers to your blog/tool homepage. (I’m mostly serious.)
If you want to impact the software engineering research, well, then I hope you can adequately dress up/obfuscate your findings to make it into such a conference. Another alternative, which seemed to work well for my last empirical line of work, is to call out prior work as having wrong/shaky assumptions, and then reveal your empirical evidence for/against the assumptions.

Reply
Fabian Christ

2012-01-06

Thanks Dirk for this nice post. I agree with your view on this. Additionally, I find it frustrating when reviewers judge a paper in this way. Too many reviewers think that their role is to decide whether or not an idea or a work is worth of being accepted. In my opinion reviewers should only look for solid scientific work presented in the paper and not judge the content that much. This is not the reviewers job. The content should be discussed as part of a scientific discourse within the community.

Reply
James

2012-01-06

Ahh Dirk, well I’m in good company!

Reply
1. Dirk Riehle
  
  2012-01-06
  
  Yep, this was the real goal of the post: To gain and share some sympathy! 🙂 As a senior ICSE leader once confided in me: “I’m looking back on a long and distinguished history of ICSE submission rejections.” 🙂
  
  Reply
Sergut

2012-01-05

(SE is not my field so what I am going to say might not make perfect sense.) I wonder if this is an issue of computer science vs software engineering. Scientists describe the world while engineers change it, scientists search truth while engineers create things that solve problems.
Maybe the SE venues are not the right venues for a “scientific” paper on the sociology/psychology of programming practices?

Reply
1. Dirk Riehle
  
  2012-01-06
  
  I think the problem is the same in all of computer science, which many question as “a real science”. (Anything with the name “science” in it is not… so the proverb goes.) We will find a suitable venue, I have no doubts about it. The larger issue though is that by comparison computer science is still a young discipline and its notion of what is proper research is still evolving.
  
  Reply
Andy J. Ko

2012-01-05

To play devil’s advocate, it’s possible that the strong bias towards intervention in the software engineering research community has something to do with the fact that it’s practice is still evolving. In physics, we don’t really expect particle behavior to change, and so investigating how they behave is something reasonable to do over the course of several decades. Researchers may view open source communities more as a moving target and therefore one to intervene in, rather than describe.
Personally, I have the same frustrations with the SE research community. Far too many brittle automated tools, far too little science in search of theories that explain and predict how software engineering projects unfold.
That said, ICSE has always been pretty conservative. There are lots of other great SE-related venues that not only accept descriptive work, but know how to review it properly, including ESEM/ISESE, VL/HCC, the CHASE workshop (which is frequently co-located with ICSE), and even CHI, which has a fair number of papers about open source communities every year. These obviously aren’t the top venues in SE (CHI is the top HCI venue), but if you want the work read, they’re a great fit.

Reply
1. Dirk Riehle
  
  2012-01-05
  
  Naturally, we’ll move on. But it bugs me that the “leading” conference is really so conservative, as you say to be the trailing conference.
  The particular result is pretty fundamental so I don’t expect it to change. However, in any science, speaking as a pragmatist, folks know that knowledge is only considered validated as long as the proof/validation stands. Theories fall all the time despite initially promise, in physics, psychology, and elsewhere.
  I also think that SE is suffering from too small a research method repertoire. We should borrow and adapt from the natural and social sciences. But then, many reviewers are uncomfortable with that. So it will take time.
  
  Reply
Jesus M. Gonzalez-Barahona

2012-01-05

I cannot agree more with your post. I also feel that in many cases we’re trying to build a discipline without knowing almost anything about the ‘natural laws’ of what we try to build. We’re always pressed for the ‘So what?’ syndrome, and it is very difficult to publish a mere descriptive (yet detailed, data-backed, novel, important) paper about how reality works…

Reply