Once again natural vs. engineering sciences struggling over definitions #FSE2014

I’m in Hong Kong, attending FSE 2014. I had signed up for the Next-Generation Mining-Software-Repositories workshop at HKUST but missed it for (undisclosed) reasons. Apparently there were two main topics of dicussion:

• Calls by colleagues to make mining work “useful” rather than “just” interesting
• Calls by colleagues to build tools rather than “just” generate insight

Both issues are joined at the hip and an expression of a struggle over the definition of “what is good science” in software engineering. As someone who started out as a student of physics, I have an idea of science that views “interesting insights” as useful in their own right: You don’t need to build a tool that shows your insight improves the world. On the other end is the classic notion of engineering science, where there is no (publishable) research if you don’t improve the world in some tangible way.

Underlying this classic claim for having to improve the world right away is also a power struggle about what is a publishable result. By requiring researchers to show how some insight also improves the world, proponents of this view implicitly lower the bar on the required validity of their results. If you build a tool and it looks useful, you can get away with a less than perfect empirical validation as to its usefulness. Software engineering conferences are still full of papers with questionable validation.

I contend that you need both (a) interesting insights into how software engineering works and (b) operationalization of those insights, typically as atools. Both should be acceptable papers in their own right; you shouldn’t have to roll them into one. So you can first publish a paper about an insight and follow-up with a second paper on a tool that utilizes this insight. This does not free the author of the tool paper from having to validate the usefulness of this tool, say by experiment, but he or she does not have to develop the original insight themselves any longer.

Both types of papers, one about an empirical insight and one about a tool that utilizes the insight, are useful research contributions and by separating them out, we make the work much more tractable.