How not to ask your research question (and what to do about it)

In software engineering, the structure of research theses, most notably dissertations, is straightforward: (1) Formulate a research question, choose a method, build a theory, then (2) generate at least one interesting hypothesis, choose a method, and test the hypothesis as part of the theory’s attempted validation. A dissertation can do both parts 1 and 2 or just one of them, relying on or leaving stuff to others. The benefit of this structure is that it will be easily recognized by other researchers and make it eas(ier) to write great papers.

Many students find it hard to cast their work in this light. I suspect this is, because most have been trained as engineers and not as scientists. The (anti-)pattern is always the same: From some practical experience, the student has a hunch about an interesting problem and then tries to cast his or her research as building a solution that solves the problem. Here is how it might be expressed in a discussion:

• I got this really cool idea for a tool and want to implement it
• From my experience, programming this way creates less bugs
• I think this workflow will make developers more effective

The assumption is that finding a solution constitutes research in itself, enough of it, so that the problem solver can be awarded the desired title. However, the only proof that their idea is worth something is in its appeal to practitioners or experts. The first person to say “I don’t believe this improves anything” can throw the student off-balance. For this reason, the work needs to framed differently. Such reframing is not hard and also follows a pattern.

For part (1) introduced earlier, ask yourself: What problem is my tool trying to solve? Then, if this is a problem, who could tell me more about it? Now switch to theory building: You explore the problem by asking others, where asking others can mean literature review, materials gathering, and interviews. Such theory building should follow a defined method or someone will claim you are making up stuff. Fortunately, there are plenty of methods to chose from, starting with simple qualitative surveys all the way to grounded theory and case study research. Pick any one, just don’t invent your own method.

For part (2), look at your theory. If you derived it well, it should already give you ideas for propositions. A good proposition or hypothesis can be answered with yes or no, lending itself to some of the “hard” validation methods like controlled experiments or hypothesis-testing surveys. If they are softer and don’t lead to a binary answer, you can probably still evaluate them continuing on with case study research, of which some researchers claim it does not only let you build theories but also evaluate them. The key insight here is, again, that you should not define your own method but use one off the shelf.

It is a supervisor’s job to help students frame their thoughts in such a way that they fit the structure described above and generate useful (and publishable!) research.