From idea to science: Knowing when you’ve got a good idea

http://arstechnica.com/science/2013/03/from-idea-to-science-the-human-factor/

by Aurich Lawson

One of the great untold stories in science is the process of science itself. I don't mean stories about what scientists have discovered and what that discovery tells us; we (and many others) cover those every day. I also don't mean stories about the pure joy of discovery and the excitement of finding out that everything you thought you understood was total bollocks. We cover that here at Ars occasionally, and there are plenty of books on it if you're hungry for more.
What's missing is the background for these stories of discovery. How do you take an idea from its very beginning as a casual musing through to an actual research program? What's involved in that process? How do you sort out good ideas from bad and choose what to pursue and what to abandon? That is the story that I want to tell.
Since this is the story of science-as-a-process rather than science-as-a-result, I will be using myself as an example. I am, as some of you may know, a tenure track faculty member at a research institute in the Netherlands. Being a researcher in the Netherlands is not that different from being a researcher anywhere else, so a lot of what I discuss will be familiar to scientists everywhere. Since I recently hopped on the tenure track, I have the next few years to prove that I am able to not only carry out research, but to start and manage entire research programs. And, as yet, I have no research program to manage.
What will follow is a series of posts that document my success or failure in this particular endeavor. It will not be a blog as such; instead, I am aiming to give you a flavor of what goes through our minds when we come up with an idea, and what happens afterwards. It's not enough just to have an idea—it has to meet all sorts of criteria, only some of which have anything to do with science. As such, we have to refine and structure an idea into something that could become a coherent body of research. Then we have to convince other people that it's a good idea.
This article will mainly be background: what does it mean to be a physics researcher in the Netherlands? What conditions do I have to meet? What sort of time-scale are we talking about in terms of viable ideas? As for the rest of the series:

• The next post will discuss some of the specifics of the research program I want to build, the sort of physics that gets me out of bed in the morning.
• The third post will be about getting the resources I need to carry out that research. How do I sell my idea, and to whom?
• The fourth post will eventually describe the outcome of my salesmanship, and the perpetual question: what next?

There will be quite a delay between the latter posts in the series (it takes a while for grants to be evaluated). And, if I have not failed in the most abject manner possible, the process may continue from there. Success never ensues immediately after the money arrives.

The human factor

To understand how I choose between good ideas and bad ideas, we need to step back from actual physics and science and take a look at the structure of research community that I work in. Research takes resources. I don't mean money—all right, I do mean money—but it also requires time and people and lab space and support. There is a human and physical infrastructure that I have to make use of. I may be part of a research organization, but I have no automatic right of access to any of this infrastructure.
In the Netherlands, doctoral candidates are not students (nor are they, as many think of students, free labor). Instead, they are full-time staff with four-year contracts. What does this mean? First, it is very difficult to organically grow an idea from small scale research projects into something larger that has a doctoral student attached to it. The timing just doesn't work out with that four-year limit. It is difficult to begin a project with a masters or undergraduate student who will "just take a look" and then hand that over seamlessly to a PhD student should it appear promising.
This also has implications for scale. A PhD student has the right to expect a project that generates a decent body of work within those four years. A project that is going to take eight years of construction work before it produces any scientific results cannot and should not be built by a PhD student. On the other hand, a project that dries up in two years is equally bad. In other words, no matter what idea I come up with, I need to be able to say that all the candidates I hire should find enough material to write a thesis and graduate—no matter what the experimental outcome.
This means that any big idea I come up with also needs to be partitioned into chunks of the right size. If it can't, then it doesn't work in an academic institution.
Since all experimental results need to be thesis-worthy, the questions I want to answer should be open enough to accommodate failure. For instance, my ideas are often based on a single experiment: if we conduct experiment "a," we could measure property "b," and that would be so cool! But, what if "a" doesn't work? Does the student go home?
So, the core idea also needs to be structured so, should certain experiments not work, they still build something that can lead to experiments which do work. Or, if the cool new instrument we want to build can't measure exactly what I intended, there are other things it can measure. One of those other things must be fairly certain of success.
To put it bluntly: all paths must lead to results of some form.

A ticking clock

So I owe it to my students to come up with research ideas that will generate some success in the right time frame  But there's another human side—mine. As a tenure tracker, time is a big boundary condition. If I choose to forego PhD students, I could come up with an idea that involves eight years of construction before the first results might be expected (and first students hired). That would be acceptable in terms of meeting their requirements.
Unfortunately, my time would be up by then. Part way through the eight years, the director of the institute would look at my performance and promptly tell me to seek work elsewhere. The tenure track is there to give researchers a limited time to prove that they can do everything a tenured researcher should be able to. I must succeed with medium scale projects or many small scale projects rather than a big, long-term project.
That doesn't mean, however, these projects can't be pieces of some longer-term big project. I simply have to ensure that the project delivers results at all time-scales. In my case, the projects should be in the one-to-six PhD student range and should not require more than a year of instrumentation building (unless building the instrument can be counted as doing science). It should, from today, deliver lots of results within a four-year period. Or I'll be looking for another job.

Fitting in

Finally, there are institutional goals and resources. At the moment, I am in an institute that is going through a major restructuring and relocation. The institute will split up, with pieces moving to three different universities. Each piece will have a very different focus: energy research at Eindhoven, soft X-Ray optics at the University of Twente, and a free electron laser facility at University of Nijmegen. At the moment, I am part of the soft X-Ray optics group, so, my research should fit within that theme. On the other hand, as a tenure track researcher, I need to demonstrate some independence. My research still needs to be distinct from what the group already does.
These considerations, which are largely political in nature, are surprisingly important. They make the difference between enthusiastic institutional support (above and beyond what you are entitled to) and grudging assistance, limited to exactly what you are entitled to and delivered on someone else's schedule. In other words, the enthusiasm with which an institute supports a research program may very well be the difference between success and failure.
In the next installment, I will talk about the idea that I am planning: how it originated, and how I expanded it into a program that meets the requirements outlined in this post. As you will see, the raw idea, as expressed in its original form, doesn't fit well to the goals of the research group that I am a part of. My job became making it fit.