back to the Eurescom home page

 

 

mess@ge home

Table of contents
of the current issue

Selected Highlights
Mobile applica-
tions - More
than a hype

 

Apple's App
store and
beyond

 

A platform for
portable apps -
Open source
initiative BONDI

 

Mobile Web applications -
The approach
of Research in
Motion

 

Challenges for
mobile applica-
tions - Strategic considerations by eMobility

 

Wanted: the unbiased reviewer

                                                                           

David Kennedy
Eurescom
kennedy@eurescom.eu

There is a growing problem in the assessments of project proposals in public funded research programmes: the search for unbiased reviewers is in fact making the process biased. At least this is the case in the ICT world. 

We now accept that the networks of the future will have tight bindings to the services of the future, and the services of the future will be bound to the application domains. Then, who is independent in this picture any more? 

Limitations of academic reviewers 

The academics have long been a major group in this process. They can surely add technical competence and a long term view into the assessment process. However, while they do their best to appreciate other viewpoints, they do lack the business dimension.  

One key problem with the academic community is that they consider many things from the new science perspective and the newness has a very high value in their communities. However, for industry, much of the innovation is in the application of emerging sciences and technologies. Here we have a problem. Can you fairly assess my proposal, if your perspective of innovative work is inherently different from mine? 

This problem is systematic across many research programmes, as the level of innovation, and therefore applied research, that is needed to take an idea from the point where the academic community has fully explored it to the point, where it might be taken up and used in the marketplace, is often way more than any side can predict. This can take time and can often result in academics being dismissive of proposals as they have been familiar with the technology discussion for several years and believe that there is no more research to be done, while the industrial community does not have it in a form that can be directly used. 

Biased reviewers from industry 

Industry players are also biased in that they are directly involved, and they might not be the best to impartially evaluate their own or their competitors proposals. However, when the industrial evaluators declare their interest in any proposal and therefore exclude themselves form assessing it, we are logically increasing the risk of their competitors then assessing the proposal.  

So where do we go? Some have said that the blind evaluation of proposals would be appropriate, where the project is stripped of identifying characteristics and is assessed as an abstract proposal of work. The risk here is that it might be the right work, but it is possible that the project is being proposed by people who cannot deliver on the promise or do not really have the business relevance to exploit the results in the end. 

Are consultants then the least evil party here, who can be trusted to be fair and impartial?  Many would argue they are not, as they have an obvious interest in ensuring they continue to secure work in the future and therefore may have more agendas than any of the other players. 

So we can rapidly reach a point where we can persuade ourselves that there is no fair way to assess proposals for public programmes, as a set of impartial reviewers is impossible to find. Clearly this is not going to work ,so what can we do? 

A new approach evaluating proposals

Well I think we need to take a step back and consider the way these programmes are funded. First the percentage of public money is significant so the agents of the authorities need to assess where their money is invested.  They chose to do this through a panel of assessors but, in doing this, they have a responsibility to ensure the panel are qualified and the interests are mixed so that we do not get the bias of any one interest group showing. This is being achieved to a fair degree today. 

What is not being achieved is the recognition of the other half of the investment in the programme. Here the industrial players invest significant time and effort in creating programmes and project proposals and they, because they are making similar level investments, should provide their panel of appropriate experts and share in the reviewing. 

If we could have all proposals evaluated by a mix of evaluators, where some represent the values of the authority and some represent the values of the industry, then we will have both investing parties well represented in the decision on the investment.   

As we now enter the new era of Public Private Partnerships, we should strive to ensure the partnerships start with the programme definition and the selection of the projects and not just after the projects are selected.

Please send us your comments on this article.