In spite of more than a decade of policy and discussion around research impact and industry engagement, the prevailing paradigm for career progresion and researcher KPIs in Australian universities is still focused on retrospective - ex-post - counting of outputs, such as the number of publications, grants received, and scholarly citations. While this approach served well in a world where research quantity and quality were the primary concerns, it falls short when it comes to managing research for impact and engagement with industry partners.
As we move into an era where the value of research will increasingly be determined by its ability to generate real-world impact and address complex societal challenges, and wherein research dunsing will increasingly be contested across public and private sectors, there is a pressing need to rethink our approach to research performance management inside universities. Rather than merely counting the outputs if research, we must shift towards a more holistic assessment that takes into account both ex-ante and ex-post factors, encompassing the entire research lifecycle.
This new paradigm requires a fundamental reorientation of our understanding of research management, moving away from discrete projects and towards ongoing programs of research. By adopting a more dynamic approach that includes planning, monitoring, assessment, learning, and adjusting, we can create a more effective and meaningful system for evaluating research performance.
The Limitations of the 'Counting' Paradigm
The current paradigm of research performance management inside universities predominantly focuses on counting research outputs, such as publications, grants, and scholarly citations. This has several limitations that hinder its effectiveness in fostering research impact and industry engagement.
Firstly, the emphasis on counting outputs creates a narrow view of research success, prioritising quantity and so-called quality over impact. This approach inadvertently encourages researchers to focus on producing more publications, at the expense of other university missions, like tralsation of research. The publishing-centric culture that follows is detrimental to the long-term health of the research ecosystem.
Secondly, the current counting paradigm overlooks the diverse range of activities that contribute to research impact and industry engagement, such as collaborations, knowledge exchange, and technology transfer. By focusing solely on a discrete set of easily quantifiable outputs, the rich and multifaceted nature of research impact pathways is not adequately captured. Moreover, the reliance on counting research outputs can lead to an oversimplification of research performance, ignoring the nuances and complexities of individual research projects and disciplines. This approach fails to account for the different contexts and requirements of various research fields, leading to misleading comparisons and assessments.
Thirdly, while we may have a relatively coherent picture of quality - via proxies such as peer reviewed funding and citations - there is an much wider set of activities that fall jnder the banner of engagement and impact. The diversity alone makes it resistant to counting based assessment. Let alone the intircate system of equivalences that would need to be sustained to make this work
The counting paradigm is ill-suited for measuring industry engagement, as the success of such collaborations cannot be easily reduced to a simple metric or number. Industry partnerships involve a wide range of actors, activities, objectives, and outcomes, and reducing these complex relationships to a single countable output does not accurately reflect their true value and impact.
Embracing a New Approach to Research Performance Management: Combining Ex-Ante and Ex-Post Assessment
As the limitations of the current paradigm become apparent, it is essential to explore new approaches that better capture the true value and impact of research. This requires the combination of ex-ante and ex-post assessment, which shifts the focus from merely counting outputs to a more comprehensive understanding of research impact and industry engagement.
The new approach to research performance management encompasses five key elements:
- planning
- monitoring
- assessment
- learning, and
- adjusting.
By placing a strong emphasis on planning, research teams can set clear objectives and goals, which serve as a foundation for monitoring progress and assessing progress throughout the research process. This allows for a more proactive approach to research management, where the focus is on anticipating and addressing potential challenges and opportunities rather than outputs.
Monitoring therefire is a critical component of the new approach, as it enables research teams to track progress against their objectives and adapt their strategies as needed. This ongoing monitoring process, in turn, feeds into the assessment phase, where research performance is evaluated not just by counting outputs, but also by considering the extent to which objectives have been met, decisions have been taken, obstacles overcome, plans refined etc.
Learning, then, is another essential aspect of the new research performance management approach, as it encourages researchers and institutions to reflect on their experiences, identify areas for improvement, and share best practices. This continuous learning process facilitates the growth and development of research teams and fosters a culture of innovation and excellence within research institutions.
Finally, this allows for continual adjustments, which involves making changes to research strategies and objectives based on the insights gained from the previous stages. This continuous cycle of planning, monitoring, assessment, learning, and adjusting enables research teams to remain agile and responsive to the evolving research landscape.
Viewing research as an ongoing program rather than a series of discrete projects promotes a more holistic understanding of research performance and impact. It acknowledges the complexities and inter-dependencies of research activities and recognises that achieving meaningful impact often requires long-term, sustained efforts, changes of plan and direction, and a high degree of complex decisions being taken along the way.
Embracing a combination of ex-ante and ex-post assessment in research performance management offers numerous benefits, chief amongst them a greater alignment between research activities and strategic objectives. By adopting this approach, research institutions can better position themselves to address complex, real-world challenges and deliver research outcomes that have a lasting, positive impact on society
Adapting to the Changing Research Landscape
The most significant change that needs to be embraced is the shift away from annual review cycles to ongoing monitoring, which better reflects the dynamics of research.
Annual cycles fail to capture the full scope of research and squander learning opportunities. By contrast, real time monitoring and feedback enables a dynamic understanding of research progress, allowing for timely adjustments and interventions as needed. This approach requires a substantially more proactive and responsive research management approach, more akin to coaching than assessment.
As always, measuring research impact poses insurmountable challenges - the challenge for esample posed by the infinite number of activities required to achieve meaningful impact, which makes it difficult to establish a standard set of metrics that can be universally applied. Or the challenges of determining equivalences between different activities when the nature and value of research activities can vary greatly depending on the context and objectives of the research.
To overcome these challenges, research institutions must simultaneously move away from the idea of performance measurement altogether, and adopt a flexible and adaptive approach to performance management. This entails embracing qualitative assessments that complement quantitative metrics, as well as recognising the importance of context and objectives in evaluating research. By doing so, institutions can develop a more nuanced understanding of research performance that takes into account the diverse range of activities and their respective contributions to achieving impact.
Insights
Adapting to the changing role of university research requires a shift towards ongoing management and a shift away from measurement. By acknowledging the complexities and challenges inherent in measuring research performance, institutions can develop more effective performance management strategies that ultimately drive more impactful research outcomes.
The following guides can help institutions in implementing the new paradigm by focusing on qualitative assessment, research objectives, and continuous improvement.
- Embrace qualitative assessment: Quantitative metrics do not provide a complete picture of research impact. Institutions must incorporate qualitative elements in oerformance management to consider the broader societal and economic benefits of research. Adopting a mixed-methods approach, institutions can gain deeper insights into the full range of research outcomes.
- Focus on research objectives: Establishing clear and well-defined research objectives is essential for effective performance management. These should serve as the foundation for planning, monitoring, and evaluating research activities. By aligning activities with objectives, institutions can ensure their research is purpose-driven and therefore can be managed against its objectives.
- Foster a culture of continuous improvement: Institutions must cultivate an environment that supports coaching and ongoing learning and improvement. Encourage researchers to engage in regular reflection and self-assessment, identifying areas of growth and opportunities for development. Additionally, provide access to professional development opportunities that enable researchers to enhance their skills and knowledge as they shift towards the new paradigm.
- Develop supportive structures and processes: Effective implementation of the new paradigm requires the development of structures and processes that facilitate interdisciplinary collaboration, ongoing monitoring, and adaptability. Create cross-cutting organisational units and implement flexible research management systems that enable researchers to work together, share knowledge, and respond to emerging opportunities and challenges.
- Communicate and engage stakeholders: Open communication and active engagement with stakeholders are crucial for the successful adoption of the new approach. Share the institution's vision for research performance management, highlighting the benefits of the new paradigm for researchers, funders, and the wider community. Collaborate with stakeholders to develop shared understanding and commitment to the new approach and acfively include rheir feedback into the process.
These are just some of the many steps institutions need to take to effectively start to implement the new research performance paradigm.
The shift from a traditional ex-post counting paradigm to a more holistic approach that combines ex-ante and ex-post assessments in research performance management is crucial. This new paradigm allows institutions to better understand research impact and industry engagement, and to become more adept over time.