Ideally, science is performed by associations of people, having an “authoritarian-free discourse” (Eitel & Mlynek, 2014: 33; cf. Habermas,1968). This ideal roots back to the medieval university, which “has had a dominant position in modernized versions until about 1980” (Bleiklie, 2018: 2). On the one hand, autonomy has always been important to be creative and perform good research, on the other hand people, who work towards the same goal, need some kind of leadership and organisational structures (Eitel & Mlynek, 2014). This tension between autonomy and leadership is amplified by the fact that in recent decades, the institution of a university has become ever more market-like under the neoliberalist paradigm (Lorenz, 2012). Researchers need to be accountable towards taxpayers, politicians and other stakeholders. In order to receive funding and proceed in their careers, they face fierce competition, wherein quantitative indicators measure their performance (e.g. Anderson et al., 2007; Moosa, 2018). That is why the focus of research evaluation has been on accountability of the individual.

Evaluating performance based on solely a few quantitative indicators, serving as a proxy for the quality of work that researchers produced, has faced critique in recent years (for an overview see Fochler & De Rijcke, 2017). In socio-technical systems, performance measures do not only describe, but also prescribe behaviour (Desrosières, 1998). Instead of working towards a goal out of autonomous motivation (identification with the action), people find themselves acting out of controlled motivation (the need to hit the target; Heuritsch, 2021c)[1]. Acting out of controlled motivation to hit a target tempts people to adopt innovative gaming strategies to hit the aspired target and therefore bereaves the measure that is targeted of its meaning (Goodhart’s law[2]). Some studies report that gaming, including research misconduct, already constitute the norm in science (e.g. De Vries et al., 2006; John et al. 2012). Hence, the “bad apple narrative” has been rendered as too simplistic to explain behaviour which threatens research integrity (Haven & Woudenberg, 2021). In other words, while individual dispositions may increase the likelihood of research misbehaviour, the structural conditions and culture of an organisation, such as the scientific system, are much more predictive in shaping researcher’s behaviour on a large scale (e.g. Crain et al., 2013; Martinson et al., 2013 & 2016; Wells et al., 2014). For example, if members of an organisation feel treated fairly in terms of research allocation (distributive justice) and how processes are handled (procedural justice), they are more likely to respond with favourable behaviour (Martinson et al., 2006). With the increasing dependence on “external resources, whether from government or private industry” (ibid.: 3), and undergoing neoliberalist-inspired reforms to stay competitive for these resources, the university has changed from a “republic of scholars” to a “corporate enterprise” (Bleiklie, 2018: 2). Since the resulting research culture seems to foster the normalisation of scientific misconduct and a decrease in research integrity, it is time to scrutinise this culture, rather than the individual.

Action research is a social science methodology which is an interactive inquiry, aimed at understanding organisational culture and -processes and transforming them at the same time. In contrast to conventional social science methods, which aim at studying social settings without interfering in order to generate “objective” knowledge (Fricke, 2014: 222), action research is inherently participative and normative. It is participative in the sense that employees of the organisation under scrutiny are involved in many steps of the research, which may range from selecting the methods to analysis and learning from the findings. Action research inspires a collective (self-) reflection on otherwise tacit and unquestioned organisational norms, values, processes and structures. It is normative in the sense that action research is based on democratic values [3; Box 1]. The aim is the development of democratic structures and fostering democratic behaviour, by facilitating participative dialogues (ibid.: 215). Action research is a reflexive endeavour in that – through reflection together with those being studied – it shapes the practices and perspectives of everybody involved. That is how through action research organisations may transform towards more democratic institutions.

<aside> 💡 Box 1 – Deliberative & participative democracy

Most democratic nations employ a representative democracy, where representatives are elected and entrusted to carry out the business of governance. These representatives form the government. Dahler-Larsen (2019) refers to this type of democracy as the government model of democratic systems.

As opposed to representative democracy, in a deliberative & participative democracy, all members of the democratic institution may directly participate in decision making processes through participative procedures, such as dialogue (e.g. Pateman, 1970). Dahler-Larsen (2019) refers to this type of democracy as the governance model of democratic systems.

Deliberation and pluralism are key democratic principles, and even more important in a participative democracy (e.g. Habermas, 1968; Dahl, 2006; Sunstein, 2018). The legitimate diversity of opinions and interests, resulting in debates and controversy, is democracy’s biggest challenge at the same time (e.g. Foroutan, 2019). Especially in a world, which becomes increasingly volatile, uncertain, complex and ambiguous (VUCA), a democratically competent agent is required to have ambiguity tolerance (Habermas, 1968: 128). This refers to the acceptance that in a (socially) complex world there may not just be an either/or, but a plurality of seemingly contradicting perspectives.

As compared to the government model of democratic systems, involving hard laws, the modes of regulation in the governance model are soft laws. These involve guidelines, recommendations, agreements, indicators, standards, and benchmarks, none of which are binding (Dahler-Larsen, 2019: 191). Governance networks may therefore be more dynamic in navigating in complex environments, while depending on the active consent of their participants and their ongoing deliberation (ibid.: 191).

</aside>

The evaluative inquiry (EI [5]) may be understood as an application of action research (e.g. Coghlan & Brydon-Miller, 2014). Researchers from the Center for Science and Technology studies (CWTS) have adapted the EI to the context of the evaluation culture in academia and describes the methodology in a series of blog articles [6]. While “mainstream evaluation metrics tend to understand academic value as performance”, the EI also investigates “other valuable elements of academic value trajectories” [6]. It does so by shifting the focus from “talking about” to “thinking with” [7] scientists and stakeholders and thereby employing a combination quantitative and qualitative methods, depending on what fits the specific evaluation purpose best. The EI is not only a participatory approach to evaluation, but also a reflexive one. It is so in two ways: First, those conducting an EI reflect together with those involved in a respective evaluation on the context they are producing knowledge and what they value about their work (output). The EI “supports academic units in crystalizing goals, missions, visions and strategy, taking stock of the diversity of output, making the multitude of stakeholder relations visible, and listing staff opinions about the academic organization.” [6]. Hence, finding a common purpose [8] and narrative on how the organisation performs research and organises itself, is of crucial value of this step. Second, the EI promotes a continuously ongoing reflection on the reflexiveness of evaluation procedures (i.e. their feedback effects). The evaluative inquiry therefore “values both, the processes and the outcomes of evaluation” (Coghlan & Brydon-Miller, 2014: 1) and delivers valuable learning opportunities for everybody involved.

“Evaluative inquiry is a way of fostering individual and team learning within an organization about issues that are critical to its purpose and what it values. It involves an intentional process of framing important questions, systematically gathering information relevant to the questions, and using the information to draw credible conclusions that can shape practice.” (Parsons, 2009)

The study of feedback effects of the use of metrics in performance measurement on the research behaviour, -culture, knowledge production processes and research quality is called reflexive metrics. Reflexive metrics is a branch of science studies and comprises of the sociology of (e)valuation and the sociology of quantification. There have been a series of studies under the umbrella of reflexive metrics which may set the basis for an EI and a transformation of research culture in astrophysics. Heuritsch (2019) investigated what astronomers themselves value as good quality research practices and output. The author found three quality criteria: (1) good research needs to push knowledge forward, which includes studying a diversity of topics and making incremental contributions; (2) The research needs to be based on clear, verifiable and sound methodology that is (3) reported in an understandable and transparent way. This includes the sharing of data and reduction code.

In a follow-up study, Heuritsch (2021a) studied the “organisational hinterland” (Dahler-Larsen, 2019) of astronomy to understand what performance indicators measure, how they diverge from the astronomers’ definition of quality and how this discrepancy affects research behaviour. In a nutshell, Heuritsch (2021a) finds evidence for the over-emphasis on performance measured by publication rate, reception of external grants and telescope time, leading to gaming strategies to score well on those indicators. These are found to be a response to the dissonance between cultural values (producing qualitative research that genuinely pushes knowledge forward) and the institutional objectives imposed to have career in academia (scoring well on indicators). In other words, there is a discrepancy between what indicators measure and the astronomers’ definition of scientific quality – the so-called evaluation gap. Gaming strategies then give the appearance of compliance with cultural values, while using institutionalised means to achieve a good bibliometric record in innovative ways, such as salami slicing, cutting corners or going for easy publications. The author finds evidence for astronomers prioritizing quantity over quality of publications.

Both studies (Heuritsch 2019 & 2021a) were conducted by means of qualitative interviews. Following up on Heuritsch (2021a), Heuritsch (2021b) performed a quantitative study, surveying international astronomers worldwide. The author found that publication pressure explains 10% of the variance in occurrence of misconduct and between 7 and 13% of the variance of the perception of distributive & organisational justice as well as overcommitment to work. Further, the survey showed that the epistemic harm of questionable research practices on research quality should not be underestimated.

The finding that publication pressure decreases the perception of a fair organizational culture and increases the likelihood of scientific misconduct stems from the fact, that “printed publications have been considered the sole publishing objective of re­search for centuries” (Breuer & Trilcke, 2021: 5). We therefore consider a meta-reflection on publication formats as a vital aspect for an evaluative inquiry. A shift of a focus away from the traditional format of publication is already under way due to digital technologies (ibid.). It seems vital to include researchers themselves to investigate how “digital technologies can support and change the growth of scientific knowledge, its reproducibility and the accessibility of said knowledge” (ibid.: 4). Such an endeavour aims at the development of context-/ discipline specific formulation of “clear criteria for the recognition of different dig­ital publication formats as attributable and remunerable scientific practice of the type ‘publishing’” (ibid.: 4; italics in the original).

The aim of this study is to reimagine cultural aspects of research in astronomy, such as publication formats and ways of being assessed. While previous literature has pointed to structural problems in academia and how they affect knowledge production processes and research integrity, the search for contextual solutions is a novel approach. Building up on these studies, especially the ones conducted about the field of astronomy, this study is based on qualitative interviews and open survey questions. The goal is to understand astronomers’ perspectives on cultural changes that are desired and/or already under way and to work out specific recommendations about how to support these. These may provide a basis for any future action researchers, working directly together with astronomers to transform research culture into a democratic and flourishing one.

This paper is structured as follows: First, in section 2 we introduce studies regarding the status quo of academic culture, which operates in a neoliberalist paradigm entailing metrics as performance measures. This section also includes an overview of recent calls for a culture change and what that could entail. The method section refers to the sample selection & procedure and describes the (interview) questions this study is based on. The result section is structured according to the three “miracle questions” asked to our study participants: reimagining alternative output formats, evaluation criteria and research. The discussion section introduces recommendations, whose implementation may lead to a more participative research culture, based on this study’s results, literature and existing initiatives. Finally, we draw our conclusions, implications thereof and give an outlook for future research in section 6.

Footnotes

[1] In short, autonomous motivation comes from identifying with the value of the activity, while controlled motivation is instrumental to acquire or avoid external rewards or punishments (Gagné et al., 2010).

[2] “When a measure becomes a target, it ceases to be a good measure.” (https://en.wikipedia.org/wiki/Goodhart's_law)

[3] For our understanding of “democracy” see Box 1.

[5] On a side note: In participation research, the abbreviation EI actually denotes “employee involvement”, which is a convenient coincidence.