This study is based on 15 semi-structured interviews with international astronomers from various career stages and a web-based survey, distributed among astronomers worldwide. The sample selection and procedure is described in detail elsewhere (Heuritsch, 2021a & 2021b, respectively).
Interview questions used for this study involved questions, such as “What issues do you think need to be improved to guarantee better science?”, “Do you feel that you are given the chance to question how science is performed?”, “When did you have your first encounter with the way science is performed and assessed? How did that compare with your initial motivation to become a scientist?” and “When it comes to evaluation of research(ers), what would you suggest to be measured?”. The web-based survey included, next to a comprehensive collection of quantitative questions, which results can be found elsewhere (Heuritsch, 2021b & 2021c), three so-called miracle questions. Miracle questions are open questions, which inspire the respondent to think outside the box. These miracle questions aimed at reimagining output formats (MQ1: “In an ideal world, how would you like to present your research (results) if it didn't have to be in form of a paper?”), reimagining evaluation criteria & procedures (MQ2: “In an ideal world, what would be the best way to assess your work performance?”) and reimagining research (MQ3: “Imagine you were given 1 Billion Dollar for research - how would you do research differently?”). 2011 astronomers completed the survey fully and we received the following number of responses to the miracle questions: 1201 (MQ1), 1177 (MQ2) & 1205 (MQ3). Both, the responses to the interview- and the miracle questions, were coded in MaxQDA according to Mayring’s qualitative content analysis (Mayring, 2000).
Interviewees (numbered IW1 to IW15) and survey respondents may be referred to as study participants or respondents. The quotations underlining the results were not chosen based on their representativeness, but their expressiveness. Hence, the number of represented quotations is also not proportional to the number of responses to a particular subject. The source of the quotations (interviewee or answer to a specific miracle question) are stated in square brackets. Since we performed a qualitative analysis, it doesn’t make sense to state exact respondent numbers when introducing topics that study participants mentioned. We therefore only mention rounded numbers for illustration, where we find it appropriate.
With the limited time and resources available, we put our best effort to follow our own recommendations from the discussion section to make this paper a bit less linear. The more interactive version of this paper can be found here, which includes many more quotations and anecdotal statements in order to do justice to the rich and many responses, study participants invested their valuable time in.
Table of Contents
<click on the content to navigate to the respective section>
2.1. Indicators as Quality inscriptions
2.2. Publish-or-Perish – a part of a neoliberalist academic culture
2.3. Calls for a transformational culture change
2.4. Democracy calls for Reflexive Evaluation: Evaluative Inquiry
4.1. MQ1 – Imagining different outputs: Re-evaluing the paper
4.2. MQ2 – Re-evaluing evaluation
4.3. MQ3 – Transformation: Re-evaluing values