Quality assurance
Quality assurance (or ‘QA’) is about verifying that the analysis is correct - are we using the right methods, does it give the numbers we expect, etc.
Quality assurance should take place for all pieces of analysis that go beyond the team - where they go to customers or stakeholders. And especially when it is going (or will ultimately be going) to Ministers or Senior Officials. There is no single set way to undertake QA, for each piece of work you should consider what the appropriate level of QA - but as a rule of thumb the larger or more important the work, the more QA you probably need to do (but it’s not linear!).
The central guidance on quality assurance in government analysis is set out in the Aqua Book. The Aqua Book mandates that each department has a process in place for managing key analytical models, search “business critical models” (including the quotation marks) on the intranet for the Cabinet Office’s process and guidance.
The Aqua Book sets out four principles for QA:
- Proportionality of response: The extent of the analytical quality assurance effort should be proportionate in response to the risks associated with the intended use of the analysis. These risks include financial, legal, operational and reputational impacts. In addition, analysis that is frequently used to support a decision-making process may require a more comprehensive analytical quality assurance response.
- Assurance throughout development: Quality assurance considerations should be taken into account throughout the life cycle of the analysis and not just at the end. Effective communication is crucial when understanding the problem, designing the analytical approach, conducting the analysis and relaying the outputs.
- Verification and validation: Analytical quality assurance is more than checking that the analysis is error-free and satisfies its specification (verification). It must also include checks that the analysis is appropriate, i.e. fit for the purpose for which it is being used (validation).
- Analysis with RIGOUR: Quality analysis needs to be repeatable, independent, grounded in reality, objective, have understood and managed uncertainty, and the results should address the initial question robustly. In particular, it is important to accept that uncertainty is inherent within the inputs and outputs of any piece of analysis. It is important to establish how much we can rely upon the analysis for a given problem.
The Aqua Book’s RIGOUR model provides a helpful framework for thinking about how to go about QA’ing your project.
| Element | Aqua Book explanation | Possible QA considerations |
|---|---|---|
| Repeatable | For an analytical process to be considered ‘valid’ it might reasonably be expected that for the “same” inputs and constraints the analysis produces the “same” outputs. It is important to note that different analysts will consider the analytical problem differently, potentially resulting in differing results, however if any one approach is repeated the results should be as expected. | Can a colleague repeat the analysis? If they use a different (but equally valid) approach can they get the same figures (e.g. do a+a and 2*a give the same result)? |
| Independent | To produce analysis that is free of prejudice or bias. In doing so, care should be taken to appropriately balance the views across all stakeholders and experts. | Would the analysis stand up to external scrutiny? If someone else approached the initial question or problem would they use a similar approach to conduct the analysis |
| Grounded in reality | Quality analysis takes the commissioner and analyst on a journey as views and perceptions are challenged and connections are made between the analysis and its real consequences. Connecting with reality in this way guards against failing to properly grasp the context of the problem – which is being analysed. | What (pre-)conditions influence the problem, policy area, or data? Are there any key variables or contextual facotrs that are missing? |
| Objective | Effective engagement and suitable challenge reduces potential bias and enables the commissioner and the analyst to be clear about the interpretation of the analytical results. | What explicit and implict assumptions are there in the analytical approach? |
| Uncertainty-managed | Uncertainties have been identified, managed and communicated throughout the analytical process. | Is there a clear log or description of uncertaininities and their potential impact on the input data or results? Are key caveats or limitations on interpreation clear in the write-up of the analysis? |
| Robust | Provide the analytical result in the context of residual uncertainty and limitations in order to ensure it is used appropriately. | Has sensitivity analysis been conducted to test assumptions and test uncertainty or model parameters? |
Some example resources have been published on GOV.UK.