safety.inge.org.uk

James Inge (2019), Oxford, UK. (MSc dissertation incorporating reviews' comments).

Abstract

Defence Standard 00-055 is the UK Ministry of Defence’s contracting standard for safe software. In the 1990s, it mandated specific practices that contractors saw as too onerous and prescriptive, such as formal design and specification methods. By 2005 it was technically outdated and declared obsolescent, but now it has been resurrected. The new Issue 4 tries to avoid the original criticisms by being less prescriptive. But is it still a good document? How can it be improved? This dissertation explores how to answer these questions.

Software safety standards have a vital role in delivering safe products, services and systems. In the defence industry, software failures can lead to significant loss of life. This makes it especially important that such standards are well understood by their users. Yet, they are often verbose, lengthy documents written by committees; hard for the uninitiated to immediately digest and understand, and awkward to implement as written. Anecdotal feedback suggests Def Stan 00-055 Issue 4 may suffer from these problems, implying that its review process was not entirely effective. The author will shortly be involved in organizing the standard’s next periodic review. His experience of reviewing similar documents consists mainly of being asked to read a text and provide comments, without any detailed guidance or methodology being provided. This seems a dull chore, and perhaps not the best way to detect flaws and potential improvements in a standard. Our motivation is to find a better standards review method that will lead to better safety standards and safer systems.

In this project, we develop a guidance framework to help make future standards reviews more effective. We build on the concept that software safety assurance standards are themselves artefacts of the software engineering process: part of the decomposition of organizational goals into software requirements and designs. With this idea, we examine the software engineering and software safety literature to identify potential review methods that might work for standards. To evaluate these methods, we experiment by applying them to Def Stan 00-055 to compare their effectiveness and practicality. While some of the methods, such as argument modelling, have previously been applied to reviews of other standards, Def Stan 00-055 has not previously been modelled. Novel aspects of this project include using assurance framework meta-models in standards reviews, and making a comparison of the effectiveness of model-based and other review methods.

We find that methods inspired by software engineering can indeed improve the quality of software assurance standards like Def Stan 00-055. The guidance framework describes how to do this using a combination of methods. We evaluate the usefulness of the guidance by seeking feedback from its potential users, finding that both the guidance and the experimental results are likely to be useful in the upcoming review of Def Stan 00-055. The guidance also appears broadly applicable to review of other assurance standards beyond the software safety field, and has implications for the preparation of standards.