Introduction

The terms 'validation' and 'verification' and 'V&V' are often used and often misused so it's worth spending a little while trying to understand the terms and the potential effects of misuse. As ever it has a lot to do with the context of the use (or misuse) of the terms.

Definitions

It's always worth starting with some authoritative sources when looking at definitions. The following table shows how 'validation' and 'verification' are defined in:-

• The Oxford English Dictionary [1]
• ISO 9000:2015 Quality Management Systems - Fundamentals and Vocabulary [2]
• ISO/IEC/IEEE 15288:2015 Systems and Software Engineering - System Life Cycle Processes [3]

Table 1 - Comparison of the Definitions of Validation and Verification

Validation

When we validate something we are establishing its correctness. For example when we validate a model we are comparing the model or model predictions against the real world. This is exactly the sense of the OED definition.

It's a lot harder to get a proper sense of this from either the ISO 9000:2005 to ISOE/IEC/IEEE 15288:2015 definitions. Both sources refer to comparison against requirements - this is confusing, particularly because of the overlap with the definition of verification. ISO/IEC/IEEE 15288 at least qualifies this with 'the right system was built'.

We always have to be mindful of the context. What is it that we are validating? Is it a model of a system, is it the build of the system, is it the design of the system? We can't just say that we are "validating a system" because the methods / sources for validation will differ because the we are comparing different things. For example we might validate a design by using the results of a design review. We might validate a build by comparing the physical build against one or more sets of drawings, a bill of materials or similar.

Verification

When we verify something we are comparing it against its input requirement document(s). Usually this involves providing evidence from simulation, test, inspection, analysis or similarity (with reference to these sources for a previous version of the same thing). Again this agrees with the sense of the OED definition.

It's a lot harder to get this sense from the ISO 9000:2005 definition because both 'validation' and 'verification' are defined in terms of evidence of requirements having been met - the difference being that validation refers to 'requirements for a specific use' whereas verification refers to 'specified requirements'. The latter probably should be 'specific requirements' (because specified requirements adds another level to the requirement trace and what is the mechanism for specifying requirements?). Even then it isn't a particularly clear definition.

ISO/IEC/IEEE 15288:2015 builds on this by adding 'the system was build right'. It does, however, make the point that verification is concerned with comparison against 'required characteristics'.

As with validation, verification only makes sense when use in context - we verify a thing - a design using evidence gathered for one or more examples that are typical of the class. In the UK Defence domain this design assurance used to be referred to as 'Design Certification', in the Nuclear domain it is 'Generic Design Assessment', elsewhere such as in Rail it might be Type Approval. All of these are approaches to verify the design of a system. These are all design-assurance processes.

Note that we don't normally verify a requirement document - we validate the requirements within it and hence the requirement document (collection) itself. We can verify a requirement document but this would involve a comparison of the requirement document against its requirements - possibly process requirements specifying the structure, format and contents or an external standard such as ISO/IEC/IEEE 29148 [7].

Design Verification Follows Validation

Normally we validate the requirements for the system. This then establishes that we have the correct (right) set of requirements.

We then verify the system design against this validated set of requirements. This produces evidence from which we identify where the design meets, complies or conforms or where there are limitations or exceptions in the system design compliance against the requirements (summarised using a compliance matrix which can be derived from a Claim - Argument - Evidence basis).

All we then need to is to ensure that a particular system build conforms to the build defined as part of the design of the type or class for which we have the Design Certificate or Type Approval. This validation of the system build against its defining documents such as the bill of materials. A Physical Configuration Audit (PCA) and baseline will be used to define the basis of the physical design. Note that we aren't 'validating the system' - we are 'validating the system build'. Validation of the system design needs to be done before verification of the system design to ensure that we are collecting verification evidence against the correct system design.

So What's in a Definition - Does it Matter?

Open Services for Lifecycle Collaboration (OSLC) Metamodel - Incorrect 'Validates'

The Open Services for Lifecycle Collaboration (OSLC)[8] is an OASIS open project that has produced a set of specifications to support integration of tools. It is based on a metamodel to standardise relationships between common engineering artefacts. The specifications produced enable tool vendors to implement the various integration mechanisms needed, for example to allow the creation, update or deletion or preview of an element in a modelling tool or requirement tool from outside the tool. The OSLC project describes this as 'achieving the digital thread':

Figure 1 - Source: OSLC - "Achieving the Digital Thread" [8]

The effort is organised into a set of domains, each of which defines a part of the metamodel in a specific subject or work area. The Requirements Management domain contains Requirement and RequirementCollection. The Quality Management domain contains Test Plan, Test Case, Test Script, Test Execution Record and Test Result [9].

Figure 2 - Source: Open Services for Lifecycle Collaboration - Quality Management Specification Version 2.0 - Relationships Between the QM Resources

The relationships between the two domains are defined as:-

• Test Script validates Requirement
• Test Case validates Requirement
• Test Plan validates Requirement Collection

The relationship validatesRequirement [9] is defined as:-

Requirement that is validated by the Test Case. It is likely that the target resource will be an oslc_rm:Requirement but that is not necessarily the case.

which doesn't actually define what is meant by 'validates'.

In the reverse direction the relationship is validatedBy e.g. Requirement validatedBy Test Script.

This isn't an accidental slip up. An expected scenario for Requirements Validation linking test results to requirements is described in [10] as:

Requirements validation

Traceability links are established as described in previous scenarios

Developer implements change request and notifies tester

Tester creates execution record for the plan item

Tester executes test and creates an execution result

Architect searches for test cases associated with the requirement

Architect navigates to a test case and examines its execution records

Architect examines execution results associated with execution record

The OSLC (v2 - Quality Management [11]) has no means to describe or capture verification. This is probably because they use their OSLC:validation as if it were ISO 15288:verification.

Note - for an open consortium there appears to be no public-facing mechanism provided where the public can raise feature requests and bugs in the specifications and the decisions / outcomes tracked. This is a significant weakness in the assurance of these specifications.

...Impacts IBM DOORS Next (Generation)

This has knock-on effects in tools implementing the OSLC such as the IBM DOORS Next Generation (DOORS NG) Requirement Management tool (an evolution and rebranding of the the Rational Requirements Composer tool) which implements the OSLC Requirements Management domain and their Quality Management tool [12] which implements the OSLC Quality Management domain on the Jazz platform. The 'validates / validatedBy' relationship is the only allowed relationship between a test artefact and a requirement artefact.

Figure 3 - Source: IBM - IBM DOORS Next - browser-based tool

These IBM Jazz tools will not permit a 'verifies / verifiedBy' relationship to be created between these artefacts [13]. It isn't obvious that the OSLC specification prohibits more than one relationship between a pair of artefacts so this might be an error in the IBM interpretation of the specification.

...Affects the Process of Verifying the System Design

The restriction of OSLC:validation (verification) to testing and test artefacts is a common weakness. It also appears in modelling languages where the focus is on functional definition, use cases and functional test. It only addresses part of the functional baseline. This completely ignores non functional requirements for a system. If we want to capture all of the system design verification artefacts we have a number of problems:

• although verifies / verifiedBy is missing we can't add this between a Quality Manager artefact and a Requirement artefact
• we can't trace to validation artefacts (in the correct sense of 'validation') - only verification by test artefacts (using the semantically incorrect 'validatedBy')
• we can't trace to both validation and verification artefacts (since 'validatedBy' is defined albeit incorrectly the tool / OSLC won't permit any other use of 'validates')
• we can't use the Quality Manager tool to capture verification of non-functional requirements or requirements where the verification method is simply analysis, inspection, similarity (read-across) - Figure 4
• we can define a 'verifies' / 'verified by' wholly within DOORS Next (Requirements Management domain). This, however, leaves us with verification artefacts / evidence split between tools and at the end of different relationship paths which fragments the collection and makes it all but impossible to collect the verification status as a single entity without a lot of bespoke under the hood effort - Figure 4.

Figure 4 - Use of Incorrect 'Validated By' As 'Verified By' Only Possible for Functional Requirement - Full Verification Traceability / Analysis Fragmented

It has to be said that the older (originally QSS - Telelogic - now IBM) DOORS "classic" suffers no such problems. It doesn't enforce a standardised metamodel / schema which might mean that there is less consistency between companies. it does mean, however, that it is flexible and can adapt if something has been forgotten and it doesn't enforce an external specifier's errors - any local errors in the DOORS schema can be corrected by the local DOORS Administrator.

Figure 5 - Source: IBM - IBM DOORS Classic - desktop tool

Conclusions

1. The definitions of 'validation' and 'verification' in ISO 9000:2005 aren't sufficiently distinct and are easy to misinterpret.
2. ISO/IEC/IEEE 15288:2015 is a little better in adding footnotes - but not much.
3. 'validation' and 'verification' should never be used without qualification e.g. 'validating a model of the system', 'validating a requirement set (document)', 'verifying the system design' are correct. 'verifying a system' or 'validating a system' are ambiguous and incorrect uses.
4. The OSLC doesn't support both 'verification' and 'validation' of a system design and incorrectly uses 'validation' as if it were 'verification'

References

1. ‘validate, v. : Oxford English Dictionary’, The Oxford English Dictionary. Oxford University Press. [Online]. Available: https://oed.com/view/Entry/221192. [Accessed: 07-Aug-2019].
2. ‘verification, n. : Oxford English Dictionary’, The Oxford English Dictionary. The Oxford University Press. [Online]. Available: https://oed.com/viewdictionaryentry/Entry/222504 [Accessed: 22-July-2019].
3. ISO 9000:2015’, ISO 9000:2015 Quality Management Systems - Fundamentals and Vocabulary - Fourth Edition. p. 7, 15-Sep-2015.
4. ISO/IEC/IEEE 15288:2015’, ISO/IEC/IEEE 15288:2015 Systems and Software Engineering - System Life Cycle Processes. 15-May-2015. https://www.iso.org/standard/63711.html
5. UK MoD Requirements and Acceptance. [Online]. Available: Activity: Confirm Final Design Satisfies System Requirement Document (SRD) [Accessed: 05-Apr-2020].
6. Office for Nuclear Regulation (ONR) [Online]. Available: Generic Design Assessment (GDA) of new nuclear power stations [Accessed: 05-Apr-2020].
7. ‘ISO/IEC/IEEE 29148:2018 Systems and software engineering — Life cycle processes — Requirements Engineering’, ISO/IEC/IEEE 29148:2018 (E), Nov. 2018.
8. Open Services for Lifecycle Collaboration [Online]. Available: https://open-services.net/ [Accessed: 05-Apr-2020].
9. Open Services for Lifecycle Collaboration - validatesRequirement [Online]. Available: https://archive.open-services.net/bin/view/Main/QmVocabulary.html#validatesRequirement [Accessed: 06-Apr-2020]
10. Open Services for Lifecycle Collaboration (OSLC) Wiki. QM Scenario: Traceability [Online]. Available: https://archive.open-services.net/bin/view/Main/QmScenariosTraceability.html [Accessed: 05-Apr-2020].
11. ‘Open Services for Lifecycle Collaboration - Quality Management Specification Version 2.0’. [Online]. Available: https://archive.open-services.net/bin/view/Main/QmSpecificationV2.html#QM_Resource_Definitions. [Accessed: 05-Apr-2020].
12. : IBM Knowledge Center. Overview of Rational Quality Manager. [Online]. Available: https://www.ibm.com/support/knowledgecenter/SSYMRC_6.0.6.1/com.ibm.rational.test.qm.doc/topics/c_qm_overview.html [Accessed: 06-Apr-2020]
13. IBM Knowledge Center. Links Across OSLC Domains. [Online]. Available: https://www.ibm.com/support/knowledgecenter/SSYMRC_6.0.6.1/com.ibm.rational.rrm.help.doc/topics/r_rm_link_domains.html [Accessed: 06-Apr-2020]