Summary

In moving from NATO Architecture Framework (NAF) v3 to v4 the metamodel (NMM) has been dropped. There's only a design implementation of it left in the UAF and no longer any input requirements to implement. This isn't how engineering design and traceability should work.

History / Background

The NATO Architecture Framework (NAF) provides a set of architecture views that the system architect can use to describe the architecture of a military system. Like the US DoD Architecture Framework (DODAF) and the UK MOD Architecture Framework (MODAF) NAF has its roots in the US C4ISR Architecture Framework [1, 2]. The current version (4) was first issued in 2018 and updated in 2020.

Figure 1 - NATO Architecture Framework (NAF) Versions

NAF Version 3

Like most architecture frameworks, NAF version 3 [3] is defined by NATO using a documentation set [Figure 2].

Figure 2 - NATO Architecture Framework Version 3 Document Structure (Partial)

The NAF document set at version 3 has five parts but the relevant parts are Chapter 5, which defines the NATO architecture framework metamodel (NMM) and Chapter 4 which defines NAF architecture views and sub-views. The NMM provides the architecture description elements that may appear in the NAF architecture views.

At NAF version 3 this terminology is non-standard and doesn't match ISO/IEC/IEEE 42010 Systems and Software:

Architecture Description. A NAF::Architecture View collects together a set of NAF Sub Views, each of which is supposed to define the content of an individual 42010Architecture View.

As 'sub..' anything denotes a position in a hierarchy, not a type, we have in NAF v3, a view meaning a collection (42010::Architecture Perspective), a specification of an architecture view (42010::Architecture Viewpoint). In NAF v3, like MODAF, there is no term that refers to the product created against the NAF::subview (42010::Architecture Viewpoint).

The version 3 terminology isn't consistent:

In the Framework, there are seven major perspectives (i.e., views ...

A view is defined as a set of subviews grouped by purpose ....

This definition of the term 'subview' is similar to the term 'viewpoint' as defined in the IEEE 1471-2000 standard'

The consistency and other problems are described by Ian Bailey [4].

Most of the issues uncovered in this study are around Chapter 4 of NAF – the chapter that documents the views. This chapter has had multiple editors and was not entirely developed with Chapter 5 (the NAF Meta-Model) in mind. Furthermore, Chapter 5 has been revised since Chapter 4 was released, but there have been no corresponding updates to Chapter 4. As a result, it has a number of issues...

Classic Engineering Lifecycle - Input Requirements vs Design Implementation

NAF v 3 is defined by NATO. It is, or should be, independent of any particular notation modelling language or tool used to produce architecture views for architecture descriptions. Independence isusually also ensured because the specifying organisation (NATO) is separate from any implementing organisation. This is classic engineering practice where the agnostic requirements - "the what" represented by the NAF v3 document set - is separate from the particular implementation of the NAF v3 - "the how" - the design (of how to implement the NAF v3.

For example, if the Unified Modelling Language (UML)[5] is to be used a UML profile is created which implements the NMM elements using UML metamodel elements so that when loaded into a UML modelling tool there is a set of UML elements that can be used to create the various NAF 42010::Architecture Views in a UML modelling tool [Figure 3].

Figure 3 - Implementation of the NAF as a UML Profile for a UML Modelling Tool

The UML is specified by the Object Management Group (OMG) and any organisation can define a UML profile implementing the NAF architecture description elements. Usually the modelling tool vendor spends the effort to define the UML profile implementation for their modelling tool. There are several organisations involved - NATO who define the "master source" requirements, the OMG who define the UML and the UML modelling tool vendors who interpret the NAF requirements to produce a tailored UML profile for each too. As with any design if the input requirements (NAF documentation set) changes, the UML specification changes or their tool capabilities change the UML profile for NAF would likely change.

All of this is invisible to the system architect - they simply start the tool and are able to select a 42010::architecture view as a UML diagram and they are presented with a toolbox palette with the right node and connector elements (from the tool vendor's UML profile that interprets the NAF). The user has no choice but to trust that each part of the chain is valid and has been implemented correctly.

Improving UML Implementation Consistency

The problem was, however, that the evolution of the defence architecture frameworks meant that the UML tool vendors were constantly struggling to implement similar but different metamodels, similar but different architecture view content, with each changing at different points in time. Even metamodel elements with the same name weren't necessarily the same type of thing and could be defined very differently. It also hasn't helped that the defence architecture frameworks weren't good at defining allowed architecture view content which therefore required interpretation and clarification that then wasn't used to update the master source AF-defining documents to avoid the same need for interpretation in the future.

On its own this failure to correct increases the technical debt as the hard won in-house expertise and knowledge in interpreting these documents remains implicit and increases the fragility of the AF implementation and AF definition. Each UML tool vendor could therefore interpret the requirements differently or implement them differently (as there are many possible 'designs' or ways in which a set of agnostic requirements could be met).

As a conseqience there was a need to improve the consistency of implementation in the UML modelling tools. This is where the Universal Profile for DODAF and MODAF (UPDM) arose as a means to improve the consistency of implementation - a common UML profile for implementing DODAF and MODAF [6].

The UPDM is a UML implementation. As with any implementation the UPDM is not the DODAF and the UPDM is not the MODAF either as indicated in the UPDM 2.1 specification:

While conformance with UPDM 2.1 Core and DoDAF-specifics should greatly facilitate conformance with DoDAF 2.02, each tool vendor is still responsible for the tool's ultimate conformance with the documented architecture framework [6]

The UPDM simply aids the architect in producing DODAF-compliant and MODAF-compliant architecture views. The UPDM isn't an architecture framework. It implements metamodel elements but it doesn't define the architecture views and it doesn't implement rules in the individual architecture frameworks.

The UPDM also partly covered the NAF NMM as NAF version 3 included and extended the MODAF metamodel (M3):

NAF and MODAF have shared a common meta-model for some time. The NAF Meta-Model (NMM – Chapter 5 of the NAF Specification) is the same as v1.2 of the MODAF Meta-Model (known as M3).[4]

The NAF views are different and there is still local tool vendor adaptation required by tool vendors to implement the metamodel triples(^1), formed from the metamodel node and connector elements, that may appear in each of the architecture views as the UPDM describes views but doesn't specify what may or may not appear in each:

UPDM is not a new Architectural Framework. Instead, UPDM provides a consistent, standardized means to describe DoDAF, MODAF andNAF architectures in UML-based tools [1]

Having a common UML implementation didn't, however, solve the problems with the NAF v3 document set. As the UPDM was the result of a joint effort by the OMG it helped reduce inconsistency in UML modelling tools but it then meant that the 'design' implementation of NAF (and other AFs) was essentially accelerating away from and leading things because the various master source AF documents weren't being updated i.e. the input requirements were no longer driving things as all of the effort and interpretation was centred on the UPDM (the downstream responding design).

The UPDM 2.1 [6] makes reference to the lack of updates of the master definitions of the architecture frameworks:

the documentation set has not been changed from DoDAF Version 2.0 and is no longer definitive for without the changes. It can be downloaded from the DoDAF Archives ...

...

The DoDAF Meta Model (DM2) has changed from DoDAF Version 2.0. It can be derived as a sequential update from DoDAF MetaModel (DM2) Version 2.00 to 2.01, and 2.02. There were 94 changes in the DoDAF MetaModel (DM2) from DoDAF 2.0 (68 in Version 2.01 and (26 in Version 2.02).

At this point we have an inconsistent master NAF v3 document set which cannot be used by system architects (users) to define what should or should be in any particular NAF architecture view - the users have to rely on the design interpretation of the M3 (NMM) by the OMG and the tool vendors being a correct one. NAF still has a metamodel and there is an implementation of it within the UPDM. There is a means in the common UML profile of the UPDM to improve the consistency of implementation of the NAF. The UPDM also avoids the need for multiple UML profiles so it is simpler for the UML tool vendors to adapt and potentially for users to work with.

Improving the NAF Master Documentation... NAF Version 4 ...

Hause et al[1] identify the cost of maintaining separate defence architecture frameworks:

Creating and maintaining the mapping between the different frameworks takes a considerable amount of time, effort and money on the part of the nations, the tool vendors, and industry.

In these days of limited budgets, austerity measures and “Doing more without more”, this may be a luxury that the nations can no longer afford

At NAF version 4[7] the master documentation is significantly different. Effort has been made to use ISO/IEC/IEEE 42010 terminology - although it still suffers from using 'architecture' where it means 'architecture description':

An architecture may be used to provide a complete expression of any part of the system in an enterprise context.

It also addresses some of the inconsistency by eliminating the old conceptual model in chapter 3. NAF version 4 also introduces a Zachman-like grid to organise and classify the various architecture views based on suggestion from Ian Bailey[8].

But what about the metamodel (NMM) that forms the heart of the NAF?

NAF v4 states

The meta-model defines the essential modelling elements that can be used to describe the system in an enterprise context and its environment

so looking for the 'the metamodel' which used to be the NMM we find:

As part of the development of NAFv4 it was agreed that it should make use of commercial architecture meta-models to enable architecting across military and non-military domains. These are described in Chapter 4.

and then in Chapter 4 - Meta-Model

Chapter 4 of the NATO Architecture Framework identifies the meta-models to be used for creating NAFv4 compliant architectures.

NAFv4 compliant architectures can be creating using the following meta-models; The Open Group®’s ArchiMate® and the Object Management Group®’s Unified Architect Framework (UAF) ® Domain Meta-model (DMM)®

Figure 4 - NATO Architecture Framework Version 4 - Metamodel Removed and Structural Changes

so in NAF v 4 we have the curious situation where NAF no longer has a metamodel [Figure 4]. The NMM is dead. NAF doesn't define a metamodel. We have an architecture framework without its own metamodel.

The reference to ArchiMate is odd. ArchiMate is agnostic of the NAF so it is equally valid to list any 'vanilla' architecture description language (ADL) including the UML, BPMN etc. The problem in doing so is that it is far from clear that any vanilla ADL has all of the types and will permit them to be joined together to represent all of the NAF types and NAF architecture description triples needed to produce all the NAF architecture views. The statement represents a leap of faith with no substantiation.

NAF v4 does, however, have a set of very vague architecture viewpoint descriptions in Chapter 3 - none of which define the acceptable architecture view content. It's impossible to use the NAF v4 master documents to verify an architecture view against. It therefore looks as though the NAF itself no longer cares about view content. This is a significant change to the NATO-controlled master definition of the NAF.

Figure 5 - Implementation of the NAF v4 Now Relies Upon the Unified Architecture Framework (UAF)

'The metamodel' referred to now only exists in the implementation of the NAF or design side ("the how"). We have an implementation but the NMM being implemented no longer exists so in effect the design response is now completely cut adrift from the master source. One of these implementations, the Unified Architecture Framework (UAF) Domain Metamodel[9] is controlled by the OMG and provides a UML implementation. The concrete UAF profile (UAFP)[10](^1) provides an implementation of an abstract metamodel itself also described using the UML [Figure 5].

Of course even the abstract UML metamodel in the UAF contains an implementation of the NMM - or at least it did when the NMM existed. If the NMM has disappeared then there is no version 4 metamodel to implement - the only published but superseded one is the faulty version 3 metamodel. We then have the situation where there is only an implementation of what used to be the NMM within the design side (OMG), not within the master NAF documentation (NATO) [Figure 6].

Figure 6 - NAF v 4 UML Implementation Traceability - No Master Metamodel Definition

Of course even the abstract UML metamodel in the UAF contains an implementation of the NMM - or at least it did when the NMM existed. If the NMM has disappeared then there is no version 4 metamodel to implement - the only published but superseded one is the faulty version 3 metamodel. We then have the situation where there is only an implementation of what used to be the NMM within the design side (OMG), not within the master NAF documentation (NATO).

The UAF even refers to the NAF as a normative source even though it no longer has a metamodel at version 4 so both sides refer to each other. And what is 'normative guidance'?

The NAF itself has no definition of the AD elements and triples to be used for its own architecture views, no content or consistency rules and no mechanism to specify these.

It appears that all of the resources, energy and motivation are in the design implementation of the NAF in the UAF rather than on the root definition of the NAF. The problem then is that corrections, fixes and improvments aren’t being made in the right place – the root definition.

It's all very messy and not how the typical engineering design and development programme is supposed to work. It is difficult to imagine these changes and structure passing a formal System Requirement Review (SRR), Preliminary Design Review (PDR) and Critical Design Review (CDR) [11].

Concluding Remarks

There is no longer a NAF metamodel that acts as an implementation-free set of requirements for any implementation of it. We have a design implementation of it as a record within the UAF. There isn't any master source specification of NAF view content - we have to rely on the interpretation and agreement of tool vendors to establish what is and what isn't permitted as there is no source specification to refer to.

Although maintaining an architecture framework is costly removing most of the core definition of it and leaving its definition and expression to designers downstream is never a good idea.

Trying to solve problems with a design by deleting the input requirements doesn't work.

There are questions that arise from this:-

• What is it that the UAF implements? It isn't NAF v3 and at NAF v4 there is no metamodel. Clearly the OMG in referencing the NAF v 4 metamodel in the UAF traceability were working against a NAF metamodel that wasn’t NAF v3 and was never released as part of NAF v4. Why were the OMG working against something unreleased with no formal baseline?

• If an architecture framework no longer has a specified metamodel then is it reasonable to refer to it in the same way i.e. if it was NAF::NSOV-4' then shouldn't it now be 'UAF::NSOV-4' since the NAF without the NMM is no longer the NAF?

• If an architecture framework has no metamodel; no specification of architecture view content and no consistency rules is it an architecture framework? What is the minimum set of parts required for an architecture framework?

• What NAF master source specification does the user of NAF check their architecture view content against? Why isn't there a simple and user-understandable metamodel so that the user can check what the allowed architecture description triples (node-connector-node) are? A simple agnostic metamodel would also be easy to validate architecture and verify any implementation against.

A common UML implementation isn’t a substitute for good defining source documentation,

The next article will look at how the content of architecture views is specified, the mechanics used and their capabilities to specify vs the commonly held beliefs.

Footnotes

^1. A triple (node - relationship - node) is the smallest unit of architecture description: a node on its own cannot describe architecture i.e. 'Software' isn't architecture description but 'Software satisfies Standard' is.

^2. What was known as the Unified Architecture Framework Profile (UAFP) at UAF version 1.1 - a UML profile - is now named the Unified Architecture Framework Modelling Language (UAFML) at UAF version 1.2 even though it is still a UML profile that implements the UAF and there's only a single individual i.e. unlike any other modelling language it cannot be used for anything else and there will only ever be the single example of it.

References

1. M. Hause, G. Bleakley, and A. Morkevicius, ‘Technology Update on the Unified Architecture Framework (UAF)’, INCOSE International Symposium, vol. 26, no. 1, pp. 1145–1160, 2016, doi: DOI: 10.1002/j.2334-5837.2016.00217.x.
2. ‘C4ISR Architecture Framework. Version 1.0’, C4ISR ITF. Integrated Architectures Panel, CISA-0000-104-96, Jun. 1996. https://irp.fas.org/doddir/dod/c4isr/index.html.
3. ‘NATO Architecture Framework. Version 3.’ Architecture Capability Team, Nov-2007.
4. I. Bailey, ‘SSAC Task 83. NATO Architecture Framework - MODEM. Review’. 16-Aug-2013. https://nafdocs.github.io/assets/20130816_MODEM_to_NAF_Review_V1_1-U.pdf
5. ‘OMG Unified Modeling Language (OMG UML©). Version 2.5.1’, The Object Management Group, formal/2017-12-05, Dec. 2017. https://www.omg.org/spec/UML/2.5.1/PDF.
6. ‘Unified Profile for DoDAF and MODAF (UPDM). Version 2.1 (Normative Document)’, The Object Management Group, formal/13-08-04, Aug. 2013. http://www.omg.org/spec/UPDM/2.1
7. ‘NATO Architecture Framework. Version 4 (2020.09)’. Architecture Capability Team, Sep-2020. https://www.nato.int/cps/en/natohq/topics_157575.htm
8. I. Bailey, ‘Reorganising MODAF and NAF’. 07-Nov-2013. http://nafdocs.org/wp-content/uploads/2013/07/Reorganising-MODAF-and-NAF.pdf
9. ‘United Architecture Framework (UAF) Domain Metamodel. Version 1.1’, Object Management Group (OMG), dtc/19-11-05, Apr. 2020. https://www.omg.org/spec/UAF/1.1.
10. ‘Unified Architecture Framework Profile (UAFP). Version 1.1’, The Object Management Group, dtc/19-06-15, Jun. 2019. https://www.omg.org/spec/UAF/1.1/Beta1/UAFP/PDF.
11. ‘NAVAIR Systems Engineering Technical Review Process Handbook’, NAVAIR, NAVAIRINST 4355.19E, Feb. 2015.

The terms 'validation' and 'verification' and 'V&V' are often used and often misused so it's worth spending a little while trying to understand the terms and the potential effects of misuse. As ever it has a lot to do with the context of the use (or misuse) of the terms.

Definitions

It's always worth starting with some authoritative sources when looking at definitions. The following table shows how 'validation' and 'verification' are defined in:-

• The Oxford English Dictionary [1]
• ISO 9000:2015 Quality Management Systems - Fundamentals and Vocabulary [2]
• ISO/IEC/IEEE 15288:2015 Systems and Software Engineering - System Life Cycle Processes [3]

Table 1 - Comparison of the Definitions of Validation and Verification

Validation

When we validate something we are establishing its correctness. For example when we validate a model we are comparing the model or model predictions against the real world. This is exactly the sense of the OED definition.

It's a lot harder to get a proper sense of this from either the ISO 9000:2005 to ISOE/IEC/IEEE 15288:2015 definitions. Both sources refer to comparison against requirements - this is confusing, particularly because of the overlap with the definition of verification. ISO/IEC/IEEE 15288 at least qualifies this with 'the right system was built'.

We always have to be mindful of the context. What is it that we are validating? Is it a model of a system, is it the build of the system, is it the design of the system? We can't just say that we are "validating a system" because the methods / sources for validation will differ because the we are comparing different things. For example we might validate a design by using the results of a design review. We might validate a build by comparing the physical build against one or more sets of drawings, a bill of materials or similar.

Verification

When we verify something we are comparing it against its input requirement document(s). Usually this involves providing evidence from simulation, test, inspection, analysis or similarity (with reference to these sources for a previous version of the same thing). Again this agrees with the sense of the OED definition.

It's a lot harder to get this sense from the ISO 9000:2005 definition because both 'validation' and 'verification' are defined in terms of evidence of requirements having been met - the difference being that validation refers to 'requirements for a specific use' whereas verification refers to 'specified requirements'. The latter probably should be 'specific requirements' (because specified requirements adds another level to the requirement trace and what is the mechanism for specifying requirements?). Even then it isn't a particularly clear definition.

ISO/IEC/IEEE 15288:2015 builds on this by adding 'the system was build right'. It does, however, make the point that verification is concerned with comparison against 'required characteristics'.

As with validation, verification only makes sense when use in context - we verify a thing - a design using evidence gathered for one or more examples that are typical of the class. In the UK Defence domain this design assurance used to be referred to as 'Design Certification', in the Nuclear domain it is 'Generic Design Assessment', elsewhere such as in Rail it might be Type Approval. All of these are approaches to verify the design of a system. These are all design-assurance processes.

Note that we don't normally verify a requirement document - we validate the requirements within it and hence the requirement document (collection) itself. We can verify a requirement document but this would involve a comparison of the requirement document against its requirements - possibly process requirements specifying the structure, format and contents or an external standard such as ISO/IEC/IEEE 29148 [7].

Design Verification Follows Validation

Normally we validate the requirements for the system. This then establishes that we have the correct (right) set of requirements.

We then verify the system design against this validated set of requirements. This produces evidence from which we identify where the design meets, complies or conforms or where there are limitations or exceptions in the system design compliance against the requirements (summarised using a compliance matrix which can be derived from a Claim - Argument - Evidence basis).

All we then need to is to ensure that a particular system build conforms to the build defined as part of the design of the type or class for which we have the Design Certificate or Type Approval. This validation of the system build against its defining documents such as the bill of materials. A Physical Configuration Audit (PCA) and baseline will be used to define the basis of the physical design. Note that we aren't 'validating the system' - we are 'validating the system build'. Validation of the system design needs to be done before verification of the system design to ensure that we are collecting verification evidence against the correct system design.

So What's in a Definition - Does it Matter?

Open Services for Lifecycle Collaboration (OSLC) Metamodel - Incorrect 'Validates'

The Open Services for Lifecycle Collaboration (OSLC)[8] is an OASIS open project that has produced a set of specifications to support integration of tools. It is based on a metamodel to standardise relationships between common engineering artefacts. The specifications produced enable tool vendors to implement the various integration mechanisms needed, for example to allow the creation, update or deletion or preview of an element in a modelling tool or requirement tool from outside the tool. The OSLC project describes this as 'achieving the digital thread':

Figure 1 - Source: OSLC - "Achieving the Digital Thread" [8]

The effort is organised into a set of domains, each of which defines a part of the metamodel in a specific subject or work area. The Requirements Management domain contains Requirement and RequirementCollection. The Quality Management domain contains Test Plan, Test Case, Test Script, Test Execution Record and Test Result [9].

Figure 2 - Source: Open Services for Lifecycle Collaboration - Quality Management Specification Version 2.0 - Relationships Between the QM Resources

The relationships between the two domains are defined as:-

• Test Script validates Requirement
• Test Case validates Requirement
• Test Plan validates Requirement Collection

The relationship validatesRequirement [9] is defined as:-

Requirement that is validated by the Test Case. It is likely that the target resource will be an oslc_rm:Requirement but that is not necessarily the case.

which doesn't actually define what is meant by 'validates'.

In the reverse direction the relationship is validatedBy e.g. Requirement validatedBy Test Script.

This isn't an accidental slip up. An expected scenario for Requirements Validation linking test results to requirements is described in [10] as:

Requirements validation

Traceability links are established as described in previous scenarios

Developer implements change request and notifies tester

Tester creates execution record for the plan item

Tester executes test and creates an execution result

Architect searches for test cases associated with the requirement

Architect navigates to a test case and examines its execution records

Architect examines execution results associated with execution record

The OSLC (v2 - Quality Management [11]) has no means to describe or capture verification. This is probably because they use their OSLC:validation as if it were ISO 15288:verification.

Note - for an open consortium there appears to be no public-facing mechanism provided where the public can raise feature requests and bugs in the specifications and the decisions / outcomes tracked. This is a significant weakness in the assurance of these specifications.

...Impacts IBM DOORS Next (Generation)

This has knock-on effects in tools implementing the OSLC such as the IBM DOORS Next Generation (DOORS NG) Requirement Management tool (an evolution and rebranding of the the Rational Requirements Composer tool) which implements the OSLC Requirements Management domain and their Quality Management tool [12] which implements the OSLC Quality Management domain on the Jazz platform. The 'validates / validatedBy' relationship is the only allowed relationship between a test artefact and a requirement artefact.

Figure 3 - Source: IBM - IBM DOORS Next - browser-based tool

These IBM Jazz tools will not permit a 'verifies / verifiedBy' relationship to be created between these artefacts [13]. It isn't obvious that the OSLC specification prohibits more than one relationship between a pair of artefacts so this might be an error in the IBM interpretation of the specification.

...Affects the Process of Verifying the System Design

The restriction of OSLC:validation (verification) to testing and test artefacts is a common weakness. It also appears in modelling languages where the focus is on functional definition, use cases and functional test. It only addresses part of the functional baseline. This completely ignores non functional requirements for a system. If we want to capture all of the system design verification artefacts we have a number of problems:

• although verifies / verifiedBy is missing we can't add this between a Quality Manager artefact and a Requirement artefact
• we can't trace to validation artefacts (in the correct sense of 'validation') - only verification by test artefacts (using the semantically incorrect 'validatedBy')
• we can't trace to both validation and verification artefacts (since 'validatedBy' is defined albeit incorrectly the tool / OSLC won't permit any other use of 'validates')
• we can't use the Quality Manager tool to capture verification of non-functional requirements or requirements where the verification method is simply analysis, inspection, similarity (read-across) - Figure 4
• we can define a 'verifies' / 'verified by' wholly within DOORS Next (Requirements Management domain). This, however, leaves us with verification artefacts / evidence split between tools and at the end of different relationship paths which fragments the collection and makes it all but impossible to collect the verification status as a single entity without a lot of bespoke under the hood effort - Figure 4.

Figure 4 - Use of Incorrect 'Validated By' As 'Verified By' Only Possible for Functional Requirement - Full Verification Traceability / Analysis Fragmented

It has to be said that the older (originally QSS - Telelogic - now IBM) DOORS "classic" suffers no such problems. It doesn't enforce a standardised metamodel / schema which might mean that there is less consistency between companies. it does mean, however, that it is flexible and can adapt if something has been forgotten and it doesn't enforce an external specifier's errors - any local errors in the DOORS schema can be corrected by the local DOORS Administrator.

Figure 5 - Source: IBM - IBM DOORS Classic - desktop tool

Conclusions

1. The definitions of 'validation' and 'verification' in ISO 9000:2005 aren't sufficiently distinct and are easy to misinterpret.
2. ISO/IEC/IEEE 15288:2015 is a little better in adding footnotes - but not much.
3. 'validation' and 'verification' should never be used without qualification e.g. 'validating a model of the system', 'validating a requirement set (document)', 'verifying the system design' are correct. 'verifying a system' or 'validating a system' are ambiguous and incorrect uses.
4. The OSLC doesn't support both 'verification' and 'validation' of a system design and incorrectly uses 'validation' as if it were 'verification'

References

1. ‘validate, v. : Oxford English Dictionary’, The Oxford English Dictionary. Oxford University Press. [Online]. Available: https://oed.com/view/Entry/221192. [Accessed: 07-Aug-2019].
2. ‘verification, n. : Oxford English Dictionary’, The Oxford English Dictionary. The Oxford University Press. [Online]. Available: https://oed.com/viewdictionaryentry/Entry/222504 [Accessed: 22-July-2019].
3. ISO 9000:2015’, ISO 9000:2015 Quality Management Systems - Fundamentals and Vocabulary - Fourth Edition. p. 7, 15-Sep-2015.
4. ISO/IEC/IEEE 15288:2015’, ISO/IEC/IEEE 15288:2015 Systems and Software Engineering - System Life Cycle Processes. 15-May-2015. https://www.iso.org/standard/63711.html
5. UK MoD Requirements and Acceptance. [Online]. Available: Activity: Confirm Final Design Satisfies System Requirement Document (SRD) [Accessed: 05-Apr-2020].
6. Office for Nuclear Regulation (ONR) [Online]. Available: Generic Design Assessment (GDA) of new nuclear power stations [Accessed: 05-Apr-2020].
7. ‘ISO/IEC/IEEE 29148:2018 Systems and software engineering — Life cycle processes — Requirements Engineering’, ISO/IEC/IEEE 29148:2018 (E), Nov. 2018.
8. Open Services for Lifecycle Collaboration [Online]. Available: https://open-services.net/ [Accessed: 05-Apr-2020].
9. Open Services for Lifecycle Collaboration - validatesRequirement [Online]. Available: https://archive.open-services.net/bin/view/Main/QmVocabulary.html#validatesRequirement [Accessed: 06-Apr-2020]
10. Open Services for Lifecycle Collaboration (OSLC) Wiki. QM Scenario: Traceability [Online]. Available: https://archive.open-services.net/bin/view/Main/QmScenariosTraceability.html [Accessed: 05-Apr-2020].
11. ‘Open Services for Lifecycle Collaboration - Quality Management Specification Version 2.0’. [Online]. Available: https://archive.open-services.net/bin/view/Main/QmSpecificationV2.html#QM_Resource_Definitions. [Accessed: 05-Apr-2020].
12. : IBM Knowledge Center. Overview of Rational Quality Manager. [Online]. Available: https://www.ibm.com/support/knowledgecenter/SSYMRC_6.0.6.1/com.ibm.rational.test.qm.doc/topics/c_qm_overview.html [Accessed: 06-Apr-2020]
13. IBM Knowledge Center. Links Across OSLC Domains. [Online]. Available: https://www.ibm.com/support/knowledgecenter/SSYMRC_6.0.6.1/com.ibm.rational.rrm.help.doc/topics/r_rm_link_domains.html [Accessed: 06-Apr-2020]

Untappd Application

One of the many apps on the mobile phone is Untappd. This allows a user to ‘check in’ a beer they’ve tried and rate it. In addition you can identify where it was bought from or the pub in which you’re drinking it. As with most apps nowadays there is a social side and you can add friends, tag their check-ins and generally spy on them. It’s very good at helping maintain a list of what beers you’ve enjoyed and what you’ve had before which is useful when the memory starts to fail. Of course the emergent behaviour is that it encourages the user to explore and try more beers - no doubt something behind the design of the app and it is clear by the feedback from brewers that the data collected from the users is made available (presumably at a price) to the brewers so that they can see what users think of their beers.

I wanted to produce a list of what I’d tried but in the app and on the website there is no mechanism that simply produces a straight list - both produce a more ornate version with photos and details. What I really wanted was one in alphabetic order by the beer name and possibly by the brewer’s name. As a Supporter ( $5 for a month) you can export your data as CSV and JSON. So what sort of (data) model would best fit this?

The data header names are:-

• beer_name
• brewery_name
• beer_type
• beer_abv
• beer_ibu
• comment
• venue_name
• venue_city
• venue_state
• venue_country
• venue_lat
• venue_lng
• rating_score
• created_at
• checkin_url
• beer_url
• brewery_url
• brewery_country
• brewery_state
• flavor_profiles
• purchase_venue
• serving_type

This seems pretty straightforward once you know that IBU = International Bitterness Unit - a measure of how bitter the beer is and ABV is the percentage Alcohol By Volume.

Entities or Concepts

There appear to be some explicit concepts or entities:-

• Beer
• Brewery
• Checkin
• Venue

Although there are names for Beer and Brewery the unique identifier is probably the beer_url and brewery_url since it is possible that beers or breweries might have the same name. These URLs are internal to Untappd and are the address at which the description of the Beer or Brewery are presented.

Venue potentially appears in many guises. It might be the place at which the beer is drunk or it might be the place from which the beer was bought or both. The way in which the app works encourages the user to add the location at checkin which is presumably usually where the beer is tasted. This appears to populate the ‘venue_…’ columns. The purchase_venue is added separately by the user and looking at the data this simply holds the name so there is a potential problem where two purchase venues share the same name. Why they don’t use a venue_url is a mystery since this could be used to link purchase_venue and venue_name et al. It only seems sensible that a purchase venue should be a subset of the total set of venues (with addresses, locations et al). This might need tweaking since an online Purchase Venue might not have or show a geographical address and hence a venue_url is a sensible identifier.

The serving_type (‘draft’, ‘can’, ‘bottle’) might initially seem to be a property of the beer but it’s really the checkin or tasting event as the same beer might be tried in many different styles at different times.

As a first pass gathering the attributes of the entities we have:-

Beer

• name
• type
• abv
• ibu
• url (unique identifier)
• flavour profiles

Brewery

• name
• state
• country
• url (unique identifier)

Checkin

• date / time
• url (unique identifier)
• rating
• comment
• serving type

Venue

• name
• city
• state
• country
• latitude
• longitude

Although Untappd doesn’t state this it is best if the Purchase Venue is simply another Venue. This creates a problem because it only contains a name whereas the (Checkin) Venue is must have the combination of name + latitude + longitude as the unique identifier. We could potentially solve this by creating our own dummy identifier for Venue and some means to check / parse both purchase_venue and venue_name a) to link them and b) to spot any potential conflicts.

Relationships

Entities without relationships are nearly useless. We need relationships to establish patterns, identify boundaries and describe behaviour.

Some instinctive first choices are:-

• Brewery brews Beer
• Checkin at Venue

We need to associate Beer and Checkin and since Checkin isn’t a person we could try

• Checkin concerns Beer

and to tie the place where the beer is served (and usually consumed)

• Beer served at Venue

which is a separate assertion from

• Beer purchased at Venue

which gives us the a first-pass model.

Untappd Export Data Model - First Pass

This isn’t complete, however.

1. The Untappd app allows a Checkin without requiring either a serving or a purchase Venue
2. The rating may well be affected by how well the beer is looked after at the pub or, say, how near it is to the ‘best before’ date if in a bottle or can which affects both the purchase venue and the serving venue. The same beer might be given a different rating at a different venue so the rating is a combination of the [beer + serving venue + purchase venue].
3. We can tie these together in a graph database by adding the Checkin identifier (checkin_url) property to the ‘served at’ and ‘purchased at’ relationships.

Untappd Export Data Model - Second Pass

The flavour profile contains a set of adjectives to describe the taste of the beer. This is something set up by the Untappd database, presumably with input from each brewer, doesn’t vary with the checkin and is therefore a property of the beer but distinct from the user’s comments on drinking it (the ‘comment’ attribute of Checkin). We could leave these as a list property but might we could make them more explicit by having Taste Descriptor as a separate entity and associating it with Beer:-

• Beer described as Taste Descriptor  e.g. ‘Headwaters Pale Ale described as smooth’

Since it’s very easy to navigate / query relationships in a graph database and there are only a handful of serving types we might as well having Serving Type as a separate entity.

There are a number of possibilities:-

• simply tie it to a particular Checkin (which is already tied to a Beer and Venue(s))
• tie it to the Beer and tie this to the Checkin using the checkin_url

The last method seems to make for more natural reading as we can then assert:-

• Beer served as Serving Type at Venue e.g. ‘Postman’s Plum Porter served as draft at The Old Post Office’

In using the Untappd App many of these relationships and entities are optional since a checkin only requires the user to identify the beer and hence each entry or row need only contain:-

• Checkin (provided by the Untappd app)
• Beer
• Brewery (provided by the Untappd app)

If we want we can traverse the succession of Checkins by adding ‘Checkin follows Checkin’ .

The final model to import against is now below.

Untappd Export Data Model - Third Pass

Note that in a directed graph database such as Neo4J <http://www.neo4j.com/> the direction of the relationship does not affect the design of the entity since it is as easy to traverse a path in either direction by specifying the direction in the query, for example using the CYPHER query language:   

MATCH breweries_beers_in_cans = (br:Brewery)-[:BREWS]->(b:Beer)-[:`SERVED AS`]->(ty:Serving_Type)   
WHERE ty = ‘Can’    
RETURN breweries_beers_in_cans`

A CYPHER Graph Query Simply Follows a Path

This is unlike a traditional SQL design where a relationship is inferred by placing a foreign key identifier value in the table with which the entity has a relationship.

The other advantage is that a properly constructed graph can be read as a simple statement or assertion so queries look much more like natural language.

In any notation it is an error not to define the relationship or define a relationship since:-

1. If there is no relationship with anything it is not a description of architecture in any meaningful sense and cannot be used / queried (so what’s the point?)
2. Without a defined relationship the meaning / semantics are unknown and different readers will interpret in different and therefore inconsistent ways.
3. Now to import the Untappd data using this model to structure, cleanse/modify/transform the data to suit the queries to be run ....

External References

• Neo4J Graph Database http://www.untappd.com/
• Untappd http://www.untappd.com/

Adelard’s Safety Assurance Metamodel

Argument supports Claim’ seems straightforward enough.

The first problem is ‘Subclaim’. A ‘sub’-anything is not a new type or metamodel element - it simply identifies a position in a hierarchy (much like ‘subsystem’ does). A Subclaim is really therefore a Claim ... beneath a Claim. We can therefore remove Subclaim and simplify.

Adelard’s Safety Assurance Metamodel - Simplified to Remove Subclaim

We then have ‘Claim is a (sub)claim of Argument’ which makes no sense. If Adelard wanted to assert this they should really have the relationship between Claim and Argument.

What is a Claim of an Argument? An Argument makes no Claims. It may support or oppose a Claim. If an Argument makes Claims then it isn’t an Argument - a single type - it is a composite type which is no longer simply an Argument. If you absolutely needed to you could state ‘Argument has part Claim’.

The definition on the Adelard site for Claim is:

Claim…  ‘a statement asserted within the argument that can be assessed to be true or false.

This cannot be true because it requires that a Claim is always part of an Argument - it has no independent existence (which is inconsistent with the metamodel). It does support the proposed ‘Argument has part Claim’ triple however.

We then have ‘Evidence is evidence for Subclaim’ which becomes ‘Evidence is evidence for Claim’. ‘is evidence for’ doesn’t tell us whether the evidence supports the Claim or opposes it so it really isn’t very helpful.

All of this is a bit of a muddle of a metamodel. A sub claim can be represented by ‘Claim has part Claim’. A sub-Argument is similarly ‘Argument has part Argument’ Both of these are recursive relationships involving a single entity / concept. What isn’t how the ASCAD tool really implements the original metamodel i.e. whether it constrains the allowed relationships. This is a problem because if the metamodel is wrong and is implemented in the tool then the semantics of any assurance model built will also be wrong. This is why application developers need to be transparent and share with us their underlying (implementation-free) model so we can scrutinise it. Ironically effective assurance cannot work without the underlying metamodel being publicly visible.

Another problem is that the definitions aren’t really consistent with the metamodel diagram shown. The text states:

Argument… This element is optional, but often it is good practice to include to explain the approach to satisfying the parent claim.

If the approach to supporting a claim is straightforward or well understood by the intended audience, it is permissible to simply link directly from the supporting claim.

What isn’t clear is whether the metamodel has an additional ‘Evidence supports Claim’ triple (in addition to the displayed ‘Evidence is evidence for Claim’) or whether they are inconsistent references to the same triple. From a practical level it is poor practice not to outline your arguments because in the early days of a design it helps to outline the Claims and supporting Arguments and discuss these with your regulator or Assessor, for example, to explain how and why the design has been partitioned. Having arguments is therefore a necessary part of the system design at an architectural level and also helps early design reviews internally.

Simply rationalising using the Adelard definitions then produces the following.

Adelard’s Safety Assurance Metamodel - Simplified Based on the Text Definitions

which still isn’t particularly good in terms of semantic assertions. The following is a lot clearer and caters for situations where Arguments oppose Claims and Evidence opposes Arguments. You can also then add ‘Claim opposes Claim’ to describe a counter-claim. You could also add ‘Argument opposes Argument’ - a counter-argument and ‘Claim opposes Claim’ - a counter-claim.

Improved Claim Argument Evidence (CAE) Assurance Metamodel

For many of these reasons this is why for TRAK we created a [metamodel that was more consistent and expressive for assurance][2] allowing many more statements / assertions to be made. It not only allows support for Arguments and Claims but opposition and also can describe the outcome i.e. the acceptance by the Assessor that a Claim is proved or disproved.Importantly, the Claims can be attached to any model element. A model of the system of interest using TRAK can therefore combine this with an assurance model of the system of interest - they are simply views on a single integrated model. It can also then be used to produced typical design verification artefacts such as a Compliance Matrix.

TRAK Assurance Metamodel Fragment based on Claim, Argument and Evidence (CAE)

Everyday Examples of Assurance

Typical examples of assurance which can be modelled include:-

• design assurance / design verification. In this case the claim is that the design meets requirement 12345, followed by a reasoned set of arguments. Following the verification method this is supported by evidence.
• process assurance. You might want to claim conformance for your (system engineering) processes with one or more international standards e.g. [ISO/IEC/IEEE 15288][3], [ISO/IEC/IEEE 42020][4]
• design integrity. Making a set of claims and supporting arguments concerning the architecture, separation, and resilience features to support structuring an assurance case.

References

• [1]: https://www.adelard.com/asce/choosing-asce/cae.html - Adelard Safety Assurance Metamodel. Accessed 27th-July-2018.
• [2]: https://trakmetamodel.sourceforge.io/claim_argument_evidence_CAE.html - ‘TRAK Metamodel - Claim, Argument and Evidence Tuples’. Accessed: 06-Feb-2020.
• [3]: https://www.iso.org/standard/63711.html - ‘ISO/IEC/IEEE 15288:2015 Systems and Software Engineering - System Life Cycle Processes’, ISO/IEC/IEEE 15288:2015, May 2015.
• [4]: https://www.iso.org/standard/68982.html - ‘ISO/IEC/IEEE 42020:2019 Software, Systems and Enterprise - Architecture Processes’, ISO/IEC/IEEE 42020, Jul. 2019. .

The MDG for TRAK is a plugin for the Sparx Systems Enterprise Architect UML modelling tool ('EA') that makes it easy for modellers/architects to create models and views using the TRAK Enterprise Architecture framework. It takes the form of a XMI text file (actually a UML model itself) that EA loads when it starts up.

Background

The plugin was created when developing the TRAK framework within London Underground Ltd. It is an implementation of TRAK that provides a set of objects in palettes that correspond to the TRAK specification. The aim of this was to make it easier for modellers/architects to produces models and architecture views that conform to TRAK.

Features

Quicklink - Context Sensitive Creation of Links and Elements

The MDG for TRAK plugin provides the following features:

• a set of object types e.g. ‘Architecture Task’, ‘Argument’, ‘Claim’, ‘Concern’, ‘Contract’, ‘Enterprise’, ‘Enterprise Goal’, Evidence’, ‘Mitigation’, ‘Job’, ‘Organisation’, ‘Physical’,  ‘Project’,  ‘Protocol’, ‘Requirement’, ‘Risk’, ‘Role’,  ‘Standard’, ‘Software’, ‘System’, ‘Threat syn. Hazard’, ‘Vulnerability’, etc - chosen from toolbar palettes
• a set of relationships (connectors)
• context-sensitive creation of relationships between objects - depending on the start/finish objects selected and the view
• a canned set of TRAK architecture views (diagrams)
• predefined searches e.g. ‘All Systems’, ‘All Open Concerns’, ‘Objects without Any Links and Not in a View’, ‘Objects Without a Description’
• Model Views based on the searches - list the objects in the Project Browser

Use

A short video clip of an early version of the Sparx MDG Technology for TRAK is available on YouTube.

License

The MDG for TRAK is released under the GPL (GNU Public License) as open source.

Support

Support in terms of raising bugs, support and feature requests is managed through the Sourceforge trackers at https://sourceforge.net/projects/mdgfortrak/support. This keeps everything in the open and means that any bugs etc. can be prioritised and responded to in a systematic way. There is also a MDG for TRAK wiki on Sourceforge at https://sourceforge.net/apps/mediawiki/mdgfortrak/.

Download

The MDG for TRAK can be downloaded from Sourceforge at https://sourceforge.net/projects/mdgfortrak/files/TRAK_MDG.zip/download.

External References

• MDG for TRAK Project home page
• TRAK Enterprise Architecture Viewpoints
• TRAK Enterprise Architecture Metamodel
• Sparx Systems Enterprise Architect