Using Moving Model-based Systems Engineering (MBSE) Techniques to Assist the Earliest Phase of the Lifecycle

Introduction

Fail early, fail often, but make sure when one fails one learns from ones failures while there is still time to fix the system before delivery to the customer. Yes, most agree that all systems are designed for mission success, but even today many still delivered to the customer with inherent design problems which limit effectiveness. The key to maximizing mission effectiveness is understanding why a system can fail while still in the development process. Also, understanding why they fail is more important that understanding the how the system succeeds, especially if a system can fail in a catastrophic manner. Would it not be great to have an iterative process that allows the designers to see when a system is failing to meet the users requirements early in the development process? Also, to have that process be iterative, so the impact of changes can be seen real-time as the concept is defined and the system designed? Do all this inside the engineering safety net of Model Based Systems Engineering? This research will show an alternative process to classic systems engineering and optimization analysis, where system design decisions can be statically and dynamically modeled in a Model-based Systems Engineering environment and “what if” type of changes can be answered and analyzed using  embedded simulation. This research will demonstrated the process with use case with a highly relevant real world problem of counting the threat of small commercial unmanned systems.

As systems grow in size and complexity, classic systems engineering methods and system optimization methods become increasingly cumbersome and ineffective. As an alternative to classic systems engineering and optimization analysis, system and component responses can be statically and dynamically modeled in a model-based systems engineering (MBSE) environment to gain an understanding of system effectiveness, similar to classic Discrete Event simulation and fault analysis methods predicting system suitability. The results can inform the system designer of key areas to focus effort to maximize system effectiveness and minimize failure.

Typically, system optimization for a DOD development program starts with the analysis of alternatives (AoA) phase of the acquisition lifecycle. The AoA is done by an independent team to help move a program from idea, sometimes known as concept demonstrator, to a Program of Record. AoA effort, though many times done very well the subject matter experts with what is known at the time, does not directly flow into the development process. Much to the AoA effort stops after the AoA phase is complete and key knowledge is lost. This research will demonstrate how using moving model-based systems engineering (MBSE) techniques to earliest phase of the lifecycle, including AoA and concept development phases, can enable better understanding of the problem statement and solution trade-space while allowing the knowledge to persist into the later development phases.

The supposition for this research is that optimizing solution trade space needs to be a continuous effort, like managing cost and schedule, not just done during the AoA, but throughout the development lifecycle. If effort is put in early on to capture and automate the trade space analysis process inside the core systems engineering development tool, than as assumptions, constraints, and capabilities evolve over time, the development effort can adjust accordingly to provide the best value to the customer.

To help understand how MBSE can be used as part of the AoA trade space analysis process and continue re-look at the trade space optimization, this research will build a system model of an example Counter Unmanned System (C-UxS), add scripts for automated trade-space analysis, run multiple simulations, and report on the findings.

Background

Real-World Problem

In a world of low-cost unmanned aerial systems, such as the very advanced DJI Phantom 4 available to terrorists on web stores with one-click and a thousand dollars, the threat of low cost commercial-off-the-shelf (COTS) technologies for unmanned systems is a serious concern. Thought the Phantom 4 is a small UAS, this class of technology is a multi-domain problem: air, surface, sea, and ground, which uses “UxS” as short hand for this large threat. Specifically, there are numerous open source articles citing the use of commercial small unmanned aerial systems (UAS) “out of the box” and similar COTS technologies used in unmanned surface vessels (USV) as improvised weapons. For this research, smaller sized COTS UAS, also known as Group 1–3, will be modeled. Small UAS’s selection is based on the challenging problem they create. Small COTS UAS are harder to defeat consistently due to rapid technological advancement cycle, their worldwide availability, and small bird-like profile. The improvised weapons can be used in congested airspace, which add complexity for detection and shortens engagement windows. Small COTS UAS may seem almost toy-like, but the reality is that they can easily be turned into dangerous autonomous weapons.

Concept Development

The intent for the generic product development process is to develop it in a way that meets all of the stakeholders needs and is value added to the end user, in other words the positive value is significantly greater than the negative value. The product is still desirable even if there is some inherent risk in its use. For example, cars or motorcycles can be in deadly accidents, but there is value in their transportation features. The systems engineering process is one of the principle methods used to achieve the goals of product development and will be the basis for this research. This research will focus on the intersection between products and systems, specifically on the intersection of complex products and systems.

Systems Engineering

As defined by INCOSE, “Systems engineering is an important investment in the development of products, and the higher complexity of a product, the better value of that investment. As defined, systems engineering is an interdisciplinary approach and means to enable the realization of successful systems” (INCOSE 2017). The key term in the INCOSE definition is “successful,” and there are multiple processes, methods, and tasks to help a development effort be successful. INCOSE further defines systems engineering as:

Systems engineering is an engineering discipline whose responsibility is creating and executing an interdisciplinary process to ensure that the customer and stakeholder’s needs are satisfied in a high quality, trustworthy, cost efficient and schedule compliant manner throughout a system’s entire lifecycle. This process is usually comprised of the following seven tasks: State the problem, investigate alternatives, model the system, integrate, launch the system, assess performance, and re-evaluate. These functions can be summarized with the acronym SIMILAR: State, Investigate, Model, Integrate, Launch, Assess and Re-evaluate. This Systems Engineering Process is shown in Figure 1. It is important to note that the Systems Engineering Process is not sequential. The functions are performed in a parallel and iterative manner. (INCOSE 2017)

Relating the discussion above and the systems engineering process, the first and second tasks are the initial tasks of concept development. The third task of the systems engineering process, model the System, is the primary emphasis for this research. It uses the model to allow for constant reevaluation of the design solution compared to the alternatives. For static development efforts, investment in building a model may not be required, but for dynamic efforts, it might be essential.

System Modeling

An alternative view of this linear recursive Systems engineering Process definition is shown in Figure 2 to provide additional emphasis on the modeling task. It encompasses the processes from defining the need to launching the system, promoting the modeling from a sub-task in a linear process to a recursive wrapper encompassing the primary steps of the systems engineering development process.

As shown, this is the way the systems engineering process is transitioning from a linear process, to a model-centric-based process. Now that the model is proposed to be the heart of the system development process, it is important to understand the “what and the why” of modeling.

As stated above, the goal is to design an effective desirable system; to help reduce the risk of ending up with a poor design, modeling allows the developers’ team a better understanding of the system. According to Rumbaugh, the models are important to do the following:

Capture and state requirements and domain knowledge so that all stakeholders may understand them.

1. Think about the design of the system.
2. Capture design decisions in a mutable form separate from the requirements.
3. Produce usable work products.
4. Organize, find, examine, filter, manipulate, and edit information about large systems.
5. Explore several solutions operationally, economically, and environmentally.
6. Master complex systems. (Rumbaugh 1999, 16–17)

Sussman (2000), also felt using viewpoints modeling techniques to better understand complex systems, to interactively experiment with different variations of the system, and to simulate the system in a representative environment is very important to good systems design. Adding to these insights by Rumbaugh and Sussman, this research will expand the thought that models are also important to allow for efficient detailed exploration of engineering trade space.

Expanding on the second point above, models positively influence the thought process on the design of a system. The memorable graphical views and the large amount of context the right side of the brain can process and store are illustrated by the colloquialism, “A picture is worth a thousand words” (Ramos 2012, 103). Personal experience has found that when one shows a large audience a spreadsheet with a lot of values, one gets limited response (left brain); show a graphical diagram, and audience response increases. Showing an interactive graphic where audience members can suggest inputs, however, yields an enthusiastic discussion that pushes design space (interaction between right and left brain). In addition, after a vigorous discussion, participants remember and ponder the discussion and offer additional inputs long after the event. Also, if one can extend the event into a process, then one enhances the design process and insight by one team that is captured into the single model for others to experience rather than have that experience lost into a stove pipe work environment.

It is not enough merely to model the system, but all of the developmental modeling must be done simultaneously in an integrated development environment. By requiring the systems engineering team to use what some now call a “Single Source of Truth” or a “Single Version of Truth” while refining the system reduces the chance of incompatible changes that show up later darning integration events. Both integrated development environment concepts are defined as “Single Source of Truth” (SSoT). Defined by Grealou, “It is the practice of structuring information models and associated schemata, such that every data element is stored exactly once” (Grealou 2016, 1). From a development perspective, at an engineering level, it means that engineering artifacts only have one instance, in the relevant master system, following a specific process or set of processes. “SVoT enables greater data accuracy, uniqueness, timeliness, alignment, etc.” (Grealou 2016,1) The concept of a SSoT for a development effort, is very powerful because currently many different disconnected models are used for analysis of alternatives, tracking critical capabilities, and design development, allowing multiple incompatible versions of the truth to exist. For example, based on additional constraints imposed mid development, design trades that were true during the analysis of alternatives modeling efforts, might not be true at this later stage. Additionally, the model used for analysis of alternatives was not maintained past that initial portion of the design phase, so those details of why something was true then, but not true at this stage are now lost. As a result, sub-optimal trades introduced to the system during the AoA might persist into the design phase.

Model-Based Systems Engineering

As discussed in Figure 2, a growing area in systems engineering is basing the systems engineering process on the model itself and removing the need for paper-based documentation. This is the emergence of Model-based Systems engineering (MBSE) as the primary framework for complex system development. As defined by Vaneman (2017a, 5), “Model-based Systems Engineering (MBSE) is the formalized application of modeling (both static and dynamic) to support systems design and analysis, throughout all phases of the system lifecycle, through the collection of modeling languages, structure, model-based processes, and presentation frameworks used to support the discipline of systems engineering in a ‘model-based’ or ‘model-driven’ context.”

The four tenets of this definition are as follows (modified from OMG, 2012):

• Modeling Languages – Serves as the basis of tools, and enable the development of system models. Modeling languages are based on a logical construct (visual representation) and/or an ontology. An ontology is a collection of standardized, defined terms and concepts and the relationships among the terms and concepts (Dam 2015).
• Structure – Defines the relationships between the system’s entities. It is these structures that allow for the emergence of system behaviors and performance characterizations within the model.
• Model-Based Processes – Provides the analytical framework to conduct the analysis of the system virtually defined in the model. The model-based processes may be traditional systems engineering processes such as requirements management, risk management, or analytical methods such as discrete event simulation, systems dynamics modeling, and dynamic programming.
• Presentation Frameworks – Provides the framework for the logical constructs of the system data in visualization models that are appropriate for the given stakeholders. These visualization models take the form of traditional systems engineering models. These individual models are often grouped into frameworks that provide the standard views and descriptions of the models, and the standard data structure of architecture models. The Department of Defense Architecture Framework (DODAF) and the Zachman Framework are examples of frameworks that may be encountered. (OMG, 2012)

This research effort used MBSE techniques to focus on the “model the conceptual system” phase of the program while fully understanding that the intent of modeling effort is to show relevance the complete lifecycle of a system, not merely to illustrate the concept development phase. Modeling performed early in the development process is done to refine the problem statement and support the analysis of alternative process in an iterative manner until a viable design solution is agreed on by the stakeholders and end users. The power of MBSE of complex systems is the investment in building a detailed model early in the concept development process to continue to provide value throughout the entire lifecycle of the systems through the disposal phase. This research, as stated above, focused on modeling the system so that the model can be used to drive the decision-making process during concept development. This requires a discussion about specifics on modeling the system.

Model the system: Models will be developed for most alternative designs. The model views for the preferred alternative will be expanded and used to help manage the system throughout its entire lifecycle. Many types of system models are used, such as physical analogs, analytic equations, state machines, block diagrams, functional flow diagrams, object-oriented models, computer simulations and mental models. Systems Engineering is responsible for creating a product, in this case a complex system, and also a process for producing it. So, models should be constructed for both the product and the process. Process models allow us, for example, to study scheduling changes, create dynamic PERT charts and perform sensitivity analyses to show the effects of delaying or accelerating certain subprojects. Running the process models reveals bottlenecks and fragmented activities, reduces cost and exposes duplication of effort. Product models help explain the system. These models are also used in tradeoff studies and risk management. As previously stated, the Systems Engineering Process is not sequential: it is parallel and iterative. This is another example: models must be created before alternatives can be investigated. (Bahill 2009, 2–3)

As an important part of MBSE different views, the concept of risk-informed design plays a large role during concept development. In fact, DOD 5000.2 forces the program manager to address risk as part of the analysis of alternative (AoA). The original intent of risk-informed design, as initially defined by NASA, is to make informed design trades in consciously reducing risk to the crew and the mission, but technique has been expanded to describe risk inside a MBSE approach (Moulds 2016).

The goal for both risk management and system architecting is to limit the number and effect of the unknown unknowns causing failures during the system’s lifecycle. Management of what you know and what you do not know, helps limit the chance that uncertainty drives the process and goals of development (Antunes 2015). “To people who lived centuries ago, risk was simply the inevitable nature of chance; an occurrence beyond the realm of human control.” (Mun 2015, 25). Today, with the use of modern Model-based Systems Engineering methods, all areas that impact mission effectiveness can be discovered and mitigated early in the lifecycle of a system. In terms of system development, if the “hard stuff” is deferred, then the residual risk does not change, leaving a risk of failure high, or at lease uncertain, to the very end (Maier 2009). For the purposes of this research, ineffective solutions, risks, faults, reliability issues and failures affecting mission performance are synonymous and are represented in the model in a uniform way. This means that one should first focus on correcting components and processes that have the greatest impact on mission effectiveness to have the greatest return on investment. This paper describes how the use of classic methods inside a modern system architecture modeling tool to help in the discovering, understanding, and documenting issues early enough in the program, thus optimizing resources maximizing the impact on the development program. Engineers often aim to solve the exciting problems first, delaying resolution for the more boring, yet relevant (and potentially costly) ones.

Operational Concept

To start the development process, the DOD has generated multiple urgent needs statements and has purchased several commercial off the shelf products to concept demonstration. Additionally, the DOD has hosted numerous events, such as Black Dart, where a live exercise environment allows to developers to test their C-UxS technologies against live systems. The Design Reference Mission (DRM) has not selected any specific operational concept but elected to develop a generic framework for C-UxS. This generic framework covers the entire kill chain of find, fix, track, target, engage, and assess, but it does not identify any particular technology. The operational requirement is to develop a solution to deter or defeat the representative UxS before they can enter the restrict zone as defined by the end user.

The concept and the architecture is based on the kill chain defined above but segregated so a technology in one sub-domain is not so tightly coupled with a technology in a separate sub-domain that the solution becomes vendor locked.

1. Find: This sub-domain is normally the trigger when the process transitions from a system waiting for something to happen, to one that is actively processing an event. The sub-domain has numerous names, such as detect, battlespace awareness, collection capability, and the sense function.
2. Fix: This sub-domain is the transition between a raw detection and a establish track. The sub-domain has numerous names, such as classify, processing exploitation capability, and perform tracking functions such as form track, fuse track measurements, correlate tracks, and associate tracks.
3. Track: This sub-domain is the monitoring phase where a detection event meets a minimum criteria such that it should be monitor and persist in the system. This sub-domain covers the transition from the battle space awareness capability to the command and control capability and completes the perform tracking functionality started on the Fix sub-domain described in item 2.
4. Target: This sub-domain is primarily the classic decision loop. A track is evaluated based on risk, and when the risk reaches a defined threshold, an appropriate course of action is selected. This is part of the decision or control part of the command and control capability and decision part of the mission execution functionality.
5. Engage: This sub-domain is one of the easier ones to understand and describe. It is part of the force application capability and the perform engagement group.
6. Assess: This sub-domain is the final step of the active process where all knowledge from the event is stored and analyzed. This sub-domain also completes the transition from active process back to the waiting phase. This is part of the Understand Command and Control capability and the mission analysis functionality.

Other capabilities that are part of this DRM, but not explicitly part of the kill-chain are force support, logistics, communications and computers, protection, corporate management and support, interactions with the UxS. Similarly, other functionalities also part of the DRM are mission planning, and enterprise IT, and network infrastructure.

From an implementation perspective, the C-UxS is composed on multiple interconnected capacities that together or alone provide the required functionality and as an integrated system, provide the required capability. The government has the additional requirement to “own the middle” so a solution does not create a vendor monopoly but at the same time allowing for proprietary functionality for find and engage. In addition, other functionalities such as fusion, which technically “lives in the middle,” can also be proprietary but still subject to change by the government Lead System Integrator (LSI).

PROBLEM STATEMENT

Requirements, assumptions, constraints, and stakeholders change and evolve rapidly, but the current DOD 5000.2 process does not allow for periodic reassessment of the concept, even when the concept that was initial viable as part of the AoA may not be when the program gets to preliminary design. Because of this lack of easy reassessment built into the process, many times what was initially asked for is developed, but in the end it is not what is really needed. By imbedding the assessment into the system model, mission effectiveness can be reassessed periodically against the design baseline. In this way, corrections can be made earlier in the process, at least as early as the change in the need is discovered. Some would call this requirements creep, but the goal is to make sure the requirements are valid.

RESEARCH QUESTIONS

• How can Model-Based Systems Engineering (MBSE) be used to forecast and investigate mission effectiveness, caused by material and design limitations, to inform and influence the early stages of the system design process?
• Once the architectural model is complete with embedded mission effectivity analysis, how can multiple runs of the simulation with varying the component level effectiveness probabilities of design choices be used to determine overall system sensitivity?
• How can the results of the system sensitivity results and analysis be used to optimize design and reliability requirements?
• Based on the architectural model with the simulation, how can one use sensitivity analysis techniques to adjust the project’s path forward by having a continuous positive impact on the early stages of the development process?

As depicted in Figure 3, early in the development of a complex system, the process presents the development team seemingly limitless possibilities with unknown correlation with mission effectiveness. The AoA process uses modeling and simulation linked to appropriate metrics to help the team understand cause and effect, but typically this process does not continue past the AoA stage.

“”The problem with this is that the period between AoA and preliminary design review (PDR) is when a system is locked into place. This research will see if the “what if” portion of the AoA process can be embedded into the system modeling process. The premise is that once embedded, it can persist throughout the system lifecycle, allowing consistent “what if” questions to be re-asked as the system is refined and constrained. The architectural model will provide both a common knowledge base of the system along with embedded integrity checking and mission effective analysis.

OBJECTIVES

The objective of this research is to define an MBSE process which ensures that trade space can be evaluated continuously throughout the lifecycle of a system. This process based on existing MBSE research and current Navy engineering practices. Without a complete, repeatable, and embedded method of re-evaluating trade space the development effort could end up with a sub-optimal design. This makes the real-world problem and resulting research  a very important process to define and demonstrate.

OVERVIEW OF THE APPROACH

Research

This research will look at how to merge the AoA modeling process with MBSE developmental and lifecycle processes to capture and expand the “what if” process of concept development. Many times the AoA is based on one set of constraints or view of an ideal system, but very soon after development starts, a new set of constraints emerges. The process compensates but not necessarily with the same information (model) the AoA used, allowing for sub optimization of the designed solution. The premise is that if the AoA and the following development process were based on the common model, then when additional constraints are added, the effect of the constraints would be known as a function or routine model analysis.

To demonstrate this process, updates will be made the model of the NAVAIR Counter UAS Architecture Framework using the Innoslate MBSE tools. This modeling process with embedded simulation will offer insights that classic fault analysis methods will not.

Overview

This study used the MBSE tool to define the Counter UAS capability, functionality, and integrated actions. The effectiveness of the actions was based on the component capabilities and the interactions between these components as work flows through the system and results in a positive or negative result. Based on multiple simulation runs where system input values are randomly varied, predicted system performance can be assessed. In addition, the modeling engine embedded in the tool allows for scripting the decision branches with additional randomized capabilities to varied path selection decisions based on the range provided from user input. The user can vary the inputs in order to see how effective a model is and also allow for automate variation to understand the sensitivity of the model is to specific capabilities and grouping of capabilities. Areas that as found to be highly sensitive can then be further investigated and optimized. For example, C-UxS system might be highly sensitive to a very high-end RADAR, thus driving cost and size, or it could have lower sensitivity allowing for additional trade space in the mixing of multiple lower cost sensors. The goal for a specific site is to be able to select the correct combination of sensors to match cost with required mission effectiveness. The assumption is that a solution that is optimal in one location might not be ideal for another.

Tools

The goal of the effort is to preform both the AoA and the system definition inside a MBSE tool. For this effort a MBSE tool that support both modeling, scripting flow, and simulation will be used. Since this research is a little be a head of the capabilities of current MBSE tools, so of the analysis will be performed in spreadsheet.

Scope

While the concept and understanding of predicting mission effectiveness is almost limitless, this paper focused on the idea that model-based methods currently used to define a system as part of development, can also be used to address mission effectiveness analysis and system sensitivity to specific component capabilities. The simulation effort was limited to the current capability of the MBSE tool. The classes of the sensors were limited to three and the fusion engine will be rule based. The use of a single tool with a single example system to demonstrate feasibility limited the scope of the effort to a manageable level but did not attempt a complete proof of equivalence. In addition, the following assumptions guided the development of the example C-UxS model:

(1) UxS will become smaller, cheaper, and more capable as technology evolves; proliferation will increase as UxS become more capable and less expensive, related new technologies will emerge/evolve that enhance UxS operations. (2) Future decisions will provide adequate resources and organizational structure to support C-UxS capabilities development. (3) Current and future capabilities, to include surface-to-air systems, air-to-air systems, Command and Control Systems, are adequate to deal with large UAS. (4) The cyber domain and electromagnetic spectrum will be more contested in the future. (5) Adversaries will challenge the United States in these areas due to evolving technology and proliferation (Army 2016).

The model expanded on cross-domain solutions where is makes sense, recognizing that the C-UxS mission set exists in every domain, not only in the air. The full DRM is appendix A for the C-UxS real-world problem.

ORGANIZATION

Chapter I of this research provides background for the challenge and defines the research questions and overview. Chapter II provides a detailed walk through of complex system development research in systems engineering, model-based systems engineering, and related use on MBSE tools for informing development based on risk informed decision making. Chapter III presents the tools selected, the model development, model analysis, and key decision to make based on the process. The goal of this chapter is show the reader the process so one can re-use it, but it will also be demonstrated with a simple example of a generic C-UxS system. Chapter IV reveals the conclusions and provides recommendations for further research. The Appendix A is the DRM for the C-UxS system and is the key reference for the demonstration portion of this research. Appendix B has additional snapshots and provides a link to reference to the Innoslate tool that contains the Count UxS System model.

Literature review

CONCEPT DEVELOPMENT PROCESS

Introduction

A key part of the development process is concept generation, and this is reflected in both commercial best practices and in the DOD 5000.2 acquisition process. As defined by Ulrich, “A product concept is an approximant description of the technology, working principles, and form of the product. It is a concise description of how a product will satisfy the customer needs. A concept is usually expressed as a sketch or as a rough, three-dimensional model and is often accompanied by a brief technical description. The degree to which a product satisfies customers and can be successfully commercialized depends to a large measure on the quality of the underlying concept.” (Ulrich 2012, 118) Ulrich also re-enforces the heuristic that “a good concept is sometimes poorly implemented in subsequent development process resulting in a commercial failure, but a poor concept can rarely be manipulated to achieve commercial success” (Ulrich, 2012, 118). The exception to this heuristic is that sometimes the DOD does manipulate poor concepts into operational systems, or adds constraints on the process later in the design phase that cause good concepts to morph into a poor one. A lack of continuous insight does not allow the potential path to failure to be known until it is too late. Ulrich concludes, “The development of a non-optimal concept is unfortunate because good concept generation leaves the team with confidence that the full space of alternatives has been explored.” (Ulrich 2012, 219)

AoA Guidance

From the DOD perspective, “the AoA is an important element of the defense acquisition process.” (DAU 2012) Similarly, an “AoA is an analytical comparison of the operational effectiveness, suitability, and lifecycle cost (or total ownership cost, if applicable) of alternatives that satisfy established capability needs.” (DAU 2012) An important part of an AoA is one of the first DOTMLPF (Doctrine, Organizations, Training, Materiel, Leadership and Education, Personnel, and Facilities) assessments, since understanding non-material solutions early makes it allows the AoA process minimize the and understand the feasible material solutions. The guidebook goes on to state that, “The AoA is not a point analysis, but should be revisited.” (DAU 2012) The concept of revisiting the analysis is often easier said than done, especially on large programs in which the AoA is often more of a paper based effort which is hard to revise, additional the AoA team members who can revise it, have already moved onto the next project. The ability to automate this process and meet the requirement to revisit the analysis could be very powerful. since one now only needs continuity in the tools, not necessarily in the people.

Additional guidance form the guidebook states (DAU 2012) “The AoA is used to identify the most promising end-state materiel solution, but the AoA also can play a supporting role in crafting a cost-effective and balanced evolutionary acquisition strategy. The alternatives considered in the AoA may include alternative evolutionary paths, each path consisting of intermediate nodes leading to the proposed end-state solution. In this way, the analysis can help determine the best path to the end-state solution, based on a balanced assessment of technology maturity and risk, and cost, performance, and schedule considerations. In other words, doing the AoA inside the model allows for more than just the AoA itself, but it establishes a strong foundation and memory for a cost effective and balanced accusation strategy.”

The MITRE System Engineering Guide: Performing Analyses of Alternatives provides a very good guidance for AoAs which is condensed below:

Why do we do AoAs? AoAs are performed to allow decision makers to understand choices and options for starting a new program or continuing an existing program.

Commercial industry also uses “alternative analyses,” but they are usually more focused on lifecycle cost. The plan is important. It should include the following information:

7. Understand the technology gaps and capability gaps—what needs is the intended system supposed to meet?
8. Develop viable alternatives
   1. Define the critical questions
   2. List assumptions and constraints
   3. Define criteria for viable/non-viable
   4. Identify representative solutions (systems/programs)
   5. Develop operational scenarios to use for comparisons/evaluation
9. Identify, request, and evaluate data from the representative systems/programs (determined to be viable)
10. Develop models – Work through scenarios

Know the baseline before starting the AoA, know your stakeholders, beware premature convergence, know your AoA Team, understand the mission ,obtain technical descriptions of the materiel solutions, anticipate problems and be persistent! (MITRE 2017, 438)

A recent GAO (Government Accountability Office) report on defense acquisitions, “Attributes premature focus on a particular solution or range of solutions as a failing of AoAs.” (GAO 2009) The GAO report goes on to state that, “If stakeholders are already enamored of a particular solution, completing a full AoA may be difficult.” (GAO 2009)

The current best practice from this GAO report and practical experience still only recommend and that an AoA is completed before program requirements are set, but this research will look beyond the AoA guidance to see how the AoA process can be part the full system development process.

MODEL-BASED SYSTEMS ENGINEERING

Model-based Systems Engineering was envisioned to transform systems engineering’s reliance on document-based work products to engineering environment based on models. This transformation means more than using model-based tools and processes to create hard-copy text-based documents, drawings, and diagrams. Data in a MBSE environment is ideally maintained within a single repository, and has a singular definition for any model element, and allows for the static and dynamic representations of a system from several different perspectives and levels of decomposition (Vaneman 2017a, 8). As stated above, this single repository can also be thought of as a Single Source of Truth for system development that multiple tool and process can access. An additionally discriminator between document-based work products and model-based tools is that connections between paper documents is usually in the minds of the authors, not captured in the model via interconnect views. Specically, in a MBSE environment, each entity is represented as data, only once, with all necessary attributes and relationships of that entity being portrayed. This data representation then allows the entity to be explored from the various engineering and programmatic perspectives (viewpoints). According to Vaneman’s 2017a paper, a viewpoint describes data drawn from one perspective and organized in a particular way useful to management decision-making. Vaneman defined the compilation of viewpoints (e.g., capability, operational, system, programmatic viewpoints) represents the entire system, where the system can be explored as a whole, or from a single perspective. (Vaneman 2017a, 8).

According to Topper (2013, 419), “the goal of this conceptual model, which is now to be built as a function the MBSE process, is to build a complete, coherent representation of a system and its operating domain, including interactions with other systems and with its environment that is common across the stakeholder community.” Concept development is the early phase in system development where brainstorms turn into prototypes and prototypes each can be assessed for individual and collective merit. These prototypes can be physical or modeled in a tool, which may or may not include simulation. The purpose behind conceptual modeling is garner an understand of what succeeds and what fails in the solution space. It also must show that there is at least one possible solution, including documented analysis that the proposed solutions are actually solutions to the problem. The process Topper developed and used to build the conceptual model described below, involves creating the following artifacts:

• Domain model: This artifact describes what the system and its environment are—it captures the high-level components of the system and its operating environment and establishes the normalized referential framework particularly important for multi-disciplined stakeholder organizations.
• Use cases: These written descriptions of what the system will do capture its expected behaviors and its interactions with external actors.
• Functional model: The functional model describes how the system will accomplish its goals—it breaks the use cases into greater detail and shows activity flows and state transitions among components. Complex functionality, an increasingly common characteristic of modern systems, is difficult to address using traditional assessment techniques. In conjunction with other artifacts presented in this section, new techniques, outlined in the Functional Thread Analysis section, enable and enhance analysis, testing, and evaluation of complex systems, which are difficult to assess using traditional analytical methodologies and tools.
• Structural model: This specification of system structure allocates attributes and operations to system components, expanding and adding detail to the domain model. (Topper 2013, 420)

Note that with modern MBSE tools and languages, use case and be represented in a varitiy of ways, for example UML, written use cases, sequence diagrams, and action diagrams. The Topper’s conceptual modeling process described above is shown in Figure 4.

Similar Areas of Research

The use of MBSE is an area with rapidly advancing techniques, processes, and tool capabilities. Based on research, using MBSE has been ongoing since at least 2010. The areas that are found to be the most relevant to this research are shown below.

MBSE Supporting Development of Systems Architectures in Naval Ship Design

Research by Tepper 2010 in the MBSE Supporting Development of Systems Architectures in Naval Ship Design paper shows the value of using executable models and then documenting the results inside the model as described. Tepper states, executable models can be used in an analysis of alternatives (AoA) by conducting system design trade-offs, and use cases can be incorporated into the model to verify that the system capability satisfies mission requirements. (Tepper 2010, ?)  Key decision making artifacts of MBSE process are trade-space analysis, understanding the impact of changes, and the capability to have version control of changes along with rational of making the decisions. Additionally, Tepper’s research shows that the traceability, as a function of version management of the tools, provides accountability for changes made and impact of alternatives. Tepper concludes. “the designer is able to see how a small change in one aspect of the design can drastically affect the whole.” (Tepper 2010, ?)    Though this research is based on system architectures in naval ship design, similar effects have been seen in the design of any complex system, where small changes have rippled through the design causing extensive and unneeded rework.

MBSE Supporting Risk-Informed Design Methods

Research by Perez (2014) into the application of MBSE tools and processes to Risk-Informed Design (RID) provides the capability to perform risk analysis earl in the life-cycle. The research focused on spaceflight projects, but the concept of performing risk analysis inside the model is valid in many areas. Perez describes, “risk-informed design uses a “minimum functionality” approach, whereby a minimal, single-string system design is first envisioned that only meets basic performance requirements without any regard to overall reliability or safety.” (Perez 2014, ?) A key enabler for this research is that Perez was able to model risk inside the MBSE tool for the first time, which sets the stage for moving the AoA and the analyses inside the tool. Perez’s research is a good example of a MBSE based process applied to a basic system model with scripted simulation for part of the Altair lunar lander system. (Perez 2014) Perez’s research is a good example that theoretically MBSE is capable of supporting a full risk assessment and that a tool like Innoslate could actually support it.

MBSE Supporting of Complex Systems Development

Research by Topper (2014) operated on the premise that MBSE technique facilitates complex design and documentation processes. As stated earlier, the key for the benefit based on this research states that these MBSE techniques supports complex systems the best. Topper goes on to state that “the resulting model is more useful than traditional documentation because it represents structure, data, and functions, along with associated documentation, in a multidimensional, navigable format.” Benefits extend beyond traditional system definition and documentation since, language-based models also support automated analysis methods, such as functional thread extraction. The definition of functional thread analysis is very relevant to this research and is defined by Topper (2014 Page#?) as the following: “The state of a complex system changes continuously as the designed functionality is executed within changing mission phases and environmental conditions. These systems can invoke a large number of functional threads to accomplish (or fail) a required task, and as system complexity grows, it can be difficult to identify critical threads and accurately assess key system performance requirements.” Note the emphasis on the ability to accurately assess key system performance requirements.

Topper’s conclusions state that, “The increase in system complexity precipitated by the advent of network-centric systems, MBSE techniques offer a way to capture, archive, and use information that is essential for complex system design, analysis, implementation, and test and evaluation (T&E) throughout a system’s lifecycle.” In Topper’s research the “conceptual model includes entities, their important attributes and interrelationships, how they operate and behave, and any assumptions made about them.” Topper goes on to state that, “MBSE provides a basis for future analysis studies, model development, simulation efforts, system requirements definition, and program information management.” (2014) Topper (2014) believes that a robust conceptual model does the following and documents in the following list:

• Facilitates communication and collaboration among project stakeholders by standardizing and documenting a common reference blueprint for the project. This basis allows the team to exhaustively explore the system’s conceptual and configuration spaces and identify and assess key parameters in the evaluation of system alternatives.
• Promotes reuse of components and analytical results among projects across a shared domain.
• Enables information management and integrates business and engineering processes into a single model. A conceptual model of the project, especially one that reuses components from previous projects and includes elements from the enterprise architecture as well as the system, allows managers to better estimate the scope, schedule, and resources needed to develop and deploy a complex system.
• Documents traceability from needs to results, supporting verification and validation. (Topper 2014, ?)

MODELING METHODOLOGIES

The modeling technique used will define the system from both a static and a dynamic perspective with the focus on the dynamic action views. For context, static modeling is used to represent the static constitutes of a system model such as the hierarchy, capabilities, functionality, and static interfaces between capabilities and functionalities. Examples of common static diagrams in a DODAF vernacular are Capability View-2 (CV-2), a Operation View-2 and -5a (OV-2, OV-5a). Dynamic modeling is used to represent behavior of the static representation of a system. Dynamic modeling also is used to represent interaction and emergent aspects as a system is exercised. Examples of common dynamic diagrams in a DODAF vernacular are System View-4 (SV-4), an Operation-5a (OV-5b).

Also the dynamic models allow a “glimpse into the black-box” by taking the inner-workings of the internal structure into account. They also allow for inputs from one period to result in outputs for another period (Kao, 2014).