iso15000

TECHNICAL REPORT

ISO/IEC TR 15504-4

First edition 1998-08-15

Information technology — Software process assessment —

Part 4:

Guide to performing assessments

Technologies de l’information — Évaluation des procédés du logiciel — Partie 4: Guide pour l’exécution des évaluations

Reference number ISO/IEC TR 15504-4:1998(E)

Contents

© ISO/IEC 1998

All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and micro- film, without permission in writing from the publisher.

ISO/IEC Copyright Office · Case postale 56 · CH-1211 Genève 20 · Switzerland Printed in Switzerland

•                         Model elements and indicators................................................................................................... 6

•                         Mapping...................................................................................................................................... 7

•                         Translation.................................................................................................................................. 7

•                         Criteria for selecting a model...................................................................................................... 7

•                         Using a model in an assessment................................................................................................ 8

•                         Selection and use of a documented assessment process.......................................................... 8

•                         Compatibility with the requirements........................................................................................... 8

•                         Assessment Input....................................................................................................................... 9

•                         The assessment process.......................................................................................................... 10

•                         Planning.................................................................................................................................... 10

•                         Data collection.......................................................................................................................... 11

•                         Data validation.......................................................................................................................... 11

•                         Process rating........................................................................................................................... 11

•                         Reporting.................................................................................................................................. 12

•                         Selecting a documented assessment process.......................................................................... 12

•                         Role of the competent assessor................................................................................................ 13

•                         Selection of instruments and tools........................................................................................... 13

•                         The purpose and use of instruments and tools within an assessment..................................... 13

•                         Selecting instruments and tools............................................................................................... 14

Annex A (informative)  Guidance on indicators.............................................................................................. 16

Foreword

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical Commission) form the specialized system for worldwide standardization. National bodies that are members of ISO or IEC participate in the development of International Standards through technical committees established by the respective organization to deal with particular fields of technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the work.

In the field of information technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1.

The main task of technical committees is to prepare International Standards, but in exceptional circumstances a technical committee may propose the publication of a Technical Report of one of the following types:

•      type 1, when the required support cannot be obtained for the publication of an International Standard, despite repeated efforts;

•      type 2, when the subject is still under technical development or where for any other reason there is the future but not immediate possibility of an agreement on an International Standard;

•      type 3, when a technical committee has collected data of a different kind from that which is normally published as an International Standard (“state of the art”, for example).

Technical Reports of types 1 and 2 are subject to review within three years of publication, to decide whether they can be transformed into International Standards. Technical Reports of type 3 do not necessarily have to be reviewed until the data they provide are considered to be no longer valid or useful.

ISO/IEC TR 15504-4, which is a Technical Report of type 2, was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology, Subcommittee SC 7, Software engineering.

ISO/IEC TR 15504 consists of the following parts, under the general title Information technology — Software process assessment :

•        Part 1: Concepts and introductory guide

•        Part 2: A reference model for processes and process capability

•        Part 3: Performing an assessment

•        Part 4: Guide to performing assessments

•        Part 5: An assessment model and indicator guidance

•        Part 6: Guide to competency of assessors

•        Part 7: Guide for use in process improvement

•        Part 8: Guide for use in determining supplier process capability

•        Part 9: Vocabulary

Annex A of this part of ISO/IEC TR 15504 is for information only.

Introduction

Process assessment is a means of capturing information describing the current capability of an organization's processes and is initiated as a result of a desire to determine and/or improve the capability of these processes.

This part of ISO/IEC TR 15504 provides guidance on interpreting the requirements set out in ISO/IEC TR 15504-3. As an aid to understanding, the requirements are embedded verbatim in italics at appropriate points within the text of this part of ISO/IEC TR 15504.

The guidance in this part of ISO/IEC TR 15504 is primarily aimed at the competent assessor who has the responsibility for the selection and use of models, documented assessment process and tools for the assessment. The guidance may also be of use to the developers of assessment models, documented assessment processes and tools as an aid to understanding the requirements.

The assessors and other participants in an assessment may use the guidance to gain an understanding of process assessment.

TECHNICAL REPORT  © ISO/IEC                                                                    ISO/IEC TR 15504-4:1998(E)

Information technology — Software process assessment —

Part 4:

Guide to performing assessments

•      Scope

This part of ISO/IEC TR 15504 provides guidance on meeting the requirements for performing an assessment contained in ISO/IEC TR 15504-3.

It provides an overview of process assessment and interprets the requirements through the provision of guidance on the selection and use of compatible models, documented assessment processes, and instruments or tools for assessment.

Process assessment is applicable in the following circumstances:

•        by or on behalf of an organization with the objective of understanding the state of its own processes for process improvement;

•        by or on behalf of an organization with the objective of determining the suitability of its own processes for a particular requirement or class of requirements;

•        by or on behalf of one organization with the objective of determining the suitability of another organization's processes for a particular contract or class of contracts.

•      Normative reference

The following normative documents contain provisions which, through reference in this text, constitute provisions of this part of ISO/IEC TR 15504. For dated references, subsequent amendments to, or revisions of, any of these publications do not apply. However, parties to agreements based on this part of ISO/IEC TR 15504 are encouraged to investigate the possibility of applying the most recent editions of the normative documents indicated below. For undated references, the latest edition of the normative document referred to applies. Members of ISO and IEC maintain registers of currently valid International Standards.

ISO/IEC TR 15504-9:1998, Information technology — Software process assessment — Part 9: Vocabulary.

•      Terms and definitions

For the purposes of this part of ISO/IEC TR 15504, the terms and definitions given in ISO/IEC TR 15504-9 apply.

•      Overview of process assessment

•       Process assessment

Process assessment is undertaken to understand the capability of an organizational unit's current processes. Process assessment deals with all the software related processes (e.g. management, development, maintenance, support) used by an organization. This is accomplished by assessing the organizational unit's processes against a model(s) compatible with the reference model described in ISO/IEC TR 15504-2.

The reference model defines the set of universal software engineering processes that are fundamental to good software engineering and a set of process attributes, applicable to any process, that characterize the capability of an implemented process.

Processes in the reference model are grouped according to the type of activity they address. Each process has a defined purpose describing the high-level objectives that the process should achieve. The purpose statements describe what to do, but do not prescribe how the process should achieve its objectives.

Each process attribute in the reference model represents measurable characteristics of process management providing the capability of the process to effectively achieve its purpose and contribute to meeting the business goals of the organization. The process attributes are grouped into capability levels that define an ordinal scale of process capability and provide a rational route for improvement of each individual process.

The assessment and rating framework is based upon assessing processes. The fundamental assessment output, therefore, consists of up to nine process attribute ratings for each process assessed.

Although the reference model contained in ISO/IEC TR 15504-2 covers a range of processes applicable to the software process, in many cases a subset of these processes may be selected for assessment. For instance the sponsor may wish to focus attention on one or more critical processes or on processes which are candidates for improvement actions. In process capability determination mode, an acquirer may wish to evaluate the capabilities of suppliers only for the processes related to the tender or contract requirements.

The sophistication and complexity of the implemented process will be dependent upon the context of that process within the organizational unit. For instance, the planning required for a five person project team will be much less than for a fifty person team. This process context, recorded in the assessment input, influences how a competent assessor should judge and rate the process attributes for an implemented process. The process context also influences the degree of comparability between process attribute and/or process capability level ratings.

In some circumstances it may be desirable to compare the outputs of the assessment of two or more organizational units, or for the same organizational unit at different times. A number of factors should be taken into account when comparing assessment results. These include but are not limited to;

•        the sample size used to generate the ratings which will influence the precision with which results may be compared;

•        the purposes of the assessments that generated the assessment outputs - it may not be meaningful, for example, to compare an assessment whose purpose was to identify best (or worst) practice with one whose purpose was to identify representative practice;

•        the documented assessment process or model(s) used;

•        the competency of the assessors;

•        the candour of the participants;

•        the time spent on the assessment;

•        the motivation of the assessor (i.e., internal assessor with incentives on the line based on the assessment results or a consultant with a long-term relationship with the organization);

•        the motivation of the assessment participants to be frank and forthcoming.

•       Process rating scheme

The process assessment rating framework is based on assessing processes. The guidance in clause 6 on information collection will help to increase the level of repeatability by different assessors.

Each process has a set of process attribute ratings that constitute the process profile. Process attribute ratings are expressed using the process attribute scale as defined in ISO/IEC TR 15504-2.

The process capability level model defines a six point ordinal scale of increasing process capability ranging from a process which is not capable of achieving its purpose (process capability level zero) to a process which optimizes its performance (process capability level 5). The process capability level model is described in terms of the process attribute ratings that must be achieved in order to achieve a particular level.

When more than one instance of a process is assessed, the assessor will be required to use the recorded assessment information collected on all of the instances to make a judgment on the rating of each of the process attributes assessed for that process.

•       Assessment approaches

•               Self-assessment

A self-assessment is carried out by an organization to assess the capability of its own software process. The sponsor of a self-assessment is normally internal to the organization.

•               Independent assessment

An independent assessment is an assessment conducted by assessors that are independent of the organizational unit being assessed. An independent assessment may be conducted, for example, by an organization on its own behalf as independent verification that its assessment program is functioning properly; the assessment sponsor will belong to the same organization but not necessarily the organizational unit being assessed.

The sponsor of an independent assessment may be external to the organizational unit being assessed, such as an acquirer who wishes to have an independently derived assessment output. The degree of independence, however, may vary according to the purpose and circumstances of the assessment.

•       Assessment process

The assessment shall be conducted according to a documented process that is capable of meeting the assessment purpose.

[ISO/IEC TR 15504-3, 4.4.1]

Irrespective of the type of assessment or the approach adopted, an assessment should be conducted according to a documented process. Some of the key elements of a documented assessment process are briefly described below, and described in more detail in clauses 5, 6 and 7. Note, however, that the guidance provided does not constitute a complete, documented process. Its role is to provide help  in  interpreting  the  requirements  in ISO/IEC TR 15504-2 and ISO/IEC TR 15504-3, and to provide a starting point for selecting or creating a documented process.

The key elements of the assessment process that are described are the documented assessment process, the compatible assessment model(s), and supporting instruments and tools.

4.4.1 Documented Assessment Process

The documented assessment process is the set of instructions and procedure for conducting the assessment. Depending upon the approach, a documented assessment process should provide guidance on the following topics:

•        roles and responsibilities;

•        use of tools and techniques;

•        required resources;

•        sequenced activities and procedures that fall under the following categories:

•        planning

•        data collection

•        data validation

•        process rating

•       Compatible model

A  compatible  model  is  one  that  meets  the  requirements  specified  in  ISO/IEC TR 15504-2.     In summary, a compatible model is one:

•        that is suitable for the purpose of process assessment;

•        whose fundamental elements can be and are mapped to the process and capability dimensions of the reference model in ISO/IEC TR 15504-2;

•        that is equipped with sets of indicators for use during an assessment to gather the information about processes and process attributes;

•        that has a formal mechanism for translating the information gathered using the model into process attribute ratings as defined in ISO/IEC TR 15504-2.

Clause 5 provides guidance on the selection and use of a compatible model. The model in ISO/IEC TR 15504-5 is an exemplar of a compatible model.

•       Supporting instruments and tools

In any assessment, information will need to be collected, recorded, stored, collated, processed, analysed, retrieved and presented. In general, a documented assessment process will be supported by various instruments and tools for information gathering, processing and presentation. For some assessments, the support tools and instruments may be manual and paper-based (forms, questionnaires, checklists, etc.). In some cases the volume and complexity of the assessment information is considerable resulting in the need for computer-based support tools.

Regardless of the form of the supporting instruments and tools, their objectives should be to help an assessor perform an assessment in a consistent and reliable manner, reducing assessor subjectivity and helping to ensure the validity, useability and comparability of assessment results. In order to achieve these objectives, the instruments and tools need to make the assessment model and its indicators accessible to the assessors.

•       Success factors for process assessment

The following factors are essential to a successful process assessment.

•               Commitment

The sponsor should commit himself to the objectives established for an assessment to provide the authority to undertake the assessment within an organization. This commitment requires that the necessary resources, time and personnel are available to undertake the assessment. The commitment of the sponsor and the assessors is fundamentally important to ensuring that the objectives are met.

•               Motivation

The attitude of the organization's management, and the documented assessment process by which the information is collected, has a significant influence on the outcome of an assessment. The organization's management, therefore, needs to motivate participants to be open and constructive. Process assessments focus on the process, not on the performance of organizational unit members implementing the process. The intent is to make the processes more effective to support the defined business goals, not to allocate blame to individuals.

Providing feedback and maintaining an atmosphere that encourages open discussion about preliminary findings during the assessment helps to ensure that the assessment output is meaningful to the organizational unit. The organization needs to recognize that the participants are a principal source of knowledge and experience about the process and that they are in a good position to identify potential weaknesses.

•               Confidentiality

Respect for the confidentiality of the sources of information and documentation gathered during assessment is essential in order to secure that information. If discussion techniques are utilized, consideration should be given to ensuring that participants do not feel threatened or have any concerns regarding confidentiality. Some of the information provided might be proprietary to the organization. It is therefore important that adequate controls are in place to handle such information.

•               Relevance

The organizational unit members should believe that the assessment will result in some benefits that will accrue to them directly or indirectly.

•               Credibility

The sponsor and the management and staff of the organizational unit must all believe that the assessment will deliver a result which is objective and is representative of the assessment scope. It is important that all parties can be confident that the assessors have adequate experience of assessment, are sufficiently impartial and have an adequate understanding of the organizational unit and its business to conduct the assessment.

•      Selection and use of a compatible model

This clause provides guidance on the selection and use of a compatible model as the basis for performing a software process assessment. The guidance is intended for the use of assessors and sponsors of assessments. It is not directed specifically at the developers of compatible models, though it may be of use to them.

In performing a process assessment, the practices observed in the organization unit being assessed are compared against those defined in a base model of good practice, to determine the extent to which the performance of the practices results in achievement of the attributes of capability. In order to achieve this, the model must contain descriptions of the practices to be observed, and indicators of the performance of these practices, so that the judgments of capability may be made reliably and consistently.

5.1 Compatibility with the reference model

The identity of the model(s) used within the assessment, which shall be a compatible model(s) of good software engineering practice that meet the requirements defined in ISO/IEC TR 15504-2.

[ISO/IEC TR 15504-3, 4.2.2 e]

The first criterion for the selection of a model(s) is that it achieves compatibility with the reference model. Compatibility is essential in order to provide a degree of comparability between the results of different assessments by maximizing the reliability of different approaches and achieving a greater degree of uniformity in the reporting of results.

•               Model Purpose

A model shall be based on good software engineering and process management principles and be suitable for the purpose of assessing software process capability.

[ISO/IEC TR 15504-2, 7.2]

There are many different types of modelling techniques available for describing, specifying and enacting processes. Models that have not been specifically developed for the purpose of process assessment may not yield reliable results, and their suitability for purpose should be validated before selection.

•               Model Scope

A model shall encompass all, or a non-empty subset, of the set of processes in the process dimension of the reference model.

A model shall address all or a continuous subset of the levels (starting at level 1) of the capability dimension of the reference model for all of the processes within its scope.

[ISO/IEC TR 15504-2, 7.3]

The reference model defines a set of universal software engineering processes that are fundamental to good software engineering and that cover best practice activities. Any model, to be compatible with the reference model, must contain at least a part of this scope. The model may be a sub-set of the reference model. It may be a super- set of the reference model, covering all of the defined processes together with additional process descriptions outside the standard scope. A compatible model may also include processes outside the reference model providing it encompasses at least one process from it. Finally, the scope of the model may be directly equivalent to the reference model.

For the capability dimension, a model must cover a complete set of capabilities for all of the processes in its scope. The set of capabilities must encompass the whole of the capability level scale in the reference model or a subset starting at level 1. It is permissible, therefore, for a model to claim coverage of levels 1 to 3 only, but not of only levels 3 to 5.

In selecting a model, the assessors should ensure that the scope of the model covers the intended area of interest for the assessment.

•               Model elements and indicators

A model shall be based on a set of elements that explicitly address the purposes, as defined in the reference model, of all the processes within the scope of the model, and that demonstrate the achievement of the process attributes within the capability level scope of the model.

[ISO/IEC TR 15504-2, 7.4]

In order for a model to be compatible with the reference model, it must address the purposes of the processes as defined in the reference model, and the achievement of the process attributes that constitute the capability dimension. In order to meet the requirements of the other components of ISO/IEC TR 15504, it must also document a set of indicators of process performance and process capability that enable judgements of process capability to be soundly based on objective evidence.

There is a clear expectation that the indicators will fall into two categories: factors that indicate the performance of the process, and factors that indicate its capability. In selecting a model, clear attention should be paid to the use of indicators in the model, the comprehensiveness of the indicator set, and the applicability of the indicator set. ISO/IEC TR 15504-5 comprises a model with a comprehensive set of indicators, that may serve as a guide to the extent of coverage to be expected.

•               Mapping

A model shall provide an explicit mapping from the fundamental elements of the model to the processes and process attributes of the reference model.

The mapping shall be complete, clear, and unambiguous and shall substantiate the declaration of the scope of coverage.

[ISO/IEC TR 15504-2, 7.5]

It is essential that the assessor has access to the details of the mapping of the elements of the model to the reference model. The mapping may be simple, as is the case in the model defined in ISO/IEC TR 15504-5. Where the structure of the model is significantly different from the reference model, however, the mapping may be quite complex.

An assessor should confirm that the mapping is meaningful, for example by sampling some of the lowest level components in the model, and locating them in the reference model, either as elements of a process or as contributors to a process attribute. Mappings that result in elements being identified as components of more than one process attribute may indicate problems with the model structure, which could result in ambiguous translation of results.

•               Translation

A compatible model shall provide a formal and verifiable mechanism for converting data collected against the compatible model into a set of process attribute ratings for each reference model process directly or indirectly assessed as defined in 6.7 of this part of ISO/IEC TR 15504, and in ISO/IEC TR 15504-3.

[ISO/IEC TR 15504-2, 7.5]

The output from a process assessment is a set of process profiles. A process profile is a set of up to nine ratings, one for each process attribute. Assessment results from any compatible model must be able to be converted into this form, so that a common basis for comparison exists.

The mechanism for translation may be manual, or computer based. It may require the inclusion of additional information collected during the assessment, and may involve further judgement on the part of the assessor. The rules for translating the results however, should be clear and unambiguous, and are to be provided by either the model developer or method provider.

If a model explicitly provides results in the format prescribed in ISO/IEC TR 15504-2, then there is no need for any translation mechanism.

•       Criteria for selecting a model

The model for an assessment may be selected by the assessor, or may be stipulated by the sponsor of the assessment (in which case, this should be documented as a constraint). In either case, there are criteria that will help ensure that the selection is appropriate for the use envisaged.

The major considerations in selecting a model, given that any model selected is compatible with the reference model, will be its suitability for the context of the assessment. The principal factors affecting the selection of a model will be:

•        the planned scope of the assessment;

•        the industry sector of the organization being assessed;

•        the application domain of the software components that are the focus of the assessment;

•        the inclusion of an improvement path for increasing the process maturity of an organization and

•        specific requirements for strong comparability with other assessments or organizations.

Where models exist that have been specifically developed for use in particular industry sectors - e.g. telecommunications, defence, aerospace - or for particular application domains - e.g. high security systems, safety critical systems, real time embedded software - then, when applicable, these should be considered.

When an organization wishes to conduct an assessment in an area that is not representative of its normal domain, it should take care that the model chosen is suitable. For example, an aerospace organization that wishes to assess the processes responsible for maintenance of its internal management systems might find that an industry specific model is not the most suitable for the task.

The model provided in ISO/IEC TR 15504-5 is a generic model that is designed to be applicable across all industry sectors and application domains.

•       Using a model in an assessment

A model provides the basic definitions of processes and process attributes that are the reference points against which judgements of process performance in the organization unit are made. As such, the use of a single model throughout an assessment is essential.

It therefore follows that a competent assessor should be highly knowledgeable about the specific model being used for the assessment - its structure, the basic elements of the model, and its relationship to the reference model.

Because the model also embeds a comprehensive set of indicators of process performance and capability, it is also an important reference point for the assessor in meeting the requirement to document the indicators referenced, and the justification for the ratings. Clause 7 gives guidance on the selection and use of assessment instruments.

•      Selection and use of a documented assessment process

This clause provides guidance on the selection and use of a suitable documented assessment process as the basis for performing a software process assessment. This guidance is intended for the use of assessors and sponsors of assessments. It is not directed specifically at the developers of documented assessment processes, though it may be of use to them.

In performing a process assessment, the documented assessment process used must ensure that the requirements defined in ISO/IEC TR 15504-3 are met. In order to achieve this, the documented assessment process should contain descriptions of the activities to be performed, the responsibilities of key individuals and the documentary evidence that must be recorded. It may also define specific compatible models and tools that are required to be used with the documented assessment process.

•       Compatibility with the requirements

In order to achieve a greater degree of uniformity in the approach to process assessment, so as to maximize the reliability of different approaches and provide a degree of comparability between the results of different assessments, any documented assessment process used should attempt to ensure that assessments performed are compliant with the requirements defined in ISO/IEC TR 15504-3. It may make sense to verify the requirements prior to and during the course of the assessment so that corrective actions can occur.

The documented assessment process should define the process for all required supporting activities, such as document control, quality assurance, project management, as well as for the key activities associated with the documented assessment process itself. This might be in the form of guidance material, procedures, standards etc., how competent assessors are to attain the required competencies to use the documented assessment process correctly, for example training courses and experience levels. The documented assessment process should  provide all necessary guidance including guidance on all activities to be performed in conducting an assessment as described in ISO/IEC TR 15504-3.

•       Assessment Input

Assessment purpose including alignment with business goals

[ISO/IEC TR 15504-3, 4.2.2 b]

Different types of assessments have different purposes. The purposes may vary depending upon the business needs such as facilitating internal process improvement and for the selection of suppliers (either internal or external).

The assessment scope

[ISO/IEC TR 15504-3, 4.2.2 c]

The process scope may include one or more processes together with those process attributes which are to be included in the assessment. Limiting the number of processes and process attributes used in the assessment has the effect of focussing the investigation. The factors that should be included in the assessment scope are the relationship between the assessment scope and the ability to provide ratings, the current process capability level and constraints on assessment duration.

The selection of the Organisational Unit should reflect the sponsor’s intended use of the assessment output. For example, if the output is to be used for process improvement then the Organisational Unit scope should match that of the intended improvement effort. An Organisational Unit scope could be anything from one project to the entire organisation.

The assessment constraints

[ISO/IEC TR 15504-3, 4.2.2 d]

The success of the assessment may be affected if the key resources are not available. Consideration needs to be given to minimize the disruption of assessees who may be constrained through project pressures.

The process and scope may be tailored to accommodate the available time.

It may be necessary to exclude certain parts of an Organisational Unit due to the life cycle phase, etc.

The identity of the model(s) used within the assessment, which shall be compatible model(s) of good software engineering practice that meet the requirements defined in ISO/IEC TR 15504-2.

[ISO/IEC TR 15504-3, 4.2.2 e]

For ease of application one may wish to use a single compatible model; however, depending on the purpose of the assessment selected parts of supporting compatible models may be used.

The identity of the assessors, including the competent assessor with specific responsibilities for the assessment

[ISO/IEC TR 15504-3, 4.2.2 f]

The number of assessors engaged in the assessment task may vary, however the combined knowledge and experience of the assessors fosters the confidence in the assessment results. Participation of assessors from the Organisational Unit can help to provide process context and supports ownership and buy-in of the results.

The criteria for competence of the assessor who is responsible for the assessment

[ISO/IEC TR 15504-3, 4.2.2 g]

ISO/IEC TR 15504-6 provides guidance to the sponsor regarding assessor competence. The documented assessment process needs to provide specific criteria related to who is eligible to be the competent assessor.

The identity of assessees and support staff with specific responsibilities for the assessment

[ISO/IEC TR 15504-3, 4.2.2 h]

The selection of assessees should be representative of the Organisational Unit being assessed. If the participants are representative of the Organisational Unit then the assessment results are more likely to provide an accurate view of the process capability.

Any additional information to be collected during the assessment to support process improvement or process capability determination eg. specific data (or metrics) that is needed to quantify the organization's ability to meet a particular business goal

[ISO/IEC TR 15504-3, 4.2.2 h]

Information supporting the process context, such as opportunities for improvement or risks to acquisition, should be documented.

•       The assessment process

The assessment shall be conducted according to a documented process that is capable of meeting the assessment purpose.

[ISO/IEC TR 15504-3, 4.4.1]

A documented process supports repeatability of an assessment approach and provides the basis for continuous improvement.

•               Planning

A plan for the assessment shall be developed and documented.

[ISO/IEC TR 15504-3, 4.4.3 a]

The documented assessment process should describe how all of the information required for the assessment input is collated, reviewed, approved and documented. It may be appropriate for the documented assessment process to require tool support to collect and store this information. The assessment input, together with other information and analyses, forms the assessment output.

The documented assessment process should provide support for recording or transferring the assessment input to a suitable form to become part of the assessment output.

The documented assessment process should provide guidance on:

•        obtaining the sponsor’s commitment;

•        defining ownership and distribution of the assessment output;

•        assessment planning;

•        suitable confidentiality statements and how these are fulfilled;

•        classifying the process context; and

•        verifying the requirements for the assessment.

The documented assessment process should define mechanisms to:

•        enable the assessment to be performed effectively within the constraints defined, or how the constraints and/or scope can be re-negotiated and approved if this is not possible; and

•        support the collection of any other information defined by the assessment sponsor.

The documented assessment process should:

•        provide mechanisms to allow the sponsor to ensure that the nominated competent assessor has the competencies to undertake the assessment and mechanisms to validate them; guidance on assessor qualification is given in ISO/IEC TR 15504-6;

•        define the other roles and responsibilities within the assessment and what competencies are required for each role;

•        provide mechanisms to ensure that the assessment is conformant  with  the  requirements  set  out  in  ISO/IEC TR 15504-2 and ISO/IEC TR 15504-3; and

•        define how this conformance is achieved and provide mechanisms to validate compliance with the requirements, where this is the mechanism by which conformance is achieved, identify the form of certification of compliance with the requirements.

•               Data collection

Data required for evaluating the processes within the scope of the assessment shall be collected in a systematic and ordered manner, applying at minimum the following:

[ISO/IEC TR 15504-3, 4.4.3 b]

The documented assessment process should provide guidance on data collection mechanisms such as interview techniques, document reviewing instruments . It should also provide guidance on identifying how the organizational unit’s processes are mapped to the processes defined within the compatible model. This will normally require guidance on translating the compatible model assessment profiles back to representations of the organizational unit’s process capability.

The documented assessment process should provide guidance on sampling to ensure that the set of processes selected are appropriate to the assessment purpose. The documented assessment process should provide mechanisms to retain the sampling information and rationale.

The information gathering may be organized as part of a regular manual monitoring or reporting mechanism used by one or more projects. Alternatively, information collection may be automated or semi-automated through the support of an instrument or tool. An instrument could be used continuously throughout the software development life cycle, for example, at defined milestones to measure adherence to the process, to measure process improvement progress, or to gather information to facilitate a future assessment.

•               Data validation

The data collected shall be validated. Actions shall be taken to ensure that the validated data sufficiently covers the assessment scope.

[ISO/IEC TR 15504-3, 4.4.3 c]

The documented assessment process should provide guidance on information validation which covers at a minimum, information from first hand, independent sources; how to use past assessment results; and feedback sessions to validate the information collected.

•               Process rating

A rating shall be assigned and validated for each process attribute.

[ISO/IEC TR 15504-3, 4.4.3 d]

The documented assessment process should provide mechanisms to assign ratings to the defined rating components in the model selected to support the documented assessment process. Where these rating components  are  different  from  the  process  attributes  defined  in  ISO/IEC TR 15504-2,  then  the  documented

assessment process should provide guidance on the use of the mechanisms defined in the compatible model to translate the rating components to the defined process attribute ratings.

The documented assessment process should define mechanisms to:

•        validate the ratings assigned for the processes assessed; and

•        record the ratings for all the processes assessed, ensuring that each rating record can be uniquely identified to the process to which it relates.

The documented assessment process should provide mechanisms to represent the process profiles in forms that allow straightforward interpretation of their meaning and value. This should support the representation of aggregated ratings.

In order to provide the basis for repeatability across assessments, the defined set of assessment indicators in the compatible model(s) shall be used during the assessment to support the assessors’ judgement in rating process attributes.

[ISO/IEC TR 15504-3, 4.4.3 d 2]

The requirements for constructing a compatible model ensure that the indicators are traceable to the statements of process purpose or the process attributes in the reference model in ISO/IEC TR 15504-2. Although information about the indicators does not form part of the process profile, it provides the essential evidence that supports an assessor’s judgement of the ratings assigned. The characteristics defined by the indicator data provide a detailed record of what was found in the organizational unit. The information collected is, therefore, significant both for assessor's evaluation and for subsequent analysis and planning for process improvement. For guidance on indicators see Annex A.

•               Reporting

The assessment results, including at minimum the outputs specified in Clause 4.5, shall be documented and reported to the Assessment Sponsor.

[ISO/IEC TR 15504-3, 4.4.3 e]

The documented assessment process should;

•        provide mechanisms to record information and/or ratings associated with the indicators defined in the compatible model selected;

•        specify the format that the results are to be relayed to the sponsor, assessees, managers etc.; presentation and/or report;

•        define how the records are to be retained — they may be paper-based or electronic depending upon the circumstances and tools used to support the assessment;

•        define the records retained by the sponsor, the assessor, the assessed organization, or another person or body depending upon any confidentiality agreement or access restrictions identified in the assessment input; and

•        provide mechanisms to record and retain the assessment output. These mechanisms should ensure that all confidentiality requirements are met.

•       Selecting a documented assessment process

The documented assessment process for an assessment may be selected by the assessor, or may be stipulated by the sponsor of the assessment (in which case, this should be documented as a constraint). In either case, there are criteria that will help ensure that the selection is appropriate for the use envisaged. Particular documented assessment processes may be appropriate to particular process contexts, particular assessment approaches and to particular processes. All of these factors may influence the decision to select a particular documented assessment process. Organizations may also be constrained to use a particular documented assessment process if it has been chosen as the standard to ensure the most effective use of resources.

The documented assessment process should provide guidance and models that assist in estimating the resource requirements that will be needed to perform an assessment of the scope defined.

The documented assessment process should provide support to ensure that each defined role can attain the required competencies, such as training and experience requirements.

The major considerations in selecting a documented assessment process will be its suitability for the context and scope of the assessment. The principal factors affecting its selection will be:

•        the planned purpose of the assessment;

•        the planned scope of the assessment;

•        the assessment approach selected; and

•        the process context of the selected processes.

Where documented assessment processes exist that have been specifically developed to support a particular assessment approach or approaches, then these should be used if at all possible. Larger, more complex organizations may also be constrained to select documented assessment processes that have the ability to cover the range of their business activities to ensure consistency of approach, reuse of competencies, etc.

The assessment process selected will significantly influence how the assessment is conducted and its usefulness to the organizational unit assessed. The competent assessor plays a pivotal role in ensuring that the purpose of the assessment is achieved.

•       Role of the competent assessor

The competent assessor is responsible for ensuring that the assessment achieves its purpose and that it is conformant with the requirements of ISO/IEC TR 15504-2 and ISO/IEC TR 15504-3. It is therefore imperative that the competent assessor selects an appropriate documented assessment process. Where the documented assessment process is selected by the assessment sponsor, then it is the responsibility of the competent assessor to ensure that assessors or users are competent in its use.

•      Selection of instruments and tools

•       The purpose and use of instruments and tools within an assessment

In any assessment, information will need to be collected, recorded, stored, collated, processed, analysed, retrieved and presented. Instruments and tools can provide valuable support in collating the evidence used by the assessor to assign ratings to the process attributes for each process assessed, and in recording the ratings as the set of process profiles.

There are two basic types of instrument, paper-based manual instruments and automated computer-based instruments, which have different characteristics. The appropriateness of an instrument depends on the planned mode of use and assessment methodology. To ensure optimum performance (effectiveness and efficiency), instruments and tools should be selected or designed to match the assessment process.

Instruments and tools may be used in a number of ways to support assessments. Examples of modes of use within an assessment include:

•        by assessors capturing information by a paper-based instrument, or a computer;

•        by process owners and/or organizational unit representatives during preparation for and prior to an assessment capturing information for subsequent processing;

•        by organizational unit representatives continuously throughout the software development life cycle, and at defined milestones to measure process adherence, process improvement progress or to gather information to facilitate a future assessment;

•        after the assessment to retrieve or organize the assessment information to facilitate process improvement planning or analysis for capability determination;

•        in a distributed approach for self-assessment throughout an organization;

•        when sampled work-products and process information are collected incrementally and reviewed prior to the commencement of on-site assessment activities, such as interviews;

•        to assist the assessor with the processing of the assessment information collected;

•        to store and retrieve assessment results, making the results more useable for process improvement planning or capability determination analysis;

•        to assist the assessor with post-assessment analysis of the results such as the analysis of process improvement results against past performance history, or of a supplier profile against an established target profile;

•        to collect information incrementally and in a distributed manner, to collect information incrementally at set milestone check points in the performance of a process or when a number of organizational units are to be assessed incrementally;

•        to generate result profiles or help in the performance of gap analysis.

Competence to use the selected instruments and tools is a key factor in ensuring that information is collected, recorded, processed and analysed in a reliable, repeatable and appropriate way. The assessors and other participants who will use the instruments and tools should be appropriately trained and have the necessary experience in the use of the instruments and tools. In addition to competence in operating the instruments or tools, training and/or experience should provide a good theoretical understanding of the underlying principles related to the assessment model, indicators, and rating.

•       Selecting instruments and tools

Particular tools may be specified as part of the documented assessment process. Alternatively, the intending user may need to select appropriate tools. The guidance presented here is intended to highlight some of the considerations in selecting instruments and tools for use throughout the assessment. It does not address issues related to general support tools such as word processors, although the ability of assessment instruments and tools to integrate together and to integrate with word processing/presentation tools can prove of considerable assistance in preparing reports and presentations of the outputs of the assessment.

The selection criteria for the type of instrument and tool may be influenced by:

•        the scope and purpose of assessment;

•        assistance in collecting and storing information including assembling the assessment input and recording it in a suitable form for transfer to the assessment output;

•        availability of the compatible assessment model through the defined set of indicators, at least for the scope of the assessment;

•        ability to capture  the  information  required  to  be  used  in  the  production  of  ratings  as  defined  in  ISO/IEC TR 15504-2;

•        ability to capture and maintain supporting information as defined in the assessment input;

•        support of the rating process and aggregation of the results according to the rating scheme defined  in  ISO/IEC TR 15504-2;

•        support of representation of process profiles in forms that allow straightforward interpretation of their meaning and value;

•        ability to store and retrieve assessment results for subsequent use in process improvement or capability determination;

•        provision of appropriate segregation of different classes of information and data to enable the information and/or data to be used or distributed in different ways;

•        ability to keep the captured information secure to meet confidentiality constraints;

•        ability to perform dynamic scoping and tailoring to support specific cultural, organizational, sponsor, or assessment needs;

•        in providing adequate configuration control of the instrument and the results collected;

•        ability to split by process and job function;

•        ability to tailor the assessment model as required;

•        portability considerations (useability for interviews, distributed inputs, simultaneous inputs);

•        ability to handle multiple assessors' inputs;

•        useability for interviews, self-assessment;

•        ability to integrate with other tools (metrics, case, etc.);

•        ability to maintain an audit trail of access to information input;

•        real-time performance: speed of information input and retrieval;

•        ability to call up practices required for specific interviews.

Guidance and standards for computer based tool selection are available in ISO/IEC 12119:1994, Information technology — Software packages — Quality requirements and testing and ISO/IEC 14598 (all parts), Information technology — Software product evaluation.

Annex A

(informative)

Guidance on indicators

•           Introduction

The reference model in ISO/IEC TR 15504-2 is defined in two dimensions: the process dimension and the process capability dimension which are characterized respectively by the statements of process purpose and process attributes. The process purposes and process attributes are the criteria against which an assessment is performed.

The process purpose statements and attributes represent good practice, but in order to make them applicable to all software applications and domains, they are defined as abstract, high level concepts without constraining the ways in which they may be implemented. Consequently, these purpose statements and attributes could be subject to wide interpretation, which could have an adverse effect on the repeatability and reliability of assessment results.

In order to reduce the level of subjectivity and variation of interpretation, a compatible model for assessment (clause 7 of ISO/IEC TR 15504-2) must be elaborated through a set of indicators of process performance related to the process purpose, and a set of indicators of process capability related to the process attributes. Indicators are used during an assessment to support the collection of objective evidence about the achievement of a particular process purpose or process attribute. As implied by the name, indicators do not represent requirements on a process. They represent a common starting point for assessment, which increases the consistency of assessor judgement and enhances the repeatability of the results. Since organizations use different techniques to create software, the absence of some indicators in some situations may not be significant.

The output of the assessment, in the form of a set of process profiles, shows the ratings of each of the nine process attributes for each process assessed, but it does not show why a particular rating was assigned. Indicators help to identify what is present or missing from a process or work product and provide guidance to the assessor when assigning a rating to a process or attribute. The detailed information captured during the assessment about the presence or absence of specific indicators provides the valuable input into analysis and process improvement planning.

The indicators provide a framework for assessment that helps to ensure that:

•        assessors have the ability to interpret the organization's instantiation of a process consistently against the reference model;

•        the information is captured for subsequent analysis;

•        the information needed for the organizational unit to plan and perform process improvement is captured;

•        assessment results are representative, reliable and repeatable.

•               Indicators of process performance

Indicators of process performance provide guidance to the assessor on how to judge how well a process is meeting its purpose as defined in the reference model. These indicators are practices that are performed within a specific process, as well as the work products and the characteristics of the work products produced by the practices.

The performance of relevant practices provides the first indication that an implemented process meets the purpose statement. The second indication is the existence of work-products from the performance of the practices. The characteristics of the work products assist the assessor in understanding what elements to expect in a meaningful instantiation of a particular work products type.

The apparent execution of the practice alone does not provide evidence of a sufficient implementation. The further evidence that the execution of the practice is meeting the purpose of the process is gained from the existence of the appropriate work products and their content, or work-product characteristics. The indicators should help the assessor to recognize an appropriate work product.

•               Indicators of process capability

Indicators of process capability are associated with each process attribute in capability levels 2-5. Similar to indicators of process performance, they complement the assessor’s ability to judge the attainment of the capability described by the process attributes. They help to identify the ability of the organization to manage a process effectively. Indicators of process capability provide a structured way of recording in the assessment record what was found in a particular implementation of a process attribute.

•               Relationship of indicators to ratings

Figure 1 shows how the indicators of process performance and process capability are brought together to support the rating of the nine process attributes within capability levels 1 to 5.

nine attribute ratings

Process profile of

ConsistsOf                                                                            ConsistsOf

Rating for capabilit

attributes in levels 2 - 5

f process mance

Rating for capability

attribute at y level 1

IsSupportedBy

Indicator o

perfor

purpose ment

Satisfies

Process

state

IsUsedtoAssess

IsSupportedBy

or of apability

IsApplicationOf

s attribute

Organization

Proces

Indicat process c

IsUsedtoAssess

unit’s process

Figure A.1 — Putting the elements together to determine ratings

The indicators of process performance provide the evidence to support the rating of the single attribute at capability level 1 which is concerned with how well the practices and work products of the process achieve the process purpose.

The indicators of process capability provide the evidence to support the ratings of the other eight process attributes of capability levels 2 to 5 which measure aspects of process management capability.

•           Indicators and information gathering

There are many approaches that can be used to gather information. The documented assessment process and approach will depend on many factors including:

•        the size of the organization being assessed;

•        the number of organizational units involved in the assessment;

•        the level of organizational participation in performing the assessment (collecting the information, demonstrating conformance);

•        the maturity of the supplier-sponsor relationship (the level of trust between the organization and sponsor);

•        the needs of the sponsor;

•        the expertise and ability of the assessor(s);

•        the needs of the organization.

Whatever documented assessment process is used, the defined set of indicators in the compatible model should form the basis for information gathering and must be used to support the assessors’ judgement in rating process attributes. Unless the assessment is small and limited in scope, it will generally be found useful to incorporate the indicators within an instrument or tool. In this way, the compatible model and its indicators can be made accessible to assessors during the assessment. The instrument or tool can also provide support for recording and organizing the information and evidence collected.

•           Rating support and presentation of results

Assessment  ratings  are  assigned  for  process  attributes  according  to  the  requirements  for  rating   in  ISO/IEC TR 15504-2. The output is represented as a set of process profiles.  For each process assessed, a rating is assigned for each process attribute up to and including the highest capability level defined in the assessment scope.

Instruments and tools may be useful in providing support for assigning ratings or for performing the translation from the information gathered against the compatible model to the form of ratings required by this standard.

In its basic form, a process profile is a vector of ratings. For presentation purposes, however, process profiles are typically presented graphically using tool support.

•           Tailoring Indicators

For enhanced useability within an assessment, computer-based instruments incorporating indicators could contain facilities for tailoring the indicators to suit particular circumstances. Examples include:

•        the modification of indicator format to accommodate presentation style preferences (i.e., questions, sentences, tables, on-line input screens, etc.);

•        the modification of indicator wording to accommodate synonyms or to account for cultural differences in meaning;

•        the addition of characteristics to help select a relevant sub-set of indicators for use based on e.g. process area, user, job function, application domain, software product;

•        the addition of new indicators to support new work products, new technology, or new or extended processes.

It is important, however, that any tailoring should not impair the availability of the set of indicators of the compatible model appropriate to the scope and context of the assessment.

ICS 35.080

Descriptors: data processing, information interchange, computers,  computer  software,  estimation,  process  assessment,  rules  (instructions).

Price based on 18 pages