Testing Security Functions

	Role of the Security Function Testing
	Aspects of the System Security Realization
		ATE_COV: The High Level (Security) Design
		ATE_DPT: The Low Level (Security) Design
		ATE_FUN: TSF Internals
		ATE_IND: The Implementation Representation
	Security Function Testing In EROS

Role of Security Function Testing

	Show That TOE Functional Requirements Are Satisfied
	Emphasis Is On Confirming That TOE Operates According To Its Specification
		Positive testing to show presence of required behaviors
		Negative testing to show absence of prohibited  behaviors
	Can be Applied Recursively to TOE Internal Structure

Points to Remember

	Penetration Testing Covered Under Vulnerability Assessment Class
	Functional Testing Limitations
		Only shows behavior for tested configurations
		Cannot be exhaustive for complex systems
		Cannot show absence of unspecified functions
	Main Value of Functional Testing Is to Show That There Is At Least One TOE Configuration That Meets ST SFRs

ATE_COV – Test Coverage

	Definition
	Aspects
	Components
		Evidence Of Coverage
		Analysis of Coverage
		Rigorous Analysis of Coverage

Definition

	Coverage Is A Measure Of The Extent To Which The Test Cases Demonstrate That The Specified TSF Are Present And Operate As Specified
		i.e., a measure of how many of the TSF interfaces specified in the Functional Specification are actually exercised by tests

Aspects of the Family

	Correspondence
	Completeness

Correspondence

	Provide An Analysis that Shows or Demonstrates Which TSF Interface(s) Each Test Is Exercising
	“Show” Only Requires An Assertion That A Test Exercises An Interface
	“Demonstrate” Requires An Explanation Of How The Test Exercises The Interface(s)

Completeness

	Provide An Analysis That Demonstrates That The Full Set Of Tests Exercises The Full Range Of TSF Interfaces
	Rigorous Demonstration Requires That It Be Shown That The Testing Of Each Interface Be Complete (i.e., exhaustive)
		Potentially a flawed requirement, since it is generally acknowledged that complex system interfaces cannot be exhaustively tested

Evidence Of Coverage

	“The developer shall provide evidence of the test coverage.”
	“The evidence of the test coverage shall show the correspondence between the tests identified in the test documentation and the TSF as described in the functional specification.”

Analysis of Coverage

	The developer shall provide an analysis of the test coverage.
	The analysis of the test coverage shall demonstrate the correspondence between the tests identified in the test documentation and the TSF as described in the functional specification.
	The analysis of the test coverage shall demonstrate that the correspondence between the TSF as described in the functional specification and the tests identified in the test documentation is complete.

Rigorous Analysis of Coverage

	The analysis of the test coverage shall rigorously demonstrate that all external interfaces of the TSF identified in the functional specification have been completely tested.

ATE_DPT – Test Depth

	Definition
	Aspects
	Components
		Testing: High-Level Design
		Testing: Low-Level Design
		Testing: Implementation Representation

Definition

	Depth Is The Level Of Detail To Which TSF Is Tested
		Keyed to the ADV representations
	Objective Is To Catch Both Subtle Errors That Are Only Visible At Internal Interfaces And Insertion Of Malicious Code

Aspect of the Family

	Granularity Of Testing
		Essentially specifies the resolution of the white/grey box testing that most be done

Testing: High-Level Design

	The developer shall provide the analysis of the depth of testing.
	The depth analysis shall demonstrate that the tests identified in the test documentation are sufficient to demonstrate that the TSF operates in accordance with its high-level design.

Testing: Low-Level Design

	The depth analysis shall demonstrate that the tests identified in the test documentation are sufficient to demonstrate that the TSF operates in accordance with its high-level design and low-level design.

Testing: Implementation

	The depth analysis shall demonstrate that the tests identified in the test documentation are sufficient to demonstrate that the TSF operates in accordance with its high-level design, low-level design and implementation representation.

ATE_FUN – Functional Testing

	Definition
	Aspects
	Components
		Functional Testing
		Ordered Functional Testing

Definition

	Functional Testing Is Performed By The Developer To Establish That The TSF Exhibit The Properties Necessary To Satisfy The Functional Requirements Of The PP/ST
	The Chosen ATE_COV and ATE_DPT Components Will Determine The Amount Of Evidence Required By ATE_FUN

Aspects of the Family

	Plans
	Procedures
	Expected Results
	Actual Results
	Ordering Dependencies

Plans

	Focus Of The Test Plans Is On The Security Functions Being Tested
		Essentially rationalize each test
		The plan for each test describes its goal(s) in terms of the security functions it will exercise

Procedures

	The Focus Of The Test Procedures Is On The Mechanics Of Executing The Tests
	Each Test Procedure Includes:
		Description of how to run the test
		Description of the system security state assumed by the test
		Description of other tests that must be executed first to bring system to the required security state (i.e., test ordering dependencies)

Expected Results

	Actually An Aspect Of The Test Procedures That Identifies The Output Predicted By The Specification From Which The Test Is Derived

Actual Results

	The Log Of The Results Obtained When The Developer Actually Runs The Test And A Demonstration That Those Results Are Consistent With The Expected Results And Therefore With The Specified Security Behavior Of The TSF

Ordering Dependencies

	An Analysis That Rationalizes The Test Ordering Dependencies
	Is Useful In Determining Adequacy Of Testing, Since It Might Allow Discovery Of Boundary Conditions That Can Be Concealed By Careful Ordering Of The Tests

Functional Testing

	The developer shall test the TSF and document the results.
	The developer shall provide test documentation.
	The test documentation shall consist of test plans, test procedure descriptions, expected test results and actual test results.
	The test plans shall identify the security functions to be tested and describe the goal of the tests to be performed.
	The test procedure descriptions shall identify the tests to be performed and describe the scenarios for testing each security function. These scenarios shall include any ordering dependencies on the results of other tests.
	The expected test results shall show the anticipated outputs from a successful execution of the tests.
	The test results from the developer execution of the tests shall demonstrate that each tested security function behaved as specified.

Ordered Functional Testing

	The test documentation shall include an analysis of the test procedure ordering dependencies.