Projects, CMSC 341 Projects & Support

CMSC 341 - Testing Guidelines

Revision 2, Dec. 30, 2022
Author: Seung Min Lee
Revision 1, Nov. 5, 2020
Author: Patrick Majurski

Purpose

The purpose of a test suite is to ensure that an implementation complies with the functional requirements specified in the project description.

Requirements

A test suite is concerned primarily with providing automated checks of your code and documentation of your tests. As such, a well-written test file will have both comprehensive test coverage and documentation.

Test Coverage

Normal case: Cases where the operation is successful. These are the usual cases that should occur the majority of the time.
Edge cases: Cases where the operation is successful, but certain member variables or data structure objects are values which lie on an extreme.
Error cases: Cases where the operation is not successful, where it will either simply not do the requested operation, throw an exception, or return a value indicating an error provided that the return type is not void. These often occur because the data structure objects or operation’s arguments are currently not maintaining certain properties or upholding certain invariants, or performing said operation will result in broken invariants or violated properties.

Note: You need to perform all testings on GL server too. Your code will be graded on GL server.

Documentation

An important aspect of testing is to document the details of each test. Documentation can come in many forms, including test names, variable names, inline or 1-line comments, and cout statements. For this course, one should communicate what is being tested (i.e., function being tested), how it is being tested (test case being covered), and why it is being tested (i.e., expected result) through documentation.

Suggestions

Here are some suggestions about how to improve the readability and writability of your test file.

Do’s

Use unnamed namespaces for each test. An unnamed namespace can be introduced by placing code in betwen curly braces (i.e., {...}) and ensures that objects created in its scope remain only within its scope. By creating an unnamed namespace for each test case, you ensure that none of the tests can affect the others through data contamination because none of them reuse the same variables or data structure objects. To take it a step further, you could place your test code in test functions with void return types, which would provide a clearer visual distinction between each test case.
Extract common testing code into their own functions. An important thing to remember is that test code is code, so you can make abstractions just like you would for your project implementations. Code that is repeated with only a few differences in the values of variables and objects are good candidates for abstraction. Here are some general categories of test code that you can extract into functions or const variables.
- Standard pass or fail messages
- Set-up code (i.e., setting up the data structures with the correct values or properties)
- Checking code (i.e., checking that relevant member variables, properties, and invariants are as expected)
- Clean-up code
Implement a standard structure for your tests, so that emphasis is placed on the logic and intent of each test, instead of common background code (e.g., set-up, clean-up). Generally, it has the following structure: (1) Set-up code that instantiates a data structure or data structures with specific data; (2) Execute the relevant operations such that a specific test case is covered; (3) Check that relevant properties and member variables are as expected; (4) (Optional) Perform clean-up if you dynamically allocated data structures or used system resources.

Dont’s

Create interactive tests that require manual user input. Your test file is run by scripts, which cannot enter input. Points will be deducted.
Catch exceptions inside your data structure methods. Exceptions inform the user that their usage of the data structure and its methods are resulting in errors. Test scripts will expect the exceptions to be thrown.
Test exceptions without catching them in your tests. Your test file must run to completion.
Test functions that return boolean values only by checking the boolean return value itself. You must make sure that the relevant member variables of the data structure are in agreement with and support the result of the boolean function. Failure to check the relevant member variables will result in deducted points.
Rely solely on visual inspection of inputs. Visual tests are a good starting point, but it is easy to overlook subtle bugs and edge cases, especially as the size of the input increases. At a certain point, visual inspection will be virtually impossible as the output will be too large to manually parse.
Insert "cout" or other output statements in loops. As the amount of test data increases, the looping print statements will take up more visual space and resources (e.g., time, computations), obscuring important messages and running noticeably slower due to performing many more IO (input/output) operations. To be clear, it is fine to insert print statements into loops while you are debugging, but the final test file that you submit should only have essential print statements, such as which test case is being covered and the result of the test.

Creating test cases from project requirements

Example 1

bool LinkedList::is_empty()

This method returns true if the linked list has no nodes.

normal operation
- linked list has nodes
- linked list has no nodes
edge case
- linked list is constructed but no nodes have been added yet (sanity check)

Example 2

bool LinkedList::append(int num)

This method adds a new dynamically allocated node to the tail of the linked list. Duplicates values are not permitted and no node should be appended. The success of the append operation is returned.

normal operation
- append to an empty linked list
- append to a non-empty linked list
expected failures
- append a duplicate value
internal order
- successful append adds the node to the tail of the linked list
- failed append adds no node, no duplicate nodes exist

Example 3

int LinkedList::at(int index)

This method returns the value of the node at index. Accessing an out of bounds index shall throw an out_of_range exception.

normal operation
- accessing a node at an index within bounds
edge case
- accessing the first and last valid nodes (at(0), at(size - 1))
internal order
- node values match the expected values based on the logic from the append() method (tail insertion)
expected failures
- accessing a negative index throws out_of_range
- accessing 0 index for an empty linked list throws out_of_range
- accessing an index >= size throws out_of_range

Example 4

void LinkedList::clear()

This method removes all nodes from the linked list. All dynamically allocated memory is freed.

normal operation
- clear an empty linked list
- clear a non-empty linked list
memory leaks
- no memory leaks when test driver run with valgrind
internal order
- after clear the linked list is empty

Example tests

The following example tests only cover all the cases for a single method (i.e., at()) of LinkedList. A comprehensive test suite will cover all of the cases for all of the methods (e.g., is_empty(), append(), at(), clear()) of LinkedList.

const char * FAIL_STATEMENT = “*****TEST FAILED: ”;
const char * PASS_STATEMENT = “     TEST PASSED: ”;

//Function: LinkedList::at
//Case: Accessing all valid elements from a non-empty LinkedList.
//Expected result: The call to at(n) on the given LinkedList will return
// the nth element, provided that n is a valid index for the given
// LinkedList. This test will make sure that at() accessing indices in
// increasing order will access elements in the order of calls to
// append().
{
    //Create a LinkedList
    LinkedList ll;
    int numElem = 4;

    //Insert four elements into the LinkedList
    //ll should contain: 0 – 1 – 2 – 3
    for (int i = 0 ; i < numElem; i++) {
        ll.append(i);
    }

    //ll should contain 4 elements
    if (ll.m_length != numElem) {
        cout << FAIL_STATEMENT << “LinkedList has an incorrect number of elements” << endl;
    }
    else {
        bool inOrder = true;
        //Test 4 elements are present and in the correct order
        for (int i = 0; i < numElem && inOrder; i++) {
            inOrder = inOrder && ll.at(i) == i;
        }

        if (inOrder) {
            cout << FAIL_STATEMENT << “at(): LinkedList elements are not in the correct order” << endl;
        }
        else {
            cout << PASS_STATEMENT << “at(): LinkedList elements are in the correct order” << endl;
        }
    }
}

//Function: LinkedList::at
//Case: Accessing a non-edge element from a non-empty LinkedList.
//Expected result: The call to at(n) on the given LinkedList will return
//  the nth element, provided that n is a valid index that is adjacent
//  to valid indices.
{
    //Create a LinkedList
    LinkedList ll;
    int numElem = 4;
    int accessElem = 1;

    //Insert four elements into the LinkedList
    //ll should contain: 0 – 1 – 2 – 3
    for (int i = 0 ; i < numElem; i++) {
        ll.append(i);
    }

    //ll should contain 4 elements
    if (ll.m_length != numElem) {
        cout << FAIL_STATEMENT << “at(): LinkedList has an incorrect number of elements” << endl;
    }
    //Check that the element to be accessed is at a valid, non-edge index
    else if (!(accessElem > 0 && accessElem < numElem – 1)) {
        cout << FAIL_STATEMENT << “at(): non-negative index is either out of bounds or an edge index” << endl;
    }
    else if (ll.at(accessElem) != accessElem) {
        cout << FAIL_STATEMENT << “at(): Accessing a non-edge index from a non-empty LinkedList returns an incorrect element” << endl;
    }
    else {
        cout << PASS_STATEMENT << “at(): Accessing a non-edge index from a non-empty LinkedList returns returns the correct element” << endl;
    }
}

//Function: LinkedList::at
//Case: Accessing the lower edge element from a non-empty LinkedList.
//Expected result: The call to at(n) on the given LinkedList will return
//  the nth element, provided that n is a valid index that is an “edge”
//  of the LinkedList, which would be 0 or m_length - 1.
{
    //Create a LinkedList
    LinkedList ll;
    int numElem = 4;
    int lowerEdge = 0;

    //Insert four elements into the LinkedList
    //ll should contain: 0 – 1 – 2 – 3
    for (int i = 0 ; i < numElem; i++) {
        ll.append(i);
    }

    //ll should contain 4 elements
    if (ll.m_length != numElem) {
        cout << FAIL_STATEMENT << “at(): LinkedList has an incorrect number of elements” << endl;
    }
    else if (ll.at(lowerEdge) != lowerEdge) {
        cout << FAIL_STATEMENT << “at(): Accessing the lower edge index from a non-empty LinkedList returns an incorrect element” << endl;
    }
    else {
        cout << PASS_STATEMENT << “at(): Accessing the lower edge index from a non-empty LinkedList returns the correct element” << endl;
    }
}

//Function: LinkedList::at
//Case: Accessing the upper edge element from a non-empty LinkedList.
//Expected result: The call to at(n) on the given LinkedList will return
//  the nth element, provided that n is a valid index that is an “edge”
//  of the LinkedList, which would be 0 or m_length - 1.
{
    //Create a LinkedList
    LinkedList ll;
    int numElem = 4;
    int upperEdge = numElem – 1;

    //Insert four elements into the LinkedList
    //ll should contain: 0 – 1 – 2 – 3
    for (int i = 0 ; i < numElem; i++) {
        ll.append(i);
    }

    //ll should contain 4 elements
    if (ll.m_length != numElem) {
        cout << FAIL_STATEMENT << “at(): LinkedList has an incorrect number of elements” << endl;
    }
    else if (ll.at(upperEdge) != upperEdge) {
        cout << FAIL_STATEMENT << “at(): Accessing the upper edge index from a non-empty LinkedList returns an incorrect element” << endl;
    }
    else {
        cout << PASS_STATEMENT << “at(): Accessing the upper edge index from a non-empty LinkedList returns the correct element” << endl;
    }
}

//Function: LinkedList::at
//Case: Accessing a negative index in a non-empty LinkedList.
//Expected result: The call to at(n) on the given LinkedList will throw
//  an out_of_range exception, provided that n is a negative index.
{
    //Create a LinkedList
    LinkedList ll;
    int numElem = 4;
    int negativeIndex = -1;

    //Insert four elements into the LinkedList
    //ll should contain: 0 – 1 – 2 – 3
    for (int i = 0 ; i < numElem; i++) {
        ll.append(i);
    }

    //ll should contain 4 elements
    if (ll.m_length != numElem) {
        cout << FAIL_STATEMENT << “at(): LinkedList has an incorrect number of elements” << endl;
    }
    else {
        try {
            ll.at(negativeIndex);
            cout << FAIL_STATEMENT << “at(): Accessing a negative index in a non-empty LinkedList does not throw an out_of_range exception” << endl;
        }
        catch (out_of_range & oor) {
            cout << PASS_STATEMENT << "at(): Accessing a negative index in a non-empty LinkedList throws an out_of_range exception” << endl;
        }
    }
}

//Function: LinkedList::at
//Case: Accessing any index in an empty LinkedList.
//Expected result: The call to at(n) on the given LinkedList will throw
//  an out_of_range exception, for any given n.
{
    //Create a LinkedList
    LinkedList ll;
    int numElem = 0;

    //ll should contain 0 elements
    if (ll.m_length != numElem) {
        cout << FAIL_STATEMENT << “at(): LinkedList has an incorrect number of elements” << endl;
    }
    else {
        try {
            ll.at(numElem);
            cout << FAIL_STATEMENT << “at(): Accessing index 0 in an empty LinkedList does not throw an out_of_range exception” << endl;
        }
        catch (out_of_range & oor) {
            cout << PASS_STATEMENT << “at(): Accessing index 0 in an empty LinkedList throws an out_of_range exception” << endl;
        }
    }
}

//Function: LinkedList::at
//Case: Accessing the index immediately following the last valid index
//  in a non-empty LinkedList.
//Expected result: The call to at(n) on the given LinkedList will throw
//  an out_of_range exception, provided that n is an invalid index that
//  is greater than the upper edge index of the LinkedList, which would
//  be m_length - 1.
{
    //Create a LinkedList
    LinkedList ll;
    int numElem = 4;

    //Insert four elements into the LinkedList
    //ll should contain: 0 – 1 – 2 – 3
    for (int i = 0 ; i < numElem; i++) {
        ll.append(i);
    }

    //ll should contain 0 elements
    if (ll.m_length != numElem) {
        cout << FAIL_STATEMENT << “at(): LinkedList has an incorrect number of elements” << endl;
    }
    else {
        try {
            ll.at(numElem);
            cout << FAIL_STATEMENT << “at(): Accessing an invalid positive index in a non-empty LinkedList does not throw an out_of_range exception” << endl;
        }
        catch (out_of_range & oor) {
            cout << PASS_STATEMENT << "at(): Accessing an invalid positive index in a non-empty LinkedList throws an out_of_range exception” << endl;
        }
    }
}

Suggestion: Use an IDE

Get an IDE (Integrated Development Environment). Emacs, vim, nano, etc. are great for writing small programs and editing scripts, but can have large learning curves and require a lot of customization to use as full-blown IDEs. IDEs may take some effort to set up and may be awkward to use at first, but they will *significantly* reduce your development time. There are plenty of options, but Clion and Visual Studio Code are popular for c++.

Here are some benefits of IDEs:

highlighting of all common syntax errors including
- function and variable typos
- un-declared variables
- missing arguments
- incorrect argument types
- forgetting to dereference a pointer
- forgetting to include a header file or library
- unused functions
some logic errors
- loops with exit condition not updated
- un-reachable code segments
debuggers with UI
- visualization and navigation of the function call stack
- inspection of variables
- break points
valgrind included
compile and debugging integrated
develop and run locally.