Package lisp-unit

Overview
Introduction
How to Use lisp-unit
Example
Assertion Forms
How to Organize Tests with Packages
Functions and macros for managing tests
Macros for assertions
Utility predicates
Listeners

Overview

Author

Copyright (c) 2004-2005 Christopher K. Riesbeck
Copyright (c) 2012 Dieter Kaiser

Version

This is the documenation of a fork of the package lisp-unit.

Homepage

http://www.cs.northwestern.edu/academics/courses/325/readings/lisp-unit.html

Mailing List

No mailing list.

Download and Source Code

The original code is available at
http://www.cs.northwestern.edu/academics/courses/325/programs/lisp-unit.lisp

You can download a fork of lisp-unit from the repository gitorious.org/lisp-projects/lisp-unit The fork contains this documentation. Furthermore, the facilities to control how results are reported are fully implemented.

Dependencies

lisp-unit does not depend on other libraries.

Introduction

The main goal for lisp-unit is to make it simple to use. The advantages of lisp-unit are:

• written in portable Common Lisp,
• dead-simple to define and run tests,
• supports redefining functions and even macros without reloading tests,
• supports test-first programming,
• supports testing return values, printed output, macro expansions, and error conditions,
• produces short readable output with a reasonable level of detail, and
• groups tests by package for modularity.

How to Use lisp-unit

Loading the file lisp-unit.lisp

There are two ways to load lisp-unit. The first method is simply to load the file lisp-unit.lisp. The steps are the following:

• Load (or compile and load) lisp-unit.lisp.
• Evaluate (use-package :lisp-unit).
• Load a file of tests. See below for how to define tests.
• Run the tests with run-tests.

Any test failures will be printed, along with a summary of how many tests were run, how many passed, and how many failed. You define a test with the macro define-test:

  (define-test name exp1 exp2 ...)    

This defines a test called name. The expressions exp1, exp2, ... can be anything, but typically most will be assertion forms. Tests can be defined before the code they test, even if they are testing macros. This is to support test-first programming. After defining your tests and the code they test, run the tests with

  (run-tests)    

This runs every test defined in the current package. To run just certain specific tests, use:

  (run-tests name1 name2 ...)    

Loading lisp-unit and executing tests with asdf

The second method is to load lisp-unit with the asdf facility:

  (asdf:load-system :lisp-unit)    

Furthermore, with asdf the execution of the tests can be performed with the call

  (asdf:test-system <package-name>)    

Here <package-name> is the package which is tested. To allow this feature of asdf a method perform has to be added to the system definition. This is an example for the package lisp-unit itself:

  (asdf:defsystem :lisp-unit
    :name 'lisp-unit'
    :author 'Dieter Kaiser'
    :license 'LLGPL'
    :serial t
    :depends-on ()
    :components ((:file 'lisp-unit'))
    :in-order-to ((asdf:test-op (asdf:load-op :lisp-unit-test))))


  (defmethod perform ((o test-op) (c (eql (find-system :lisp-unit))))
    (eval `(,(intern (string '#:run-all-tests) :lisp-unit) :lisp-unit-test)))    

The test package for lisp-unit has the name lisp-unit-test and has the following system definition:

  (asdf:defsystem :lisp-unit-test
    :depends-on (:lisp-unit)
    :version '1.0.0'
    :components ((:module 'test'
                  :components ((:file 'rtest-lisp-unit')))))    

In this example the test files are expected to be in a subdirectory test/ of the package itself. Because of these definitions tests in the test file rtest-lisp-unit.lisp for the package lisp-unit can be executed with the command:

  (asdf:test-system :lisp-unit)    

Example

The following example

• defines some tests to see if pick-greater returns the larger of two arguments,
• defines a deliberately broken version of pick-greater, and
• runs the tests.

First, we define some tests.

  > (in-package :cs325-user)
  #<PACKAGE CS325-USER>
  > (define-test pick-greater
      (assert-equal 5 (pick-greater 2 5))
      (assert-equal 5 (pick-greater 5 2))
      (assert-equal 10 (pick-greater 10 10))
      (assert-equal 0 (pick-greater -5 0))
    )
    PICK-GREATER      

Following good test-first programming practice, we run these tests before writing any code.

  > (run-tests pick-greater)
      PICK-GREATER: Undefined function PICK-GREATER called with arguments (2 5).      

This shows that we need to do some work. So we define our broken version of pick-greater.

  > (defun pick-greater (x y) x)  ;; deliberately wrong
  PICK-GREATER      

Now we run the tests again:

  > (run-tests pick-greater)
  PICK-GREATER: (PICK-GREATER 2 5) failed: Expected 5 but saw 2
  PICK-GREATER: (PICK-GREATER -5 0) failed: Expected 0 but saw -5
  PICK-GREATER: 2 assertions passed, 2 failed.    

This shows two failures. In both cases, the equality test returned nil. In the first case it was because (pick-greater 2 5) returned 2 when 5 was expected, and in the second case, it was because (pick-greater -5 0) returned -5 when 0 was expected.

Assertion Forms

The most commonly used assertion form is

  (assert-equal value form)    

This tallies a failure if form returns a value not equal to value. Both value and test are evaluated in the local lexical environment. This means that you can use local variables in tests. In particular, you can write loops that run many tests at once:

  > (define-test my-sqrt
    (dotimes (i 5)
      (assert-equal i (my-sqrt (* i i)))))
  MY-SQRT


  > (defun my-sqrt (n) (/ n 2))   ;; wrong!!


  > (run-tests my-sqrt)
  MY-SQRT: (MY-SQRT (* I I)) failed: Expected 1 but saw 1/2
  MY-SQRT: (MY-SQRT (* I I)) failed: Expected 3 but saw 9/2
  MY-SQRT: (MY-SQRT (* I I)) failed: Expected 4 but saw 8
  MY-SQRT: 2 assertions passed, 3 failed.    

However, the above output doesn't tell us for which values of i the code failed. Fortunately, you can fix this by adding expressions at the end of the assert-equal. These expression and their values will be printed on failure.

  > (define-test my-sqrt
    (dotimes (i 5)
      (assert-equal i (my-sqrt (* i i)) i)))  ;; added i at the end
  MY-SQRT
  > (run-tests my-sqrt)
  MY-SQRT: (MY-SQRT (* I I)) failed: Expected 1 but saw 1/2
     I => 1
  MY-SQRT: (MY-SQRT (* I I)) failed: Expected 3 but saw 9/2
     I => 3
  MY-SQRT: (MY-SQRT (* I I)) failed: Expected 4 but saw 8
     I => 4
  MY-SQRT: 2 assertions passed, 3 failed.      

The next most useful assertion form is

  (assert-true test)    

This tallies a failure if test returns false. Again, if you need to print out extra information, just add expressions after test. There are also assertion forms to test what code prints, what errors code returns, or what a macro expands into. A complete list of assertion forms is in the reference section. Do not confuse assertion forms with Common Lisp's assert macro. assert is used in code to guarantee that some condition is true. If it isn't, the code halts.

How to Organize Tests with Packages

Tests are grouped internally by the current package, so that a set of tests can be defined for one package of code without interfering with tests for other packages. If your code is being defined in cl-user, which is common when learning Common Lisp, but not for production-level code, then you should define your tests in cl-user as well. If your code is being defined in its own package, you should define your tests either in that same package, or in another package for test code. The latter approach has the advantage of making sure that your tests have access to only the exported symbols of your code package. For example, if you were defining a date package, your date.lisp file would look like this:

  (defpackage :date
    (:use :common-lisp)
    (:export #:date->string #:string->date))


  (in-package :date)


  (defun date->string (date) ...)
  (defun string->date (string) ...)      

Your date-tests.lisp file would look like this:

  (defpackage :date-tests
    (:use :common-lisp :lisp-unit :date))


  (in-package :date-tests)


  (define-test date->string
    (assert-true (string= ... (date->string ...)))
    ...)
  ...    

You could then run all your date tests in the test package:

  (in-package :date-tests)


  (run-tests)    

Alternately, you could run all your date tests from any package with:

  (lisp-unit:run-all-tests :date-tests)    

Functions and macros for managing tests

Macros for assertions

Utility predicates

Listeners

lisp-unit calls three listener functions to report test results and summary statistics. These functions can be rebound using the facilities below.

Test Listener

The test listener is called after each assertion form in a test is executed. The listener is passed

• the truth or falsity of the assertion
• a keyword for type of assertion; current valid values for type are:

  :error

  for assert-error

  :macro

  for assert-expands

  :output

  for assert-prints

  :result

  for assert-true, assert-false

  :failure

  for fail calls

  :equal

  for all other assertions

• the name of test in which assertion occurred
• the form tested
• the expected value
• the actual value
• any extra arguments passed to assertion form
• how many assertions evaluated so far
• how many assertions passed so far

Two test listener functions are exported:

show-failure-result

the default, only prints when an assertion fails. It prints the form, expected and actual values, and the values of any extra forms.

show-no-result

never prints anything.

Error Listener

The error listener is called when an error occurs in running a test. The listener is passed

• the name of the test
• the error object

Two error listener functions are exported:

show-error

the default, prints the error message. Further execution of the test forms is terminated.

count-error

prints nothing but the error count is incremented. Further execution of the test forms is terminated.

Summary Listener

The summary listener is called after

• all forms in a test have been executed
• all tests in a package have been run
• all tests in all packages have been run

The listener is passed:

• the name of the test or a package just finished
• the number of assertions evaluated
• the number of tests that passed
• the number of errors that occurred

Three summary listener functions are exported:

show-summary

the default, prints the summaries at both the test and package level.

show-package-summary

prints the summaries at only the package level.

show-now-summary

never prints summaries.

Rebinding Listeners

The three listeners are stored in the exported global variables *test-listener*, *error-listener*, *summary-listener*. So one way to change listeners is with let. For example, to show only package-level summary counts:

  (let ((*summary-listener* 'show-package-summary))
    (run-tests))    

The above would still show failures and error messages. To hide those and just get the counts:

  (let ((*error-listener* 'count-error)
        (*summary-listener* 'show-package-summary)
        (*test-listener* 'show-no-result))
    (run-tests))    

To show no individual test results and only package summaries with failures, we need to define a function that checks the number of failures.

  (defun show-failing-package (name test-count pass-count error-count)
    (when (or (< pass-count test-count) (> error-count 0))
      (show-summary name test-count pass-count error-count)))


  (let ((*error-listener* 'count-error)
        (*summary-listener* 'show-failing-package)
        (*test-listener* 'show-no-result))
    (run-tests))    

with-listeners

A simpler way to rebind listeners is with with-listeners. The example above could be done with:

  (with-listeners (show-no-result show-failing-package count-error)
    (run-tests))    

The macro with-test-listener can be used to rebind only the test listener.

  (with-test-listener (show-no-result)
    (run-tests))