Property-Based Testing

PonyTest includes property-based testing support.

In traditional unit testing, the developer must choose input examples for the unit under test and check whether the output meets expectations.

Property-based testing leaves generation of test input samples to the testing engine, which generates random examples taken from a description of how to do so, called Generators. The developer defines a Generator and describes the condition that should hold for every input sample.

Usage

Writing property-based tests is done by implementing the trait Property. A Property needs to define a type parameter for the type of the input sample, a Generator and a property function. Here is a minimal example:

class _MyFirstProperty is Property[String]
  fun name(): String =>
    "my_first_property"

  fun gen(): Generator[String] =>
    Generators.ascii()

  fun ref property(arg1: String, ph: PropertyHelper) =>
    ph.assert_eq[String](arg1, arg1)

A Property needs a name for identification in test output. We created a Generator by using a factory method defined in the Generators primitive, and we used PropertyHelper to assert on a condition that should hold for all samples.

Below are two classic list reverse properties:

use "pony_test"
use "collections"

class _ListReverseProperty is Property[Array[USize]]
  fun name(): String => "list/reverse"

  fun gen(): Generator[Array[USize]] =>
    Generators.seq_of[USize, Array[USize]](Generators.usize())

  fun ref property(arg1: Array[USize], ph: PropertyHelper) =>
    ph.assert_array_eq[USize](arg1, arg1.reverse().reverse())

class _ListReverseOneProperty is Property[Array[USize]]
  fun name(): String => "list/reverse/one"

  fun gen(): Generator[Array[USize]] =>
    Generators.seq_of[USize, Array[USize]](
      Generators.usize()
      where from = 1, to = 1)

  fun ref property(arg1: Array[USize], ph: PropertyHelper) =>
    ph.assert_array_eq[USize](arg1, arg1.reverse())

Integration with PonyTest

Register a Property with PonyTest by calling test.property in the test list:

use "pony_test"
actor Main is TestList
  new create(env: Env) =>
    PonyTest(env, this)

  fun tag tests(test: PonyTest) =>
    test.property(_MyFirstProperty)

for_all on TestHelper integrates any number of properties directly into one UnitTest:

class _ListReverseProperties is UnitTest
  fun name(): String => "list/properties"

  fun apply(h: TestHelper) ? =>
    let g = Generators

    let gen1 = recover val g.seq_of[USize, Array[USize]](g.usize()) end
    h.for_all[Array[USize]](gen1)(
      {(arg1, ph) =>
        ph.assert_array_eq[USize](arg1, arg1.reverse().reverse())
      })?

    let gen2 = recover val g.seq_of[USize, Array[USize]](g.usize(), 1, 1) end
    h.for_all[Array[USize]](gen2)(
      {(arg1, ph) =>
        ph.assert_array_eq[USize](arg1, arg1.reverse())
      })?

Additional resources

The API documentation covers property-based testing in detail. The ponyc GitHub repository has example tests.

The Pony Patterns book has a testing section covering testing more broadly.