Skip to content

Get and report return value of benchmark functions #9

@eriknw

Description

@eriknw

I would love for benchmark functions to be able to return a value. This can be used for several things:

  1. User can test benchmark functions
  2. Verify that the returned value is consistent for each implementation variant
  3. If specified, perhaps the output of the benchmark functions can be given

It is good practice for the user to provide a reference implementation of the function in the benchmark file. For example, if toolz.groupby is being benchmarked, this is as simple as doing from toolz import groupby in the benchmark file. Another option is to define it in the benchmark file, such as def groupby(key, seq). The reference implementation isn't benchmarked, but it does allow benchmark functions to be tested.

If no reference implementation is given, the output of each variant can still be compared to make sure they are all equivalent. If there is a reference implementation, then it is possible to identify which variants are incorrect.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions