Skip to content

Error in calculating the Foundation score #8

@abdulmuneer

Description

@abdulmuneer

The calculation of score for foundation benchmark has two errors:

  1. For non-generation, the total count is not updated
  2. For predictions that do not result in one of letters [A, B, C, D] in either in predict[0] or predict[-2], the total count is not updated.

Therefore, the denominator while calculating the % accuracy is much smaller than the sample space. This makes the score high and non-representative of the actual model performance.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions