Skip to content

Create process to maintain a queue of MetabaseCache migrations #21

@preaction

Description

@preaction

We have 80,000,000 reports in the old metabase cache that need to be upgraded to the new test report format. We have CPAN::Testers::Backend::Migrate::MetabaseCache to do this, but there are too many reports to migrate all in one go.

We need a way to use Minion to process these reports, at a low priority (so that new incoming test reports are processed first). This new process should:

  1. Check how many Metabase cache upgrade jobs are in the queue
  2. Look for unprocessed cache entries that do not already have a job in the queue
  3. Add new jobs to the queue until there are a set number of jobs (starting with 1000)

Every time the process is run, it should try to refill the queue with new jobs. This process will be run periodically to keep refilling the queue until every piece of data is processed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions