Replace MirrorBrain by MirrorCache or Mirrorbits

[kelson42]

Mirrorbrain is deprecated and there is a replacement http://www.mirrorcache.org/. We should probably migrate our architecture

[rgaudin]

This is becoming more important with one of our mirror (https://mirror.accum.se/mirror/kiwix.org/) hosting our files on multiple servers (it's a mirror frontend itself) making use of redirections which are not supported by mirrorbrain.

I don't know if that's supported in mirrorcache though but I know mb is not worth it.

In the mean time, I've duplicated the mirror entry so we point independently to the two offloaders I've seen files in. This wastes a lot of requests in scanMirror step but at least we can use the mirror…

[rgaudin]

See https://github.com/etix/mirrorbits as well

[kelson42]

I had tested MirrorCache a long time ago and without remembering the details it was really too short on the features.

Mirrorbits seems more mature and deserve probably to give a try.

Here are the features we like or rely on:

• Metalink (Metalink headers)
• Bittorrent files
• Magnet links
• Mirror mgmt via ftp/http/rsync
• Priorisation of mirrors
• Auto choice of mirrors based on client geo location
• Multiple hashes of files
• Easy update of mirrors file database (at file/directory level)
• Support of very large files >100GB
• IPV6 support

@benoit74 @rgaudin Do you see other features which are important to us?

Now what needs to be decided is when and how we will proceed to move forward with this POC with Mirrorbits.

[benoit74]

I have very little experience on this part of the stack.

One thing we struggle with currently is the scans of mirrors to refresh individual assets status. Currently this process has to be made one mirror at a time, it is not possible to run in parallel (at least we failed). The new solution must be able to run this scan in parallel, otherwise it is not scalable. As the number of mirrors grows, the time to scan all of them grows as well and our refresh period if getting bigger and bigger.

Currently the refresh period is getting pretty high, more than 2 hours at least: https://kiwixorg.grafana.net/d/bb0f0990-04c5-4314-8afc-6185ac49c668/mirrorbrain?orgId=1&from=1695625815425&to=1696230615425

[rgaudin]

We've decided that @benoit74 will assess mirrorbits in regard to our needs. What we want to know is:

• whether mirrorbits is suited for the job
• which mb features (that we are relying on) are missing from it:
  • if we could bring them back
  • how (what king of effort)
    The goal is to be able to assess whether it's doable with our resources.

[benoit74]

This is my comparison chart so far.

❌ Not Supported, bad thing
✅ Supported
❓Unknown (meaning probably not)

Feature	MirrorCache	MirrorBits
Metalink (Metalink headers)	✅	❓JSON file mentioned, but not compatible with aria obviously
Bittorrent files	❓	❓
Magnet links	❓	❓
Mirror mgmt via ftp/http/rsync	HTTP only❓ (no access to file)	FTP and RSYNC only
Priorisation of mirrors	❓	✅
Auto choice of mirrors based on client geo location	Geo only	Geo + AS number + custom rules
Multiple hashes of files	❓	❓
Easy update of mirrors file database (at file/directory level)	✅ Mirrorcache has been designed to fix Mirrorbrain issues around parallel scans and scans taking ages to update the DB	❓
Support of very large files >100GB	✅ Probably	✅ Probably
IPV6 support	✅	✅
Documentation	❌ Too limited	❌ Too limited
Programming Language	Perl	GoLang
Database	PostgreSQL	Redis (with persistence)
Project liveness	Project updated regularly ; Multiple PR closed on a regular basis, including last days / weeks	No update since at least one year, no code change since 2020, many very simple pending PR without responses, still based on Golang 1.13 (Sept 2019)
Developers	One main dev (Andrii Nikitin), working at openSuse (project supporter) ; another person helped a bit in the past	One single dev (etix), based in Paris, former Videolan Ops + developer, no more activity on Github / personal blog / twitter
Usage	openSuse only ?	Many websites mentioned, including some which have stopped using it

I'm really not convinced by those two solutions. I would probably prefer to stay with MirrorBrain for now until we find a better solution.

If we are forced to choose one now, I will try MirrorCache for:

• its new design which solves something our scan issues
• its liveness and hence ability for us to fix things / submit patches (if anyone is willing to contribute in Perl, I don't)
• its PostgreSQL DB, I'm still absolutely not convinced by NoSQL databases + it probably means we will have to learn how to install it properly, backup it, maintain it, ...

Effort to implement MirrorCache given all missing features is however probably significant (1 month?). I have too limited experience of Bittorrent / Magnet links to say something very pertinent on that point. But since it is written in Perl, we probably need to hire an external developer to do our stuff.