Feature: downloads leaderboard

[epoberezkin]

@pvorb The idea is to show the leaderboard on the homepage without entering any packages. Could be top 10 packages in the last week, month, year. It could be updated daily if you have concerns that it is a heavy query to hit on each home page load.
It should be relatively straightforward, unless I am missing something.

Given that downloads leaderboard doesn't exist anywhere else it could give you some traffic boost.

[epoberezkin]

Also could be the same for authors (I forgot that you can compare authors stats :)

[epoberezkin]

Found this: #35
I think there is a database now?

[pvorb]

There is a database right now, but it only serves as a cache. The problem is that I don't download the numbers for all the packages every day, but only those that are requested. The numbers are then persisted and won't be downloaded again.

For the feature you requested, the first thing to do would be to get the list of all packages every day, and then download the counts for every package.

While it sounds cool, I don't think I'll implement this feature. There doesn't seem to be a good API on npm's end to get the names of every package. If there was some sort of daily dump it could be easily done.

[epoberezkin]

I see. But it should be relatively simple to see and maintain top 10 of all requested by updating daily top 100 of all requested. A bit less simple though :)

[pvorb]

But if it only included the top X of requested packages, how helpful would that statistic be?

[epoberezkin]

The idea is that sooner or later most important packages will be requested. I meant "show top 10 of top 100 ever requested packages"

[epoberezkin]

So trying to have the full list of all packages, you'll let the visitors to create the lists of important packages and updating them daily. Or maybe a good start could be to download stats of these packages: https://gist.github.com/anvaka/8e8fa57c7ee1350e3491 and then maintain them daily (I think you can get weekly, monthly and annual in one API call each, you don't need as much calls as to do the chart.

[pvorb]

There's a complete list of all packages at https://replicate.npmjs.com/_all_docs. Maybe I can use that for downloading all package stats in a daily cron job...

[0xdevalias]

There doesn't seem to be a good API on npm's end to get the names of every package. If there was some sort of daily dump it could be easily done.

There's a complete list of all packages at replicate.npmjs.com/_all_docs. Maybe I can use that for downloading all package stats in a daily cron job...

This is basically the most recent/relevant page for updates to that replicate API:

• https://github.com/orgs/community/discussions/152515

There seem to be a few issues with it at the moment (recently migrated to a new system), I believe this is the most recent/relevant official update on that + a summary of some of the issues:

• https://github.com/orgs/community/discussions/152515#discussioncomment-13433941

But at least in theory, when it's working correctly, you'll be able to download that full snapshot once, and then use the sequence number to just download the updates since that first 'snapshot' you downloaded from:

• https://replicate.npmjs.com/registry/_changes

For many usecases, the revision included in those records is useful for matching up with the full package data from one of the other npm API's; but for this use case I don't even think the part matters, as you're mostly just wanting a list of package names; so you could probably:

• fetch the latest changes since the last snapshot
• process and de-duplicate the package entries in that snapshot into a 'reduced snapshot' (since you only really care if a package name exists, not how many times/etc)
• process and de-duplicate that 'reduced snapshot' against the main database (I separated this from the above step since I figured de-duplicating the snapshot against itself is probably cheaper than hitting the DB for each entry; but it may be a negligible cost either way)

I haven't looked deeply into the download stats API, but I think that would probably actually end up being the 'heavier' part of things; at least if you wanted it to be truly representative (and not just an approximation based on already fetched/cached data). The docs for the download count API are here:

• https://github.com/npm/registry/blob/main/docs/download-counts.md

I won't go into that too deeply, since I believe it's what this project uses currently anyway, but probably the most important part of that would be the limitation of bulk queries:

• https://github.com/npm/registry/blob/main/docs/download-counts.md#bulk-queries

  • Important: Scoped packages are not yet supported in bulk queries. So you cannot request /downloads/point/last-day/@slack/client,@iterables/map yet.

  • Bulk queries are limited to at most 128 packages at a time and at most 365 days of data.

  • All other queries are limited to at most 18 months of data. The earliest date for which data will be returned is January 10, 2015.

Given we can only fetch the details of 128 packages at a time, we can do a bit of 'back of napkin' math to figure out what would be required. Looking at the total number of rows in the replicate API:

• https://replicate.npmjs.com/registry/_all_docs
  • total_rows: 5537482

I'm not 100%, but if we assume each of those corresponds with a single npm package, and there aren't duplicates/etc in that count, then we would need ~43,262 queries to fetch all of the download data for those packages for any given time period:

• 5537482 / 128 = 43,261.578125 ~= 43,262

The real number of queries would actually be higher than that, as according to those docs, the bulk query system doesn't work for scoped packages, so each of those would need to be a single query on it's own.

From a quick search, I didn't find a lot about API rate limits/etc to see how viable it is to make that many requests over a short(ish) period of time; this was the best I found:

• Rate limit documentation for NPM Registry API npm/feedback#658
  • https://blog.npmjs.org/post/164799520460/api-rate-limiting-rolling-out.html

But if it were me implementing it, that number feels like it might be a bit too high for fetching daily stats.. so maybe weekly or monthly would be more viable in that regard.

That said, there are likely also a lot of packages in that list that aren't really used/downloaded very often; so it would be an interesting exercise to get the full list of package names, and then get their download stats for the last year or so, and then sort/filter based on that. I suspect that there is probably a large number of packages that would have 0 downloads, or at the very least less than a certain low threshold. You could potentially look at the download count on that higher time scale (eg. year, or month maybe) to filter down the full list of packages to a 'reasonably active' list (eg. above a certain download count threshold), and then only fetch the lower time-scale download counts (eg. daily) for that 'reasonably active' list of packages.

Anyway, that's my /2c of thoughts on this sort of thing based on recent explorations and my own project ponderings/musings.

Edit: See also:

• [RRFC] Expose an API for top packages by download count npm/rfcs#151 (comment)

[pvorb]

@0xdevalias Thank you for elaboring! Looks like a lot of work to do along the way. I'm not sure whether I should start working on this, although it would be very cool.

[0xdevalias]

I'm not sure whether I should start working on this, although it would be very cool.

@pvorb Yeah, that's fair enough. I figured I would just share my thoughts/notes since I came across the issue; but obviously whether you want to implement it in this project or not is up to you.