I don't see anything, normal?

[aderchox]

Is this expected that I'm only seeing an empty template (a red bar at the top and two arrows on both sides of the page) not anything else?

[allan2]

Fetch for the report JSON is returning 404. Perhaps it isn't being generated.

@taoalpha?

[taoalpha]

ya I ran out of github storage for storing all the history records, so the CI builds has been failing for quite a while... trying to purge some very old records to bring it back

[taoalpha]

the site should be back now

[hbriese]

It has broken again

[0xdevalias]

Looking at the data/ folder I see the following files:

• https://github.com/taoalpha/npm-trending/tree/master/data
  • 404s.json
  • info-2024-11-25-0.json
  • info-2024-11-25.json
  • info-2024-11-27-0.json
  • info-2024-11-27.json
  • meta.json
  • stat-2024-11-25.json
  • stat-2024-11-27.json

Looking at meta.json:

• https://github.com/taoalpha/npm-trending/blob/master/data/meta.json
  • {"last_generated_date":"2024-11-26"}

Looking at the website:

• https://taoalpha.github.io/npm-trending/

I can see that it's getting a 404 while trying to access the following URL:

• https://taoalpha.github.io/npm-trending/reports/pkg-2025-06-15.json

We can see that template/lib/main.ts is responsible for this fetch, and it will default to 'today' if not set in the query parameter:

npm-trending/template/lib/main.ts

Lines 411 to 420 in 6dd3205

	// the date, default to today if not set from querystring
	let theDate = Helpers.getParameterByName("date") || DateHelper.add(DateHelper.today, -1);
	if (isNaN(new Date(theDate).getTime())) theDate = DateHelper.add(DateHelper.today, -1);
	// fetch data
	Helpers.ready(() => {
	// get the json
	(axios as any).get(`./reports/pkg-${theDate}.json`)

Since we can see the last_generated_date in meta.json (2024-11-26), we could try manually accessing that by setting it in the URL:

• https://taoalpha.github.io/npm-trending/?date=2024-11-26

But that also seems to get a 404 unfortunately:

• https://taoalpha.github.io/npm-trending/reports/pkg-2024-11-26.json

We can look even deeper, and check what exists in the gh-pages branch, in the reports/ folder that seems to be where the /reports/pkg-YYYY-MM-DD.json is being fetch from:

• https://github.com/taoalpha/npm-trending/tree/gh-pages/reports

Looking at that, it seems that the most recent entry is pkg-2019-11-25.json:

• https://github.com/taoalpha/npm-trending/blob/gh-pages/reports/pkg-2019-11-25.json

And we can confirm that that loads on the website as expected with the following URL:

• https://taoalpha.github.io/npm-trending/?date=2019-11-25

Based on that, we would expect that we wouldn't be able to load 2019-11-26, but as you can see, that also seems to be accessible:

• https://taoalpha.github.io/npm-trending/?date=2019-11-26

So I believe that GitHub's UI is just refusing to show the full list of files when I browse it through the website. Looking a bit closer at GitHub's repository size limits:

• https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-large-files-on-github#repository-size-limits

It's probably because it violates this pattern:

• https://github.com/github/git-sizer

  • Does the repository include gigantic trees (directories)? Every time a file is modified, Git has to create a new copy of every tree (i.e., every directory in the path) leading to the file. Huge trees make this expensive. Moreover, it is very expensive to traverse through history that contains huge trees, for example for git blame. Suggestions:

    • Avoid creating directories with more than a couple of thousand entries each.
    • If you must store very many files, it is better to shard them into a hierarchy of multiple, smaller directories.

So I decided to clone the repo locally and find out what the latest entry was that way:

⇒ git clone git@github.com:taoalpha/npm-trending.git
Cloning into 'npm-trending'...
remote: Enumerating objects: 54331, done.
remote: Counting objects: 100% (140/140), done.
remote: Compressing objects: 100% (115/115), done.
remote: Total 54331 (delta 78), reused 87 (delta 25), pack-reused 54191 (from 3)
Receiving objects: 100% (54331/54331), 4.17 GiB | 8.53 MiB/s, done.
Resolving deltas: 100% (31878/31878), done.

Which as you can see, it's a fairly large repo:

⇒ du -sh npm-trending
4.4G	npm-trending

And looking at git-sizer's stats for the repo:

⇒ git-sizer
Processing blobs: 28025
Processing trees: 16171
Processing commits: 10135
Matching commits to trees: 10135
Processing annotated tags: 0
Processing references: 7
| Name                         | Value     | Level of concern               |
| ---------------------------- | --------- | ------------------------------ |
| Overall repository size      |           |                                |
| * Blobs                      |           |                                |
|   * Total size               |  95.1 GiB | **********                     |
|                              |           |                                |
| Biggest objects              |           |                                |
| * Trees                      |           |                                |
|   * Maximum entries      [1] |  2.10 k   | **                             |
| * Blobs                      |           |                                |
|   * Maximum size         [2] | 100.0 MiB | **********                     |
|                              |           |                                |
| Biggest checkouts            |           |                                |
| * Total size of files    [3] |  29.0 GiB | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! |

[1]  2cc565acaa0f66f1ad43fda447d388b353985e22 (refs/remotes/origin/gh-pages:reports)
[2]  062af3bc902ef2f26c67c0b958bce27e455f33ed (refs/remotes/origin/data-archive:data/info-2019-11-14.json)
[3]  78ebe67c38010d7fd7c5b911c87ce0529a3160b2 (2f38fb67a0735d6faba973c23e1f9157511fe23e^{tree})

Switching from master to gh-pages took a while, and was larger on disk again:

⇒ du -sh npm-trending
6.5G	npm-trending

And if we list the reports/ directory, we can see that the actual latest date seems to be pkg-2022-11-27.json:

 ⇒ ls -la reports | tail
-rw-r--r--    1 REDACTED REDACTED  1113058 Jun 16 12:47 pkg-2022-11-18.json
-rw-r--r--    1 REDACTED REDACTED  1677943 Jun 16 12:47 pkg-2022-11-19.json
-rw-r--r--    1 REDACTED REDACTED  1592835 Jun 16 12:47 pkg-2022-11-20.json
-rw-r--r--    1 REDACTED REDACTED  1398027 Jun 16 12:47 pkg-2022-11-21.json
-rw-r--r--    1 REDACTED REDACTED  1878301 Jun 16 12:47 pkg-2022-11-22.json
-rw-r--r--    1 REDACTED REDACTED  2433644 Jun 16 12:47 pkg-2022-11-23.json
-rw-r--r--    1 REDACTED REDACTED   738892 Jun 16 12:47 pkg-2022-11-24.json
-rw-r--r--    1 REDACTED REDACTED   819604 Jun 16 12:47 pkg-2022-11-25.json
-rw-r--r--    1 REDACTED REDACTED  6159624 Jun 16 12:47 pkg-2022-11-26.json
-rw-r--r--    1 REDACTED REDACTED  3862938 Jun 16 12:47 pkg-2022-11-27.json

Which we can confirm by checking on the website:

• https://taoalpha.github.io/npm-trending/?date=2022-11-27
  • https://taoalpha.github.io/npm-trending/reports/pkg-2022-11-27.json

And then we can double check that by confirming that the next day gets a 404 error, which it does seem to:

• https://taoalpha.github.io/npm-trending/?date=2022-11-28
  • https://taoalpha.github.io/npm-trending/reports/pkg-2022-11-28.json (404)

We can also look at what's in the data-archive branch, which took quite a while to checkout, and expanded to the following disk size when I did:

⇒ du -sh npm-trending
 33G	npm-trending

Looking in the data/ folder, the oldest entry seems to be:

• info-2018-08-31.json

And the latest entries seem to be:

⇒ ls -la data | tail
-rw-r--r--   1 REDACTED REDACTED   6748368 Jun 16 12:56 stat-2019-11-06.json
-rw-r--r--   1 REDACTED REDACTED   6703411 Jun 16 12:56 stat-2019-11-07.json
-rw-r--r--   1 REDACTED REDACTED   6674626 Jun 16 12:56 stat-2019-11-08.json
-rw-r--r--   1 REDACTED REDACTED   6524705 Jun 16 12:56 stat-2019-11-09.json
-rw-r--r--   1 REDACTED REDACTED   6510646 Jun 16 12:56 stat-2019-11-10.json
-rw-r--r--   1 REDACTED REDACTED   6532516 Jun 16 12:56 stat-2019-11-11.json
-rw-r--r--   1 REDACTED REDACTED   6684325 Jun 16 12:56 stat-2019-11-12.json
-rw-r--r--   1 REDACTED REDACTED   6777845 Jun 16 12:56 stat-2019-11-13.json
-rw-r--r--   1 REDACTED REDACTED   6794271 Jun 16 12:56 stat-2019-11-14.json
-rw-r--r--   1 REDACTED REDACTED   6824866 Jun 16 12:56 stat-2019-11-15.json

Looking back at the master branch, in the data/ folder's history, we can still see that there are a bunch of commits, with the most recent seeming to be for 2025-04-19:

• https://github.com/taoalpha/npm-trending/commits/master/data

But at least looking at the most recent commits, they only seem to be making changes to the following files:

• data/404s.json
• message

  • Job 1 fetch finished! 7588 packages fetched this time(2025-04-19).

• seed
  • This seems to be a CSV of package names; with about 1745 entries in the most recent commit I looked at.

A few thoughts/observations:

• In the gh-pages branch, instead of defaulting to the current date, perhaps you could store the newest data file that is available in a metadata file/similar, and then read from that to fetch the default page of data (probably also showing a warning banner if it is out of date)
• In the gh-pages / data-archivebranches; instead of having all of the json files directly in thereports//data/ folders, maybe it would be better to separate them into sub-folders; the most intuitive of which could be by year (reports/YYYY/.json, data/YYYY/.json), or even by year then month (reports/YYYY/MM/*.json, `data/YYYY/MM/*.json)
• Instead of storing all of the historical data in a single git repo, and then needing to purge old data to prevent running into GitHub's size limits; maybe you could separate the data across multiple repositories. That seems to be the approach that PyPi-Data took:
  • https://github.com/pypi-data
  • https://py-code.org/
• While I didn't compare/contrast what the latest data/ in master is to what is stored in data-archive, I assume it is basically the same format. What I was less clear about is whether the data in reports/ on gh-pages is also that same sort of format; which if so, there would seem to be a lot of duplication in the repo, that would be making it larger than it needed to be, and contributing to it crashing out due to space issues. If it is the same, I wonder if instead of inlining the data directly in the gh-pages branch, you could fetch it directly from GitHub's 'raw' view, eg.
  • For: https://github.com/taoalpha/npm-trending/blob/gh-pages/reports/pkg-2017-03-01.json
  • Instead of loading it from: https://taoalpha.github.io/npm-trending/reports/pkg-2017-03-01.json
  • Accessing it directly from the raw URL like: https://raw.githubusercontent.com/taoalpha/npm-trending/refs/heads/gh-pages/reports/pkg-2017-03-01.json
  • That way, you could read it from any branch as needed.
  • But obviously, that's a moot point if the data/ and reports/ data is different.