Migrate to Zimfarm API v2

[benoit74]

A significant upgrade of the Zimfarm is being prepared by @elfkuzco (and a bit of myself).

We will take this opportunity to make breaking changes in the API and move to a v2.

Details about the migration are at https://github.com/openzim/zimfarm/blob/zimfarm-upgrade/backend/V1_TO_V2_MIGRATION.md

Zimfarm v2 schema has now more or less stabilized, so we can start to work on adapting WP1.

You can see both v1 and v2 schemas (and they will stay there after the migration as reference) at https://tmp.kiwix.org/ci/zimfarm_v2/

Migration is for now planned for second half of August (or early September, depending how ready everybody is, and how stuff is going on when I go back from holidays).

@audiodude how would you like to perform the migration effort?

From my perspective, the simplest is maybe that @elfkuzco prepares a draft PR (since he knows quite well the migration details, and WP1 code should be pretty easy to modify) and we both review it. WDYT?

[audiodude]

@benoit74 I think @elfkuzco preparing the draft PR is a great move.

However, I worry about the timing with our GSoC project. We are already modifying the way we request ZIM files, so I don't want the work to conflict. Is there any time pressure for this migration, beyond the fact that WP1 is a "first party" tool and we control the implementation?

[benoit74]

Is there any time pressure for this migration

Once we will deploy Zimfarm with API v2, WP1 will stop to work, we do not provide any backward compatibility. That being said, there is no strong pressure to move API v2 in production. The only concern I have is that we've already accumulated quite a lot of work with @elfkuzco, and it is clearly sad we do not deploy this to production "asap". We already have many benefits expected from this migration, but this is nothing until deployed. And it will continue to get worse with @elfkuzco fixing more and more important issues.

I'm however definitely open to consider that "asap" can be a bit later than sooner. What would be the good timeframe for you? "New" WP1 should be ready by end of August? Maybe migration both Zimfarm and WP1 at the same time could be a great opportunity to have focus on these systems and quickly fix what might get broken?

[audiodude]

Yes, GSoC ends the week of Sept 1.

We could potentially start the v2 work now as a branch off of the schedule-zims branch, which is where all the GSoC code is being staged. Then, it would be merged at the end of August with everything else. So it could be prepared ahead of time and land at the same time.

[benoit74]

Looks good to me. Should we realize we have an issue deploying schedule-zims branch, we will always have time to arrange another plan.

This means that from the moment we've merge @elfkuzco work inside schedule-zims branch, you will have to use zimfarm-upgrade branch on Zimfarm repo. Beware that there are also DB changes to perform if you already have a local Zimfarm DB.

[audiodude]

@nicolapace FYI

[elfkuzco]

Opend #968 to begin the migration