Feedback on CDX Server requirements page

[ikreymer]

Hi, I wanted to give feedback on the CDX Server requirements wiki page.

There's not really a good way to comment on the page though, so rather than just editing the wiki page, I thought it'd be easier to start a conversation as an issue. Feedback follows as comments.

As part of making the CDX-Server the default index engine for OpenWayback we need to clean up and formally define the API for the CDX-Server. This document is meant as a workplace for defining those API's.

I think that's a great idea, especially this API can be shared across multiple implementations, not just OpenWayback.

The CDX-Server API, as it is today, is chracterized by a relatively close link to how the underlying CDX format is implemented. Functionality varies if you are using traditional flat CDX files or compressed zipnum clusters. One of the nice things by having a CDX Server is to separate the API from the underlying implementation. This way it would be relatively easy to implement indexes based on other technologies in the future. As a consequence we should avoid implementing features just because they are easy to do with a certain format if there is no real need for it. The same feature might be hard to implement on other technologies.

The intent was to keep it separate (and there is support for different output formats, eg. JSON lines). The zipnum cluster does provide extra APIs, such as Pagination, but that is mostly because pagination is otherwise technically difficult without a secondary index, nothing ties it to zipnum cluster implementation in particular. The 'secondary index' is presented as a separate concept and perhaps could be kept abstracted out further.

The API should also try to avoid giving the user conflicting options. For example it is possible, in the current api, to indicate match type both with a parameter and a wildcard. It is then possible to set matchType=prefix and at the same time use a wildcard indicating matchType=domain.

Sure, the wildcard query was added a 'shortcut' in place of the matchType query, 'syntactic sugar', but if people feel strongly about removing one or the other, I don't think its a big deal

The following is a list of use-cases seen from the perspective of a user. Many of the use-cases are described as expectations to the GUI of OpenWayback, but is meant to help the understanding of the CDX-Server's role. For each use-case we need to understand what functionality the CDX-Server is required to support. CDX-Server functionality with no supporting use-case should not be implemented in OpenWayback 3.0.0.

This is a work in progress. Edits and comments are highly appreciated.

The CDX Server API was not just designed for GUI access in OpenWayback, but a more general API for querying web archives. The interactions from a GUI in OpenWayback should be thought of as a subset of the functionality that the API provides. Everything that was in the API had a specific use case at one point or another.

As a starting point, the CDX Server API provides two APIs that are defined by memento:

• TimeGate Query = closest timestamp match
• TimeMap Query = list of all captures for a given url

The closest match functionality is designed to provide an easy way to provide the next closest fallback, if replay of the first memento fails, and allow for trying the next best, and so forth..

Another use case was better support for the prefix query, where the result is a list of unique urls per prefix, followed by the starting date, end date and count. The query can then be continued to get more results from where the end of the previous query.

Another important use case is parallel bulk querying, which can be used for data extraction.
For example, a user may wish to extract all captures by host, prefix, or domain across a very large archive. The user can create MapReduce job to query the CDX server in parallel, where each map task sets the page value. (Implementations of this use case already exist in several forms).

The difference between the bulk query and the regular prefix query, is that the pagination api allows you to query a large dataset in parallel, instead of continuing from where the previous query left off.
But this requires pagination support, which requires the zipnum cluster, but it would in theory be possible to support without (just requires do a lot more work to sample the cdx to determine page distribution).

Another use case was resolving revisit records, if the original was the same url, in a single pass, to avoid having to do a second lookup. This is done by appending the original record as extra fields.
This may be not as useful if most deduplication is 'url agnostic'

_

Use-cases

1. The user has a link to a particular version of a document

This case could be a user referencing a document from a thesis. It is important that the capture referenced is exactly the one the user used when writing the thesis. In this case the user should get the capture that exactly matches both the url and timestamp.

This is more of a replay system option, rather than cdx query.
What happens if the exact url doesn't exist? There is not a way to guarantee exact match just by url and timestamp, you would also need the digest, and you can filter by url, timestamp and digest with cdx server, but not with a replay (archival url) format.

2. The user selects one particular capture in the calendar

Pretty much the same as above, but it might be allowed to return a capture close in time if the requested capture is missing.

I think this is not at all the same as above, but closest capture/timegate behavior. An option could be added to remove closest match and only do exact match, but again, this is a replay system option, not cdx server option..

3. Get the best matching page when following a link

User is looking at a page and want to follow a link by clicking it. User then expects to be brought to closest in time capture of the new page.

4. Get the best match for embedded resources

Similar to above, but user is not involved. This is for loading embedded images and so on.

It seems that these all fall under the 'closest match' / Memento TimeGate use case

5. User requests/searches for an exact url without any timestamp, expecting to get a summary of captures for the url over time

The summary of captures might be presented in different ways, for example a list or a calendar.

Yep, this is the Memento TimeMap use case.

6. User looks up a domain expecting a summary of captures over time

7. User searches with a truncated path expecting the results to show up as matching paths regardless of time

8. User searches with a truncated path expecting the results to show up as matching paths regardless of time and subdomain

These are all different examples of the prefix query use case.

9. User navigates back and forth in the calendar

This is already possible with the timemap query, right?

But, could also add an "only after" or "only before" query, to support navigating in one direction explicitly.

10. User wants to see when content of a page has changed

This seems more like a replay api, as cdx server is not aware of embeds or relationships between different urls.

[johnerikhalse]

Thanks @ikreymer for the valuable input.

First of all, this is a work in progress so there are probably a lot of use-cases to be written including use-cases for processing by a Map-Reduce framework. To get started, I begun to write up some use-cases based on the end-user experience on OpenWayback. As you correctly mention, several of these use-cases can be supported by the same set of functionality from the CDX-Server. I just wanted the list to be as complete as possible.

While working with OWB and the CDX-Server, it is a challenge for me to understand all the options supported by the CDX-Server. Both because several parameters are not documented, but also because I do not understand the use-case for all parameters. I might be wrong, but my impression is that sometimes functionality has been implemented to work around shortcomings which preferably should have been fixed other places. I think that the 3.0.0 version could be a good time to clean up even if it breaks backward compatibility. After all the current CDX-Server is still marked as BETA.

I was also thinking of including a comment for each use-case if the use-case was valid or doesn't make sense. This way we would keep track of ideas that have been rejected.

The main goal with this document is to make it easier for new people (like me) to participate in coding. The use-cases written so far only reflects my understanding. I would love if you (or anybody else) add, remove or improve use-cases.

One thought I have, is whether it would be better to split up the API into more than one path and reduce the number of parameters. For example from/to parameter doesn't make sense combined with sort=closest. Instead the closest parameter is used for the timestamp. I don't think wildcard queries should be combined with closest either (at least I have not found the use-case). Maybe we could split API into something like /query which allows a timestamp parameter and a /search which allows from/to parameters and wildcards and/or matchType. This way the API would be more explicit and less code for checking combinations of parameters would be needed on server side.

I would also like to know the use-cases for parallel queries with the paging API. If i'm not mistaking, the current implementation could give different results for the paging API from the regular API if you use collapsing. This is because the paging API will not know if the starting point for a page, except for the first one, is in the middle of several lines to be collapsed. Maybe a separate path (e.g. /bulk) should added which do not allow manipulation like collapse?

For revisits, I wonder if that is a task for the CDX-server to resolve. I see the efficiency with one pass as you mention, but also the problem with url agnostic revisits. Maybe that should either go into a separate index keyed on digest, or maybe as extra fields when creating CDX-files instead of adding them dynamically.

[ikreymer]

Thanks @johnerikhalse It sounds like you'd prefer editing in the wiki directly. I can move some of my comments to the page, and add additional use cases. Another option might be to have an issue for each proposed cdx server change or suggestions, so that folks can comment on additional suggestions as well, and then at the end, modify the wiki based on the suggestions.

One thought I have, is whether it would be better to split up the API into more than one path and reduce the number of parameters. For example from/to parameter doesn't make sense combined with sort=closest. Instead the closest parameter is used for the timestamp. I don't think wildcard queries should be combined with closest either (at least I have not found the use-case). Maybe we could split API into something like /query which allows a timestamp parameter and a /search which allows from/to parameters and wildcards and/or matchType. This way the API would be more explicit and less code for checking combinations of parameters would be needed on server side.

Hm, i think that from/to can still make sense with sort closest, if you wanted to restrict the results to a certain date range (not certain if that works currently). The sort closest is definitely a special case.

I would also like to know the use-cases for parallel queries with the paging API. If i'm not mistaking, the current implementation could give different results for the paging API from the regular API if you use collapsing. This is because the paging API will not know if the starting point for a page, except for the first one, is in the middle of several lines to be collapsed. Maybe a separate path (e.g. /bulk) should added which do not allow manipulation like collapse?

Yes, the collapsing would just happen on that specific page, as that is all that is read.. I can see the benefits of a separate 'bulk' api in some sense, though removing functionality may not be best approach.

I think the cdx server APIs should be thought of as a very low level api, designed to be used by other systems and perhaps advanced users. As such, there could be combinations don't quite make sense. There could be higher level APIs, such as the 'calendar page api' or 'closest replay api' which prevent invalid combinations. Similarly, there could be a 'bulk query api' which helps the user run bulk queries.

For revisits, I wonder if that is a task for the CDX-server to resolve. I see the efficiency with one pass as you mention, but also the problem with url agnostic revisits. Maybe that should either go into a separate index keyed on digest, or maybe as extra fields when creating CDX-files instead of adding them dynamically.

Yeah, I suppose this can be removed and it would be up to the caller to resolve the revisit. Or, better yet, there could be a separate 'record lookup API' which queries the cdx API once or more times to find the record and the original (if revisit).

[johnerikhalse]

Another option might be to have an issue for each proposed cdx server change or suggestions, so that folks can comment on additional suggestions as well, and then at the end, modify the wiki based on the suggestions.

I agree that we should have an issue for each change in the cdx server. I also agree that cdx server is a pretty low level API. The document I started writing is meant as a background for proposing changes. It will hopefully be easier to see why a change is necessary when you can relate it to one or more use cases. It wasn't my intention to describe the cdx server in that document. That should go into a separate documentation of the API. Usually fulfilling the requirements of a use case will be a combination of functionality in the cdx server and the replay engine.

I think the cdx server APIs should be thought of as a very low level api, designed to be used by other systems and perhaps advanced users. As such, there could be combinations don't quite make sense. There could be higher level APIs, such as the 'calendar page api' or 'closest replay api' which prevent invalid combinations. Similarly, there could be a 'bulk query api' which helps the user run bulk queries.

Even if the API is low level, I think it's important that users get predictable results. For example, I think it's a bad idea if a function return different results if you are using paging or not. Another example would be if a function return different results for a configuration with one cdx-file from a configuration with multiple cdx-files.

I have a question about match type. I can see the justification for exact, prefix and domain, but host seems to be just a special case of prefix. Is that correct? According to current documentation you can use wildcards to indicate prefix and domain and no wildcard for exact. If host isn't necessary, it is possible to avoid the matchType parameter all together.

[johnerikhalse]

One comment to how low level the cdx server api is. I agree that it is to low level for most end users, but there is built in logic assuming certain use cases which, in my opinion, doesn't qualify for calling it very low level. For example filtering is done before collapsing (which makes sense). Given the query parameters filter=statuscode:200&collapse=timestamp:6, you could get quite different results if filtering was done after collapsing. If someone comes up with a use case where filtering should be done after collapsing, or perhaps one filter before collapsing and another filter after, that would not be possible because cdx server has this built in assumption of some use cases.

For a real low level api we should implement a kind of query language, but I think that is to overcomplicate things. I think it is better to try to document real world use cases and create enough functionality to support them. It is important tough to document those built in assumptions.

Makes sense?

[ikreymer]

Yes, a full query language may be a bit too much at this point. If collapsing is the main issue, perhaps it can be taken out, except in a very specific use case. For example, I believe the collapseTime option (not documented) was added specifically to allow a specific optimization of skipping secondary ZipNum blocks.. I think it makes sense to examine the collapsing and see if it belongs, and perhaps document certain optimizations. Most of the optimizations were created in the context of IA wayback scale cdx queries, and may not even be relevant for smaller deployments.

As a reference, in the pywb cdx server API, I have implemented CDX features only as they became needed (https://github.com/ikreymer/pywb/wiki/CDX-Server-API) resulting in a smaller subset of the current openwayback cdx server API. Collapsing has not really been needed, and it is something that I think may best be done of the client.

For example, a calendar display may offer dynamic grouping as needed as user switches different zoom levels, w/o having to make a server-side request each time.

Also, taking a look at http://iipc.github.io/openwayback/api/cdxserver-api.html I would strongly advise against going in this direction. The /search /query and /bulk distinctions are extremely confusing,
(not to mention search and query are often used synonymously). It appears that under this scheme, there would be 3 different ways to query the same thing:

http://cdxserver.example.com/cdx/main/search?url=foo.com
http://cdxserver.example.com/cdx/main/query?url=foo.com
http://cdxserver.example.com/cdx/main/bulk?url=foo.com

which would make things more complicated. I think that most of the CDX server features can be combined (collapsing perhaps being an exception), and when they can't, that can be documented.
I don't think there is a need for multiple endpoints at this time.

[johnerikhalse]

Thanks for the comments @ikreymer

For example, I believe the collapseTime option (not documented) was added specifically to allow a specific optimization of skipping secondary ZipNum blocks.

If it is just an optimization not altering the results compared to the general collapse function, then I think such optimizations should be done by analyzing the query and not by adding more parameters.

Collapsing has not really been needed, and it is something that I think may best be done of the client.

For example, a calendar display may offer dynamic grouping as needed as user switches different zoom levels, w/o having to make a server-side request each time.

That's true if we by client understand the browser. If the client is OpenWayback, then I think a new roundtrip to the Cdx Server is better to avoid keeping state in OWB.

The /search /query and /bulk distinctions are extremely confusing,
(not to mention search and query are often used synonymously).

I agree that the names 'search' and 'query' are not good names (suggestions welcome). But I do think the split makes sense. The query path is ment to support the Memento TimeGate use case and the search path supports the Memento TimeMap as described in your first posting on this issue.

It appears that under this scheme, there would be 3 different ways to query the same thing:
http://cdxserver.example.com/cdx/main/search?url=foo.com
http://cdxserver.example.com/cdx/main/query?url=foo.com
http://cdxserver.example.com/cdx/main/bulk?url=foo.com

No, they are not the same.

• The first one would return a list of all captures for foo.com sorted in ascending order and could be mainpulated by filters and collapse, and the list could be reversed with sort=desc
• The second is similar to sort=closest&closest=<today> which will get the captures sorted closest to the timestamp. But since no timestamp is submitted (and then by default using current time) it will in effect sort it like sort=desc (or sort=reverse in current implementation). It might be that the default limit should be set to 1 to only return the best match. This path allows filtering to for example only return 2xx responses. Collapse is not allowed since it is not clear in which order the captures are returned.
• The bulk request will return a list of urls to batches which could be retrieved by subrequests. The reason to return a list of urls is to allow the Cdx Server to spread load to several instances if such functionality is to be implemented.

[ikreymer]

If it is just an optimization not altering the results compared to the general collapse function, then I think such optimizations should be done by analyzing the query and not by adding more parameters.

The optimization does alter the results, I believe it allows for skipping of secondary index zipnum blocks when the data was too dense (eg. over 3000+ captures of same url within one minute/hour/day, etc...)

I agree that the names 'search' and 'query' are not good names (suggestions welcome). But I do think the split makes sense. The query path is ment to support the Memento TimeGate use case and the search path supports the Memento TimeMap as described in your first posting on this issue.

Well, you could use timegate and timemap, but the cdx server does a lot more than memento, so that may only add to confusion. The difference between these is 'list all urls' and 'list all urls sorted by date closest to x', so in this case closest=<date> makes the most sense to user.

Even if the names were different, switching between query <-> search just changes the default value of the sort field, right? I think just makes it more complicated then allowing user to specify sort, as it does now (and the user can still change the sort order). This doesn't seem to solve any problem.

The separate bulk endpoint might make sense, if there is a good reason that the page= counter would not suffice.

The bulk request will return a list of urls to batches which could be retrieved by subrequests. The reason to return a list of urls is to allow the Cdx Server to spread load to several instances if such functionality is to be implemented.

That is a very dubious reason for such a big change. Modern load-balancers can spread the load automatically amongst worker machines. This is like arguing for going back to the era of using www2. www3. prefixes to spread the load on a web site.

Also, when is a query considered 'bulk'? Suppose a user wants to get all results for a single url, there could be 1 result or there could be, say 50,000. Should they use the bulk query or the 'non-bulk' query?
If they use the non-bulk query, will they see all the results at once, or will it be cut-off at some arbitrary limit (since there is no pagination).
If it's all the results, but there is no pagination, the result may be much larger then expected and will take longer to load. If there is a cut-off, then the user may not get all the results.
Thus, it seems like there is no reason to use the non-bulk query at all when the pagination supporting bulk query is available..

In the current system, page=0 is implied, so the user will see the first page of results. It is also possible to query and set the page size, so the user has an idea of the max number of results to expect. (This can be improved too).

Again, I would suggest ways to improve the current API rather than creating distinct (and often conflicting) endpoints that only add more user decision (bulk vs non-bulk, search vs query, etc..) and do not actually add any new features.

[johnerikhalse]

The optimization does alter the results, I believe it allows for skipping of secondary index zipnum blocks when the data was too dense (eg. over 3000+ captures of same url within one minute/hour/day, etc...)

I'm not sure if I understood this correctly, but I wondered if the result was altered different from using ordinary collapse or if collapsetime is just an optimisation for collapse=time. In the latter case I think the server should be smart enough to optimize just by looking at what fields you are collapsing on.

Even if the names were different, switching between query <-> search just changes the default value of the sort field, right?

Looking at http://iipc.github.io/openwayback/api/cdxserver-api.html, I realized that it might not be obvious that the methods (the blue get boxes) are clickable to get a detailed description of the properties. I added a label above the column to make it clearer. I also updated the descriptions somewhat. The reason I mention this is because there are other differences than just the sort which hopefully should be clear from the documentation and not needed to be repeated here. If that is not the case, then feedback is valuable to enable me to enhance it.

The bulk request will return a list of urls to batches which could be retrieved by subrequests. The reason to return a list of urls is to allow the Cdx Server to spread load to several instances if such functionality is to be implemented.

That is a very dubious reason for such a big change. Modern load-balancers can spread the load automatically amongst worker machines. This is like arguing for going back to the era of using www2. www3. prefixes to spread the load on a web site.

Well, I might have been a little too fast when stating that as the primary reason, it should be more like a possibility. Another possible use could be to hand bulk requests over to dedicated servers for that purpose. Anyway, the primary reason was to get closer to the guidelines for REST. REST advocates the use of absolute urls for references. That leaves less work on the client and ease the evolution of the api. Even though the Cdx Server is not restful, I think following the guidelines where it is possible is reasonable.

Also, when is a query considered 'bulk'? Suppose a user wants to get all results for a single url, there could be 1 result or there could be, say 50,000. Should they use the bulk query or the 'non-bulk' query?
If they use the non-bulk query, will they see all the results at once, or will it be cut-off at some arbitrary limit (since there is no pagination).

If it's all the results, but there is no pagination, the result may be much larger then expected and will take longer to load. If there is a cut-off, then the user may not get all the results.

The bulk api is not meant to be used by OpenWayback, but by processing tools, and yes it definitely supports paging which the other apis in the proposal don't.

To not discuss everything in one issue, I created a separate issue for the bulk api (#309) where I try to describe it in more detail.

[ldko]

In response to the CDX Server API, I think I understand the motives behind having both a /get and a /search but don't wholly agree with the need. It looks like the main difference is that /get is meant to look up an "exact URL," but because of the nature of web archives in which a URL in itself does not clearly reference a single resource, requests to /get are still negotiating a set of results by use of query parameters very similar to those used at /search.

Also, if I understand, you are saying for /get requests:

Collapse is not allowed since it is not clear in which order the captures are returned.

Not having collapse as a parameter for /get seems to somewhat go against what is in the examples in the README of wayback-cdx-server-webapp where collapse is explained to play a significant role in calendar pages. Based on the current API proposal, is /get what would be used for calendar pages?

Regarding /bulk:

Anyway, the primary reason was to get closer to the guidelines for REST. REST advocates the use of absolute urls for references.

Many RESTful systems use pagination. If the next and previous pages are discoverable from a page, then it does not violate REST. Previous and next links can be given in HTTP Link headers and/or in the body of the response (would be relatively normal looking in a JSON response); the total number of results could also be returned in the headers or the response body. What I see currently in the proposal for /{collection}/bulk looks similar to the sort of request you would have if you just used pagination on /search, so I am not sure a separate path needs to be used.

[ikreymer]

I agree with everything @ldko stated.
There just does not seem to be a good reason to split api into several endpoints at this point.

Most of the options can interact with all the other options, including collapse
As I recall, it was specifically implemented to support grouping by date, and perhaps the collapse and collapseTime confusion is definitely one area that can be improved... (and its been many years since I looked at it in quite detail)

My point was that if /bulk has pagination why shouldn't /search? Saying an API is only for external use also doesn't seem like a good enough reason.. Suppose someone wants to use the bulk API with openwayback directly. For example, a UI could be added to allow users to get a quick estimate of the number of pages.. (IA wayback had this at one point).

There can definitely be improvements to the showNumPages query though. One is to make it a JSON object, as has been done in the pywb implementation.

An example of it is seen here:
http://index.commoncrawl.org/CC-MAIN-2015-48-index?url=*.com&showNumPages=true
{"blocks": 444486, "pages": 88898, "pageSize": 5}

For IA, this query: http://web.archive.org/cdx/search/cdx?url=*.com&showNumPages=true returns
16 million+ pages. If you add a page listing, you would have to list all 16 million urls, where the only thing that changes is the page number... Sure, you can add pagination to that, but why bother making this change?

A lot of thought went into making the original API as it is, and there are already a few tools that work with the current api (for bulking querying) so I think any significant changes should have a clear positive benefit.