<div dir="ltr"><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">yeah, clients always need to make that extra call and it does seem a little clunky... that's the same pattern (and necessity) in several object store APIs I have been working with so vospace isn't alone here. <br></div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">On the other hand, over in DAL services we put a bit of metadata at the end of a VOTable (after the rows) to say that the result was truncated (DALI "overflow")... so the caller knows whether there are more records they didn't see.</div><div class="gmail_default" style="font-size:small"><br clear="all"></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div>--<br></div><div>Patrick Dowler<br></div>Canadian Astronomy Data Centre<br></div>Victoria, BC, Canada<br></div></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 21 Jun 2021 at 01:57, Zorba, Sonia <<a href="mailto:sonia.zorba@inaf.it">sonia.zorba@inaf.it</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Sorry, obviously I missed this too.</div><div><br>Maybe also expanding the first example could help.<br>I would add a first call with limit=100 and no uri parameter, to explain that if the response contains 100 child nodes the client knows that it can fetch the next results by setting the last child node URI in the uri parameter of the next call.</div><div><br></div><div>What I don't like about this approach is that if there were exactly 100 child nodes the next call would return a single result, so it could be avoided. It would be nice to have a "total count" parameter in the response, to know the exact number of remaining pages, but I don't know if this complicates too much the current implementations.</div><div><br></div><div>Cheers,</div><div>Sonia</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno dom 20 giu 2021 alle ore 05:30 Dave Morris <<a href="mailto:dave.morris@metagrid.co.uk" target="_blank">dave.morris@metagrid.co.uk</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Apologies, I forgot this was in the specification.<br>
The text describing how this works is buried in the Response section of <br>
the getNode method, which makes it easy to miss.<br>
<br>
As a start, I've added an issue to revise the text to make this clearer.<br>
<a href="https://github.com/ivoa-std/VOSpace/issues/4" rel="noreferrer" target="_blank">https://github.com/ivoa-std/VOSpace/issues/4</a><br>
<br>
This wouldn't change the technical definition, just promote the <br>
description of how pagination works into a separate sub-section clearly <br>
labelled 'pagination'.<br>
<br>
If we want to take it further, possibly by adding the exception that Pat <br>
proposes below, then that would be a new issue.<br>
<br>
-- Dave<br>
<br>
On 2021-06-18 22:00, Patrick Dowler wrote:<br>
> The current spec does support pagination when listing child nodes of a<br>
> container (*uri* and *limit* params), but implementation is complex. We<br>
> have two VOSpace implementations that illustrate quite well.<br>
> <br>
> Impl 1: relational database + object store<br>
> Here, it is easy enough to implement pagination because it is just a <br>
> couple<br>
> extra things injected into the SQL query to the DB. The server picks <br>
> the<br>
> default order, but we also added support for a custom optional param so <br>
> the<br>
> client could control the order: name, lastModified date, or <br>
> contentLength.<br>
> <br>
> Impl 2: only a posix file system<br>
> Here, it is really hard to implement pagination because the posix <br>
> directory<br>
> listing APIs don't have any concept of order (iirc, I determined it <br>
> lists<br>
> in inode order so you could get some strangeness if an inode is re-used <br>
> --<br>
> rename? -- during listing). It also looks more or less impossible to <br>
> scale<br>
> paginated listing with many children: with each request, you have to <br>
> start<br>
> at the beginning of the list and skip over previously seen entries so <br>
> it<br>
> gets slower and slower with each "page" of children. This service <br>
> cannot<br>
> support the custom sorting on the server side either.<br>
> <br>
> So, I would also like to improve the spec here but would like to see<br>
> something where a service that cannot support pagination (just stream<br>
> output) can be effectively used: clients will need to be able to figure <br>
> out<br>
> which to expect or at least if they got all the rows or not. That <br>
> really<br>
> means support for the *uri* parameter would be optional and maybe just<br>
> responding with an error with a specified "fault" term would suffice. <br>
> The<br>
> *limit* param is easy enough to implement (like MAXREC in DAL <br>
> standards) in<br>
> both cases.<br>
> <br>
> --<br>
> Patrick Dowler<br>
> Canadian Astronomy Data Centre<br>
> Victoria, BC, Canada<br>
> <br>
> <br>
> On Wed, 16 Jun 2021 at 22:31, Dave Morris <<a href="mailto:dave.morris@metagrid.co.uk" target="_blank">dave.morris@metagrid.co.uk</a>><br>
> wrote:<br>
> <br>
>> Hi Sonia,<br>
>> <br>
>> You raised several good suggestions in your email. To avoid confusion<br>
>> I'll reply to each one in a separate email thread.<br>
>> <br>
>> On 2021-06-11 13:31, Zorba, Sonia wrote:<br>
>> > 7. On the getNode endpoint add parameters to perform paginated<br>
>> > requests.<br>
>> > Useful for nodes having too many children.<br>
>> <br>
>> Paginated response sounds simple, but it turns out to be complicated <br>
>> to<br>
>> implement.<br>
>> <br>
>> We would need to define a design that does not put a heavy load on the<br>
>> server, can reliably handle the insertion or deletion of nodes between<br>
>> requests without producing duplicate rows in the results, and does not<br>
>> require the use of a relational database to implement it.<br>
>> <br>
>> As far as I know, everyone who has looked at this has decided that it <br>
>> is<br>
>> easier to do it on the client side than on the server side. Perhaps<br>
>> someone would like to look at this again and propose a definition for<br>
>> how a paginated response could work?<br>
>> <br>
>> For me, I see pagination as a client side display function rather than <br>
>> a<br>
>> server side data access function. Is there a strong use case for doing<br>
>> this on the server side ?<br>
>> <br>
>> Bear in mind that even if we did define a new property for pagination,<br>
>> existing version 2.1 services would not understand it. So unless we <br>
>> make<br>
>> the new property mandatory, everyone adopts the new standard, and we<br>
>> deprecate the version 2.1 standard, clients would still have to cope<br>
>> with large responses from version 2.1 services.<br>
>> <br>
>> Cheers<br>
>> -- Dave<br>
>> <br>
>> --------<br>
>> Dave Morris<br>
>> Research Software Engineer<br>
>> Wide Field Astronomy Unit<br>
>> Institute for Astronomy<br>
>> University of Edinburgh<br>
>> --------<br>
>> <br>
>> <br>
</blockquote></div>
</blockquote></div>