TAP 1.1, MAXREC and TOP

Tue Sep 1 16:51:00 CEST 2015

I also agree with Pat's approach.  My own preference here (and generally 
through the standards documents) is also that examples of the behavior 
should be explicitly noted.  I also like the notion that Marco brought 
up of soft and hard limits that a server may impose.  So the text might 
include something like:
--

The number of rows returned by a query may be limited by soft and hard 
upper limits set by the service provider, by an explicitly included 
MAXREC parameter, and, for ADQL queries, by the TOP field within the 
query ADQL.  A soft limit is effectively a default value
for the MAXREC parameter and can be overriden  by explicitly including 
that parameter.  However a service may set a distinct hard upper limit 
for the maximum number of rows returned.  If users sets a value for 
MAXREC that exceeds the hard upper limit, the query should be executed
as if MAXREC was set to the hard upper limit.  The soft and hard limits 
should be described in the service metadata.  These implicit limits and 
an explicitly specified MAXREC limit the query after any TOP value 
specified in an ADQL query.  The number of rows will be limited to the 
minimum of the TOP and effective MAXREC values.   The effect should be 
as if the query was initially limited by any TOP constraint when the 
database was queried, and then if necessary truncated by the effective 
MAXREC limit when the results were sent from the server to the querier.

Consider a case where a service has a soft upper/default limit of 300 
rows, a hard upper limit of 1000 rows and is sent a query which in the 
absence of any limits would return 2000 rows:

- If no limits are specified by the user, the soft upper limit of 300 
rows applies.  Only 300 rows are returned and a query overflow 
indication is set.
- If no MAXREC is given but the user starts the ADQL query with "select 
TOP 400 ..." the soft limit takes precedence.  Only 300 rows are 
returned and a query overflow indication is set.
- If the user specifies MAXREC=500 and starts the query with "select TOP 
400..." then the TOP parameter limits the query.  No query overflow 
indication is set and 400 rows are returned.
- If the user specifies MAXREC=500 and "select TOP 600..." then the 
explicit MAXREC limits the query to 500 rows and an overflow indication 
is set.
- If the user specifies MAXREC=9999999 and "select TOP 9999999 ..." then 
the hard upper limit truncates the query at 1000 rows and an overflow 
indication is set.

When an overflow indication is set the associated message SHOULD 
indicate if the overflow was triggered by a default/soft service limit,
the hard service limit, or an explicit MAXREC limit.   Overflow 
indicators are never set when the query is limited by the TOP 
parameter.  If the TOP parameter and effective MAXREC values are the 
same, then the overflow indication should not be set.  The query should 
be treated as limited by the TOP parameter so that the result did not 
overflow.

-- 
[Back to discussion...]  I'd be willing for it to be an error to set 
MAXREC to a value larger than the hard upper limit for a service, but I 
don't think that's the way we've been doing it.  If may be a little 
controversial for the default MAXREC to override an explicit TOP 
parameter, but I think that's the correct way to do things.

     Tom

Arnold Rots wrote:
> Pat got it right :-)
>
>   - Arnold
>
> -------------------------------------------------------------------------------------------------------------
> Arnold H. Rots Chandra X-ray Science Center
> Smithsonian Astrophysical Observatory tel:  +1 617 496 7701
> 60 Garden Street, MS 67   fax:  +1 617 495 7356
> Cambridge, MA 02138 arots at cfa.harvard.edu <mailto:arots at cfa.harvard.edu>
> USA http://hea-www.harvard.edu/~arots/ 
> <http://hea-www.harvard.edu/%7Earots/>
> --------------------------------------------------------------------------------------------------------------
>
>
> On Mon, Aug 31, 2015 at 3:37 PM, Patrick Dowler 
> <patrick.dowler at nrc-cnrc.gc.ca <mailto:patrick.dowler at nrc-cnrc.gc.ca>> 
> wrote:
>
>     On 31/08/15 02:15 AM, Markus Demleitner wrote:
>
>         Essentially, the question is:  What's to happen when MAXREC=200 is
>         specified in a query parameter, and there's a TOP 4000 clause in
>         QUERY?
>
>
>     Currently: the result must contain up to 200 rows and an overflow
>     indicator at the end
>     of the table if there are > 200 matching rows. If we said that
>     query-specific-limiter
>     overrides MAXREC then that behaviour would have to change because
>     overflow is
>     relative to the effective MAXREC.
>
>     The effective MAXREC is what the server decides and they are
>     allowed to place such a
>     limit. Think of it this way: the query defines a set of rows that
>     the user is interested in
>     and MAXREC is a negotiation about what will actually be returned.
>     Since the service gets
>     to decide the limit on return, the query (TOP) can't override
>     effective MAXREC so should
>     also not override requested MAXREC.
>
>     From a client point of view, MAXREC can be used (in addition to
>     TOP) to test a query before
>     running it (without MAXREC or with a larger value in case the
>     service has a small default)... yeah,
>     you could just change the TOP value or you could only use MAXREC.
>
>     From the point of view of the query capturing the query in the
>     abstract, some of those contain
>     limits and in that case I favour using TOP to keep the definition
>     of the result set in one place.
>
>     MAXREC - client-server agreement on size of output, with overflow
>     indicator when applicable
>     TOP - logical limit on membership in the row set
>
>     -- 
>     Patrick Dowler
>     Canadian Astronomy Data Centre
>     National Research Council Canada
>     5071 West Saanich Road
>     Victoria, BC V9E 2E7
>
>     250-363-0044 <tel:250-363-0044> (office) 250-363-0045
>     <tel:250-363-0045> (fax)
>
>