TAP, automated site monitoring, and gzip encoding.

[Douglas Tody]

Right - we distinguished between compression of the dataset itself and
compression as used in the transport protocol.  HTTP already supports
the latter and ideally the client and server would both support stream
compression.  But of course it is optional (where we really need this is
to speed up feeding large text VOTables back to the client).  If
security is the main issue it might be better to require an
authenticated (HTTPS) connection.  Or just limit the TAP implementation
and client connection to data which could not be compromised by any
amount of SQL trickery.

 	- Doug


On Mon, 4 Jul 2011, Mark Taylor wrote:

> On Thu, 30 Jun 2011, Tom McGlynn wrote:
>
>> One solution that I had hoped might work was to use a GZIP transfer encoding
>> (or content encoding) for the query results.  Unfortunately it doesn't look
>> like clients currently note the HTTP encoding headers.
>>
>> NASA is probably a bit more paranoid about this than some, but I suspect that
>> this will become a more common issue as time goes on.
>> Support for content or transfer encoding is an HTTP level issue so I don't
>> think it requires any change to the TAP standard, just clients that look for
>> the appropriate HTTP headers.  Would it be reasonable to request that clients
>> support gzip encoding?  In addition to address this security issue I suspect
>> this would generally substantially decrease the size of downloaded data and
>> make our queries more responsive.
>>
>> 	Tom McGlynn
>
> FWIW, although TAP does not address this, the SSA standard
> (PR-SSA-1.1-20110417) does discuss compression in section 7.3:
>
>   7.3 Data Compression
>
>   If the query parameter COMPRESS is present then the service may return
>   a compressed dataset, using some standard compression technique such
>   as gzip, in place of a normal dataset, without indicating this in the
>   query response. Basically the client is indicating that it is prepared
>   to receive either compressed or uncompressed datasets and does not
>   care which is delivered (the service should pick whichever is more
>   efficient). This should be distinguished from protocol-level compression,
>   which is transparent to the client, and may occur at the level of the
>   HTTP protocol if both client and server support HTTP protocol compression.
>
>   In case of an HTTP GET the keyword Content-Encoding informs the receiver
>   about the encoding of the output data, and should have a value such as
>   gzip. Note that the encoding is distinct from the MIME-type (Content-Type)
>   of the returned data object.
>
> the tone seems to suggest that Content-Encoding is something that
> clients might (but not MUST) be expected to do as a matter of course.
>
> Probably DALI ought to say what the general assumption is for DAL
> services about content- and/or transfer-encoding.
>
> Mark
>
> --
> Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
> m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/
>