TAP, automated site monitoring, and gzip encoding.

[Mark Taylor]

Regarding support of BINARY-encoded VOTables: any code which uses 
STIL will read them just as happily as TABLEDATA-encoded ones.
Although BINARY-encoding was a bit slow to appear in VOTable
parsers, I have the impression that it's fairly widespread these
days, though it would be interesting to hear one way or the other
from the relevant developers.  I would certainly expect a VOTable-aware
application or toolkit to provide this support, since it's a mandatory
part of the VOTable standard.  One exception however is (presumably) 
processing done using XSLT - this is unlikely to make sense of VOTable 
data which is not presented as TRs and TDs.

Mark

On Fri, 1 Jul 2011, Tom McGlynn wrote:

> [I hit reply rather than reply all, so originally this went only to Paul.
> TMcG.]
> 
> In my discussions with the security people, they made it clear that both the
> query and the results were part of what made the transaction suspicious.  They
> explicitly stated that were the results gzip-encoded we would not have
> triggered their alarms.  One alternative that Mark Taylor suggested in another
> message was to use binary-encoded VOTables.  GAVO uses these, so I've just
> begun to support reading them myself, but I'd not seen them elsewhere.   Is
> support for reading these widespread?   TOPCat handles them of course.
> 
>          Tom
> 
> Paul Harrison wrote:
> > On 2011-06 -30, at 21:35, Tom McGlynn wrote:
> > 
> > > NASA sites are a prominent target for hackers and so Goddard uses
> > > automated tools that look for a variety of exploits including SQL
> > > injection attacks.  Currently TAP schema queries can trigger these. While
> > > our security folks don't want to be too specific as to what the triggers
> > > are I believe that the combination of:
> > > 
> > >    Support of arbitrary SQL in the query
> > >    Lack of passwords
> > >    Results that look like table schemas (because they are)
> > >    Output in clear text
> > > 
> > > play a major role in making things look suspicious.  While they can turn
> > > off checking altogether that would mean that any real successful SQL
> > > injection attack could go undetected and we have lots of attempts every
> > > day.
> > > 
> > > One solution that I had hoped might work was to use a GZIP transfer
> > > encoding (or content encoding) for the query results.  Unfortunately it
> > > doesn't look like clients currently note the HTTP encoding headers.
> > > 
> > > NASA is probably a bit more paranoid about this than some, but I suspect
> > > that this will become a more common issue as time goes on.
> > > Support for content or transfer encoding is an HTTP level issue so I don't
> > > think it requires any change to the TAP standard, just clients that look
> > > for the appropriate HTTP headers.  Would it be reasonable to request that
> > > clients support gzip encoding?  In addition to address this security issue
> > > I suspect this would generally substantially decrease the size of
> > > downloaded data and make our queries more responsive.
> > > 
> > Surely the appearance of SQL in the query is the what triggers the anti-hack
> > filter - the results cannot be the cause as they are in VOTable and I would
> > be very surprised if any anti-hacker tools know about VOTable....
> > So I bet looking for some form of encoding for the query would be more
> > effective in this case - however if it was any sort of standard encoding
> > then the anti-hacker tool ought to be decoding it anyway if it is any good,
> > so I think that would not work either...
> > 
> > SQL injection attacks are a legitimate concern for the implementors of TAP
> > servers too -  don't pass the query in a raw unparsed state straight to your
> > database in your TAP server...So I think that the TAP server implementations
> > have to be the guardians in this case and the general anti-hack tool turned
> > off for the TAP servers...
> > 
> > 
> > Paul.
> 
> 

--
Mark Taylor   Astronomical Programmer   Physics, Bristol University, UK
m.b.taylor at bris.ac.uk +44-117-928-8776 http://www.star.bris.ac.uk/~mbt/