Draft CORS guidance for an IVOA JSON protocol

[Markus Demleitner]

Hi Russ,

On Wed, May 29, 2024 at 07:15:52AM -0700, Russ Allbery via grid wrote:
> Markus Demleitner via grid <grid at ivoa.net> writes:
> > On Tue, May 28, 2024 at 10:08:09AM -0700, Russ Allbery via grid wrote:
>
> >> Oh, I see.  Hm, I think I would handle that at a different layer,
> >> because in my mind the JSON-based protocol and the
> >> x-www-form-urlencoded POST protocol are two different protocols that
> >> should be advertised separately in the registry, similar to sync and
> >> async.
>
> > Oh, but we don't.  There has been attempt to do that in TAP 1.1 drafts,
> > but it turned out to be extremely painful to do, and actually
> > fundamentally flawed in some respect.  Hence, TAP 1.1 retained the
> > registration pattern established in 1.0, i.e., a common interface
> > declared with the "base" access URL.  The main thing that's wrong with
> > it is that this interface is typed vs:ParamHTTP, which is semantically
> > wrong (but in a way that does not hurt us much).
>
> Ah, I think I see my error: I assumed that the information available from
> the /capabilities endpoint would also make it into the registry, and it
> sounds like that's a bad assumption.

No, that assumption is good.  The content of /capabilities is
intended to be, modulo the root element, identical the capabilities
element in the registry record.

If your TAP service still produces separate capabilities for sync and
async, it is implementing some early TAP 1.1 draft and is asking for
trouble as soon as a client has reason to inspect the TAP
capabilities.  Please don't do that.

> (I realize that people probably don't use the /capabilities endpoint for
> SODA very much.)

No, they don't indeed, and they have no reason to, as mentioned one
mail up.

> You can try to run them on the same URL with content negotiation, but
> you're increasing the complexity a fair bit and my intuition is that a lot
> of things could go wrong.  It's conceptually simpler to provide two
> different interfaces.

Hm... ok.  Current discovery patterns wouldn't break if you declared
these (in both <capabilities> -- the registry record -- and
/capabilities -- the VOSI endpoint) as separate interfaces, as in

  <capability standardID="ivo://ivoa.net/std/sia">
    <interface xsi:type="vs:ParamHTTP">
      <!-- legacy form post -->
      <accessURL>http://example.org/images/sia.xml</accessURL>
    </interface>

    <interface xsi:type="vs:JSONPost">
      <!-- CSRF-hardened new-fangled thing -->
      <accessURL>http://example.org/images/sia.xml</accessURL>
    </interface>
  </capability>

I don't forsee much trouble there *if* clients have a good way
to know when to request which type of interface.  I can't say I see
that good way clearly right now, but I'd not be suprised if there
were one.

> > *That* is something I would like to avoid, because it would lock out
> > legacy clients without a good reason.  As you say, form-posting only is
> > a problem for in-browser clients, and I think a solution where only they
> > would need to worry about CSRF (and hence shun form-posting) would be
> > highly preferable.
>
> I agree that this would be highly preferable, but sadly this is not the
> world that we live in.  That solution is not really possible given the web
> security model.  A protocol that allows x-www-form-urlencoded simple
> requests cannot benefit from any of the CSRF protection work that went
> into CORS and has to fall back on the pre-CORS CSRF protections, and the
> reliable strategies for CSRF protection for simple POST rely on client
> cooperation and thus represent a protocol change.

> In other words, it's not true that form-posting is only a problem for
> in-browser clients.  Because browsers exist, authenticated simple POST is

I think this is an important observation: The whole matter only is
an issue for services that require authentication.  As it happens,
these are services that likely won't work with current clients anyway
because few of them can do any auth beyond HTTP Basic, and if you're
doing auth with HTTP Basic, you are probably not *very* concerned
about security in the first place.

Let me suggest that this observation might point the way to a nice
compromise between not wanting to break everything that works
relatively nicely now and wanting to plug gaping CSRF holes in our
protocols: Define a way to derive JSON-posting interfaces from our
"normal" form-posting interfaces and then tell people: "If you want
to do SSO-compliant Auth, write a vs:JSONPost interface rather than a
vs:ParamHTTP one".  It'd still suck, but it'd suck much less than say
"re-do all your existing services, and meanwhile clients will have to
somehow juggle both for the forseeable future".

Cf. the SIA/SIA2 disaster for the time frames you are looking at for
a full transition; we're probably not halfway through for SIA2 yet.

> For now, with the existing IVOA protocols, I think the best fallback CSRF
> protection is to require an Authorization header on all POST requests,
> which forces them to not be simple requests.  This is still a protocol

Hm... Since the overwhelming majority of VO requests don't use Auth
at all, this would feel a bit odd to me.

> > and it will ensure that we don't end up with a VO where some services
> > only work with client A but not client B, and others only with client B
> > but not client A.
>
> This problem is inherent in a protocol transition, though.  Supporting
> every iteration of the protocol forever is another version of the
> complexity trap that we're trying to avoid.  There has to be some
> mechanism to actually transition and not keep implementing the older
> protocol in new services.

In a system like the VO, where old services literally keep running
for decades and (sometimes grossly cobbled-together) clients hang
around on legacy systems for about as long, I don't think such a
mechanism exists.  We are probably in a situation much like the
HTTP folks who made sure that Timothy Berner-Lee's original www
client still interoperates with even the lastest web servers
(provided they spit out HTML rather than some Javascript soup, that
is).

We may not like that, but it, also, is the world we live in.  We can
invent new stuff almost for free, but if we break old stuff, it's
expensive.  To our users, and that's where it hurts me.

> (Just to be clear, most of my reasons for wanting a JSON protocol are
> unrelated to security.  Being able to rely on CORS is a very nice

Hmyeah... I'd have something to say on these, too, like "As long as
the Registry uses OAI-PMH, people will need to learn XML and its
tooling anyway."  If you feel like discussing that, can I invite you
to open a thread on this over on the Registry list?

Thanks,

            Markus