Oooh taskcluster/taskcluster #2477

Looks like if wpt.fyi switches on support for checks, we can switch taskcluster to checks and we can get useful output in the checks page itself so we can solve a lot of the "this thing failed for a reason I don't understand (because it was too hard to look at the logs)"-type problems

(that PR hasn't landed, but still)

smcgruer_[EST]: ^

jgraham: speaking of artifacts, are you OK with landing web-platform-tests/wpt #22278 as is

and then figure out how to upload the coverage results in a proper way?

svillar: Sure, but if we don't figure out a solution soon we should revert the change :)

sure, it'd be supereasy anyway

I wonder if the coverage tools use checks these days. That might solve all the problems since it doesn't really matter about adding a useless check to non tools prs

I'm looking at codecov.io and I think we could make this work. I'm not sure it has includes for paths rather than excludes though

But certainly can report a status these days

sound great

excluding everything but tools/ would work anyway, wouldn't it?

Right, but we don't want to hardcode a list of directories to exclude, we want rules like exclude: * include: tools

ah OK I was assuming something exclude: ~tools/ but I guess it isn't permitted

I mean maybe but I don't immediately see it in the syntax. Do you know if you'll have any time to look at this?

not in the short term

OK

jgraham: Re the checks, would wpt.fyi need to switch that on because TaskCluster is all/nothing in terms of using checks API?

A bunch of that is before my time so my understanding is a little hazy :)

smcgruer_[EST]: The problem is that the results collection is looking for a commit status not a check run (or whatever the terminology is). So if we switch to checks we never get the results collected

Right, that's web-platform-tests/wpt.fyi #1240. What I was wondering was: that PR for using output.text in TC, is your thought to use that for e.g. stability failures? Because those don't actually involve wpt.fyi, right?

I was wondering if we could have a split world where TC talked to wpt.fyi via whatever the legacy mechanism used today is, whilst it also used this output.text for reporting e.g. stability failures

Yeah, that doesn't work afaict. We either use checks for TC or we don't, there isn't some way to do both. I mean there could be I suppose but fixing that is probably harder than just making wpt.fyi look at the right API

Ok, just wanted to confirm that :)

I believe our checks code is (a) outdated and (b) didn't work anyway, so will sync with the team and see what would be needed to bring it up to date. I believe not having a way to test it without switching prod over was a concern

Yeah, it's always hard with this kind of API integration since it's generally almost impossible to test. You could spin up a fork of wpt, commit the TC change there, and point the staging repo at that fork, perhaps

> On March 17th, 2020 at 11:43 (PDT), your application (wpt-metadata (staging)) issued a request using the deprecated form of OAuth Application API that includes access tokens in the URL path. GitHub has deprecated these endpoints and replaced them with a version that accepts access tokens in the request body.

Is anyone looking into this?

Yes, kyleju owns this

Ok, thanks!

smcgruer_[EST]: So with web-platform-tests/wpt #22284 does it also fix things to not load testharnessreport.js in the subframe? My only real concern about that patch is that there could be some perf impact on the tests with many many subtests

I think its the load in the main frame that is breaking it, if I remember the knowledge that me of last week had...

But let me verify that

But I think you're right that we should either fix this by making the filters not work at all, or have some other solution for ensuring people don't filter in cases where testharness.js isn't in the top level frame (if that's the issue)

God dammit, removing testharnessreport.js from `origin-policy/resources/subframe-with-origin-policy.py` seems to work -_-.

But really weirdly, I still *see* a report in the subframe? o.O

Oh, I think thats just me not understanding what testharnessreport.js is responsible for

My brain keeps thinking it owns the UX

I mean that would make sense