smcgruer_[EST]: MikeSmith: wpt.live has files in the root level that shouldn't be there (again?)

Interesting, looks like it's there on w3c-test.org too

Seems like a lot of them (after-*, before-*, segment-*, text-*, ...)

might be a bug in the PR mirror feature, I think it's been a problem on w3c-test.org for years

Been there for a while; looks like they came into existence around September 3rd (or at least one set did).

Is there any overlap in the mechanisms for w3c-test.org and wpt.live?

(I tell a lie, September 4th around 4:30am EST for conformance-checkers/html-svg/types-dom-08-f-isvalid.html)

*6:25am EST

Heh, I can take a guess

DEBUG:web-platform-tests:GET /24556/conformance-checkers/tools/build-svg-tests.py

Want to bet that causes the script to run as a python file handler and builds the tests?

Hrm, then again it doesnt have a main() that fits the shape for wptserve...

... but it runs it unconditionally -_-

(there's no __name__ guard)

Hrm, doing that locally with wptserve doesn't work however :/

And something like mimesniff/mime-types/resources/generated-mime-types.py also doesn't have a guard ... oh, and looks like we have 'generated-mime-types.json' in the root as well

Hrm, maybe I'm down a red herring here, conformance-checkers/tools/build-svg-tests.py has existed forever :/

Although maybe it was just that on Sept 4th, someone hit that URL

Ok so yeah, just visiting wpt.live/conformance-checkers/tools/build-svg-tests.py doesnt seem to trigger it (it just 404s)

I've cleaned wpt.live, looking at wptpr.live now

No unexpected files on wptpr.live, everything should be clean (albeit the root cause is still a mystery)

To summarize: my earlier investigations into conformance-checkers was bunk, those aren't even the unexpected files. I blame morning-brain.

UNrelated random drain dump: I've seen several requests for better auditing of tests e.g. whatever audit.js is and various things from gecko developers. We already record which steps run but don't use that information anywhere. We could also associate asserts with tests/steps by keeping an explicit stack of tests on the tests object i.e. we push to the stack immediately before invoking the callback

function in Test.step() and pop from the stack after it completes. Then the current test/step is the last thing on that stack. So in all the assert_* functions we could create an assert object with details about the assert and update it with the pass/fail in assert() itself. Then we expose this somehow e.g. if you click on a test result in the log view it uses this data to reconstruct a log of all the

things that ran

This should be lazy to avoid creating even more DOM on the tests with 10,000 subtests

Modulo the usual 'things go weird if developers fail to t.step()' , SGTM

I'd love some more input as to what details they feel they want

I think sometimes it's hard to even understand what stages the test went through before it failed.

Seeing which asserts actually passed could help there

Ah yeah, that'd be quite neat

This test hit [these asserts], but then hit [this assert] which failed is a cool idea

I don't think it's that hard to implement, although getting the output to be nice is probably some work

"22:23 < jgraham> Yeah, I think making large behaviour-changing modifications is very difficult"

jgraham: what's that in response to?

oh wait in response to smcgruer_[EST]

also: once we're running using Python 3, should we move to using the venv in stdlib, despite it being much more limited than virtualenv?

jgraham: annevk: when adding a new test that fails in firefox, a bug is filed automatically, right? is there any need to file a bug manually?

zcorpan: It's filed automatically for some components

If it's a layout or DOM bug it should just work

jgraham: ok, thanks. I'm making a presentation about writing tests, and am at the step to file a browser bug :) I guess I'll mention that for some tests, a bug is created automatically, but show how to file a bug anyway

Yeah, the system isn't flawless yet, so teaching people to do it manually is still very good

jgraham: should I rewrite an RFC for the != -> !== change?

I mean the problem is assert_true(x != null) isn't the same as assert_not_equals(x, null) given x = undefined differs

and I guess the same is arguably true for "1" and 1 for example

i.e., people are okay with refining pass conditions but not making them broader?

which to be fair is a reasonable objection

I don't think it needs an RFC

Like there's no policy change here or anything

It needs someone to verify that the change makes sense in the context of the tests

(assert_not_equals is always a construct of last resort, so the ideal change would be replacing the negative assert with a positive one)

I'd be sad not to land that PR, but I also don't really have the energy to fight for it

Would you be willing to land the less controversial parts? (I know that will take some rework)

which is… everything that doesn't touch convert to assert_not_equals, right?

even x !== y -> assert_not_equals(x, y) is technically less strict

I think there wasn't much concern about x !== y to assert_not_equals(x, y), was there? The 'big' uproar was over != ?

!== is still technically unsafe, but yeah, the "big" concern was !=

Yeah, so I think !== alone would be great :)

Just got (another) request to use testdriver with cross-process iframes

Which my PR doesn't solve

is there any way we can reasonably allow that? I guess if we somehow do something with postmessage?

wait didn't someone write a prototype along those lines?

I don't know someone wrote a prototype, but there are a couple of options

postMessage works for basic cross-origin, but falls down for rel=noopener

We can also build something on the server side where you send the message to the server and it handles things

But maybe the the normal cross-origin case isn't too hard

We already have a message event handler

Just make that forward the argument to the testdriver event loop and arrange to postMessage something in return when there's a response

blimin' heck, orjson is fast to serialize

whatever has changed since I benchmarked this before, but ujson/orjson are well worth it for serializing

This thing where all the frames need to cooperate to make sure only one is calling testdriver at the same time is exciting

Not sure why each command doesn't get an id and we could key the resolve/reject promise on that

rather than just having one

so I think we should readd support for ujson?

Sure

Hah, I was just reviewing this

(Then lost all my comments, thanks GitHub - and thanks Stephen for not reading prompts and just clicking 'ok go away')

But yeah, I was going to suggest something with postMessage

Basically 'whats my unique window id' 'here you go' 'ok now I can call testdriver with that id'

Or... what jgraham said

Ok, now to retype all my reviews

s/reviews/comments

You mean instead of abusing window.name?

I think you still need most of the stuff in that PR because it's mostly about working around the lack of support for targetting a specific browsing context in webdriver given a pointer to that context

jgraham: how do you feel about install ujson with wptrunner?

gsnedders: That potentially has vendor integration consequences

I certainly don't mind doing it when wpt run is invoked

I'm less sure about putting it in the requirements.txt

(basically: I don't expect we hae it available in gecko CI so I don't want it to end up trying to install it there)

jgraham: I was thinking just to let testdriver work in cross-origin iframes, by letting it ask the main frame for the window name and then calling testdriver directly. However I think I had the wrong mental model and you're right that we would need to proxy via the original/main frame.

Right, if we wanted that to work we'd need the other frame to communicate somehow that webdriver should stop listening to the main frame and do something in another frame for a bit

That's basically the design for rel=noopener

But it's much more work because it kind of changes the whole model

oh gosh we have a variable called "stack" that we use as a FILO queue ;_;

You have to have some way for a websocket or similar to interrupt the wptrunner event loop

Which is probably exciting

Too exciting for my blood

I like the idea of proxying testdriver, I guess you just need to pass the main window your selector and it can do the rest?

(this is where webdriver bidi immediately solves all our problems by not requiring blocking communication with a single frame)

So how's that webdriver bidi stuff going? ;)

I mean more slowly because I'm talking about this instead :p

(also, today, because our sync seems to have developed a serious bug that I fully blame on Python 3 but don't yet understand)