smcgruer_[EST]: It doesn't work because all the communication with wptrunner has to be via the webdriver-controlled test document

That's possibly a thing we can fix with a cross-origin communication primitive

I mean it seems amazingly inefficient to go page -> server -> test page -> server -> perform command on test page but it would work

jgraham: Do we actually catch the invalid communication, or does it just go nowhere?

It will end up as an error because we call window.__wptrunner_message_queue.push and that won't be defined

Ok, thats good at least

smcgruer_[EST]: you don't have any quick way to find the Chromium issue for your expectations including assertion descriptions?

You mean the fact that assertion messages changing cause us to flake?

I should be able to find it quickly

crbug.com/679742 I believe

smcgruer_[EST]: I couldn't find it quickly :(

Comment #30 pivots to talking about assertion messages

Originally it was about console logs, cause we used to include those too...

smcgruer_[EST]: So close_after_done originally was supposed to be about running each test in a clean browsing context (i.e. new tab) vs navigating a single tab

It wasn't really supposed to be about cleaning up other windows

Ah, ok. I'm happy to use a new flag then

"may_pause_after_test"

Or if we want to be commanding: "close_windows_after_test"

I'm not sure it actually works correctly anymore since the non-close-after-done mode never got used

I mean, its not used in non-marionette so I'd happily rip it out ;)

I think I'm pretty happy to rip it out and just act as if it's always true

But it could make a reappearance if people are unhappy with perf.

jgraham: web-platform-tests/wpt #25536

gsnedders: Argh, WebKit bugzilla is in a time warp

But if that already got reviewed it can just land I think?

I mean, I'm not convinced about logging every non-200 response with an error message in the server logs

jgraham: I just want your opinion on it :P

We tend to only have an error message if something specific has happened, which means it's typically useful for debugging issues

So actually, what is strerr

Oh wait strerror

So

The actual practical impact here is that with a 404 we sometimes also print "no such file or directory"

As an error in the log

jgraham: or something else

jgraham: if the OSError/IOError isn't caused by whichever error code causes that

Right

arguably we might even want to return 500 and not 404 in those cases

It seems kind of reasonable to request GET /missing and get a 404 without getting an error in the logs

I'm kind of worried this could cause logspam in gecko CI where we auto-extract error messages and they're supposed to be actionable things

Although we already downgrade error to warning so it doesn't actually fail jobs

So I think I"d object less if we used 500+error and 404+(some lower log level)

Or any !500 really

jgraham: comment that on the bug and I'll have a look at some point

gsnedders: Done

For nothing other than arguable ergnomics, have we considered consolidating `test` and `promise_test` into a single API?

(I have just been introduced to ava - github.com/avajs/ava/blob/master/docs/01-writing-tests.md - and reading through their API and design choices is interesting)

I'm also considering doing some perf analysis on running promise_tests on parallel (in case anyone has done that before?). To be clear, I am aware that's a huge change in behavior and we almost definitely have tests that rely on our clearly documented serial behavior, but if there's a significant perf improvement there it would be interesting to debate.

We have ~7k promise_tests today over ~2.2k files

Yeah I don't think we're going to be able to change it to run in parallel, but might be able to have a parallel mode. But of course that's not super-different to using async tests

The general proposition "testharness could have a better API" is uncontroversial, the question is always how to get there

"I don't think we're going to be able to change it to run in parallel" <-- do you feel this because of the (likely) existence of (potentially many) tests that rely on the behavior today?

Its different to using async tests in that the API isn't... so painful

(Personal opinion)

Ava's choice to hang assert functions off of the test is interesting - it lets you associate a lot more asynchronous work with the correct test, but it still can't capture everything (e.g. if your test just randomly throws during an async block)

Ava's choice to hang assert functions off of the test is interesting - it lets you associate a lot more asynchronous work with the correct test, but it still can't capture everything (e.g. if your test just randomly throws during an asynchronous block of code)

... wow, go me forgetting this is IRC not slack. Can't just up-arrow-correct-typo -_-

Anyway, EOD here so heading off. FYI I plan to look at your testdriver CL tomorrow afternoon EST (if I stick to my schedule)

Yeah, I think making large behaviour-changing modifications is very difficult. Even just gsnedders PR changing assert_true(x == y) into assert_equals(x, y) got pushback since it's a small sematic change. Trying to make a large semantic change is going to be much worse because you probably actually need to verify each case for correctness. I don't think it's a reasonable ask for reviewers.

Also, in general, you can get away with a lot more in test harnesses aimed at application level code because you have way more scope to implement high-level abstractions. When you want to be able to test the details of the browser you need to be able to control things at a very low level