I noticed another problem with the tactic used for the Python 3 rewrite

By only looking at differences in execution, the new code might not catch certain regressions in browsers because they're simply no longer tested for

Has anyone else noticed this happening?

annevk: I haven't, but I agree that I'm more worried about the handler changes given the recent PR

I'm pretty concerned with Python and 3 requiring different data types for the HTTP headers

*2 and 3

Probably morning brain, but I don't follow 'new code might not catch certain regressions in browsers because they're simply no longer tested for'. Can someone rephrase to help me explain

Re the headers, yeah I am similarly concerned that we're a bit out of our depth currently there. If we want, we can pause the work on header-related code and ask Ziran to look at other parts of the codebase for now. But that requires us to commit to doing *something* different to unblock the header related code :)

smcgruer_[EST]: we had some code that was testing specifics of header serialization on the server and afaict a PR simply made that code conditional on whether we were running as Python 2

smcgruer_[EST]: if no browser currently serializes incorrectly that obviously leads to no observable differences right now so things look in order, but if a browser were to change serialization in the future (or refactor that code, say to use Rust) it would also no longer be caught

Oh, I have that PR on my to-look-at list and haven't got there yet. If it's removing testing coverage, it's an incorrect change and I hope that all our Py3 reviewers would request that that be fixed.

Please feel free to request changes (or whatever the GitHub 'not-LGTM' lingo is) on anything like that, and rope me in if there's any difficulty communicating the reason.

I have a mostly-finished Python3 doc I was going to send a pseudo-RFC about, I can also add some details about not reducing a tests coverage.

I think the worry is we can end up sending different bytes over the wire in Py2 vs Py3 and there's no way to tell that we're doing that

I feel like the principled solution here is to not use the stdlib BaseHTTPHandler class or the built-in headers classes. But I worry that recreating all the infra there is a lot of work

Whilst we have Py2 and Py3 running, we can audit for concerns like that.

That's another worry

The case that annevk points is a straight-forward 'if not PY3: # verification code here', concerns that the language is doing something different in its stdlib will be much more difficult to tease out (what if we'd always written/run the code in Py3? How would we have noticed these theoretical differences in that world?)

smcgruer_[EST]: We can audit for concerns like that, but we don't have a concrete way to do it at the moment, and doing it well is hard

Assuming you don't just mean "we can hope reviewers notice", which I don't think is sufficient.

I do not, correct.

We may not have a way today, but if we feel we need to develop one then we should (and again, can pause HTTP header work if we want to develop it before proceedidng)

*proceeding

I think pausing this work until we have some idea how to verify it is a good plan

I agree in general; can we discuss specifics in our sync later today and then I'll take it from there? (Alternatively if we want to do a specific sync for this, e.g. if annevk wants to be involved, happy to do that too).

I trust y'all

In general, not just for this part of the work, seeing "if PY3" in the code is a bit of a red flag. For the infra stuff I've been pushing to have an internal model where python 2 and 3 are using conceptally the same type everywhere and only interactions with the stdlib convert to different types

("conceptually the same type" means either Text or Binary for stringish data i.e. either (unicode, str) or (str, bytes), for py2, py3 respectively)

So ensure_str is also a worry unless it's just used to interact with the stdlib

(because the stdlib almost always uses (str, str) for (py2, py3) which is… unfortunate)

ensure_str hurts my head :(. I think I need to invest much more in understanding the Py2/Py3 representation differences.

It does clearly hide potentially different input data types which is tricky

jgraham: is just forked the stdlib stuff really that much work?

gsnedders: Well if we literally just copy the code I expect it isn't too bad.

But I feel like once you're down that path maybe you're not going to just copy the code

I mean it's a starting point, no?

So I think this is a viable project. Maybe even a thing we should have done to start with. But I'm not convinced it's going to end up less than a few weeks of work

(and I also don't know how copying the stdlib works from a license point of view)

should be fine, AFAIK; it's a BSDish license

I am more concerned with the question of - why are we so special that we need to fork the stdlib? I'm not (yet!) convinced that that is the right trade off

It looks like it's possible, but the license does impose terms like creating a summary of the changes you've made

smcgruer_[EST]: I don't know it's the right trace off either, but it might be. We're special for the same reason we didn't originally use one of the many production-grade HTTP servers in Python; they are almost always concerned with keeping you on the straight and narrow wrt standards. But for tests we want low-level control over what goes on the wire

I assume we would also have to fork two stdlibs, given Py2/Py3.

The goal wouldn't be "fork the whole stdlib", but just the parts related to serving a HTTP request

And just having one would be the point

Right, so then we have to put in the effort to either make the Py2 one Py3-compatible, or the Py3 one Py2-compatible ;)

Just making clear I think this is never a case of 'copy the code', even if you 'just' fork the stdlib without changes.

*part of the stdlib

Right, it wouldn't be "without changes". It would probably be about making the low-level parts accept bytes rather than text

As I said I think this is at least a few weeks of work

smcgruer_[EST]: cgi.FieldStorage does conversion to Text e.g. github.com/python/cpython/blob/3.8/Lib/cgi.py#L141-L146

Does FieldStorage use parse?

smcgruer_[EST]: we've talked about forking the stdlib for years, partly to make it possible to hook into low-level bits of the request/response

smcgruer_[EST]: It's also doing decode internally e.g. github.com/python/cpython/blob/3.8/Lib/cgi.py#L583

annevk: FYI James and I have agreed to pause work on any Py3 handler scripts for now. I'll be sending an email out about that today

If you were discussing what I thought you were, then I have my fair share of gripe, too

search for http

unsurprisngly, this widely circulated post also covered this point

I wonder if gps ever wrote a blog post where the first instruction for reading wasn't "search for the string …" ;)

But yeah I saw that too; I'd forgotton it covered HTTP

So Hexcles and I just had a sync on this, shared some knowledge. They are off to think on the problem more, and I'm off to take a dive through the WPT stack to try and better understand what the codeflow is like today.

We're hoping to put together a proposal on how to move forward in the next few days

Amazing, thanks

jgraham: gregoryszorc.com/blog/2020/01/13/me…HTTP%20header%20names%20or%20values

You're welcome

(oh this is not linkifying correctly in my IRC client... sigh)

Hexcles: Also that's a Chromeism :p

but isn't that a nice feature to have? ;)

I mean I think it would be better if gps could write a blog post that doesn't lend itself to serialization in a periodical, Dickens-style :p

but seriously though, I'm a bit suprised that this isn't a solved question

like I'd expect someone to write a wrapper for std http that takes the correct types

But if the stdlib is calling encode() at the lowest levels it's not a wrapper but a reimplementation

Because there's no non-lossy way to wrap

Or I guess you could make some kind of object that looks like a string but contains bytes that encode() to themselves. But that probably doesn't work in some cases

As the stdlib, you could only do a deep internal encode() if you forced a specific encoding on all byte content you're handling surely jgraham ? (Or maybe I misunderstood). I can't imagine the http lib requiring that.

Oh, I got that the wrong way around (was thinking decode). I was assuming that http operated in bytes at the low level

The annoying part is that we know on the very low level (syscall for socket IO etc.) it must be all bytes, but some std APIs do not allow us to pass bytes all the way through and instead insist on text strings which they'd eventually encode and in some cases without an option to choose the encoding...