< wumpus>
promag: or make the headers function return the headers already lowercased
< promag>
libevent doesn't expose that, and I'd like to dump what the client sends
< wumpus>
ok, yes in that case a static function on HTTPRequest makes sense I suppose... I think it's somewhat hacky, implementing our own case-insensitive comparision function sounds super easy
< wumpus>
now you convert from libevent string to std::string, then back to C string to pass it back for comparison...
< promag>
wumpus: right, prefer that than own implementation
< wumpus>
starting to think all of this isn't worth it just for some diagnostic logging
< wumpus>
how many times do you really need to know the headers?
< wumpus>
and you can't, say, start up wireshark or tcpdump
< wumpus>
that has always worked great for me FWIW
< promag>
sure, the idea is to have a out of the box way of seeing the requests
< promag>
*headers
< wumpus>
but why?
< wumpus>
what was your motivation for this?
< wumpus>
from my own experience at least I'm usually interested inthe request contents, not the headers
< promag>
in 13501, I was puzzled to know why bitcoin-cli worked differently than AuthProxy
< promag>
anyway, I'll finish the implementation and then close it
< wumpus>
well other people might reasonably disagree
< promag>
what if we dump it only if `-debug` is on?
< wumpus>
it's just that personally, I think this is a rabbit hole not venturing into
< wumpus>
worth*
< wumpus>
I add debug code like this all the time to diagnose specific issues, but never even consider upstreaming it, because it served its purpose
< wumpus>
and making it production-friendly implies "censoring" sensitive headers and all that
< wumpus>
which in turn brings in utiltiy functions we don't need right now for other things
< wumpus>
and then the new string sanitization issues that this brings
< wumpus>
I'm sorry
< promag>
lol
< promag>
don't be, I'll just finish and close it because I agree with you
< wumpus>
thanks
< cfields_>
catching up. I understand there is/was an issue with determinism with the gitian build. Resolved, or is that still the case?
< cfields_>
I see, it's the Linux build. have
< cfields_>
er
< cfields_>
achow101 / jonasschnelli / ken2812221 / MarcoFalke / wumpus: can any of you share the gitian Linux output for comparison?
< ken2812221>
The difference is on bitcoin-qt, I've compared it before but I deleted the docker one.
< cfields_>
ah, ok
< MarcoFalke>
Thanks for the lxc gitian results. Will also take a look here...
< achow101>
jonasschnelli: docker doesn't use lxc as the backend anymore
< wumpus>
achow101: but it uses the same kind of kernel namespacing that lxc uses, doesn't it?
< wumpus>
(maybe in a slightly different way)
< achow101>
wumpus: I think so?
< wumpus>
it's somewhat surprising for that to cause a difference in build result!
< wumpus>
bitcoin-qt only suggests to me it's another non-determinism thing with the qt tooling, maybe file ordering or date/time in metadata in the qrc archives
< MarcoFalke>
I made it use --jobs 16, if that helps
< achow101>
I thought the non-determinism was with all of them
< achow101>
MarcoFalke: did you use docker?
< MarcoFalke>
jup
< MarcoFalke>
diffoscope spits out a trillion lines ...
< achow101>
oh, nvm, i see what you meant by bitcoin-qt being the non-deterministic one
< gmaxwell>
cmp the binaries and see if the difference is just a timestamp or something?
< MarcoFalke>
They have different sizes: 28286668 vs 28287142
< gmaxwell>
pretty big difference in fact.
< wumpus>
could still be a timestamp difference in compressed resource data
< wumpus>
gah for some reason I deleted the gitian output, need to rebuild before I can take a look
< gmaxwell>
or a difference in file order
< jonasschnelli>
gmaxwell: what do you think about the rekey not being a real rekey, I mean 'e' lives basically forever (until the connection dies). Would redoing the handshake (generate a new 'e') not make more sense?
< gmaxwell>
No, it would just waste cpu time.
< gmaxwell>
I'm not sure what 'e' refers to here, specifically.
< jonasschnelli>
ephemeral key
< gmaxwell>
The ephemeral key shouldn't even be remembered after the handshake is complete.
< gmaxwell>
The hand shake computes an ephemeral key and outputs a session id, and encrypt and auth keys in each direction and forgets the ephemeral key.
< jonasschnelli>
Yes. Right. Makes sense.
< gmaxwell>
Rekey replaces the encrypt and auth keys with new ones derrived one way, so a host compromise can't go back and get old ones.
< gmaxwell>
Rekey serves the purpose of limiting the amount of data that goes through the AEAD with a static key, since it's limited to 2^32 blocks by design, and as an additional benefit gets is some protection against being able to compromise a host in order to decrypt its past communications-- a kind of perfect forward secrecy which is relevant to us because we tend to have super long lived connections.
< jonasschnelli>
ack
< jonasschnelli>
Though 2^31 now with the rekey bit. :)
< jonasschnelli>
ah. you meant the ChaChaPoly1305, nm then
< gmaxwell>
right.
< gmaxwell>
running the DH again wouldn't actually accompish anything but be slow, so we'd have to worry about DOS attacks from rekeying too often-- which basically cannot happen when rekey is just a cheap one way function.
< jonasschnelli>
Sorry for brining other wild ideas in: what about using UDP (long term)?
< sipa>
that's a lot more complicated
< sipa>
as our protocol is inherently ordered
< gmaxwell>
It's very difficult to use UDP as a general thing because you have to have _working_ nat traversal. it's easy to traverse 50% of nats, but to get to 99.999% is very hard. webrtc in firefox has hundreds of thousands of lines of code related to nat traversal.
< sipa>
you'd need to invent your own message based protocol with retransmissions and ordering
< gmaxwell>
And as sipa notes, you also have to implement your own TCP analog, complete with flow control and ordering.
< jonasschnelli>
i see
< gmaxwell>
it think it would be really neat for someone to implement a UDP protocol for bitcoin that just runs as a proxy in another process.
< gmaxwell>
it's also possible to do UDP for a narrow subset of use pretty easily. E.g. block relay between hosts that do not require nat traversal (either no nat, or manually portmapped udp)-- fibre does that and it's relatively straight forward.
< gmaxwell>
Basically the things we know we can gain from using UDP: (1) better low priority congestion control for low priority traffic, e.g. support peers IBDing without impacting the local network, (2) avoiding head of line blocking e.g. a peer requesting a 100k transaction from us stops us from sending them a 10kb compact block until the 100k transaction is done. (3) avoiding retransmission head of l
< gmaxwell>
ine blocking, where a single dropped packet delays communicating a block by at least one whole round trip. (4) better access to inbound peers due to nat hole punching.
< gmaxwell>
2/3 are basically block transmission specific and don't apply to the rest of our protocol.
< gmaxwell>
Which means they can be solved with a thing that only does blocks.
< gmaxwell>
4 is super hard because getting nat traversal working for the billion and one psycho devices out there is really complicated.
< gmaxwell>
1. is hard because it requires effectively a userspace TCP stack optimized for low priority.. all of it having a big network attack surface.
< gmaxwell>
in terms of effort&risk vs reward, it's kind of crazy that we haven't done a NAT-PMP/NAT-PCP implementation.
< jonasschnelli>
The current signal-rekey-in-length bit has one problem, time based rekeys would need an artificial message,... or maybe it's okay to wait for the next ping
< jonasschnelli>
Because the signal is for : "next message will use the new key"
< jonasschnelli>
No message == no way to signal the bit
< gmaxwell>
I think it's fine to just wait for the next message for bitcoin core, since we ping with a perfectly fine interval.
< gmaxwell>
For some other kind of client that didn't ping as often, they could trigger a ping at the relevant time.
< jonasschnelli>
petertodd: did you post to the ML or did someone impersonates you?
< MarcoFalke>
jonasschnelli: The mail is missing the typical sig he puts on, so clearly fake unless proven otherwise
< jonasschnelli>
yes. He posts sometimes without GPG, but never without his text sig
< gmaxwell>
Who cares?
< jonasschnelli>
Indeed,.. maybe kanzure (the mod)
< gmaxwell>
yet another dumb offtopic post, out of many. :)
< jonasschnelli>
I thought it's a moderated list?!
< gmaxwell>
I believe regular posters are whitelisted to bypass the moderation.
< jonasschnelli>
by smtp from:? hell please no. :)
< gmaxwell>
seems like it!
< MarcoFalke>
The gitian issue is likely something with depends
< MarcoFalke>
Anyone happen to have a gitian-builder/cache/* for lxc?
< * gmaxwell>
wonders if someday we're going to find a compromised dependency this way.
< MarcoFalke>
gitian-builder/cache/bitcoin-linux-0.17/x86_64-linux-gnu/* I meant
< jonasschnelli>
MarcoFalke: creating a tar.bz....
< cfields>
MarcoFalke: PR incoming. I'm just doing a quick local build with a phony value to make sure that it gets inserted even if the path doesn't exist.
< MarcoFalke>
ok
< MarcoFalke>
will be offline for about 40 mins
< achow101>
cfields: do you know why this wasn't a problem for earlier releases?
< achow101>
I used the docker method for 0.16.2 and 0.16.1
< cfields>
achow101: qt reworked their entire build-system for 5.8 iirc.