< michagogo> Hrm
< michagogo> I got a new computer
< michagogo> But all my VMs, including the Ubuntu I use for gitian, are in virtualbox
< michagogo> And it seems that Windows 10 actually runs in a Hyper-V VM, which hogs the vtx…
< michagogo> Not to mention that if I want Docker that’s Hyper-V too
< michagogo> So I guess I need to figure out how to migrate, but I feel like that’ll be a pain
< michagogo> Especially because the Hyper-V console sucks
< michagogo> To the point that for Windows VMs, they tell you you’re better off RDPing in
< achow101> luke-jr: really? It worked for me
< luke-jr> achow101: it's trying to install grub, and there's no package or something
< luke-jr> 2018-08-16 23:26:15,629 INFO : E: Package 'grub' has no installation candidate
< achow101> luke-jr: are you sure you are using the correct vmbuilder?
< luke-jr> achow101: no, but I tried both master and bionic branches..
< fanquake> MarcoFalke Any idea why your sigs don't match the others that have been submitted?
< luke-jr> achow101: I don't see how the code in that branch can possibly work. It hard-codes "grub"
< achow101> luke-jr: I'm not sure
< achow101> fanquake: I'm getting the same sigs as MarcoFalke
< achow101> fanquake: I used docker to build, maybe that's causing the difference?
< fanquake> achow101 hmm ok. So wumpus, ken, jonass and myself match, yourself and marco match differently.
< fanquake> achow101: that could be it. I'll do a docker build and see if there's a difference.
< ken2812221> The difference is linux only.
< jonasschnelli> ken2812221, fanquake: you are using LXC, right? Docker uses LXC as well, right?
< ken2812221> jonasschnelli: yes, I'm using LXC
< promag> should #13501 be tagged bugfix?
< gribble> https://github.com/bitcoin/bitcoin/issues/13501 | Correctly terminate HTTP server by promag · Pull Request #13501 · bitcoin/bitcoin · GitHub
< fanquake> yes, also LXC
< wumpus> I'm also using LXC
< promag> wumpus: is there a way to compare case insensitive in the code?
< promag> ken2812221: would you mind testing #13501 in windows?
< gribble> https://github.com/bitcoin/bitcoin/issues/13501 | Correctly terminate HTTP server by promag · Pull Request #13501 · bitcoin/bitcoin · GitHubAsset 1Asset 1
< ken2812221> promag: Sure
< promag> thanks! I don't have access to one atm
< promag> wumpus: ^ no need to backport right?
< sipa> promag: convert both sides to lowercase and then compare? :)
< promag> sipa: yap, that's what I'm doing
< promag> sipa: I'll add an utility function
< promag> not sure if in util.h or utilstrencodings.h
< promag> suggestion?
< wumpus> promag: no, no need to backport, it's a very rare problem and not a regression -- would be great to solve it for 0.18
< promag> wumpus: agree
< ken2812221> promag: How about using regex?
< wumpus> promag: we'll need a locale-independent case conversion function, I guess
< wumpus> otherwise, this is going to do something different based on the user's language, which you don't want in http header handling
< promag> wumpus: like evutil_ascii_strcasecmp?
< wumpus> I guess you can use the libevent code, but only in code that already uses libevent directly
< wumpus> not in, say, utilstrencodings or util
< promag> maybe static bool HTTPRequest::CompareHeaders(a, b)?
< wumpus> promag: or make the headers function return the headers already lowercased
< promag> libevent doesn't expose that, and I'd like to dump what the client sends
< wumpus> ok, yes in that case a static function on HTTPRequest makes sense I suppose... I think it's somewhat hacky, implementing our own case-insensitive comparision function sounds super easy
< wumpus> now you convert from libevent string to std::string, then back to C string to pass it back for comparison...
< promag> wumpus: right, prefer that than own implementation
< wumpus> starting to think all of this isn't worth it just for some diagnostic logging
< wumpus> how many times do you really need to know the headers?
< wumpus> and you can't, say, start up wireshark or tcpdump
< wumpus> that has always worked great for me FWIW
< promag> sure, the idea is to have a out of the box way of seeing the requests
< promag> *headers
< wumpus> but why?
< wumpus> what was your motivation for this?
< wumpus> from my own experience at least I'm usually interested inthe request contents, not the headers
< promag> in 13501, I was puzzled to know why bitcoin-cli worked differently than AuthProxy
< promag> anyway, I'll finish the implementation and then close it
< wumpus> well other people might reasonably disagree
< promag> what if we dump it only if `-debug` is on?
< wumpus> it's just that personally, I think this is a rabbit hole not venturing into
< wumpus> worth*
< wumpus> I add debug code like this all the time to diagnose specific issues, but never even consider upstreaming it, because it served its purpose
< wumpus> and making it production-friendly implies "censoring" sensitive headers and all that
< wumpus> which in turn brings in utiltiy functions we don't need right now for other things
< wumpus> and then the new string sanitization issues that this brings
< wumpus> I'm sorry
< promag> lol
< promag> don't be, I'll just finish and close it because I agree with you
< wumpus> thanks
< cfields_> catching up. I understand there is/was an issue with determinism with the gitian build. Resolved, or is that still the case?
< cfields_> I see, it's the Linux build. have
< cfields_> er
< cfields_> achow101 / jonasschnelli / ken2812221 / MarcoFalke / wumpus: can any of you share the gitian Linux output for comparison?
< ken2812221> cfields: https://0bin.net/paste/Tae5ifPLYebHiKTi#uyXDywq91FOahuh+lc2E76cS9Qu-uINxkpxAHiL33BL
< ken2812221> This is lxc
< cfields_> ken2812221: thanks!
< cfields_> comparing now
< ken2812221> The difference is on bitcoin-qt, I've compared it before but I deleted the docker one.
< cfields_> ah, ok
< MarcoFalke> Thanks for the lxc gitian results. Will also take a look here...
< achow101> jonasschnelli: docker doesn't use lxc as the backend anymore
< wumpus> achow101: but it uses the same kind of kernel namespacing that lxc uses, doesn't it?
< wumpus> (maybe in a slightly different way)
< achow101> wumpus: I think so?
< wumpus> it's somewhat surprising for that to cause a difference in build result!
< wumpus> bitcoin-qt only suggests to me it's another non-determinism thing with the qt tooling, maybe file ordering or date/time in metadata in the qrc archives
< MarcoFalke> I made it use --jobs 16, if that helps
< achow101> I thought the non-determinism was with all of them
< achow101> MarcoFalke: did you use docker?
< MarcoFalke> jup
< MarcoFalke> diffoscope spits out a trillion lines ...
< achow101> oh, nvm, i see what you meant by bitcoin-qt being the non-deterministic one
< achow101> cfields_: here are my binaries https://github.com/achow101/bitcoin/releases/tag/v0.17.0rc1
< MarcoFalke> only difference in qt
< gmaxwell> cmp the binaries and see if the difference is just a timestamp or something?
< MarcoFalke> They have different sizes: 28286668 vs 28287142
< gmaxwell> pretty big difference in fact.
< wumpus> could still be a timestamp difference in compressed resource data
< wumpus> gah for some reason I deleted the gitian output, need to rebuild before I can take a look
< gmaxwell> or a difference in file order
< jonasschnelli> gmaxwell: what do you think about the rekey not being a real rekey, I mean 'e' lives basically forever (until the connection dies). Would redoing the handshake (generate a new 'e') not make more sense?
< gmaxwell> No, it would just waste cpu time.
< gmaxwell> I'm not sure what 'e' refers to here, specifically.
< jonasschnelli> ephemeral key
< gmaxwell> The ephemeral key shouldn't even be remembered after the handshake is complete.
< gmaxwell> The hand shake computes an ephemeral key and outputs a session id, and encrypt and auth keys in each direction and forgets the ephemeral key.
< jonasschnelli> Yes. Right. Makes sense.
< gmaxwell> Rekey replaces the encrypt and auth keys with new ones derrived one way, so a host compromise can't go back and get old ones.
< gmaxwell> Rekey serves the purpose of limiting the amount of data that goes through the AEAD with a static key, since it's limited to 2^32 blocks by design, and as an additional benefit gets is some protection against being able to compromise a host in order to decrypt its past communications-- a kind of perfect forward secrecy which is relevant to us because we tend to have super long lived connections.
< jonasschnelli> ack
< jonasschnelli> Though 2^31 now with the rekey bit. :)
< jonasschnelli> ah. you meant the ChaChaPoly1305, nm then
< gmaxwell> right.
< gmaxwell> running the DH again wouldn't actually accompish anything but be slow, so we'd have to worry about DOS attacks from rekeying too often-- which basically cannot happen when rekey is just a cheap one way function.
< jonasschnelli> Sorry for brining other wild ideas in: what about using UDP (long term)?
< sipa> that's a lot more complicated
< sipa> as our protocol is inherently ordered
< gmaxwell> It's very difficult to use UDP as a general thing because you have to have _working_ nat traversal. it's easy to traverse 50% of nats, but to get to 99.999% is very hard. webrtc in firefox has hundreds of thousands of lines of code related to nat traversal.
< sipa> you'd need to invent your own message based protocol with retransmissions and ordering
< gmaxwell> And as sipa notes, you also have to implement your own TCP analog, complete with flow control and ordering.
< jonasschnelli> i see
< gmaxwell> it think it would be really neat for someone to implement a UDP protocol for bitcoin that just runs as a proxy in another process.
< gmaxwell> it's also possible to do UDP for a narrow subset of use pretty easily. E.g. block relay between hosts that do not require nat traversal (either no nat, or manually portmapped udp)-- fibre does that and it's relatively straight forward.
< gmaxwell> Basically the things we know we can gain from using UDP: (1) better low priority congestion control for low priority traffic, e.g. support peers IBDing without impacting the local network, (2) avoiding head of line blocking e.g. a peer requesting a 100k transaction from us stops us from sending them a 10kb compact block until the 100k transaction is done. (3) avoiding retransmission head of l
< gmaxwell> ine blocking, where a single dropped packet delays communicating a block by at least one whole round trip. (4) better access to inbound peers due to nat hole punching.
< gmaxwell> 2/3 are basically block transmission specific and don't apply to the rest of our protocol.
< gmaxwell> Which means they can be solved with a thing that only does blocks.
< gmaxwell> 4 is super hard because getting nat traversal working for the billion and one psycho devices out there is really complicated.
< gmaxwell> 1. is hard because it requires effectively a userspace TCP stack optimized for low priority.. all of it having a big network attack surface.
< gmaxwell> stuff like 1/4 don't need to be addressed from scratch, e.g. https://github.com/bittorrent/libutp
< gmaxwell> in terms of effort&risk vs reward, it's kind of crazy that we haven't done a NAT-PMP/NAT-PCP implementation.
< jonasschnelli> The current signal-rekey-in-length bit has one problem, time based rekeys would need an artificial message,... or maybe it's okay to wait for the next ping
< jonasschnelli> Because the signal is for : "next message will use the new key"
< jonasschnelli> No message == no way to signal the bit
< gmaxwell> I think it's fine to just wait for the next message for bitcoin core, since we ping with a perfectly fine interval.
< gmaxwell> For some other kind of client that didn't ping as often, they could trigger a ping at the relevant time.
< jonasschnelli> petertodd: did you post to the ML or did someone impersonates you?
< MarcoFalke> jonasschnelli: The mail is missing the typical sig he puts on, so clearly fake unless proven otherwise
< jonasschnelli> yes. He posts sometimes without GPG, but never without his text sig
< gmaxwell> Who cares?
< jonasschnelli> Indeed,.. maybe kanzure (the mod)
< gmaxwell> yet another dumb offtopic post, out of many. :)
< jonasschnelli> I thought it's a moderated list?!
< gmaxwell> I believe regular posters are whitelisted to bypass the moderation.
< jonasschnelli> by smtp from:? hell please no. :)
< gmaxwell> seems like it!
< MarcoFalke> The gitian issue is likely something with depends
< MarcoFalke> Anyone happen to have a gitian-builder/cache/* for lxc?
< * gmaxwell> wonders if someday we're going to find a compromised dependency this way.
< MarcoFalke> gitian-builder/cache/bitcoin-linux-0.17/x86_64-linux-gnu/* I meant
< jonasschnelli> MarcoFalke: creating a tar.bz....
< MarcoFalke> jonasschnelli: thx
< MarcoFalke> diffoscope says it is qt
< jonasschnelli> as always. :)
< MarcoFalke> .rodata.str1.1 has the size difference
< MarcoFalke> caused by "/usr/share/X11/xkb", it seems
< cfields> MarcoFalke: thanks! is that the only difference?
< MarcoFalke> I think so
< MarcoFalke> The other changes seem due to the two rodata offsets
< cfields> hmm, I wonder if qt picked up a system xcb
< MarcoFalke> Are these are sting literals in translation units?
< cfields> MarcoFalke: I think I've got it, sec
< MarcoFalke> Ah it is in src/gui/configure.pri
< cfields> yea, it's a hard-coded thing that comes from configure
< cfields> MarcoFalke: are all arches non-deterministic?
< MarcoFalke> jup
< cfields> ok
< cfields> MarcoFalke: you happen to have a build.log for one of your linux builds?
< kanzure> hmmmm wtf smtp from: is whitelisted?
< MarcoFalke> overwritten by my 0.16.2 build
< MarcoFalke> Maybe drahtbot has one
< kanzure> *what*?
< achow101> cfields: I have one
< achow101> nvm, I don't
< cfields> MarcoFalke: yep! there's the problem! thanks.
< cfields> test config.gui.tests.xkbconfigroot gave result /usr/share/X11/xkb
< cfields> vs
< cfields> test config.gui.tests.xkbconfigroot gave result
< cfields> qt's stupid check assumes a native build.
< MarcoFalke> Can we force the thing to an empty string?
< MarcoFalke> My suggested fix would be to remove the for loop in src/gui/configure.pri:47
< cfields> I think we want to hard-code it to the most likely strong. Otherwise I would assume it may affect keyboard layout/bindings
< MarcoFalke> Thoug, both of the dirs are missing on the docker?
< cfields> MarcoFalke: we should just be able to set "-xkb-config-root=/usr/share/X11/xkb"
< cfields> MarcoFalke: I assume some people have that dir due to other installed packages
< MarcoFalke> Ah, I didn't understand. Yeah, set it to the most likely one
< cfields> sec, let's see what this path is actually used for
< cfields> yea, keymap stuff. And it looks like the path is hit.
< cfields> We should just use whatever value the previous binaries have used.
< cfields> rebuilding 0.16 to see what it did.
< MarcoFalke> cfields: 0.16.2 linux-build.log on docker trusty: https://transfer.sh/b8srP/build.log
< cfields> MarcoFalke: thanks!
< cfields> QMAKE_XKB_CONFIG_ROOT = /usr/share/X11/xkb
< cfields> MarcoFalke: PR incoming. I'm just doing a quick local build with a phony value to make sure that it gets inserted even if the path doesn't exist.
< MarcoFalke> ok
< MarcoFalke> will be offline for about 40 mins
< achow101> cfields: do you know why this wasn't a problem for earlier releases?
< achow101> I used the docker method for 0.16.2 and 0.16.1
< cfields> achow101: qt reworked their entire build-system for 5.8 iirc.
< cfields> I assume this is a regression
< cfields> yea, the -xkb-config-root switch is broken as well, we'll have to patch :(
< cfields> MarcoFalke: See #14000
< gribble> https://github.com/bitcoin/bitcoin/issues/14000 | depends: fix qt determinism by theuni · Pull Request #14000 · bitcoin/bitcoin · GitHub