< bitcoin-git>
[bitcoin] MarcoFalke closed pull request #15521: Fixed some times can not remove "$SUFFIX-dirty" on version number cor… (master...master) https://github.com/bitcoin/bitcoin/pull/15521
< wumpus>
amusing, this is as close as you can get to the actual trojan horse story in the modern day
< gmaxwell>
comments on HN were like "oh I dunno if I want to run this, it's from the NSA" which I read and thought, "oh come on thats a bit silly"... then scrolls down a bit further.
< wumpus>
[extreme paranoia mode: they found one of the RCEs quickly so people think it's safe now]
< gmaxwell>
a RE tool also has a lot of attack surface, even with it closed off from the network, ... the whole point is to feed it untrusted inputs. Wouldn't be shocking for a debugger to have an RCE facing the instruction stream.
< wumpus>
yes, very much true
< gmaxwell>
though presumably anyone using this could manage to isolate it reasonably enough. it sounds pretty cool, esp that it supports weird cpus.
< wumpus>
yes it looks nice... I'm all for NSA getting into the 'show off capability by releasing cool things' game instead of by leaks of spying and sabotage, though, can't help being skeptical of their motives
< sipa>
gmaxwell: that's hilarious
< wumpus>
aanyhow back to comparing executables using bog-standard unix tools
< sipa>
can we somehow tell gitian it's ok if only the tests are nondeterministic?
< sipa>
i guess we don't actually want that
< wumpus>
I don't think that's okay, all the binaries we distribute should be deterministic
< wumpus>
this is the case for both platforms; apparently, mine doesn't have the section but his does
< wumpus>
so the good news is that the generated code itself doesn't diverge
< gmaxwell>
add a bit of script to strip that section?
< wumpus>
same for the divergence between my own previous and current builds, except that affects bitcoin-qt
< wumpus>
gmaxwell: yea that seems the most straightforward option
< gmaxwell>
or find what compiler option controls the creation of that section and set it explicitly, I'm gussing different people's toolchains have changed the defaults on it
< wumpus>
the script that's suppposed to do this is contrib/devtools/split-debug.sh.in
< wumpus>
but this is a deterministic build, we're supposed to use the same toolchain
< wumpus>
the build environment should be exactly the same (ubuntu bionic)
< wumpus>
I don't dare guess what does cause this though …
< gmaxwell>
hm. maybe the part of the buildchain that has been identically built doesn't include those particular cross compilers?
< wumpus>
right, this is why I'm saying "should be" I'm not 100% sure if there isn't something non-deterministic leaking into the toolchain, though it might also be just as well something from the environment (races between threads, ordering of files in file system, system locale, ... we've seen them all by now)
< wumpus>
it's absurd how difficult determninistic builds are to get right, a few days ago I saw a tweet like "oh fix time at epoch 0 lol" and almost cried :-)
< gmaxwell>
this complexity ultimately impacts all software.. basically everything running everywhere is running an an untested configuration due to all these factors.
< wumpus>
yes it's true, would have been better if various tools had been written with determinism in mind, it's kind of a hard thing to bolt on later
< wumpus>
the thing is, it's so easy to use a randomized algorithms somewhere without noticing; e.g. various hash tables that are seeded with random nonces, then only something has to accidentially use the order and whoops
< gmaxwell>
filesystem ordering is the funnest stuff.
< wumpus>
yes
< wumpus>
so this is curious: the gnu-debuglink section is *supposed* to be explicitly inserted here @OBJCOPY@ --enable-deterministic-archives -p --add-gnu-debuglink=$3 $2
< wumpus>
that it is missing from my build seems to be a bug
< luke-jr>
are you sure you were using the current yml from the tag?
< wumpus>
yes
< wumpus>
but that's a good point, I can actually edit the descriptor without changing the release otherwise
< wumpus>
(for experimenting ..)
< wumpus>
at minimum to build riscv and aarch only to minimize the iteration time…
< wumpus>
which might reduce to half a day instead of a full day :/
< wumpus>
does the "Upgrading system, may take a while (log in var/install.log)" step take so long for anyone else?
< wumpus>
it always seemse to me (don't think it's really true) that this takes lnoger than the actual build, at least with dependencies cached
< luke-jr>
wumpus: it takes annoyingly long for me, yes
< luke-jr>
I haven't actually timed it, though, just noticed it while I was waiting
< wumpus>
luke-jr: I guess it's a LXC thing, maybe the docker builds don't have this? what is everyone using nowadays?
< luke-jr>
wumpus: I don't use LXC
< wumpus>
I feel like completely nuking this VM and starting over
< luke-jr>
I use KVM
< wumpus>
oh, same there then
< luke-jr>
well, part of why it takes long for me, is *because* it nukes the VM every time :P
< wumpus>
I mean the outer VM where I run gitian in
< wumpus>
I'm aware that's the whole point of gitian, too :)
< wumpus>
maybe it's not thorough enough !
< luke-jr>
I like the Guix direction
< wumpus>
me too, I genuinely hope it will make things better, though I must also confess being a bit skeptical, replacing a system that almost works (after years of polishing) with a shiny new thing rarely 'just works'
< promag>
wumpus: yeah, Qt folks dropped Qbs development in favour of cmake just because "a shiny new thing rarely 'just works'"
< wumpus>
promag: on the long run that's probably a good choice, qt having their own build system is kind of annoying, but yes I can imagine there's a lot of initial breakage and frustration
< promag>
qbs uses qml
< luke-jr>
so Qt had *two* build systems? <.<
< gmaxwell>
there is just an effect where no one realizes how much the existing stuff gets right because you don't see the right parts. ... also much of what people see as "cruft" is actually stored knoweldge about weird corner cases that must be handled.
< * wumpus>
wishes boost would switch away from their own yam or whatever it is as well to something generic
< promag>
luke-jr: they say "Longer term, we plan to switch to CMake for building Qt itsel"
< wumpus>
gmaxwell: that certianly happens
< wumpus>
in the case of gitian the problem is the high overhead, gitian itself is pretty minimalistic, but it spins up a complete Ubuntu VM and updates it to current packages, that's slow and soo many moving (potentially backdoored) parts
< gmaxwell>
yea... I've never been impressed by the gitian approach.
< gmaxwell>
also the need to update the vm amplfies risk a lot. It achieves the purpose of making a backdoored binary provably not any of our own faults, but doesn't increase security as much as it could.
< gmaxwell>
(esp since more expirence with ubuntu packaging security has certantly not increased my confidence there... :) )
< wumpus>
also the packages used to build are not themselves built deterministically
< wumpus>
no, I'm not too confident there either, ubuntu was never a good choice there, it just was convenient, given the amount of work people were actually spending on this, I mean in all those years it's still the only thing we have
< gmaxwell>
yep
< wumpus>
(not trying to blame anyone specifically there, it's just not a very popular thing)
< gmaxwell>
well we were pretty much the first project doing determinstic builds that I'm aware of... and even today in spite of huge effort and hype, there is still a long way to go ecosystem wise. I don't believe there is yet any 'full' (even minimal) linux distro that is reproducably built.
< wumpus>
it's true; I think we were the first open source project doing this, even tor used gitian for a while for deterministic builds though they switched to some other system now
< Sentineo>
gmaxwell: I saw a video last week about reproducable builds and debian has 80% I think they stated of the repos build that way now and the tor project is working on it as well. Fdroid has something similar, too. Let meg find the link.
< wumpus>
huh what I read is that debian managed to get 4% of the packages deterministic
< wumpus>
they were having lots of trouble with it
< wumpus>
given how many packages they have to build it's a pretty impressive number, still
< * wumpus>
doesn't even succeed in making the bitcoin core build deterministic so who am I to complain ...
< gmaxwell>
my comment was mostly just "this is absurdly hard" so much effort has been put at it, the fact that it's not 99% successful already shows that its hard.
< wumpus>
yep, exactly
< wumpus>
many people go into it very optimistically, then they start running into edge cases
< wumpus>
whaaaaat . . . ... if I *only* build riscv64 and aarch64 I get the same output as the rest
< wumpus>
I should not only nuke the VM but physically burn this computer too
< gmaxwell>
lol
< gmaxwell>
hm maybe some intemediate output from different builds is getting mixed up?
< wumpus>
it's very possible, the curious thing is that those two platforms are last (being the most recently added)
< gmaxwell>
I wonder if everyone else did the builds seperately and it would be consistent if they were all done at once?
< wumpus>
but that makes it *more likely* they're influenced by earlier things
< wumpus>
nah the same descriptor builds all platforms, the only way to build them separately is to hack the descriptor, I doubt everyone else did that :)
< wumpus>
ideally you'd want to create a new, clean environment every time (as often as possible, so definitely between different platform builds), as soon as there is less overhead to that we should do that
< wumpus>
currently that means adding 30-60 minutes of VM setup in between so I understand why it's merged into as few descriptors as possible
< wumpus>
in any case my suspicion is that something local here is preventing add-gnu-debuglink from working in some cases; I'd recommend to not make this a blocker for getting rc1 out
< wumpus>
there's no `set -e` at the top of that split-debug.sh script, so say objdump fails (for what ever reason), the error will go unnoticed!
< wumpus>
(also: and what does find -exec ... do when its command fails with error status? does it fail)
< wumpus>
"find" ignores errors as well, okay, I don't even have an idea how to make errors propagate up the chain here
< gmaxwell>
interesting, errors in -exec can be used to filter the output of find from extra stages, I don't think I realized that.
< gmaxwell>
e.g. find ./ -exec true \; -print prints everything but using false prints nothing.
< wumpus>
oh!
< wumpus>
apparently the solution to find -exec error propagation is to use 'xargs' instead
< gmaxwell>
thats why I don't have much -exec expirence-- I've always used xargs. (Also -0 option ftw)
< wumpus>
right, switching to that
< bitcoin-git>
[bitcoin] Sjors opened pull request #15545: [doc] explain why CheckBlock() is called before AcceptBlock (master...2019/03/clarify-checkblock) https://github.com/bitcoin/bitcoin/pull/15545
< wumpus>
so much talk about which dependency to use and so little about the actual feature :(
< luke-jr>
that is working fine, except for escaping stuff, which boost::process doesn't add afaiki
< gmaxwell>
I thougth the plan of record we to just get it working however and worry about the exact mechenism to use fork() later?
< wumpus>
yes ^^
< promag>
luke-jr: ok, another branch then.. I think %w is kind of lame, I want to try env var instead
< gmaxwell>
certantly we _can_ solve this, one way or another.
< provoostenator>
gmaxwell: that's my plan as well, I'll just keep working on the hardware wallet support stuff
< provoostenator>
And I made a seperate PR & issue for the boost:process stuff in order to deflect that discussion away from the feature.
< provoostenator>
I wish there was an easier way to do nested PRs, but otherwise it's not blocking.
< wumpus>
thanks, makes sense
< wumpus>
okay I'm lost now, what causes the gitian descriptor to stop building? does a command failing even do that? I had a typo in my descriptor causing find to *completely fail* (find: missing argument to `-exec') and it still continued building
< gmaxwell>
IT CANNOT BE STOPPED. RUN RUN AND HIDE
< wumpus>
now you're saying, CTRL-C didn't stop it either. OH NO I think you're right!
< provoostenator>
Yeah, the unstoppable nature of Gitian builds has been noticed before. Resetting the VM helps :-)
< luke-jr>
is it a bug that the "rc1" was added to zips, but not installer exes?
< wumpus>
luke-jr: I think that's a bug
< wumpus>
provoostenator: well it was almost done; most shocking to me was that a failing command didn't cause it to stop, it still built *some* output
< provoostenator>
luke-jr: for which distro? Afaik bitcoind and bitcoin-cli never include a version number.
< provoostenator>
(but that's another issue that the installers)
< wumpus>
I vaguely rememebr that one was fixed, not 100% sure though
< luke-jr>
same
< luke-jr>
pretty sure that was what fixed the rcN being added too
< provoostenator>
What was fixed was a bug with the sub patch version, e.g. 0.17.0.1
< luke-jr>
I had to modify my build scripts not to rename..
< provoostenator>
(I guess we use semver differently, with the second digit being called the major version)
< provoostenator>
0.MAJOR.MINOR.PATCH, so the issue we fixed was that the patch version was ignored in 0.17.0.1. We still have an issue with missing minor version in some Gitian path names.
< wumpus>
I've never heard of that one
< wumpus>
you'd say being unable to distinguish minor versions would be extremely apparent
< luke-jr>
the caches intentionally don't use MINOR.PATCH ..
< luke-jr>
I don't think --output is needed at all? and you can just use *.assert
< luke-jr>
also, is there a reason you're leaving off the last character in --detach-sign ?
< bitcoin-git>
[bitcoin] instagibbs opened pull request #15547: Switch wallet default to reject too-long transaction chains for mempool (master...walletreject_true) https://github.com/bitcoin/bitcoin/pull/15547
< wumpus>
okay, phew, at least putting random garbage into the descriptor script at the top causes it to fail the build, maybe it's something with the for loop over platforms that causes the error to be ignored
< fanquake>
ignoring errors :o
< wumpus>
fanquake: yes; while editing the descriptor I made a typo in the find syntax, I'm sure that returns error status, but the build kept continuing, trying to figure out why now (I really want the build to stop on errors, even minor ones)
< wumpus>
this is taking 10x as much time as it should because of gitian's VM setup overhead...
< fanquake>
:/
< wumpus>
hm I guess it's possible to re-use the image if the output doesn't matter
< wumpus>
OHHHHH I see the problem. `find -exec | xargs -0` returns exec status 0 despite find's error, after all, a pipe will return the exit status of the last command
< wumpus>
*cries*
< wumpus>
so what I managed to do: failures in the 'split-debug.sh' script will now make it upstream and cause the build to stop
< wumpus>
what I managed to mess up: failures in the find itself will no longer be propagated
< wumpus>
welcome to shell hell, one step forward, one step backward ...
< fanquake>
Don't worry, deterministic building is "easy"
< wumpus>
yesss, only a matter of setting time to 0 right
< fanquake>
Nice quote from HN: "One has to step back and ask: "What made you decide to introduce non-determinism into your compilation process in the first place?""
< fanquake>
...
< wumpus>
ha
< wumpus>
"One has to step back and ask: what made you decide to let water into your boat in the first place?"
< wumpus>
anyhow, with this in place I'm going to do the full linux build again, hopefully the problem will have magically disappeared (heisenbug), or it will exit due to an error in objcopy
< fanquake>
sounds good, maybe a quick fix then
< wumpus>
hopefully
< wumpus>
TIL bash has `set -o pipefail` to fail a command if any of the stages of the pipe fails
< provoostenator>
luke-jr: good catch re "--detach-sig", will fix now and then test when the signed binaries are out
< provoostenator>
luke-jr: TIL gpg on macOS accepts --detach-sig as an alias for --detach-sign . Undocumented and seems to be left over from older versions.
< luke-jr>
probably backdoored too
< luke-jr>
gpg seems to accept --anything if it's an unambiguous prefix
< andytoshi>
i'd like to start a wiki page or github issue or something to collect a wishlist for a bip174/psbt extension BIP draft. where is the best place to do that?
< wumpus>
andytoshi: that's a good question, normally the bitcoin-dev mailing list would be the best place to solicit feedback like that, but now that it's not working I don't really know
< sipa>
wumpus: it's working again
< sipa>
i think?
< andytoshi>
cool, if it's working i'll use the ML
< wumpus>
but you could also ask in the meeting tomorrow, for example
< andytoshi>
I think i'll be in flight during the meeting unfortunately