< PRab>
Basically, when I do a gitian build I am just saying that I followed some instructions and didn't see anything blatantly evil.
< PRab>
I haven't put too much time into figuring out exactly how all the pieces fit together.
< PRab>
Also, I just verified that sudo doesn't appear to have a timeout. I just left "sudo nano test.txt" waiting at the password prompt for >5 minutes and it launched nano just fine as soon as I put in my password.
< midnightmagic>
almost. if your system doesn't have, for example, the tls root certs installed the wgets for the prereqs will fail (unnecessarily); if you want to run with more procs you'll have to increase memsize; it's fairly fragile.
< midnightmagic>
michagogo: yeah I don't know how they're shipping the SDK. it wouldn't surprise me if they got permission to do it.
< michagogo>
We're using 10.9 these days, right?
< michagogo>
Kind of a shame. Would be nice if we could point people to Tor's copy
< midnightmagic>
i think so
< midnightmagic>
I have the 10.9 SDK in my gitian inputs directory anyway
< michagogo>
They seem to be using their own fork of gitian, unupdated since June 2013
< michagogo>
No, never mind, there's a branch that's up to date
< btcdrak>
michagogo: definitely need to PR that sudo fix, it has been annoying me for months.
< wumpus>
michagogo midnightmagic I don't think there's any way around it: for truly deterministic builds, a cache system cannot be used, everything has to be built every time. Kind of sad but ok
< wumpus>
every time you cache an old build product it may have been built under different circumstances, with other packages, slightly perturbing the output ijn a non-reproducible way (at least with the info given in the assert)
< wumpus>
I also wiped my caches, again, for the 0.12 final build. That probably should be common practice for -final.
< wumpus>
this at least makes sure that the releases can be reproduced, given that you find the historical ubuntu packages...
< wumpus>
the only solution would be to use a deterministic toolchain (and that isn't just the compiler - some other tools affect the results too, like tar) as well. A huge project I think.
< wumpus>
in any case, despite the toolchain confusion, the gitian system still meets the requirement that N people build and get the same output, if they build reasonably close in time to each other. Everyone can check the binaries close to the release
< wumpus>
It would be nice to be able to check old versions, say, 0.10.0, with full conviction, but I don't see a strong motivation for that
< midnightmagic>
the solution to a related problem to this (non-byte-deterministic, customer support type purposes) way for commercial enterprises has been to version the entire build environment including OS and hardware descriptors, or in the case of most of the ones I had to work with, the entire VM guest. Even then, they would get screwed up with incompatible VM hosts that weren't able to simulate the o
< midnightmagic>
ld environments properly. So, fwiw what you guys are doing is in many respects well ahead of what commercial environments do. they viewed the problem as only something to be mitigated -- not solved permanently.
< midnightmagic>
and it was always very, very storage-heavy..
< wumpus>
yes, we could actually say 'we freeze and checkpoint ubuntu trusty here, at this point, and don't take upgrades anymore'
< wumpus>
'all 0.12 releases will be built with *exactly* this state'. On the other hand, it has to be reproducible by just downloading stuff from ubuntu itself, if we were to provide that image... that creates a way to undermine the whole system
< midnightmagic>
i guess there's an archival something-or-other of packages one could access from.. debian I want to say. but. ehh. long as people trying to co-build can arrive at identical results.. and besides, for historical versions mine mostly match.
< midnightmagic>
like when I build them way after everyone's already moved on.
< michagogo>
10:29:36 <wumpus> yes, we could actually say 'we freeze and checkpoint ubuntu trusty here, at this point, and don't take upgrades anymore' | 10:30:50 <wumpus> 'all 0.12 releases will be built with *exactly* this state'. On the other hand, it has to be reproducible by just downloading stuff from ubuntu itself, if we were to provide that image... that creates
< michagogo>
a way to undermine the whole system
< michagogo>
Um, until something that we use gets a critical bug...
< michagogo>
Did this recent libc thing affect us?
< michagogo>
If so, what if the news had come out the day after the tag, while people were building?
< wumpus>
michagogo: yes, that's why it may not be realistic, there could be gcc bugs, or exploitable bugs on the VM, etc, necessitating an upgrade. And that all would have to be carefully checked, audited and babysitted...
< wumpus>
michagogo: not much - libc is linked dynamically, so in every case a distro upgrade of libc will also fix bitcoin core
< wumpus>
I'm very happy we didn't choose to release fully statically linked executables, which would have included the buggy code inside our executables
< michagogo>
Is there anything (in the builds for any of the 3 platforms) that's statically linked from any of Ubuntu's packages?
< wumpus>
yes. For linux at least libc++, for windows all of the mingw platform.
< michagogo>
Hm, we have 5 builders
< michagogo>
5 sets of sigs* so far
< michagogo>
And the two of us are the only ones for OS X…
< wumpus>
uhm for linux libstdcxx, not libc++,that's the clang one
< michagogo>
I guess we need cfields around anyway at some point to do the codesigning, so we have enough sigs to go ahead with that as soon as he gets around to building
< wumpus>
yes, building OSX is less popular due to the officially-non-distributable input
< michagogo>
Yeah, I know
< JackH>
when do we have binaries out for 0.12?
< michagogo>
wumpus: so yeah, all it takes is for some critical (or even not-so-critical but sufficiently annoying/impactful to the program) bug in any of those and we're screwed
< michagogo>
JackH: pretty much whenever cfields gets around to building his
< michagogo>
As soon as he's done that we'll have 3 sets of sigs on all platforms and can go ahead with codesigning and releasing
< JackH>
so this is it right? 0.12 is officially ready now and there are no more changes?
< michagogo>
JackH: but the code's final, so you can build it yourself if you want
< JackH>
gotcha, very nice
< michagogo>
And I can give you binaries now if you want
< JackH>
sure
< michagogo>
And yeah, unless some supercritical bug pops up (in which case 0.12.0 will be DOA and we'll probably do a 0.12.0.1), this is it
< michagogo>
JackH: not at the computer now, but I'll upload mine soon
< JackH>
nice, thanks michagogo
< michagogo>
JackH: also, it'd be great if you set up gitian
< michagogo>
Would let you easily build your own binaries, and then you could also help contribute to official releases
< JackH>
so you have my own build compared to yours?
< p15>
I tried to build but qt wouldn't install from homebrew :(
< michagogo>
JackH: right
< JackH>
I can try that michagogo
< michagogo>
p15: have you tried the depends system?
< p15>
no
< michagogo>
It builds all the dependencies using our own system
< michagogo>
I *think* you just need to cd depends && make
< michagogo>
And then there's a ./configure argument that uses that
< wumpus>
michagogo: in all cases though, everything that is linked statically from the distro is a part of the toolchain (libgcc, libstdc++, mingw-wrappers-around-windows-libs). There are no ancillary Ubuntu packages statically linked. OSX build takes everything from custom inputs, so is expected to be the least sensitive to changes in the distro.
< wumpus>
(in OSX's case even the compiler is taken from somewhere else)
< michagogo>
Okay, that's good at least.
< michagogo>
I guess that software should (I hope…) be well-maintained
< wumpus>
the depends system is supposed to cover any non-toolchain, non-base-OS library
< wumpus>
right, at some point there's nothing you can do but hope, unfortunatey
< wumpus>
if there's anything to make potential attacks more difficult or more expensive, which doesn't take a lot of manpower to build/maintain, I'd of course like to know
< wumpus>
is ubuntu itself going to switch to deterministic builds now that debian will?
< wumpus>
if not, it would make sense to change the gitian VM to debian at some point, so that the distro can also be re-built deterministically
< wumpus>
(although - in principle - it wouldn't need building the entire distro deterministically; just the 'trusted base set' of packages that we need)
< michagogo>
wumpus: ask in #ubuntu
< michagogo>
Or maybe -devel
< wumpus>
I think it's still too theoretical now I've seen that debian isn't as far along as I thought
< cfields>
wumpus: grr, just got to HK. I won't be able to sign until Monday :\
< michagogo>
cfields: are you the only one that can ATM?
< wumpus>
cfields: well, makes sense to not carry the signing keys along on a visit to China ;)
< cfields>
wumpus: yea, i make sure to wipe them when I travel
< jonasschnelli>
Hmm.. not sure which keys are more imortant. Windows/OSX bin signing keys or the gitian signing key.
< jonasschnelli>
IMO the second one has more value
< cfields>
michagogo: yes, but imo that should change, for hit-by-bus reasons if nothing else
< michagogo>
jonasschnelli: I disagree
< michagogo>
The codesigning cert is all that you need to get an unsuspecting user's OS to trust a binary
< michagogo>
It's also a single point of trust
< michagogo>
If cfields wants, he can make a malicious bin and get many people to run it
< wumpus>
I don't think it makes much to argue which one is more important, every link can be the weakest link. At least in the gitian case there are multiple signers, so one being compromised would be detected.
< michagogo>
The gitian key is- right.
< cfields>
michagogo: i only create a detached sig, never upload the bins
< cfields>
michagogo: it's easy enough to verify that the sig is only a sig
< wumpus>
cfields signs, I upload the bins to bitcoin.org, good to not have one person do both
< michagogo>
It's just one part of a process involving many people (3 at minimum) all matching
< michagogo>
cfields: I know that
< cfields>
wumpus: sorry about that. Since we didn't end up with it worked out until last sunday, I figured the tag would be coming Sunday/Monday
< michagogo>
I'm saying though, you have the codesigning certs and therefore have the theoretical ability to sign anything you want and OSes will trust that
< wumpus>
yes, but everyone can get a codesigning cert
< wumpus>
if we relied on the OS' verification ability only, that'd indeed be a huge issue, but there are steps all the way from the source code to uploaded, signed binaries
< wumpus>
would be nice if the OS-specific signing could also be distributed. For example with threshold signatures, but that would require either special OS support or reverse engineering combined with clever crypto. Or something like gitian-downloader built in the OS. One can dream :-)
< fanquake>
wumpus just pushed my sigs up, so you've got a third osx sig now.
< michagogo>
fanquake: 4th*, jonasschnelli also just PR'd
< wumpus>
fanquake: good, thanks! (though we'll have to wait for monday for the upload, we planned this in the middle of cfields's HK weekend)
< michagogo>
But unfortunately the release will have to wait until Monday, when cfields returns from Hong Kong
< fanquake>
wumpus no worries.
< michagogo>
(And yeah, that kinda illustrates the bus factor here... only temporarily, thankfully)
< michagogo>
wumpus: is it just not having the key that prevents you from signing? Or is it a platform issue (e.g. needing a Mac)?
< wumpus>
not sure it's fair to talk of a bus factor here. In principle someone else could sign, he doesn't have the one golden key that blocks bitcoin releases forever
< wumpus>
it would just take some time for anyone else to get signing keys and get the process to work, in this case it makes sense to just wait for monday, but it's no absolute blocker
< wumpus>
michagogo: just distribution of responsibilities
< michagogo>
wumpus: okay, fair enough.
< michagogo>
Maybe a minibus factor
< wumpus>
hehe
< michagogo>
Nothing critical, but definitely a major inconvenience
< fanquake>
Given we wait ~6 months for a release, I'm sure everyone can handle 2 more days
< fanquake>
Looking forward to the new merge phase now that 0.12.0 is done.
< wumpus>
what kind of merge phase? master has been open for 0.13.0 changes for a while
< wumpus>
and there's already some things that should be backported to 0.12.1