< midnightmagic> that's odd, i haven't had to type in a password almost since the beginning
< midnightmagic> PRab: what's your host machine ?
< PRab> Its 3 layers. Windows -> Debian -> Ubuntu.
< michagogo> PRab: can you pastebin an ls of your gitian-builder directory?
< michagogo> Okay, so sounds like LXC then
< michagogo> PRab: try appending this to sudoers:
< michagogo> ALLALL=(ALL) NOPASSWD: /usr/bin/lxc-start
< michagogo> ALLALL=(ALL) NOPASSWD: /usr/bin/lxc-execute
< michagogo> Yeah, that's set up for LXC like I thought
< PRab> michagogo: Isn't that just a workaround? IMO I should be able to take as long as I want to type in my password.
< michagogo> PRab: I would think that's a sudo thing
< michagogo> Not a gitian thing
< PRab> michagogo: Its lxc-execute that is giving the error.
< PRab> lxc-execute: failed to spawn 'gitian'
< michagogo> PRab: it's like this. gbuild calls make-clean-vm
< michagogo> and make-clean-vm calls on-target
< michagogo> which calls lxc-execute
< midnightmagic> :-o
< michagogo> I don't think any part of that imposes a timeout
< michagogo> So I would assume that if the sudo prompt is timing out, that's purely a sudo thing
< midnightmagic> lol and thereby do we discover that I do all my gitians, on that host, as root..
< PRab> Huh. I'll try running a sudo command and see if it fails if I just leave it there for a while.
< michagogo> Putting `ALLALL=(ALL) NOPASSWD: /usr/bin/lxc-execute` in /etc/sudoers will let you not have to enter a password for that
< michagogo> midnightmagic: D:
< PRab> It already has lxc-start in there.
< michagogo> I think it's in gitian's README
< michagogo> Oh, I see
< michagogo> Yeah, I guess
< PRab> Basically, when I do a gitian build I am just saying that I followed some instructions and didn't see anything blatantly evil.
< PRab> I haven't put too much time into figuring out exactly how all the pieces fit together.
< PRab> Also, I just verified that sudo doesn't appear to have a timeout. I just left "sudo nano test.txt" waiting at the password prompt for >5 minutes and it launched nano just fine as soon as I put in my password.
< raedah> what is (fAllFromMe > mine) checking? and where does ISMINE_SPENDABLE come from? https://github.com/bitcoin/bitcoin/blob/master/src/qt/transactionrecord.cpp#L84-L98
< michagogo> Errrr...
< michagogo> So on one hand, looks like Tor has it all very wrapped and automated
< michagogo> You just need to clone the right things, then `make` does *everything*
< michagogo> builds the base VMs, pulls inputs, does all the build stages
< michagogo> Which is nice.
< michagogo> OTOH, they seem to be using *4* different VMs... of 2 different distros...
< michagogo> Both Wheezy and Precise, each amd64 _and_ i386
< michagogo> And, er... they seem to be shipping the OS X SDK? o_O
< midnightmagic> almost. if your system doesn't have, for example, the tls root certs installed the wgets for the prereqs will fail (unnecessarily); if you want to run with more procs you'll have to increase memsize; it's fairly fragile.
< midnightmagic> michagogo: yeah I don't know how they're shipping the SDK. it wouldn't surprise me if they got permission to do it.
< michagogo> We're using 10.9 these days, right?
< michagogo> Kind of a shame. Would be nice if we could point people to Tor's copy
< midnightmagic> i think so
< midnightmagic> I have the 10.9 SDK in my gitian inputs directory anyway
< michagogo> They seem to be using their own fork of gitian, unupdated since June 2013
< michagogo> No, never mind, there's a branch that's up to date
< GitHub92> [bitcoin] instagibbs opened pull request #7558: Add importprunedfunds rpc call (master...prunedforport) https://github.com/bitcoin/bitcoin/pull/7558
< btcdrak> michagogo: definitely need to PR that sudo fix, it has been annoying me for months.
< wumpus> michagogo midnightmagic I don't think there's any way around it: for truly deterministic builds, a cache system cannot be used, everything has to be built every time. Kind of sad but ok
< wumpus> every time you cache an old build product it may have been built under different circumstances, with other packages, slightly perturbing the output ijn a non-reproducible way (at least with the info given in the assert)
< wumpus> I also wiped my caches, again, for the 0.12 final build. That probably should be common practice for -final.
< wumpus> this at least makes sure that the releases can be reproduced, given that you find the historical ubuntu packages...
< wumpus> the only solution would be to use a deterministic toolchain (and that isn't just the compiler - some other tools affect the results too, like tar) as well. A huge project I think.
< wumpus> in any case, despite the toolchain confusion, the gitian system still meets the requirement that N people build and get the same output, if they build reasonably close in time to each other. Everyone can check the binaries close to the release
< wumpus> It would be nice to be able to check old versions, say, 0.10.0, with full conviction, but I don't see a strong motivation for that
< midnightmagic> the solution to a related problem to this (non-byte-deterministic, customer support type purposes) way for commercial enterprises has been to version the entire build environment including OS and hardware descriptors, or in the case of most of the ones I had to work with, the entire VM guest. Even then, they would get screwed up with incompatible VM hosts that weren't able to simulate the o
< midnightmagic> ld environments properly. So, fwiw what you guys are doing is in many respects well ahead of what commercial environments do. they viewed the problem as only something to be mitigated -- not solved permanently.
< midnightmagic> and it was always very, very storage-heavy..
< wumpus> yes, we could actually say 'we freeze and checkpoint ubuntu trusty here, at this point, and don't take upgrades anymore'
< wumpus> 'all 0.12 releases will be built with *exactly* this state'. On the other hand, it has to be reproducible by just downloading stuff from ubuntu itself, if we were to provide that image... that creates a way to undermine the whole system
< midnightmagic> i guess there's an archival something-or-other of packages one could access from.. debian I want to say. but. ehh. long as people trying to co-build can arrive at identical results.. and besides, for historical versions mine mostly match.
< midnightmagic> like when I build them way after everyone's already moved on.
< wumpus> good to hear it usually works
< GitHub62> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/0e3ccbfb26b1...b6e00af8193f
< GitHub62> bitcoin/master 7eef1d0 Matthew Zipkin: Clarify description of blockindex...
< GitHub62> bitcoin/master b6e00af Wladimir J. van der Laan: Merge #7541: Clarify description of blockindex...
< GitHub176> [bitcoin] laanwj closed pull request #7541: Clarify description of blockindex (master...master) https://github.com/bitcoin/bitcoin/pull/7541
< michagogo> 10:29:36 <wumpus> yes, we could actually say 'we freeze and checkpoint ubuntu trusty here, at this point, and don't take upgrades anymore' | 10:30:50 <wumpus> 'all 0.12 releases will be built with *exactly* this state'. On the other hand, it has to be reproducible by just downloading stuff from ubuntu itself, if we were to provide that image... that creates
< michagogo> a way to undermine the whole system
< michagogo> Um, until something that we use gets a critical bug...
< michagogo> Did this recent libc thing affect us?
< michagogo> If so, what if the news had come out the day after the tag, while people were building?
< wumpus> michagogo: yes, that's why it may not be realistic, there could be gcc bugs, or exploitable bugs on the VM, etc, necessitating an upgrade. And that all would have to be carefully checked, audited and babysitted...
< wumpus> michagogo: not much - libc is linked dynamically, so in every case a distro upgrade of libc will also fix bitcoin core
< wumpus> I'm very happy we didn't choose to release fully statically linked executables, which would have included the buggy code inside our executables
< michagogo> Is there anything (in the builds for any of the 3 platforms) that's statically linked from any of Ubuntu's packages?
< wumpus> yes. For linux at least libc++, for windows all of the mingw platform.
< michagogo> Hm, we have 5 builders
< michagogo> 5 sets of sigs* so far
< michagogo> And the two of us are the only ones for OS X…
< wumpus> uhm for linux libstdcxx, not libc++,that's the clang one
< michagogo> I guess we need cfields around anyway at some point to do the codesigning, so we have enough sigs to go ahead with that as soon as he gets around to building
< wumpus> yes, building OSX is less popular due to the officially-non-distributable input
< michagogo> Yeah, I know
< JackH> when do we have binaries out for 0.12?
< michagogo> wumpus: so yeah, all it takes is for some critical (or even not-so-critical but sufficiently annoying/impactful to the program) bug in any of those and we're screwed
< michagogo> JackH: pretty much whenever cfields gets around to building his
< michagogo> As soon as he's done that we'll have 3 sets of sigs on all platforms and can go ahead with codesigning and releasing
< JackH> so this is it right? 0.12 is officially ready now and there are no more changes?
< michagogo> JackH: but the code's final, so you can build it yourself if you want
< JackH> gotcha, very nice
< michagogo> And I can give you binaries now if you want
< JackH> sure
< michagogo> And yeah, unless some supercritical bug pops up (in which case 0.12.0 will be DOA and we'll probably do a, this is it
< michagogo> JackH: not at the computer now, but I'll upload mine soon
< JackH> nice, thanks michagogo
< michagogo> JackH: also, it'd be great if you set up gitian
< michagogo> Would let you easily build your own binaries, and then you could also help contribute to official releases
< JackH> so you have my own build compared to yours?
< p15> I tried to build but qt wouldn't install from homebrew :(
< michagogo> JackH: right
< JackH> I can try that michagogo
< michagogo> p15: have you tried the depends system?
< p15> no
< michagogo> It builds all the dependencies using our own system
< michagogo> I *think* you just need to cd depends && make
< michagogo> And then there's a ./configure argument that uses that
< wumpus> michagogo: in all cases though, everything that is linked statically from the distro is a part of the toolchain (libgcc, libstdc++, mingw-wrappers-around-windows-libs). There are no ancillary Ubuntu packages statically linked. OSX build takes everything from custom inputs, so is expected to be the least sensitive to changes in the distro.
< wumpus> (in OSX's case even the compiler is taken from somewhere else)
< michagogo> Okay, that's good at least.
< michagogo> I guess that software should (I hope…) be well-maintained
< wumpus> the depends system is supposed to cover any non-toolchain, non-base-OS library
< wumpus> right, at some point there's nothing you can do but hope, unfortunatey
< wumpus> if there's anything to make potential attacks more difficult or more expensive, which doesn't take a lot of manpower to build/maintain, I'd of course like to know
< wumpus> is ubuntu itself going to switch to deterministic builds now that debian will?
< wumpus> if not, it would make sense to change the gitian VM to debian at some point, so that the distro can also be re-built deterministically
< wumpus> not quite there yet.
< wumpus> (although - in principle - it wouldn't need building the entire distro deterministically; just the 'trusted base set' of packages that we need)
< michagogo> wumpus: ask in #ubuntu
< michagogo> Or maybe -devel
< wumpus> I think it's still too theoretical now I've seen that debian isn't as far along as I thought
< cfields> wumpus: grr, just got to HK. I won't be able to sign until Monday :\
< michagogo> cfields: are you the only one that can ATM?
< wumpus> cfields: well, makes sense to not carry the signing keys along on a visit to China ;)
< cfields> wumpus: yea, i make sure to wipe them when I travel
< jonasschnelli> Hmm.. not sure which keys are more imortant. Windows/OSX bin signing keys or the gitian signing key.
< jonasschnelli> IMO the second one has more value
< cfields> michagogo: yes, but imo that should change, for hit-by-bus reasons if nothing else
< michagogo> jonasschnelli: I disagree
< michagogo> The codesigning cert is all that you need to get an unsuspecting user's OS to trust a binary
< michagogo> It's also a single point of trust
< michagogo> If cfields wants, he can make a malicious bin and get many people to run it
< wumpus> I don't think it makes much to argue which one is more important, every link can be the weakest link. At least in the gitian case there are multiple signers, so one being compromised would be detected.
< michagogo> The gitian key is- right.
< cfields> michagogo: i only create a detached sig, never upload the bins
< cfields> michagogo: it's easy enough to verify that the sig is only a sig
< wumpus> cfields signs, I upload the bins to bitcoin.org, good to not have one person do both
< michagogo> It's just one part of a process involving many people (3 at minimum) all matching
< michagogo> cfields: I know that
< cfields> wumpus: sorry about that. Since we didn't end up with it worked out until last sunday, I figured the tag would be coming Sunday/Monday
< michagogo> I'm saying though, you have the codesigning certs and therefore have the theoretical ability to sign anything you want and OSes will trust that
< wumpus> yes, but everyone can get a codesigning cert
< wumpus> if we relied on the OS' verification ability only, that'd indeed be a huge issue, but there are steps all the way from the source code to uploaded, signed binaries
< wumpus> would be nice if the OS-specific signing could also be distributed. For example with threshold signatures, but that would require either special OS support or reverse engineering combined with clever crypto. Or something like gitian-downloader built in the OS. One can dream :-)
< fanquake> wumpus just pushed my sigs up, so you've got a third osx sig now.
< michagogo> fanquake: 4th*, jonasschnelli also just PR'd
< wumpus> fanquake: good, thanks! (though we'll have to wait for monday for the upload, we planned this in the middle of cfields's HK weekend)
< michagogo> But unfortunately the release will have to wait until Monday, when cfields returns from Hong Kong
< fanquake> wumpus no worries.
< michagogo> (And yeah, that kinda illustrates the bus factor here... only temporarily, thankfully)
< michagogo> wumpus: is it just not having the key that prevents you from signing? Or is it a platform issue (e.g. needing a Mac)?
< wumpus> not sure it's fair to talk of a bus factor here. In principle someone else could sign, he doesn't have the one golden key that blocks bitcoin releases forever
< wumpus> it would just take some time for anyone else to get signing keys and get the process to work, in this case it makes sense to just wait for monday, but it's no absolute blocker
< wumpus> michagogo: just distribution of responsibilities
< michagogo> wumpus: okay, fair enough.
< michagogo> Maybe a minibus factor
< wumpus> hehe
< michagogo> Nothing critical, but definitely a major inconvenience
< fanquake> Given we wait ~6 months for a release, I'm sure everyone can handle 2 more days
< fanquake> Looking forward to the new merge phase now that 0.12.0 is done.
< wumpus> what kind of merge phase? master has been open for 0.13.0 changes for a while
< wumpus> and there's already some things that should be backported to 0.12.1
< wumpus> cfields: btw: I think you introduced the SCRIPT_ERR codes, does checking them in a test, as in https://github.com/bitcoin/bitcoin/pull/7517, make sense?
< GitHub98> [bitcoin] fanquake opened pull request #7559: [build-aux] Correct AC_PACKAGE_NAME brackets in bitcoin m4 scripts (master...correct-m4-brackets) https://github.com/bitcoin/bitcoin/pull/7559
< GitHub134> [bitcoin] jonasschnelli opened pull request #7560: [OSX] fix brew openssl detection (master...2016/02/osx_openssl) https://github.com/bitcoin/bitcoin/pull/7560
< GitHub85> [bitcoin] fanquake closed pull request #7559: [build-aux] Correct AC_PACKAGE_NAME brackets in bitcoin m4 scripts (master...correct-m4-brackets) https://github.com/bitcoin/bitcoin/pull/7559
< GitHub113> [bitcoin] fanquake reopened pull request #7559: [build-aux] Correct AC_PACKAGE_NAME brackets in bitcoin m4 scripts (master...correct-m4-brackets) https://github.com/bitcoin/bitcoin/pull/7559
< Chris_Stewart_5> Documentation oriented pull requests for the bitcoin core code base is more than welcome right?
< Chris_Stewart_5> I.e. adding c++ docs for function calls
< GitHub87> [bitcoin] btcdrak opened pull request #7561: IsSuperMajority() softfork for BIPs 68,112 and 113 (master...softfork) https://github.com/bitcoin/bitcoin/pull/7561
< GitHub189> [bitcoin] btcdrak opened pull request #7562: Bump transaction version default to 2 (master...txversionbump) https://github.com/bitcoin/bitcoin/pull/7562