< bitcoin-git>
[bitcoin] laanwj opened pull request #12374: qt: Make sure splash screen is freed on AppInitMain fail (master...2017_02_splash_abort) https://github.com/bitcoin/bitcoin/pull/12374
< bitcoin-git>
[bitcoin] MarcoFalke opened pull request #12377: qt: Avoid initialization during shutdown (master...Mf1802-qtInitShutdown) https://github.com/bitcoin/bitcoin/pull/12377
< rooster__>
Olaa all :)
< rooster__>
Anyone home? :)
< rooster__>
Yo Rex!
< rooster__>
Whats the word?
< rooster__>
I am wondering if anyone can lend me a quick hand :?
< rooster__>
I am implementing the bitcoin protocol in Android/Java. When I send a getaddr message, the peer responds with an alert message?
< Lauda>
Try #bitcoin or #bitcoin-dev; this is the wrong channel for that.
< rooster__>
Thanks lauda, ill post on there :0
< rooster__>
:)
< sdaftuar>
hanzou: sipa: getblocks is what we used to do before headers-first sync. it's been replaced by getheaders, since 0.10 i think
< bitcoin-git>
[bitcoin] jnewbery opened pull request #12379: [WIP] Better stderr testing in functional tests (master...test_full_stderr2) https://github.com/bitcoin/bitcoin/pull/12379
< bitcoin-git>
bitcoin/master 9ad6746 practicalswift: Use static_cast instead of C-style casts for non-fundamental types...
< bitcoin-git>
bitcoin/master 0277173 MarcoFalke: Merge #10498: Use static_cast instead of C-style casts for non-fundamental types...
< bitcoin-git>
[bitcoin] MarcoFalke closed pull request #10498: Use static_cast instead of C-style casts for non-fundamental types (master...static_cast) https://github.com/bitcoin/bitcoin/pull/10498
< cfields>
dongcarl: what's the issue with gitian and newer LXC ?
< dongcarl>
cfields: I'm just writing up something right now actually haha.
< cfields>
dongcarl: ah, ok
< dongcarl>
cfields: in a nutshell at the very least the config file has to be changed
< dongcarl>
cfields: I'm also getting a tty error for all lxc-execute commands
< cfields>
dongcarl: isn't the config written on-the-fly ?
< dongcarl>
yes, but they changed the key names in 2.1
< dongcarl>
kallewoof actually ran into the tty errors back in December
< dongcarl>
and through GitHub digging it seems like an upstream glib thing, but that could be wrong
< dongcarl>
these issues are with LXC 2.1.1 right now, but I'm guessing non-rolling distros will move over to 2.1 soon
< cfields>
dongcarl: I assume you tried passing "-t" to ssh ?
< dongcarl>
cfields: I'm using LXC so there's no ssh, just lxc-execute
< dongcarl>
look in libexec/on-target:53
< cfields>
ah, right
< dongcarl>
I'm looking through and it seems like some of the stuff hasn't been touched for quite a while
< dongcarl>
and most "touching" have been just workarounds
< dongcarl>
correct me if I'm wrong
< cfields>
gitian, you mean?
< dongcarl>
cfields: yes
< cfields>
dongcarl: sounds about right, i haven't looked lately though
< dongcarl>
cfields: gimme a bit, gunna dump a writeup in a sec
< cfields>
dongcarl: does passing -c to sh/bash help at all?
< cfields>
ok
< dongcarl>
cfields: sipa recommended that I talk to you about this. I've been looking into making
< dongcarl>
gitian more robust over the last few days and was wondering what you and others would t
< dongcarl>
hink of these possible changes. I refer to the container actually building the Bitcoin
< dongcarl>
binary as the "build container" and the host/container running the aforementioned conta
< dongcarl>
iner the "container host."
< * dongcarl>
shakes head at tmux
< dongcarl>
1. Using vagrant to spin up a "container host" with a provisioner to spin up a "build container." This is similar to what is done for zcash (https://github.com/zcash/zcash-gitian). This solution has the advantage of not breaking existing builds and being easy to implement, but wastes cycles for containerization/virtualization.
< dongcarl>
2. Replacing gitian's direct interfacing with LXC and KVM with vagrant. As in, let vagrant handle communicating with its provider (LXC, KVM, VirtualBox, VMWare, etc.), and gitian will interface with vagrant. This solution has the advantage of wasting less cycles, but it might break existing builds, and a slight annoyance is relying on third party repos to code up the integration between vagrant
< dongcarl>
and the provider of choice, but looking through the ones people would use (e.g. vagrant-lxc) it seems like they're well maintained.
< dongcarl>
3. Same as #2 but have libvirt instead of vagrant.
< sipa>
from what i hear, it sounds like we may end up relying on a pre-built image somehow
< cfields>
dongcarl: I'm working on a process that builds deterministic toolchain and rootfs for us to use. In theory, gitian isn't even necessary. But in practice, we'll need a very thin wrapper that just runs a script and gathers the results
< sipa>
cfields: how far along are you with that?
< cfields>
sipa: pre-built image?
< sipa>
i was assuming that may not happen overnight
< cfields>
sipa: I was on a roll for a while, but then got stuck working on static pie for gcc. I think I could have it up and going relatively quickly if it's holding up progress.
< sipa>
hmmm, static pie </homer>
< cfields>
haha
< dongcarl>
cfields: so sort of like what coreboot does?
< sipa>
cfields: am i right that this would mean we can build deterministic binaries just from depends make, without any vm?
< cfields>
I only mentioned the toolchain stuff because Mark mentioned reworking gitian a while back, and his plans would've conflicted
< cfields>
sipa: correct. The only variables are the running kernel and filesystem
< cfields>
dongcarl: I'm not familiar
< sipa>
cfields: what do you mean by variables?
< cfields>
sipa: sorry, sources of non-determinism
< sipa>
hmm, how so?
< sipa>
how does the filesystem matter?
< sipa>
order it lists files etc?
< cfields>
yup
< cfields>
I'm hoping to just slowly patch those things upstream as they arise
< dongcarl>
so we'll end up needing some sort of container to normalize that anyways? I'm trying to decide whether or not I should spend time on solution no. 2 right now...
< sipa>
well that sounds like a major improvement over what we have
< cfields>
gcc/binutils is in good shape these days. busybox is the one I'm unsure about
< sipa>
it would also let us pin a particular gcc version more easily, without depending on what's available in a particular build env
< cfields>
dongcarl: realistically yes, I think
< arubi>
cfields, you've seen landley's mkroot right?
< cfields>
sipa: yes, very much so. Our own config as well. the stupid mingw-w64 thread issue is a good example of how we're at the mercy of Ubuntu atm
< sipa>
cfields: do you have any idea about realistic timelines on the self-built compiler environment stuff?
< cfields>
sipa: just for using locally or for actually building releases?
< dongcarl>
actually building?
< dongcarl>
does sound cool
< sipa>
cfields: i'd say something in between, as in "actually usable for building releases if we wanted to"
< sipa>
even if we don't immediately decide to switch to it
< cfields>
sipa: .17 seems reasonable...
< dongcarl>
cfields: do you have a repo up somewhere where I can also take a look and contribute?
< cfields>
sipa: I think the biggest issue is that I'm targetting gcc 8.1, which hasn't even been released yet. And I'm sure we want to wait for some hardening
< cfields>
dongcarl: not atm, but I can push something up in a week or two
< dongcarl>
cfields: that would be fantastic, I'd love to help get it across the line in whatever way I can
< cfields>
dongcarl: great :)
< cfields>
dongcarl: thinking about it a bit, an abstraction around lxc/kvm/vbox/whatever is kinda just re-introducing the same non-determinism we're looking to avoid.
< dongcarl>
cfields: how so?
< cfields>
dongcarl: presumably they'd all have different methods for creating the target image, meaning that each may have (for ex) a separate filesystem
< cfields>
s/separate/differing/
< dongcarl>
cfields: well, actually, for vagrant they share the same image!
< dongcarl>
cfields: right, and if we can finish this in .17, doesn't make sense to spend time on moving to that
< sipa>
i see
< dongcarl>
that = vagrant and no. 2
< cfields>
dongcarl: right
< * dongcarl>
is excited for cfield's code to be pushed up
< cfields>
dongcarl: basically, we're defining a reference system. We really can't get around that. So I'm apprehensive about attempts to abstract away the reference.
< cfields>
Again, I realize that's what gitian does now.
< dongcarl>
You're absolutely right. I think for my own building needs I'll just make my own version of no. 1 for now. Basically what's in the guides for virtualbox but with vagrant. Super easy to setup.
< cfields>
sounds good
< cfields>
dongcarl: what I'd be curious to see would be the thinnest possible wrapper on top of some vm. Just setup the cache, run the script, report the results
< dongcarl>
cfields: check out the linked zcash README
< dongcarl>
(not a big fan of the number of build dependencies tho)
< cfields>
well ours will have no dependencies, so that's easy
< dongcarl>
woot woot
< cfields>
hmm
< cfields>
I guess using vagrant with some static config is no different from rolling our own, in terms of determinism
< gmaxwell>
gitian abstracting away the reference mostly is just introducing significant insecurity and pretending we don't need to worry about it.
< dongcarl>
cfields: true, but vagrant references boxes by name, not hashes, and those boxes can be maliciously built.
< dongcarl>
cfields: what'd be nice is a script that builds the vagrant box (like gitian does with debootstrap) then launches it with a static config
< dongcarl>
gmaxwell: Agreed.
< dongcarl>
cfields: btw what did you mean by "rolling our own"?
< cfields>
dongcarl: can we not just use a generic ubuntu-minimal image ?
< dongcarl>
cfields: from ubuntu or from vagrant boxes? Vagrant has its own repository of pre-built boxes that are referred to by name that everyone uses.
< cfields>
(I've never used vagrant, though I assume it's not hugely different from the other similar vm builders/launchers)
< dongcarl>
so the "vagrant boxes" are supposed to just immediately spin up with no install process
< cfields>
gmaxwell: the insecurity being "trusting ubuntu", you mean?
< cfields>
dongcarl: by "rolling our own", I meant writing something like gitian
< cfields>
dongcarl: I see. Looks like I need to read up.
< gmaxwell>
cfields: right, and their ongoing updates. For users that already run ubuntu arguably there is not much increase in surface over what they already face, but that doesn't apply to people who run other things than ubuntu.
< cfields>
gmaxwell: sure, completely agree. Just wanted clarification.
< gmaxwell>
AFAIK we don't have any reason to think ubuntu's build process is particularly secure either, e.g. could be internet connected hosts that a hundred people have remote shell access on.
< gmaxwell>
And certantly whatever it is, it runs a lot of effectively unreviewed potentially malicious code.
< dongcarl>
cfields: yeah if we end up needing something to control the two variables you mentioned earlier, vagrant would probably work well with a fixed provider, static config, and building our own boxes instead of fetching from the repo.
< dongcarl>
gmaxwell: :-/
< gmaxwell>
if or once we get to the point where our build process builds the same binaries when run from different systems at least we'll be able to crosscheck against something else, even if the standard build enviroment remains an ubuntu vm.
< dongcarl>
gmaxwell: "where our build process builds the same binaries when run from different systems" <- as in when cfields finishes the work on unVM'd deterministic builds?
< gmaxwell>
right.
< gmaxwell>
I think it's likely though that the standard process for that will still have people use a standard vm image... just because it'll be a baseline system that we know works.
< * dongcarl>
looks forward to the day when determinist builds work on FreeBSD
< dongcarl>
gmaxwell: yeah, we were talking about that earlier, kernel and filesystem are the two variables that are most easily controlled by VMs I guess.
< cfields>
gmaxwell: yes. So far I have a minimal rootfs that's capable of building bitcoin. It's basically just busybox/make/toolchain. Dealing with kernels is just too far out of scope, though. Ideally, we'd just boot some minimal image in a vm, chroot, and make
< gmaxwell>
cfields: are you finding that the kernel and whatnot breaks determinism
< cfields>
(that minimal rootfs build is deterministic and would first be built in gitian itself)
< cfields>
gmaxwell: no, I was just enumerating the issues at play. Almost all issues we've had have come from filesystem.
< gmaxwell>
ah good.
< cfields>
dongcarl: I should think that fbsd deterministic builds would come pretty close to working as-is
< dongcarl>
cfields: with virtualbox you mean?
< cfields>
dongcarl: cross build from linux with gitian
< dongcarl>
cfields: Oh. I mean, DOING deterministic builds from FreeBSD haha
< cfields>
oh. why? :)
< dongcarl>
cfields: Because... I'm a sadist who loves making things work on *BSD.
< gmaxwell>
(1) because some people run it, and (2) hey, distinct build enviroment is better for crosschecking determinism.
< gmaxwell>
But it's okay if people have to suffer to make it work.
< dongcarl>
let's pretend I thought of (2) there
< dongcarl>
next stop: plan 9 deterministic builds
< dongcarl>
then: seL4
< gmaxwell>
nah, build on AT&T unix on SIMH pdp11. :P
< dongcarl>
<3
< cfields>
sure, no complaints if someone wants to try
< cfields>
I plan to attempt the same for osx, but just because I want to see where the differences end up coming from
< sipa>
OS/2, please.
< gmaxwell>
it would be kinda interesting if the project there were at least two distinct project recommended base VMs, and people were told to use one based on hashing their name.