< gmaxwell>
I don't know why we're suddenly talking about this in here. Nothing has changed. We knew that sha1 collisions could be constructed with ~2^64 work, and the prefix that google constructed it for can't (AFAIK) be used to trouble git.
< gmaxwell>
If github were to move to a propritary and incompatible version of git from git itself I would urge we drop github immediately.
< jeremyrubin>
gmaxwell: i would expect it to be FOSS, although it's unclear what you mean by 'propritary'
< jeremyrubin>
my fault for offtopicing
< sipa>
gmaxwell: even if it was FOSS, it would require them to start maintaining a fork of gut
< sipa>
eh
< sipa>
jeremyrubin: even if it was FOSS, it would require them to start maintaining a fork of git
< sipa>
something i don't expect them to be remotely capable of (nor should they)
< jeremyrubin>
I thought they already do that? I think I read a while back they have an internal git implementation, could be wrong
< gmaxwell>
It took weeks of nagging just to get them to not display incorrect authorship information on our reposititory recently. Yet that didn't get 1:10th the conversation in here.
< jeremyrubin>
(could be misinformed there, can't find a link)
< luke-jr>
jeremyrubin: not one every user needs; in any case, unless they actually do it, it doesn't matter
< sipa>
jeremyrubin: i'm sure they have some patches
< sipa>
gmaxwell: not as dramatic :)
< sipa>
jeremyrubin: i don't understand why you think _github_ should be doing anything at all, besides sponsoring git development
< sipa>
(which i hope they're already doing)
< bitcoin-git>
[bitcoin] jtimon opened pull request #9882: RPC: Introduce -rpcamountdecimals for the RPC to use other units than BTC (master...0.14.99-rpc-amounts) https://github.com/bitcoin/bitcoin/pull/9882
< bitcoin-git>
[bitcoin] jtimon closed pull request #9855: RPC: Use integer satoshis instead BTC with decimals (master...0.15-rpc-satoshis) https://github.com/bitcoin/bitcoin/pull/9855
< gmaxwell>
it just seems odd to me, we had an actual ongoing exploit of the kind that you'd worry about in the abstract from second preimage attack (much less a hash collision). Yet it could be fixed by some simple action on github's part (not a protocol rework)-- and silence. Yet it isn't even clear if the kind of common prefix hash collision construction that people know how to do with sha1 could /ever
< gmaxwell>
/ be much of an interesting attack beyond perhaps undermining signed commits, and everyone is talking about it... and now there are suggestions of drastic things like total github lock-in.
< gmaxwell>
This seems cultish.
< gmaxwell>
"omg sha1 is broken. we must do something! this is something! it must be done!"
< jeremyrubin>
jfc you're acting like github making a choice implies total github lock in
< jeremyrubin>
precisely, github is free to offer improvements on the code as they see fit
< sipa>
yes, but why github?
< jeremyrubin>
they are a company that offers git based services
< sipa>
and not say, the linux foundation
< jeremyrubin>
the largest one!
< jeremyrubin>
Does linux foundation offer git services? (honestly not sure? support?)
< sipa>
15:41:32 < jeremyrubin> Is anyone else kinds surprised that github didn't have a ready to go sha256 <- i find that an incredibly offensive statement
< jeremyrubin>
offensive to who?
< sipa>
it sounds like you're saying "does anyone else think the git maintainers are idiots for ignoring this issue, and that github hasn't decided to fork off"
< sipa>
for something that may never be exploitable
< BlueMatt>
hmmmm, segfault in bitcoin-qt on startup :(
< jeremyrubin>
so it's offensive to... github? or git maintainers?
< sipa>
to me
< jeremyrubin>
Well git is named after a stupid person, so it may be fair to call them idiots :p
< gmaxwell>
jeremyrubin: if your question were why hasn't github subbmited some patches upstream to implement it, I would have found it less alarming. But asking about them switching to an incompatible version? They can barely keep their site running competently.
< gmaxwell>
were they to do that it would sound a lot like embrace-extend-extinguish.
< jeremyrubin>
I don't think I ever once said incompatible
< jeremyrubin>
you guys keep injecting that
< BlueMatt>
so, sipa, any thoughts on my segfault?
< gmaxwell>
nothing in the immediate backtrace of that crash appears to have been recently changed.
< gmaxwell>
BlueMatt: I assume this is a build on your system? (not a gitian binary)
< gmaxwell>
BlueMatt: if it's 100% reproducable can you bisect it?
< BlueMatt>
gmaxwell: it is not reproduceable, happened once
< BlueMatt>
i can go restart that thing all day and see.......
< gmaxwell>
BlueMatt: maybe try adding a shutdown shorty after start and leave it going in a loop?
< BlueMatt>
well it crashed in the same place on ~ the 4th try
< BlueMatt>
so its not exactly non-reproduceable
< gmaxwell>
BlueMatt: gitian binary or not.
< BlueMatt>
not
< BlueMatt>
locally-built
< gmaxwell>
can you stick a shutdown right after init completes and see if you can still reproduce it?
< BlueMatt>
lemme see if I can rebuild and still reproduce it first :P
< gmaxwell>
yea, I guess I was also suggesting that.
< BlueMatt>
yes, fresh build crashes in the same way, so thats...good?
< gmaxwell>
The fresh build contains potassium benzoate.
< BlueMatt>
gmaxwell: yay hisenbug :(
< BlueMatt>
yup, just adding a StartShutdown() 30 seconds after start in the middle of the net code appears to fix it
< cfields>
BlueMatt: is there anything in the wallet?
< BlueMatt>
yes, lots 'o things
< BlueMatt>
(nothing unconfirmed, though)
< BlueMatt>
yup, and now even the old binary wont crash
< BlueMatt>
ffs
< cfields>
BlueMatt: seems likely to me that re-acceptance is advertising a tx early in the startup process, before the TransactionView is fully setup. So I assume you'd need a sufficiently interesting persisted mempool in order to repro.
< cfields>
(catching up, not sure if you guys already looked down that path)
< BlueMatt>
cfields: no one looked down any path
< BlueMatt>
cfields: strange, there is definitely nothing in mempool which applies to that wallet
< cfields>
BlueMatt: still reproducible if you disable mempool loading?
< cfields>
i don't really see where else it'd be coming from, so i assume not
< BlueMatt>
cfields: well it stoped being reproduceable, so maybe mempool dump changed
< cfields>
BlueMatt: right. If you repro again, I'd be sure to make a backup
< BlueMatt>
still, makes no sense as no mempool txn should apply to that wallet
< BlueMatt>
sure
< cfields>
i have no idea what gets blasted around, will try to trace the paths
< bitcoin-git>
[bitcoin] TheBlueMatt opened pull request #9884: Add Pieter's old signed commits to revsig-commits (master...2017-02-pieter-revsig) https://github.com/bitcoin/bitcoin/pull/9884
< BlueMatt>
^ will need to be merged before anything else or travis will barf
< BlueMatt>
well, travis will barf in an obvious way and only on master, so nbd
< BlueMatt>
but will barf
< bitcoin-git>
[bitcoin] gmaxwell opened pull request #9886: In feerate ties, prefer larger packages first. (master...prefer_bigger_packages) https://github.com/bitcoin/bitcoin/pull/9886
< jonasschnelli>
Can I still normally merge a PR or did we already witch to the SHA512 merge script?
< wumpus>
I'm testing the SHA512 merge script, but we haven't merged it yet
< wumpus>
so feel free to not use it yet
< jonasschnelli>
okay... i'll wait until we have merged the SHA512 tree change
< wumpus>
there's still an open issue about symlinks on that ,but as we (AFAIK) don't have symlinks in bitcoin repo it's somewhat theoretical
< bitcoin-git>
bitcoin/master 83ac719 Marijn Stollenga: Change bitcoin address in RPC helpaddress to an invalid address, so people don't accidentally send coins there (like I did).
< bitcoin-git>
bitcoin/master 30bdcfc Wladimir J. van der Laan: Merge #9865: Change bitcoin address in RPC help message...
< bitcoin-git>
bitcoin/0.14 289204f Marijn Stollenga: Change bitcoin address in RPC helpaddress to an invalid address, so people don't accidentally send coins there (like I did)....
< bitcoin-git>
bitcoin/0.14 ad24256 Russell Yanofsky: Fix importmulti returning rescan errors for wrong keys...
< wumpus>
apparently. THanks for mentinoning, I was subliminally ignoring that one for some reason because I assumed it still had nits open
< wumpus>
* [new tag] v0.14.0rc3 -> v0.14.0rc3
< bitcoin-git>
[bitcoin] MarcoFalke opened pull request #9888: travis: Verify commits only for one target (master...Mf1702-travisCommits) https://github.com/bitcoin/bitcoin/pull/9888
< bitcoin-git>
bitcoin/master fa32a16 MarcoFalke: travis: Verify commits only for one target...
< bitcoin-git>
bitcoin/master 36afd4d MarcoFalke: Merge #9888: travis: Verify commits only for one target...
< bitcoin-git>
[bitcoin] MarcoFalke closed pull request #9888: travis: Verify commits only for one target (master...Mf1702-travisCommits) https://github.com/bitcoin/bitcoin/pull/9888
< achow101>
whoever finishes osx next, let me know if your hashes match mine: 9b247d80bb79f3b96d49e9d61b1977e07260c788022714e670dc7673a35fbf25 bitcoin-0.14.0-osx-unsigned.dmg
< achow101>
jonasschnelli: linux and windows match
< jonasschnelli>
Yes. Linux and Win matches with achow101
< jonasschnelli>
But not OSX
< jonasschnelli>
Same descriptor and SDK
< jonasschnelli>
source tarball file matches..
< jonasschnelli>
So.. this is strange
< achow101>
just like the issue with the last two rcs...
< jonasschnelli>
ah.. haven't followed that. OSX build of rc1-2 wasn't deterministic?
< achow101>
are you using kvm or lxc?
< jonasschnelli>
lxc
< achow101>
jonasschnelli: yeah, my osx builds for rc1 and 2 did not match everyone elses except under certain circumstances. it was really weird and the cause is unknown
< achow101>
(rc1 each time I built it alternated between matching and mismatching, rc2 the first build mismatched and all subsequent ones after a new base vm matched)
< achow101>
jonasschnelli: try remaking with a new base vm?
< jonasschnelli>
achow101: I made the base vm last week.
< achow101>
ok, so I just rebuilt and this one matches jonasschnelli's
< achow101>
:/
< jonasschnelli>
achow101: okay. Thanks for removing your OSX backdoor. :)
< achow101>
cfields must have the same backdoor too :D
< jonasschnelli>
yeah... maybe you are the same person as well. :)
< wumpus>
did anyone ever compare the executables?
< achow101>
wumpus: I think cfields did. IIRC I sent him the osx binaries and all of the compiled object binaries pulled oof of the vm
< achow101>
s/oof/out/
< cfields>
wumpus: yes, i compared a ton.
< cfields>
couldn't track it down to anything. Makes no sense.
< cfields>
wumpus: all object files were the same, only difference is the executables. And They're created with our self-built linker
< cfields>
only thing i can come up with (and how it looks) is symbol sort order
< wumpus>
sounds a bit like the issue we had on windows, where some section was left uninitialized
< cfields>
achow101 / jonasschnelli: mind retrying with 1 build job?
< wumpus>
a linker map eventually helped me narrow down the issue
< wumpus>
sort order? hm, a locale issue?
< achow101>
cfields: sure.
< jonasschnelli>
cfields: Yes. I can do that.
< cfields>
wumpus: my only theory so far (based on absolutely nothing) is the $(wildcard) in the osx makefile, which could maybe lead to a different link-order
< cfields>
wumpus: oh, i left out.. only the osx binary differs
< wumpus>
cfields: that sounds like a good theory to me, various filesystems will return files in different order. May make sense to pass that through a (locale independent) sort as well.
< achow101>
wumpus: what doesn't make sense is that my build would alternate between two different hashes every time I built
< cfields>
wumpus: yes
< cfields>
achow101: hence the -j1 suggestion :)
< sipa>
9
< wumpus>
achow101: it makes perfect sense if it's filesystem non-determinism
< achow101>
wumpus: how so?
< sipa>
the filesystem is recreated for each build, no?
< wumpus>
achow101: there's just no guarantee that the file system is deterministic
< wumpus>
especially in the presence of multple threads, the order in which things are written, i/o scheduling, etc
< achow101>
but shouldn't that also effect linux and windows builds?
< wumpus>
no, in linux and windows builds we don't use any wildcards in the linker AFAIK
< achow101>
oh
< wumpus>
also even if they did, they use a different linker which may be sorting things differently, for all we know
< wumpus>
osx is kind of an odd duck out in the toolchains as it uses the clang-based stuff
< sipa>
also, any idea why Travis fails on the rebase of #9791 ?
< bitcoin-git>
bitcoin/0.14 5e70912 Matt Corallo: Add Pieter's old signed commits to revsig-commits...
< cfields>
jonasschnelli: woohoo, match
< jonasschnelli>
cfields: yes. So the build jobs break the determinism
< cfields>
jonasschnelli: certainly not definitive, but seems very plausible
< cfields>
jonasschnelli: for rc1/rc2, achow101's builds alternated between 2 outcomes, one of which was a match
< wumpus>
what should I use to re-sign my subkey?
< wumpus>
without getting trouble like that
< BlueMatt>
sipa/wumpus: yea, I mean its not clear to me that keyservers will update their revocation cert for keys which changed from "superceded" to "compromised"
< BlueMatt>
so I dont want to do that unless there is some assurance there
< BlueMatt>
wumpus: so I dont think anyone figured out how to re-cross-certify using a new hash, so I'm just gonna enforce non-sha1 on new sigs
< BlueMatt>
wumpus: meaning just create new subs for signing and use those instead of the old ones
< BlueMatt>
jonasschnelli: may need to do so as well, or needed to as of yesterday
< BlueMatt>
wumpus: (though your sigs were fine - you werent signing with your subkeys)
< wumpus>
BlueMatt: okay, thanks
< jonasschnelli>
BlueMatt: I'd like to create a new Ed25519 key (HWW) and add them to my key package under contrib/... would that be a problem?
< BlueMatt>
jonasschnelli: hmm? do you need new keys or just a new subkey?
< MarcoFalke>
jonasschnelli: A subkey?
< jonasschnelli>
BlueMatt: I'd like to do a new key
< BlueMatt>
you can put a subkey in your key and put it on the keyservers and everything will work without any other updates
< BlueMatt>
ahh, ok, yea, I mean just put it in contrib/ then, make sure to sign the new key with your old one (and preferably get someone else local to sign)
< jonasschnelli>
BlueMatt: but I would need to add the new key ID into the verify-git script?
< BlueMatt>
yes
< jonasschnelli>
Maybe I just do a subkey for now
< sipa>
note that github doesn't seem to support ed25519 subkeys
< jonasschnelli>
sipa: hmm.. they support ed25519 "main key" but not sub?
< sipa>
i don't think so either
< sipa>
but i didn't try creating a new main key
< jonasschnelli>
Hmm.. yes. I have only tried git/ssh access keys with Ed25519... this works.. haven't tried the gpg part though
< wumpus>
maybe the 64k stack frame needs some extra support, so one brace cannot hold it?
< BlueMatt>
wumpus: .......
< sipa>
possibly
< BlueMatt>
i assume that was cfields trying not to have too large a diff
< sipa>
subsequent refactorings that removed locks
< BlueMatt>
or, at least, I blame cfields either way
< sipa>
i assume
< wumpus>
I could understand that for one level, but three? anyhow, no big deal, I just had to laugh
< cfields>
heh yes, i think 3 rounds of scopes have been eliminated now. Got tired of stomping on myself with whitespace :)
< cfields>
there are a few places like that. It's probably a good time to do a whitespace cleanup on those.
< wumpus>
I guess all that code is going to be nuked and replaced with libevent anyhow
< * sipa>
casually mentions adding ?w=1 to github urls
< bsm117532>
If anyone knows a way to make github do that by default...I would love you eternally...
< cfields>
sipa: sure, but towards the end, i can only imagine the initial reaction to "sigh, another +520,-500 diff from cfields" :)
< wumpus>
bsm117532: get in the habit of diffing locally instead of in gh, or at least that's what I did
< bsm117532>
Oh I do...but then there's code reviews...
< wumpus>
I usually review and take notes offline
< wumpus>
then pester all pulls at once
< sipa>
offline?
< wumpus>
well outside-of-the-browser
< wumpus>
I don't generally go as far as to disconnect my network :p
< sipa>
i imagine you now printing PRs on a line printer, laying the meters-long paper on the floor, and drawing arrows all over it
< wumpus>
good idea
< cfields>
heh
< wumpus>
outoing connections aren't checked against -whitelist are they?
< sipa>
i believe not
< wumpus>
I'm trying to whitelist an outgoing connection (want to push blocks from a node that is not completely up-to-date to a node without any blocks, and trying to get around "Ignoring getheaders from peer=1 because node is in initial block download")
< cfields>
wumpus: set the ibd date way back
< cfields>
sec
< cfields>
wumpus: -maxtipage
< wumpus>
the node is in a sandbox that cannot make outgoing connections so I can't do it the other way around
< wumpus>
setting maxtipage is not working for some reason, set it to 400000000 which should be enough to put us in 2004, but it still sees it as initial block download. Checking why...
< cfields>
wumpus: not past a checkpoint yet, maybe?
< sipa>
i read that as max ti page, and was wondering why we're supporting configuring the page size on ti calculators
< cfields>
!InitialBlockDownload() added in #6172
< gribble>
Error: "InitialBlockDownload()" is not a valid command.
< sdaftuar>
sipa: wumpus: i think because of checkpoints you can partition a node that is syncing off from the honest network by feeding it a forked chain before the last checkpoint
< wumpus>
hehe, because ti calculators are super fast at ecdsa computations of course!
< cfields>
er.. #6172
< sdaftuar>
and then it responds to it's outbound's peers requests with the bogus chain, and gets disconnected
< wumpus>
didn't we stop using checkpoints for that purpose?
< sdaftuar>
wumpus: we still lock in the chain once we've gotten to each checkpoint, and will ban nodes that send us an alternate chain after that point
< wumpus>
anyhow, just going to comment out the check locally
< wumpus>
sdaftuar: okay
< wumpus>
sdaftuar: yes in that case it'd make sense to replace that with a checkpoint-based check
< wumpus>
I'm not entirely sure I understand this: why would responding to getheaders before the last checkpoint (or during initial block download) cause it to be potentially fed a forked chain?
< sipa>
because you may be on a forked chain without knowing it
< sipa>
so a peer asking you for headers, and then downloading them, and then discovering those headers don't align with a checkpoint
< bitcoin-git>
[bitcoin] ericshawlinux opened pull request #9890: Add a button to open the config file in a text editor (master...master) https://github.com/bitcoin/bitcoin/pull/9890
< sipa>
(i'm speculating)
< wumpus>
right, okay, that makes sense
< wumpus>
why didn't we add a comment for this :/
< wumpus>
it's not exactly trivial to think though
< gmaxwell>
Please no checkpoint based check. Use the work based check.
< gmaxwell>
also failing to respond to getheaders is a DOS attack against the peer. If we're not going to respond we should disconnect.
< wumpus>
the point is that the check that causes the ban (on other nodes) uses a checkpoint based check
< wumpus>
not sending the headers is trying to preempt that
< sdaftuar>
wumpus: i think gmaxwell is right though -- a work based check is more reasonable long-term and just as effective for this purpose
< gmaxwell>
The minimum work check has vastly more work than the highest checkpoint deployed. If there is another chain with more work than our minimum work chain that is incompatible with the checkpoints then we have big problems.
< wumpus>
yes, but what work threshold? if it is that threshold that is updated every version, it's not going to be any better than checking IsInitialBlockDownload
< wumpus>
(which already checks against that)
< gmaxwell>
I think checking IsInitialBlockDownload should be fine as is now. It used to be stupid and only check the height against the checkpoints.
< sdaftuar>
what do you think of allowing nMinimumChainWork to be a hidden command line option? i would find that useful for my testing at least, and sounds like it'd be useful for what wumpus is trying to do as well.
< wumpus>
it's fine but incredibly annoying if you want to sync blocks from a node that is not up to date to a new node
< gmaxwell>
Well one downside of IsInitial is that it's not eager enough to turn on.
< wumpus>
which I apparently do regularly and spend time troubleshooting then realize it's that check again
< gmaxwell>
Which would be a reason to use a straight work check instead of IsInitial.
< gmaxwell>
sdaftuar: I wouldn't complain about that. But perhaps a switch to more directly change the behavior would be more what you'd want?
< sdaftuar>
gmaxwell: yeah, that might be called for here. i guess what i often want to do is actually make IBD end sooner (eg for my simulations)
< wumpus>
sdaftuar: cfields's proposal of using maxtipage was great for that, unfortunately it doesn't work as the fixed work check is before that
< wumpus>
gmaxwell: it's already possible to avoid the check by whitelisting a node, but in my case I couldn't use that because there's no way to whitelist outgoing connections :)
< gmaxwell>
Seperate from the bypass, -- We should probably work to get rid of that ban-on-inconsistent-with-checkpoints behavior.
< sdaftuar>
gmaxwell: agreed. i also might put getting rid of checkpoints on my list of goals for 0.15.
< gmaxwell>
I think eliminating them will require a soft-fork to increase the minimum difficulty or something similar.
< gmaxwell>
(but not the kind of softfork that ever need to get activated, a flag height softfork)
< wumpus>
it seems like a lousy reason to ban
< wumpus>
banning should be reserved for things that consume a lot of resources
< gmaxwell>
We should have a general mechenism for punting peers that don't agree with our consensus... that should be rather lazy.
< cfields>
wumpus: interesting, your osx sigs mismatched as well
< cfields>
seems we can all flip-flop
< BlueMatt>
lolwtf...someone is trying to connect to the fibre network using a testnet node, connecting to port 8333
< sipa>
we need a proxy that forwards connections to one of two bitcoind, based on network magic
< BlueMatt>
or they could just use the right port.....
< BlueMatt>
(well, those nodes dont have testnet, so dunno wtf they think they're doing, but whatever)
< BlueMatt>
suppose its good that at least one miner has a full copy of their configs running on testnet
< profall>
if I rpcbind to a different address then localhost, when I try to use bitcoin-cli it cannot connect to the server. Before you assume, it's binded to a private IP on a VLAN that only I control.
< BlueMatt>
profall: i believe there is a rpcconnect option for that case
< BlueMatt>
(which you would pass into bitcoin-cli to tell it where to connect)
< profall>
ok
< profall>
Figured it out, thanks :)
< gmaxwell>
BlueMatt: probably me.... common config for testnet and not.
< BlueMatt>
gmaxwell: looks like ckpool, maybe
< BlueMatt>
sipa: whats the formula for the 15 pointers of size for mapTx? Can we get a comment to describe that?
< sipa>
3 pointers per sorted list
< sipa>
s/list/index/
< sipa>
i think
< sipa>
will do
< BlueMatt>
thanks
< cfields>
ok, I think i nailed down the gitian issue
< BlueMatt>
nice!
< cfields>
wumpus: still around, by any chance?
< cfields>
wumpus: it's not worth a new tag, but i can't sign your osx bin :(
< cfields>
osx's ld64 uses threads for linking. It then either doesn't sort the output, or produces a graph that can't actually be sorted deterministically. Not sure which.
< achow101>
but at least it will be fixed for final
< bitcoin-git>
[bitcoin] theuni opened pull request #9891: depends: make osx output deterministic (master...fix-osx-link-determinism) https://github.com/bitcoin/bitcoin/pull/9891
< achow101>
cfields: what hash should we get with the osx patch?
< luke-jr>
checking whether to build with support for UPnP… configure: error: "UPnP requested but cannot be built. use --without-miniupnpc"
< luke-jr>
^ for reference, this means ccache dir is read-only
< luke-jr>
(or *can* mean it, anyway)
< bitcoin-git>
[bitcoin] luke-jr opened pull request #9892: Bugfix: Only install manpages for built programs (master...bugfix_man_onlybuilt) https://github.com/bitcoin/bitcoin/pull/9892
< cfields>
luke-jr: where are you seeing that?
< cfields>
luke-jr: ah. We should do a quick compile check.
< luke-jr>
cfields: Gentoo's build system supports ccache, but apparently doesn't manage permissions correctly (root owns ccache dir)
< luke-jr>
not normally an issue, but I was testing out an updated ebuild as non-root
< cfields>
luke-jr: ah. There's probably a use flag that's supposed to signal to export a var for the path, then?