< bitcoin-git> [bitcoin] maaku opened pull request #9981: Bloom: Consider witness script pushes in bloom filter check. (master...segwit-bloom-filter) https://github.com/bitcoin/bitcoin/pull/9981
< Hisham> Hello ?
< Hisham> Any core developer online please ?
< sipa> maybe!
< bitcoin-git> [bitcoin] maaku closed pull request #9981: Bloom: Consider witness script pushes in bloom filter check. (master...segwit-bloom-filter) https://github.com/bitcoin/bitcoin/pull/9981
< Hisham> Pieter Wuille
< Hisham> At first, it's my pleasure to talk to you :)
< Hisham> If you don't mind, I would like to discuss something with you regarding SegWit UASF
< Hisham> Can we talk please ?
< BlueMatt> dear god, dont ask to ask, just ask
< Hisham> :S
< Hisham> Guys, I know you are nice to people, c'mon, what happened ?
< Hisham> Maybe you are busy or something, does politeness and taking permission is considered a crime nowadays ?
< BlueMatt> it is generally discouraged on irc
< BlueMatt> text-based communication is slow enough already :(
< Hisham> Ok I see and understand, no problem :)
< BlueMatt> engineers, you know
< Hisham> Here is one as well, Mechanical and a CNC programmer :)
< Hisham> I was going to be a colleague, but I had a nasty doctor who let me hate CS, Java to be precise.
< gmaxwell> google for "don't ask to ask"-- beyond being wasteful, it's considered rude because it asks people to commit to an open ended conversation which they may not be interested in when they know the details.
< gmaxwell> as far as your question, all the discussion on that has been on the mailing list, I don't think anyone here has been involved in it (though perhaps I'm incorrect, as I haven't been following it)
< sipa> Hisham: just ask whatever you want to ask
< Hisham> Thanks for the clarification Gregory.
< Hisham> But guys, for God sake take it easy on me.
< sipa> yes, get to the point :)
< Hisham> I will.
< Hisham> I'm a normal old Bitcoiner since the days of CPU and browser mining but I've got highly involved at 2014, I've read a lot about Bitcoin and you can I'm my life is oriented to it for many reasons.
< Hisham> When I saw the blocksize debate, I felt that I can't stand on the sidelines and try to help or at least understand and engage in conversations, maybe I can do something for all the BS happening around us.
< achow101> Hisham: please just ask your question. none of this preamble is necessary. I hate to break it to you, but no one cares about your backstory. Please just get straight to the point and ask your question.
< Hisham> No problem, I understand, sorry.
< Hisham> As some of you may witness, Shaloin Fry made what we can call a SegWit UASF.
< Hisham> As I understand, it needs consensus, if I'm not mistaken.
< Hisham> How can we get this implemented if there is a chance for it to be implemented ?
< sipa> no politics here, please
< BlueMatt> Hisham: if you want a UASF to happen, go start talking to the community - talk to businesses and users
< BlueMatt> such things are not for us to decide
< BlueMatt> Hisham: relevant is bluematt.bitcoin.ninja/2017/02/28/bitcoin-trustlessness/ (and also #bitcoin, not really #bitcoin-core-dev)
< * BlueMatt> does the shameless-self-promotion dance
< Hisham> I understand, but from technical POV, is there a chance for intergation ?
< Hisham> *integration ?
< sipa> implementing it is absolutely trivial
< sipa> the only question is whether we should
< sipa> and generally the answer to that is "if it's clearly uncontroversial"
< BlueMatt> i believe code was already written, as for people actually using that code....<BlueMatt> such things are not for us to decide
< Hisham> Pieter, may you elaborate more please on this "if it's clearly uncontroversial"
< Hisham> ?
< BlueMatt> maybe this is a better topic for #bitcoin, its not really a development topic
< Hisham> Ok, since I'm not welcomed form the beginning.
< BlueMatt> Hisham: I'm happy to go discuss it further on #bitcoin
< Hisham> Ok, can we talk there now please along with Pieter ?
< bitcoin-git> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/7e58b41bd7ce...f8a709161f37
< bitcoin-git> bitcoin/master e007b24 Matt Corallo: Fix shutdown hang with >= 8 -addnodes set...
< bitcoin-git> bitcoin/master 819b513 Matt Corallo: Add missing braces in semaphore posts in net
< bitcoin-git> bitcoin/master f8a7091 Wladimir J. van der Laan: Merge #9953: Fix shutdown hang with >= 8 -addnodes set...
< bitcoin-git> [bitcoin] laanwj closed pull request #9953: Fix shutdown hang with >= 8 -addnodes set (master...2017-03-exit-with-addnode) https://github.com/bitcoin/bitcoin/pull/9953
< bitcoin-git> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/f8a709161f37...afcd7c0e52d8
< bitcoin-git> bitcoin/master af61d9f Russell Yanofsky: Add COutput::fSafe member for safe handling of unconfirmed outputs...
< bitcoin-git> bitcoin/master dcf2112 NicolasDorier: Add safe flag to listunspent result
< bitcoin-git> bitcoin/master afcd7c0 Wladimir J. van der Laan: Merge #9830: Add safe flag to listunspent result...
< bitcoin-git> [bitcoin] laanwj closed pull request #9830: Add safe flag to listunspent result (master...listunspenttrusted) https://github.com/bitcoin/bitcoin/pull/9830
< bitcoin-git> [bitcoin] laanwj pushed 4 new commits to master: https://github.com/bitcoin/bitcoin/compare/afcd7c0e52d8...2cc0df1fcecc
< bitcoin-git> bitcoin/master 0068361 Cory Fields: release: add win detached sig creator and our cert chain...
< bitcoin-git> bitcoin/master f642753 Cory Fields: release: create a bundle for the new signing script...
< bitcoin-git> bitcoin/master 09fe2d9 Cory Fields: release: update docs to show basic codesigning procedure
< bitcoin-git> [bitcoin] laanwj closed pull request #9514: release: Windows signing script (master...win-signing-script) https://github.com/bitcoin/bitcoin/pull/9514
< bitcoin-git> [bitcoin] laanwj pushed 5 new commits to master: https://github.com/bitcoin/bitcoin/compare/2cc0df1fcecc...fa99663bec1d
< bitcoin-git> bitcoin/master fd5e905 Matt Corallo: Make verify-commits.sh non-recursive
< bitcoin-git> bitcoin/master 8ed849f Matt Corallo: Fix travis failing to fetch keys from the sks keyserver pool...
< bitcoin-git> bitcoin/master efc06c2 Matt Corallo: If GNU sha512sum is missing, try perl shasum in verify-commits
< bitcoin-git> [bitcoin] laanwj closed pull request #9940: Fix verify-commits on OSX, update for new bad Tree-SHA512, point travis to different keyservers (master...2017-03-verify-commits-no-recursion) https://github.com/bitcoin/bitcoin/pull/9940
< bitcoin-git> [bitcoin] laanwj opened pull request #9983: tests: Convert selected tests to using named RPC arguments (master...2017_03_rpc_tests_named_arguments) https://github.com/bitcoin/bitcoin/pull/9983
< bitcoin-git> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/fa99663bec1d...8040ae6fc576
< bitcoin-git> bitcoin/master 3b092bd Wladimir J. van der Laan: util: Properly handle errors during log message formatting...
< bitcoin-git> bitcoin/master b651270 Wladimir J. van der Laan: util: Throw tinyformat::format_error on formatting error...
< bitcoin-git> bitcoin/master 8040ae6 Wladimir J. van der Laan: Merge #9963: util: Properly handle errors during log message formatting...
< bitcoin-git> [bitcoin] laanwj closed pull request #9963: util: Properly handle errors during log message formatting (master...2017_03_handle_exception_tinyformat) https://github.com/bitcoin/bitcoin/pull/9963
< MarcoFalke_lab> wumpus: I think you need to enable the verify commits pre push hook.
< MarcoFalke_lab> Alternatively, someone could fix the bug in github-merge
< wumpus> which bug?
< MarcoFalke_lab> It will calculate the hash of the files in the working directory, instead of the ones in git
< wumpus> right, that should be fixed
< wumpus> without that it's kind of useless and we should disable the hashing completely
< wumpus> seems easy enough to do though, just request the data from git
< MarcoFalke_lab> Why not do the hard reset early? I mean at some point the script does the hard reset anyway, so the data is lost regardless
< wumpus> it shouldn't do any hard resets
< wumpus> okay, yeah I don't really like that solution. It should hash what is in git, not what is in the current working directory
< MarcoFalke_lab> ok, fine.
< wumpus> otherwise there's always a window for races
< MarcoFalke_lab> jup, it is the cleaner solution.
< wumpus> working on it now - apparently ls-files gives the blob ids, that can be passed as-is to git cat-file
< BlueMatt> wumpus: yes, iirc that was annoying but i dont remember why now....verify-commits does cat-file $COMMIT_ID:$FILE_PATH
< wumpus> BlueMatt: ah, that likely works just as well
< BlueMatt> wumpus: if you can get it to work some other way, though, use that...I kinda liked how merge did the hashes via checkout and verify-commits used a different approach, but it really doesnt matter all that much
< * BlueMatt> doesnt have to deal with the merge script, so I get to argue for it to be harder to use all day long with no repurcussions, though :P
< wumpus> well how the sha is computed doesn't affect how hard it is to use, it does affect whether the output is correct, which is arguably very important
< BlueMatt> indeed
< * BlueMatt> is ok with the sha being less likely to match, if that also means its more likely to trip up if something strange happens (symlink detection gets broken, etc)
< BlueMatt> anyway, this is too inconsequential to nitpick over :P
< wumpus> I'm kind of annoyed at travis tripping up all the time though, if that keeps happening I'll hvae to disable the verify-commits check there. SO let's please just get it working
< BlueMatt> i think it should be good now
< BlueMatt> it failed a number of times just downloading the pubkeys because the sks pool was broken
< BlueMatt> but we changed to subset which appears to possibly be more stable
< BlueMatt> i do not believe we have otherwise failed in some time
< BlueMatt> hmmm, no, failed again this morning
< BlueMatt> I'll go take a look
< wumpus> according to MarcoFalke_lab it failed due to my merge script miscomputing the SHA
< MarcoFalke_lab> BlueMatt: It used the 'old' server url
< MarcoFalke_lab> The other failure was due to the wrong sha
< BlueMatt> that was once, looks like it also once failed to retreive pubkeys
< BlueMatt> wtf
< BlueMatt> how did it do that? it now points to subset???
< MarcoFalke_lab> BlueMatt: IIRC the subset-patch was not merged at that time
< BlueMatt> yes it was, the very top commit failed and the second-to-top merge was the subset change
< wumpus> what subset patch?
< BlueMatt> im super confused
< BlueMatt> wumpus: part of #9940 was to change the server gpg fetches the keys from
< gribble> https://github.com/bitcoin/bitcoin/issues/9940 | Fix verify-commits on OSX, update for new bad Tree-SHA512, point travis to different keyservers by TheBlueMatt · Pull Request #9940 · bitcoin/bitcoin · GitHub
< wumpus> ok, confusing
< MarcoFalke_lab> Right
< MarcoFalke_lab> Probably just a travis network hickup
< petertodd> BlueMatt: why not have the merge script get the pubkeys from the repo itself?
< BlueMatt> MarcoFalke_lab: no, look at the build, it tried to fetch from pool., not subset.pool.
< BlueMatt> petertodd: yea, I think thats the next step, I was thinking subset might help, but...
< BlueMatt> still need to do a --refresh-keys
< BlueMatt> but need a backup
< petertodd> BlueMatt: better to make the whole thing deterministic; if verify commits doesn't run because a key is missing, fix that!
< BlueMatt> oh, no, I'm wrong, github is listing me commits out of order
< BlueMatt> wtf
< wumpus> yes it does that, it uses a stupid sort ordering for commits
< wumpus> by date instead of chronologically according to the repo
< BlueMatt> oh, ok, yes, the failure occurred before the subset merge, and then another afterwards due to bad sha512
< BlueMatt> wumpus: well i was only looking at merge commits, i think github actually just had a hiccup
< BlueMatt> anyway, I restarted the build on the >=8 -addnode fix merge, which should pass this time, if there are any non-sha512-related failures we'll switch to pulling keys from in-repo
< wumpus> argh, git cat-file blob <blob> is slow compared to the current script
< wumpus> oh it can work in batch mode, that's cool
< BlueMatt> wumpus: its hella slow
< BlueMatt> like, insanely slow
< wumpus> also in batch mode? I think the overhead right now is spawning a process for every file
< BlueMatt> hmm, not sure, didnt try, but you need to separate out different files for our current hash format
< wumpus> it prints a header per file
< wumpus> "84e7eb60d70d9fae3bdaae7f04f9f08fecaf2052 blob 150829"
< BlueMatt> ahh, ok
< wumpus> ooh it's fast in batch mode
< BlueMatt> argh, great, now i need to make verify-commits use batch mode
< BlueMatt> :P
< bitcoin-git> [bitcoin] laanwj opened pull request #9984: devtools: Make github-merge compute SHA512 from git, instead of worktree (master...2017_03_merge_hash_git) https://github.com/bitcoin/bitcoin/pull/9984
< bitcoin-git> [bitcoin] NicolasDorier opened pull request #9985: [QT] Show more descriptive label for pay to yourself entries (master...watchonlylabel) https://github.com/bitcoin/bitcoin/pull/9985
< gmaxwell> Apparently IBD on 0.14 OOMs on 2GB (at least on an odroid c2, as expirenced by sipa and someone on reddit); due to the mempool loaning allowing it to achieve peak memory at IBD rather than sometime later.
< gmaxwell> I suppose it's better to crash during IBD rather than later...
< achow101> gmaxwell: someone has reported that in an issue and on bitcointalk too
< gmaxwell> achow101: well the answer for them right now is to lower their dbcache/maxmempool, I recommended 150/150.
< gmaxwell> It's not really a regression in 0.14, the same devices would eventually crash with 0.13.x, just later.
< sipa> well, not necessarily, but they would with -dbcache=600
< sipa> so we've effectively just raised the default dbcache during IBD
< gmaxwell> sipa: say dbcache is 300, and mempool is 300.. eventually dbcache will be full and mempool will be full.... sooo similar peak memory usage, no?
< gmaxwell> (or would you argue that because the leveldb batch can result in 2x memory, that we should be adding half the mempool to the dbcache?)
< sipa> gmaxwell: right, that's what i'm saying
< sipa> with 300 MB dbcache + 300 MB mempool, you can use up to 900 MB at flush time
< sipa> with 600 MB dbcache + 0 MB mempool, you can use up to 1200 MB at flush time
< gmaxwell> so perhaps a patch we should do quickly and backport is to add half the unused memory to the dbcache. Can you see if that would make your odroid c2 successfully sync?
< sipa> BUT, we should probably halve the -dbcache setting as well, because someone who configures 2300 MB of dbcache does not expect the application to suddenly use 4600 MB, regardless of mempool borrowing
< gmaxwell> hm. so perhaps half the units, double the default?
< gmaxwell> halve*
< sipa> or just fix leveldb to not require the whole batch to be stored in a single std::string.
< gmaxwell> yea, but would we backport that for 0.14.1?
< sipa> depends on how trivial it is, i think
< gmaxwell> (I think we need to do something for 0.14.1 but it could be something dumb like halving the mempool loan)
< sipa> yeah, that would fix the 'unexpectedness' of the memory loaning
< sipa> but not the incorrectness of the estimate
< luke-jr> is there a reason to use dbcache post-sync?
< sipa> yes, much faster block verification
< sipa> even with sigcache
< luke-jr> but why not commit it after each post-sync block immediately?
< luke-jr> it doesn't grow that much from just one block, does it?
< sipa> it can grow by 20 MB or so from one block
< sipa> and flushing slows down future blocks
< luke-jr> hmm
< sipa> with per-txout caching, the max (and typical) growth of the cache per block will be much less
< gmaxwell> much (most?) of dbcache's gains come from avoiding writes entirely for utxo that never need to make it to disk.
< sipa> indeed.
< sipa> though avoiding writes can always be moved out of the critical path of block validation
< sipa> in synced state, it still prevents recent utxos from needing to be read from disk
< nemgun> Hello guys, thank you for 0.14, i am resyncing, and it uses far less CPU
< luke-jr> sipa: aside from validation, that's probably useful for external tools accessing them via RPC?
< sipa> luke-jr: i believe gettxout will use the cache
< sipa> luke-jr: gettxoutsetinfo always flushes to disk, and then iterates over the leveldb state
< TD-Linux> <gmaxwell> I suppose it's better to crash during IBD rather than later... <- yeah though super annoying when pruning.
< gmaxwell> oh my we should probably only be pruning after sync flushes.
< jouke> w/win 24
< jouke> >_<
< sipa> gmaxwell: we do
< sipa> pruning always triggers a flush
< TD-Linux> sipa, were you pruning on your c2? if so did you hit https://github.com/bitcoin/bitcoin/issues/9001
< sipa> TD-Linux: i'm not pruning (yet)
< sipa> 128GB microsd card ftw
< luke-jr> I wonder if there's a safe way to catch bad_alloc, drop dbcache/mempool stuff, and resume cleanly?
< bitcoin-git> [bitcoin] practicalswift opened pull request #9987: Remove unused code (master...remove-unused-code) https://github.com/bitcoin/bitcoin/pull/9987