< bitcoin-git> [bitcoin] hebasto opened pull request #16077: docs: Follow ISO/IEC 14882 terms and definitions (master...20190523-iso-parameter) https://github.com/bitcoin/bitcoin/pull/16077
< stevenroose> Does bitcoind have an RPC that either (1) gives the xpub of the master priv (the one that dumpwallet gives) or (2) allows calculating the xpub of an xpriv?
< bitcoin-git> [bitcoin] LongShao007 opened pull request #16078: replace tx hash with txid in test rawtransaction (master...pytest) https://github.com/bitcoin/bitcoin/pull/16078
< achow101> stevenroose: no
< stevenroose> achow101: thanks
< bitcoin-git> [bitcoin] stevenroose opened pull request #16079: wallet_balance.py: Prevent edge cases (master...wallet-balance-test) https://github.com/bitcoin/bitcoin/pull/16079
< dongcarl> cfields: Trying to understand your fork of cctools-port... Out of the 5 commits that you made that's no upstream, they've taken care of openssl and uuid (the first 2 commits), but I'm having trouble understanding the remaining 3... Namely: 3a2152b8e4d74601d9976f75aa7c30fd2b54733f, 81589bba090b29989dee31d8f04a78d917b34589, and ee31ae567931c426136c94aad457c7b51d844beb
< bitcoin-git> [bitcoin] MarcoFalke pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/12fd4bbd1ed9...b4223dd5f113
< bitcoin-git> bitcoin/master bb41e63 Steven Roose: wallet_balance.py: Prevent edge cases
< bitcoin-git> bitcoin/master b4223dd MarcoFalke: Merge #16079: wallet_balance.py: Prevent edge cases
< bitcoin-git> [bitcoin] MarcoFalke merged pull request #16079: wallet_balance.py: Prevent edge cases (master...wallet-balance-test) https://github.com/bitcoin/bitcoin/pull/16079
< bitcoin-git> [bitcoin] MarcoFalke pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/b4223dd5f113...0b058ba69d85
< bitcoin-git> bitcoin/master e23809a Chris Capobianco: [rpc] deriveaddresses: Correct descriptor checksum in RPCExamples
< bitcoin-git> bitcoin/master 0b058ba MarcoFalke: Merge #16024: [rpc] deriveaddresses: Correction of descriptor checksum in ...
< bitcoin-git> [bitcoin] MarcoFalke merged pull request #16024: [rpc] deriveaddresses: Correction of descriptor checksum in RPC example (master...2019-05-14-deriveaddresses-example-checksum) https://github.com/bitcoin/bitcoin/pull/16024
< bitcoin-git> [bitcoin] jonatack opened pull request #16080: build/doc: update bitcoin_config.h packages, release process (master...bitcoin_config-and-release_process-updates) https://github.com/bitcoin/bitcoin/pull/16080
< bitcoin-git> [bitcoin] MarcoFalke pushed 5 commits to master: https://github.com/bitcoin/bitcoin/compare/0b058ba69d85...65c4bbe629bb
< bitcoin-git> bitcoin/master cc25885 practicalswift: Move LockAnnotation from threadsafety.h (imported code) to sync.h (our cod...
< bitcoin-git> bitcoin/master 3a80944 practicalswift: Move LockAnnotation to make it reflect the truth
< bitcoin-git> bitcoin/master de9b5db practicalswift: Make sure the compile-time locking promises given via LockAnnotation:s hol...
< bitcoin-git> [bitcoin] MarcoFalke merged pull request #16034: refactoring: Rename LockAnnotation to LockAssertion and add run-time check to it (master...make-sure-LockAnnotation-promises-are-truthful) https://github.com/bitcoin/bitcoin/pull/16034
< bitcoin-git> [bitcoin] fanquake reopened pull request #13442: Convert the 1-way SSE4 SHA256 code from asm to intrinsics (master...201806_sse4intrin) https://github.com/bitcoin/bitcoin/pull/13442
< wumpus> #startmeeting
< lightningbot> Meeting started Thu May 23 19:00:30 2019 UTC. The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
< lightningbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
< sipa> ohai
< wumpus> #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier jl2012 achow101 meshcollider jnewbery maaku fanquake promag provoostenator aj Chris_Stewart_5 dongcarl gwillen jamesob ken281221 ryanofsky gleb moneyball kvaciral
< achow101> hi
< promag> hi
< fanquake> Hi
< achow101> only here for 30 min or so (if the meeting even goes that long)
< meshcollider> hi
< jamesob> hi
< gwillen> hi
< wumpus> proposed topics? (none on moneyball 's list)
< phantomcircuit> hi
< jonasschnelli> hi
< Chris_Stewart_5> hello
< wumpus> hello
< wumpus> #topic High priority for review
< wumpus> current PRs: #15427 #15741 #15759 #15024 #15649
< gribble> https://github.com/bitcoin/bitcoin/issues/15427 | Add support for descriptors to utxoupdatepsbt by sipa · Pull Request #15427 · bitcoin/bitcoin · GitHub
< moneyball> hi
< gribble> https://github.com/bitcoin/bitcoin/issues/15741 | Batch write imported stuff in importmulti by achow101 · Pull Request #15741 · bitcoin/bitcoin · GitHub
< gribble> https://github.com/bitcoin/bitcoin/issues/15759 | [p2p] Add 2 outbound blocks-only connections by sdaftuar · Pull Request #15759 · bitcoin/bitcoin · GitHub
< gribble> https://github.com/bitcoin/bitcoin/issues/15024 | Allow specific private keys to be derived from descriptor by meshcollider · Pull Request #15024 · bitcoin/bitcoin · GitHub
< gribble> https://github.com/bitcoin/bitcoin/issues/15649 | Add ChaCha20Poly1305@Bitcoin AEAD by jonasschnelli · Pull Request #15649 · bitcoin/bitcoin · GitHub
< kanzure> hi
< jamesob> can I ask that #15976 be added? it's prone to rebase conflicts and pretty close to ack-threshold for merge (I think)
< gribble> https://github.com/bitcoin/bitcoin/issues/15976 | refactor: move methods under CChainState (pt. 1) by jamesob · Pull Request #15976 · bitcoin/bitcoin · GitHub
< jamesob> (and blocks the assumeutxo project)
< wumpus> jamesob: of course
< jamesob> wumpus: thanks
< fanquake> done
< dongcarl> Could we add #16059? Small but without it linux gitian builds are broken right now
< gribble> https://github.com/bitcoin/bitcoin/issues/16059 | configure: Fix thread_local detection by dongcarl · Pull Request #16059 · bitcoin/bitcoin · GitHub
< wumpus> dongcarl: sure
< wumpus> fanquake: thanks
< jnewbery> hi
< fanquake> I think that can be merged quite soon. I'm testing it at the moment.
< fanquake> Have added to high prio.
< dongcarl> thank you
< wumpus> why are we using thread_local again btw?
< jamesob> for thread names
< wumpus> I thought that was removed at some point
< jamesob> cory and I came up with a way to do without at some point but it was pretty convoluted
< fanquake> It's also linux and openbsd? only atm. Broken on win, freebsd and we can't use it on macOS until we are using a newer sdk.
< wumpus> doesn't pthread have a way to keep track of thread names?
< wumpus> I thought that was how we did that, anyway
< jonasschnelli> fanquake: you mean SDK for the depends build?
< wumpus> that'll also show up in top and such
< jamesob> there's some posix key-value store API we were using with the pthreads id to avoid thread_local, but it ended up being a lot of code IIRC
< wumpus> pthread_setname or such
< wumpus> pthread_getname_np
< fanquake> jonasschnelli yes. Although now I think about it, we are building against 10.11 which I think means we can use it. Have to double check.
< jonasschnelli> Also since this is nice for debugging, the depends build matters not too much
< wumpus> do we have a whole util to keep track of thread names? wow
< ryanofsky> #16059 is a straightforward fix, simpler than rewriting thread names imo...
< gribble> https://github.com/bitcoin/bitcoin/issues/16059 | configure: Fix thread_local detection by dongcarl · Pull Request #16059 · bitcoin/bitcoin · GitHub
< jamesob> ryanofsky: agree
< wumpus> I had no idea it was such a big issue, anyhow, any other topics?
< * luke-jr> for one finds thread names useful sometimes
< jamesob> we avoided doing pthread_getname because there are supposedly implementation problems (https://stackoverflow.com/questions/2369738/how-to-set-the-name-of-a-thread-in-linux-pthreads/7989973#7989973). see also the notes at the bottom of this PR description: https://github.com/bitcoin/bitcoin/pull/13099
< gribble> https://github.com/bitcoin/bitcoin/issues/7989973 | HTTP Error 404: Not Found
< wumpus> jamesob:thanks
< meshcollider> Not a topic, but I won't be able to attend tomorrows wallet meeting so could someone please volunteer to host it?
< wumpus> anyonw?
< sipa> i'll be here, if there's interest
< fanquake> Maybe #15993 for a topic? Has been through a few iterations but seems to be more ready now?
< jnewbery> I'll be there
< gribble> https://github.com/bitcoin/bitcoin/issues/15993 | net: Drop support of the insecure miniUPnPc versions by hebasto · Pull Request #15993 · bitcoin/bitcoin · GitHub
< wumpus> fanquake:I think that should just be merged?
< wumpus> fanquake: is there anything to discuss about it?
< luke-jr> did configure finally get updated?
< luke-jr> not that I see..
< wumpus> I don't know...
< MarcoFalke> Seems like it is just shuffling things around and not actually dropping support
< fanquake> wumpus I haven't tested any of the changes, just seemed there was discussion about which versions to drop support for, and wether other versions had actually been patched.
< wumpus> it's dropping support for <10, which is fine, the only thing controversial was <14
< luke-jr> looks like it would detect older miniupnpc libraries, then fail to compile
< wumpus> because that's still in debian table
< wumpus> stable
< fanquake> ok.
< fanquake> One other topic is suggestions for 0.18.1 backporting that aren't already in #16035
< gribble> https://github.com/bitcoin/bitcoin/issues/16035 | 0.18.1: Backports by MarcoFalke · Pull Request #16035 · bitcoin/bitcoin · GitHub
< * wumpus> would prefer to drop support for miniUPnP completely but, don't feel like having that discussion
< wumpus> #topic backport suggestions for 0.18.1
< wumpus> anything?
< promag> maybe #14984?
< gribble> https://github.com/bitcoin/bitcoin/issues/14984 | rpc: Speedup getrawmempool when verbose=true by promag · Pull Request #14984 · bitcoin/bitcoin · GitHub
< wumpus> it's not really a bugfix
< MarcoFalke> It is not strictly a bugfix.
< fanquake> Apparently that isn't actually a clean backport, and maybe not worth it if it's just perfomance gain.
< wumpus> unless performance is *really* horrible right now
< MarcoFalke> Only when the mempool is large ;)
< wumpus> if it's non-trivial to backport too then better not to, I think
< promag> right, large mempool, about 30% faster
< promag> no problem, just asked
< wumpus> no problem, thanks for suggesting something
< MarcoFalke> I am mostly waiting on those to get merged: https://github.com/bitcoin/bitcoin/milestone/41
< wumpus> are we planning on doing a 0.18.1 soon btw?
< wumpus> I mean, is there anything motivating it?
< MarcoFalke> nothing urgent, no
< promag> ops, forgot about #15453
< gribble> https://github.com/bitcoin/bitcoin/issues/15453 | Starting bitcoin-qt with -nowallet and then opening a wallet does not show the wallet · Issue #15453 · bitcoin/bitcoin · GitHub
< fanquake> A couple of bug fixes for potentially confusing behaviour, like #15952, but nothing catastrophic
< gribble> https://github.com/bitcoin/bitcoin/issues/15952 | Cant open wallet · Issue #15952 · bitcoin/bitcoin · GitHub
< MarcoFalke> oh, maybe the gcc compile bug?
< wumpus> MarcoFalke:yes maybe that
< MarcoFalke> But it seemed to only affect the tests, so ... flip a coin?
< wumpus> would be good to have that out of the way
< wumpus> we don't *know*
< MarcoFalke> jup
< fanquake> #15983
< gribble> https://github.com/bitcoin/bitcoin/issues/15983 | build with -fstack-reuse=none by MarcoFalke · Pull Request #15983 · bitcoin/bitcoin · GitHub
< fanquake> Maybe wait another week or two then, assuming outstanding PRs are merged, cut an rc1 ?
< wumpus> that definitely needs backport to 0.18
< MarcoFalke> It is the first thing in #16035
< gribble> https://github.com/bitcoin/bitcoin/issues/16035 | 0.18.1: Backports by MarcoFalke · Pull Request #16035 · bitcoin/bitcoin · GitHub
< wumpus> it's labeled for 0.17.2 but that's not very likely to happen
< wumpus> oh okay
< sipa> has anyone done general benchmarks to see the affect of -fstack-reuse=none ?
< fanquake> jamesob
< jamesob> I tried but I don't think I was doing it right
< MarcoFalke> sipa: We couldn't find any significant changes on our hardware
< wumpus> I doubt it affects performance at all
< sipa> MarcoFalke: that's not surprising, but good to confirm
< wumpus> just stack use
< gmaxwell> My only concern was that it would break some other performance optimization, sounds like it doesn't.
< wumpus> anyhow, upgrade to a compiler that doesn't have the bug (when there's one)
< wumpus> performance loss is preferable to random unpredictable corruption
< wumpus> certainly for something like bitcoin
< MarcoFalke> right
< wumpus> (I mean it's obviously different if say, validation becomes two times as slow or something extreme like that, but we'd have noticed)
< gmaxwell> if the performance loss were truly significant though, we might want to look for other alternative workarounds, good that we don't need to.
< wumpus> of course
< sipa> agree
< wumpus> any other topics?
< wumpus> avoiding hashing is always the best sha256 optimization :)
< jonasschnelli> ;-)
< wumpus> (but if it's worth it depends on what slice of the total the 2.44s is, if it's on the whole initial sync it's neglible)
< fanquake> If anyone can reproduce #16027 that might be handy. Has come up a couple times now.
< gribble> https://github.com/bitcoin/bitcoin/issues/16027 | client 0.18.0 crashes when computer wakes up from hibernation · Issue #16027 · bitcoin/bitcoin · GitHub
< MarcoFalke> Yeah, sync is ~hours, so a second doesn't matter at all
< wumpus> MarcoFalke: right
< promag> wumpus: right, agree at the moment saving a couple of seconds isn't worth it
< gmaxwell> that sounds like it maybe more more relevant to propagation at the tip.
< wumpus> and that's within the measuring noise isn't it?
< gmaxwell> e.g. time from getting a block to sending the first non-HB peer.
< wumpus> gmaxwell: yes, that latency would be useful to measure
< wumpus> if it makes a significant difference there then it could still be worth it
< gmaxwell> the fact that we're rehashing in general suggests we've got something kinda wrong, layout wise.
< wumpus> true
< promag> ok, I'll keep doing those profiles
< MarcoFalke> Could just cache the hash in the data structure?
< wumpus> depends also on how much more complicated it makes the code, or whether it increases memory use, etc
< wumpus> how much it increases memory use, of course caching increases memory use
< * MarcoFalke> ducks
< sipa> jamesob: mr profiling... do we have any way to generally benchmark block-propagation-speed-at-synced-tip ?
< jamesob> mmm short of parsing verbose debug.log, I don't think so. not built into bitcoinperf yet, at any rate :)
< promag> MarcoFalke: the hash is only necessary while processing/validating
< sipa> jamesob: i think it would be very valuable to have; especially for scenarios where the block's transactions have already been validated in the mempool
< sipa> as there are very relevant performance improvements there that'd generally be dwarfed by script validation
< MarcoFalke> sipa: Would that require two nodes?
< MarcoFalke> or are you talking about a micro benchmark
< jamesob> sipa: yeah, agree that's a metric worth tracking
< jamesob> happy to build something for it
< sipa> perhaps we should do some brainstorming about that, but maybe outside the meeting
< wumpus> jamesob: cool!
< fanquake> If anyone is interested in guix building, dongcarl has been doing a lot of work in #15277. Would be good to get some more builds to compare hashes with.
< gribble> https://github.com/bitcoin/bitcoin/issues/15277 | [Help Wanted] contrib: Enable building in Guix containers by dongcarl · Pull Request #15277 · bitcoin/bitcoin · GitHub
< MarcoFalke> sipa: If it is for bitcoinperf: https://github.com/chaincodelabs/bitcoinperf/issues
< gmaxwell> sipa: there are tests in bitcoinfibre.
< wumpus> fanquake: if there's clear instructions to test, I'm happy to try
< promag> ah btw, one thing that takes a while is base_uint<BITS>& base_uint<BITS>::operator>>=(unsigned int shift)
< dongcarl> wumpus: I'll update and link you
< wumpus> dongcarl: great !
< sipa> promag: inside the division logic for retargetting that's expected
< sipa> most of the time is in shifts
< gmaxwell> MarcoFalke: The logical place to cache hashes in block objects themselves, doing so there is irrelevant memory use wise (32 bytes in a 1.2MB object), but complicates constness.
< fanquake> I have a setup/container guide for a quick Guix setup here as well: https://github.com/fanquake/core-review/tree/master/guix
< gmaxwell> promag: if thats actually taking a non-trivial amount of some real usecase, the division could be made faster... it uses a fairly naieve algorithim right now.
< wumpus> agree it's irrelevant on block objects, it's mostly CBlockIndex where it counts because so many of them are permanently in memory
< jamesob> fanquake: cool!
< wumpus> fanquake:nice
< wumpus> any other topics?
< wumpus> #endmeeting
< lightningbot> Meeting ended Thu May 23 19:47:00 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< jamesob> gmaxwell: were those fibre tests you mentioned in the microbench or somewhere else?
< gmaxwell> microbench, IIRC
< gmaxwell> BlueMatt: I summon thee
< BlueMatt> hmm
< luke-jr> is it just me, or did gmaxwell's nick change colour?
< BlueMatt> gmaxwell: note that CBlock const-ness is already fucked (yay CheckBlock), not to say we should make it worse, though
< kanzure> btw, i'm still seeking submissions for topics for upcoming physical meetup.
< promag> gmaxwell: effective
< BlueMatt> whats the question?
< gmaxwell> BlueMatt: 12:38:07 < sipa> jamesob: mr profiling... do we have any way to generally benchmark block-propagation-speed-at-synced-tip ?
< gmaxwell> 12:41:03 < gmaxwell> sipa: there are tests in bitcoinfibre.
< gmaxwell> basically promag has an optimization that avoids rehashing the whole block when accepting it.
< gmaxwell> her rehashing the header perhaps actually.
< promag> gmaxwell: no the block
< promag> yap, the blockheader
< BlueMatt> when do we hash the block?
< BlueMatt> yea, blockheader, ok
< BlueMatt> there was a patch like 5 years ago that removed most of the duplicate hashes in acceptance path
< BlueMatt> how many are left?
< gmaxwell> header yea, so question is it worthwhile? obviously not against IBD, but maybe at tip relay for non-hb peers.
< bitcoin-git> [bitcoin] MarcoFalke pushed 3 commits to master: https://github.com/bitcoin/bitcoin/compare/65c4bbe629bb...fe47ae168b57
< bitcoin-git> bitcoin/master 3ee28c5 Jon Atack: build: bump bitcoin_config.h packages to v0.18
< bitcoin-git> bitcoin/master 8afca32 Jon Atack: doc: add bitcoin_config.h PACKAGE updates to release process
< bitcoin-git> bitcoin/master fe47ae1 MarcoFalke: Merge #16080: build/doc: update bitcoin_config.h packages, release process...
< BlueMatt> hmm, only things I have that would test this is looking at debug.log (which I have scripts that do, but its fibre-specific)
< gmaxwell> BlueMatt: I thought one of your fibre unit tests tested accepting a block.
< bitcoin-git> [bitcoin] MarcoFalke merged pull request #16080: build/doc: update bitcoin_config.h packages, release process (master...bitcoin_config-and-release_process-updates) https://github.com/bitcoin/bitcoin/pull/16080
< BlueMatt> they do not, I believe
< BlueMatt> they do test the equivalent of getting a block in via fibre and converting it to a CBlock for acceptance
< BlueMatt> but nothing further
< gmaxwell> ah, okay, my mistake then.
< jamesob> so if we were to bench acceptance-at-tip, would it make sense to have different fixtures for HB vs. LB?
< gmaxwell> Yes. Both figures are important and they're radically different.
< gmaxwell> Because HB relay happens before EVERYTHING, including validation.
< gmaxwell> (okay maybe my everything was still to uppercase)
< gmaxwell> too*
< jamesob> hm, in net_processing:SendMessages?
< gmaxwell> when I'm benchmarking I'm normally looking at logs with microsecond time on for the difference between these lines:
< gmaxwell> 2019-05-23T20:07:22.551652Z received: cmpctblock (16841 bytes) peer=1
< gmaxwell> 2019-05-23T20:07:22.571894Z sending cmpctblock (16841 bytes) peer=5
< gmaxwell> 2019-05-23T20:07:22.654285Z SendMessages: sending header 00000000000000000013886cc5bfa15d88db56f024a66b92f4b10ecf46922ff5 to peer=2
< gmaxwell> The first is getting the cmpct block, the second is sending the first HB block, and the third is the first non-HB announcement.
< jamesob> gmaxwell: ah okay, that helps
< gmaxwell> it might be interesting to setup a little stats datastructure that you can get at with an RPC.
< gmaxwell> That keeps a couple relevant times --- like the above two deltas for the last block, and maybe a couple EWMA averages of them. Then the python test framework could be used to do some benchmarking without relying on log parsing (which we break with frustrating frequency)
< gmaxwell> one thing to keep in mind is that cmpctblock is only the relevant start point when there are no missing txn, otherwise the blocktxn message is.
< gmaxwell> (if you measure cmpct->cmpct when there are missing txn, you'll include the response time of the peer and network latency-- which is itself also very interesting but perhaps doesn't make for a clean test :) )
< jamesob> gmaxwell: thanks for the pointers. I was actually about to start working on a bitcoind log parsing lib, but the RPC idea's interesting too. I wonder if maintaining some of those stats datastructures under an #if defined(DEBUG) wouldn't be too invasive
< jamesob> err probably not DEBUG; we might want a separate configure flag I guess
< gmaxwell> Ifdefs are kind of invasive, I don't think some operational stats would be bad to have at runtime regardless.