< bitcoin-git> [bitcoin] zenosage opened pull request #16422: test: remove redundant setup in addrman_tests (master...addrman_tests) https://github.com/bitcoin/bitcoin/pull/16422
< bitcoin-git> [bitcoin] fanquake pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/e5abb59a9a66...89d7229c9c18
< bitcoin-git> bitcoin/master 024ecd7 Jonas Schnelli: QA: Fix race condition in wallet_encryption test
< bitcoin-git> bitcoin/master 89d7229 fanquake: Merge #16420: QA: Fix race condition in wallet_encryption test
< bitcoin-git> [bitcoin] fanquake merged pull request #16420: QA: Fix race condition in wallet_encryption test (master...2019/07/wallet_enc_test_fix) https://github.com/bitcoin/bitcoin/pull/16420
< aj> yay!
< meshcollider> achow101: There is a wallet meeting scheduled tomorrow
< meshcollider> But i will be on a bus so I won't be able to host it
< meshcollider> Someone else want to volunteer?
< kallewoof> Would it be horrible to remove the assume valid stuff from bitcoin core? It feels like signature validation is pretty fast these days.
< mryandao> why not set the default to off instead?
< sipa> kallewoof: really?
< sipa> i think it's around a week of CPU time to verify all historical signatures
< sipa> on a modern x86 cpu
< sipa> if you have 32 cores that's perhaps acceptable
< kallewoof> It's that big of a difference? I must have misheard numbers then.
< sipa> it's 50 microseconds or so per signature check
< sipa> assuming a billion sigcheck (rough guess) in the chain, that's 5 days of CPU time (divided by the number of threads to get real time)
< mryandao> is there a plot that tracks number of sigcheck ops to date?
< jonasschnelli> MarcoFalke: are you sure CCACHE_SIZE is the right directive: https://github.com/bitcoin/bitcoin/blob/master/.travis.yml#L49?
< jonasschnelli> maybe it's travis special
< jonasschnelli> But it looks like that the ccache env var would be CCACHE_MAXSIZE (instead of CCACHE_SIZE) https://ccache.dev/manual/3.4.html#_cache_size_management
< fanquake> jonasschnelli: hard to tell. Looks like there are usages of either VAR in .travis.yml files on github. i.e https://github.com/search?l=yaml&q=CCACHE_MAXSIZE&type=Code
< jonasschnelli> maybe set both *duck*
< kallewoof> CCACHE_MAXSIZE is correct according to man ccache
< kallewoof> Just me or is travis not creating new jobs when pushing to a PR branch..? (it creates local instances for my own repo but not for bitcoin).
< bitcoin-git> [bitcoin] fanquake pushed 4 commits to master: https://github.com/bitcoin/bitcoin/compare/89d7229c9c18...59ce537a4994
< bitcoin-git> bitcoin/master 5efcb77 Matt Corallo: Disable bloom filtering by default.
< bitcoin-git> bitcoin/master f27309f Matt Corallo: Move DEFAULT_PEERBLOOMFILTERS from validation.h to net_processing.h
< bitcoin-git> bitcoin/master bead32e Matt Corallo: Add release notes for DEFAULT_BLOOM change
< bitcoin-git> [bitcoin] fanquake merged pull request #16152: Disable bloom filtering by default. (master...2019-06-fix-dos) https://github.com/bitcoin/bitcoin/pull/16152
< jonasschnelli> Would someone mind if I add a webhook to bitcoin/bitcoin (a "readonly" webhook) to drive the CI i have built in the last weeks?
< jonasschnelli> It's non-invasive (as said, readonly)
< jonasschnelli> ^ wumpus, MarcoFalke, fanquake
< jonasschnelli> the CI is currently under heave development: https://bitcoinbuilds.org/
< jonasschnelli> But I want to test it with some load and integrate building all PR pushes
< fanquake> jonasschnelli: Interesting. Is the source for the site going to live here: https://github.com/jonasschnelli/bitcoin-core-ci?
< fanquake> What hardware are you running the builds on atm?
< jonasschnelli> Yes. The source code will end up at that repo (as soon as its ready)
< jonasschnelli> Its running on a relatively powerful physical host
< jonasschnelli> It's based on KVM
< jonasschnelli> Full custom software though
< fanquake> The site looks pretty nice. At least looking at large logs is more responsive than Travis 👍
< fanquake> I assume you'll be adding macOS and other builds later on?
< jonasschnelli> Yeah... travis logs are a nightmare.
< jonasschnelli> fanquake: Yes. All possible. Just caping to three platforms right now then later expand
< fanquake> Cool. I'm not opposed to a read-only hook if you want to stress / load test for a week or two. Probably worth bring up at a meeting after that.
< jonasschnelli> Yes. We can discuss that next thursday
< kallewoof> Weird. Github claims "all checks passed" for #16411 but in reality, the travis check never actually executed, only the appveyor one. :/
< gribble> https://github.com/bitcoin/bitcoin/issues/16411 | Signet support by kallewoof · Pull Request #16411 · bitcoin/bitcoin · GitHub
< fanquake> Travis has a habit of doing weird things like that. Sometimes it'll show Travis failing when in fact all tests have passed.
< bitcoin-git> [bitcoin] MarcoFalke pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/59ce537a4994...c7b7cf299a9e
< bitcoin-git> bitcoin/master 5c3c24c zenosage: test: remove redundant setup in addrman_tests
< bitcoin-git> bitcoin/master c7b7cf2 MarcoFalke: Merge #16422: test: remove redundant setup in addrman_tests
< bitcoin-git> [bitcoin] MarcoFalke merged pull request #16422: test: remove redundant setup in addrman_tests (master...addrman_tests) https://github.com/bitcoin/bitcoin/pull/16422
< bitcoin-git> [bitcoin] laanwj pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/c7b7cf299a9e...f4b1fe7165c8
< bitcoin-git> bitcoin/master a52818c tecnovert: net: Make poll in InterruptibleRecv only filter for POLLIN events.
< bitcoin-git> bitcoin/master f4b1fe7 Wladimir J. van der Laan: Merge #16412: net: Make poll in InterruptibleRecv only filter for POLLIN e...
< bitcoin-git> [bitcoin] laanwj merged pull request #16412: net: Make poll in InterruptibleRecv only filter for POLLIN events. (master...bitcoin-poll) https://github.com/bitcoin/bitcoin/pull/16412
< bitcoin-git> [bitcoin] laanwj pushed 1 commit to 0.18: https://github.com/bitcoin/bitcoin/compare/3f76160087c0...063c8ce7a054
< bitcoin-git> bitcoin/0.18 063c8ce tecnovert: net: Make poll in InterruptibleRecv only filter for POLLIN events.
< MarcoFalke> 0.18.1 can go
< wumpus> MarcoFalke: what about #16414?
< gribble> https://github.com/bitcoin/bitcoin/issues/16414 | 0.18: wallet: Fix -maxtxfee check by moving it to CWallet::CreateTransaction by promag · Pull Request #16414 · bitcoin/bitcoin · GitHub
< wumpus> (I think that is the last one)
< wumpus> going to do pre-rc1 translations update
< bitcoin-git> [bitcoin] laanwj pushed 3 commits to 0.18: https://github.com/bitcoin/bitcoin/compare/063c8ce7a054...a6cba19831da
< bitcoin-git> bitcoin/0.18 641b2ff Wladimir J. van der Laan: qt: pre-rc1 translations update
< bitcoin-git> bitcoin/0.18 aa2d12a Wladimir J. van der Laan: build: Bump version to 0.18.1rc1
< bitcoin-git> bitcoin/0.18 a6cba19 Wladimir J. van der Laan: doc: Update manpages for rc1
< BlueMatt> wumpus: plz2merge #15681
< gribble> https://github.com/bitcoin/bitcoin/issues/15681 | [mempool] Allow one extra single-ancestor transaction per package by TheBlueMatt · Pull Request #15681 · bitcoin/bitcoin · GitHub
< elichai2> sipa: kallewoof that's not right
< wumpus> looks like recent changes in master broke compatibility with c-lightning
< wumpus> BlueMatt: looking
< elichai2> I did it yesterday. did a full IBD with assumbalid=0 from scratch in ~6 hours
< elichai2> *assumevalid
< wumpus> user@medea:~ % ./launch-lightning.sh
< wumpus> bitcoin-cli getblockchaininfo: invalid response
< wumpus> not sure why yet
< elichai2> (fair point that I do have 8 cores and 16 threads, NVMe and a ~600MiB internet connection and I set dbcache=4096)
< achow101> Could someone help me figure out where the memory leak being reported here is: https://travis-ci.org/bitcoin/bitcoin/jobs/560780816 ?
< achow101> It seems like it's something happening in std::map but I don't know why it happens here
< bitcoin-git> [bitcoin] laanwj pushed 2 commits to master: https://github.com/bitcoin/bitcoin/compare/f4b1fe7165c8...51a6e2c41929
< bitcoin-git> bitcoin/master 50cede3 Matt Corallo: [mempool] Allow one extra single-ancestor transaction per package
< bitcoin-git> bitcoin/master 51a6e2c Wladimir J. van der Laan: Merge #15681: [mempool] Allow one extra single-ancestor transaction per pa...
< BlueMatt> woooo
< bitcoin-git> [bitcoin] laanwj merged pull request #15681: [mempool] Allow one extra single-ancestor transaction per package (master...2019-03-lightning-policy) https://github.com/bitcoin/bitcoin/pull/15681
< sipa> elichai2: what kind of hardware?
< sipa> ah, just saw the later message
< sipa> elichai2: 16 * 6 hours = 4 days
< sipa> i don't think i'm far off
< elichai2> Most laptops these days have at least 4 threads. And I'm not even entirely sure what was the bottleneck because nothing maxed out
< elichai2> Probably the peers internet connection
< sipa> with 4 threads it would be a day...
< sipa> 6 hours is already pretty terrible on itself
< sipa> though admittedly there is little we can do about the time to download the chain (except assumeutxo like security model changes)
< sipa> most laptop cpus will also downclock when many cores are used at once, so it may not be fair to assume more threads linearly oncreases speed
< elichai2> one thing I saw is that there's no logic that chooses prefers peers by their bandwidth
< elichai2> so when I manually when to https://bitnodes.earn.com and looked for closer peers that share the same ISP and added then via `addnode` I got faster download
< elichai2> and if everyone had good cpus we could try and use compression algorithms before sending the blocks around to move some of the weight from the internet to the cpu
< sipa> elichai2: not explicitly, though there is the "stall detection" logic which tends to kick out the slowest peers from time to time in many cases
< sipa> elichai2: yeah, we have someone working on transaction compression (though primarily aimed at satellite links, where a large amount of CPU to gain some bandwidth is much more acceptable than on the general P2P network), still some of the results are likely useful for a P2P compression mechanism too
< elichai2> cool. I was actually impressed by the 6 hours. really thought it will take me at least a day with the `assumevalid=0`
< wumpus> so no-one else has had issues with c-lightning and recent bitcoind yet?
< wumpus> trying to bisect it
< dongcarl> wumpus: Thinking about implementation of addrv2... I'm thinking: rework CAddress to be more enum-y and less weird-IPv6-y, and do the weird-IPV6-y tricks in the serialization code only if `s.GetVersion() < GOSSIP_ADDRV2_VERSION`
< dongcarl> lmk if that makes sense
< bitcoin-git> [bitcoin] MarcoFalke opened pull request #16424: build: Treat -Wswitch as error when --enable-werror (master...1907-buildSwitchError) https://github.com/bitcoin/bitcoin/pull/16424
< bitcoin-git> [bitcoin] MarcoFalke opened pull request #16425: addrman: Add sleep to while(1) loops (master...1907-addrmanNomNomCpu) https://github.com/bitcoin/bitcoin/pull/16425
< wumpus> dongcarl: rust enums would be really nice here
< dongcarl> wumpus: Haha yeah... But, alas...
< wumpus> dongcarl: but I don't think you need to change the internal representation for this
< wumpus> oh wait, you do, ofc
< * dongcarl> almost got gaslighted
< wumpus> sorry
< sipa> boost::variant?
< * sipa> ducks
< BlueMatt> lets just use rust?
< wumpus> tfw no std::variant
< BlueMatt> dont we already have rust build support? time to start using it :p
< BlueMatt> (only like 90% joke, fwiw)
< * dongcarl> is going to do it the stupid way first
< sipa> just a uint256 and an enum?
< sipa> how will you deal with serialization in addrman.dat?
< sipa> peers.dat, i guess
< dongcarl> yes just a uint256 and an enum (do we have uint256's?)
< * sipa> points to uint256.h
< dongcarl> oh cool!
< sipa> we use 256-bit values here and there in bitcoin :)
< dongcarl> lol
< dongcarl> I haven't looked at peers.dat serialization...
< dongcarl> What are potential problems?
< dongcarl> Oh... we need to maintain back-compat I guess?
< sipa> yes
< sipa> well, not backward compatibility
< sipa> that'll be impossible anyway
< sipa> but you do need forward compatibility
< sipa> (new code will need to be able to read an old peers.dat)
< dongcarl> Right, sorry that's what I meant... What's usually done in this case? One-off migration of peers.dat?
< sipa> up to you
< dongcarl> Cool. Thanks for tips!
< sipa> if there is a way to serialize a CADdress (haven't checked if there is) that would be compatible, that would be great
< dongcarl> sipa: You mean serialize an addrv2 in a way that's back-compatible with addv1?
< sipa> yes
< sipa> oh, there is
< sipa> SER_DISK mode for CAddress stores a version number
< bitcoin-git> [bitcoin] MarcoFalke closed pull request #16425: addrman: Add sleep to while(1) loops (master...1907-addrmanNomNomCpu) https://github.com/bitcoin/bitcoin/pull/16425
< sipa> you can leverage that to just use old serializtion for things compatible with it
< sipa> and new serialization otherwise
< sipa> (which you can define i guess identical to how the addrv2 bip encodes addressrs)
< sipa> that would be automatically forward compatible
< sipa> and backward compatible as long as no new style addresses are stored
< dongcarl> sipa: Ah... So if I understand you correctly, this way disk serialization of ipv4, v6, and torv2 will be in the older format... Which means if I generate a peers.dat in a newer client and feed it to an older one, the older client will pick up everything except torv3, i2p, and CJDNS?
< sipa> it will just fail to deserialize entirely if it contains torv3/i2p/cjdns
< sipa> as it has no idea about the length of the field it's trying to read
< sipa> so i think there are two ways:
< * dongcarl> listening
< sipa> 1) have a global peers.dat-wide field (it has an nVersion for this) that says "this file uses new encoding", and then all addresses in it use new encoding; you'd have read support for the old format, but always write in the new format
< sipa> 2) you use the per-address nVersion field to distinguish between old and new (right now the nVersion field there is just the client version number which is a terrible practice anyway; define some cutoff equivalent to say 0.99.0 which means "new format" and stick to 0.18.x for old format), but use old encoding when writing tor/ipv4/ipv6
< sipa> the first idea is perhaps simpler and much more compact
< sipa> as ipv4 is hugely wasteful in the old serialization
< dongcarl> Yeah that makes sense
< dongcarl> I'm gunna aim for #1 first
< gribble> https://github.com/bitcoin/bitcoin/issues/1 | JSON-RPC support for mobile devices ("ultra-lightweight" clients) · Issue #1 · bitcoin/bitcoin · GitHub
< dongcarl> sipa: Thanks for helping think this thru!
< sipa> to switch between old and new encoding, i suggest not actually using nVersion directly (it's messy...), but instead ORing some field flag into like, like SERIALIZE_TRANSACTION_NO_WITNESS for transactions
< sipa> then you can also use OverrideStream in the P2P code to select between old and new encoding based on what the peer supports
< sipa> dongcarl: i mean, don't use the p2p protocol nVersion directly to select between features
< sipa> that feels like an enormous layer violation, and makes things messy in addrman (where nVersion is the _client_ version, while on the wire it's the protocol version...)
< sipa> even if at connection time nVersion is used to signal readiness for the new protocol, set a flag in CNode or so to remember what addr protocol is used, and then explicitly choose a flag to set when serializing addr messages based on that
< * dongcarl> reading and trying to understand
< sipa> dongcarl: basically my opinion is that it's a mistake that there is a single "stream version" number that influences serialization
< dongcarl> sipa: You mean like our existing logic for CADDR_TIME_VERSION?
< dongcarl> That's the mistake?
< sipa> yes
< sipa> well, that's historical, and dates from a time when client versions and protocol versions were the same thing
< sipa> but right now those two are distinct things, and it's not the CAddress that should know exactly which protocols require which serializations
< sipa> it should be the protocol implementations that decide that, and just tell CAddress whether to use v1 or v2 (which is something CAddress should know about)
< dongcarl> Okay, I think I understand that. What's a good way of passing the v1 vs v2 info to CAddress so it knows how to serialize?
< sipa> by essentially using the high bits of nVersion as a bit field, and using one bit in it to indicate which encoding to use
< sipa> like SERIALIZE_TRANSACTION_NO_WITNESS (which is equal to 0x40000000)
< dongcarl> Oh I see... Seems hacky but there's no way we're serializing a transaction while we're also serializing an address so that's okay, right?
< sipa> right, they don't need to live in the same namespace
< sipa> for segwit, based on whether segwit support was negotiated with the peer (and/or whether the witness flag in a CInv request is set), a wrapper makes the stream temporarily report that flag in its nVersion
< dongcarl> Okay that makes sense!
< dongcarl> sipa: Glad you pointed this out because I was about to copy the CADDR_TIME_VERSION logic
< sipa> for example, see the line in net_processing:
< sipa> connman->PushMessage(pfrom, msgMaker.Make(SERIALIZE_TRANSACTION_NO_WITNESS, NetMsgType::BLOCK, *pblock));
< sipa> or
< sipa> int nSendFlags = (inv.type == MSG_TX ? SERIALIZE_TRANSACTION_NO_WITNESS : 0);
< sipa> connman->PushMessage(pfrom, msgMaker.Make(nSendFlags, NetMsgType::TX, *mi->second));
< dongcarl> Ah! Very good examples... I should probably define a SERIALIZE_ADDR_AS_V2 or something
< sipa> right
< sipa> in the v2 stuff you can probably drop the nVersion/SER_DISK/... logic too
< sipa> no need to encode a per-addr version number if that version is being communicated externally
< dongcarl> sipa: True!
< * dongcarl> is glad he understood
< sipa> much of this is my personal opinion btw, i don't want to give a false impression that this approach is some commonly agreed upon way of doing things
< dongcarl> sipa: It's certainly given me a solid direction for implementation... Much better than me designing in the dark, and I'm happy to iterate once I have an MVP working. :-)
< sipa> great
< bitcoin-git> [bitcoin] MarcoFalke closed pull request #16409: Remove mempool expiry, treat txs as replaceable instead (master...1907-noExpiryButReplace) https://github.com/bitcoin/bitcoin/pull/16409