< kallewoof>
CCACHE_MAXSIZE is correct according to man ccache
< kallewoof>
Just me or is travis not creating new jobs when pushing to a PR branch..? (it creates local instances for my own repo but not for bitcoin).
< fanquake>
What hardware are you running the builds on atm?
< jonasschnelli>
Yes. The source code will end up at that repo (as soon as its ready)
< jonasschnelli>
Its running on a relatively powerful physical host
< jonasschnelli>
It's based on KVM
< jonasschnelli>
Full custom software though
< fanquake>
The site looks pretty nice. At least looking at large logs is more responsive than Travis 👍
< fanquake>
I assume you'll be adding macOS and other builds later on?
< jonasschnelli>
Yeah... travis logs are a nightmare.
< jonasschnelli>
fanquake: Yes. All possible. Just caping to three platforms right now then later expand
< fanquake>
Cool. I'm not opposed to a read-only hook if you want to stress / load test for a week or two. Probably worth bring up at a meeting after that.
< jonasschnelli>
Yes. We can discuss that next thursday
< kallewoof>
Weird. Github claims "all checks passed" for #16411 but in reality, the travis check never actually executed, only the appveyor one. :/
< bitcoin-git>
bitcoin/master a52818c tecnovert: net: Make poll in InterruptibleRecv only filter for POLLIN events.
< bitcoin-git>
bitcoin/master f4b1fe7 Wladimir J. van der Laan: Merge #16412: net: Make poll in InterruptibleRecv only filter for POLLIN e...
< bitcoin-git>
[bitcoin] laanwj merged pull request #16412: net: Make poll in InterruptibleRecv only filter for POLLIN events. (master...bitcoin-poll) https://github.com/bitcoin/bitcoin/pull/16412
< bitcoin-git>
bitcoin/master 50cede3 Matt Corallo: [mempool] Allow one extra single-ancestor transaction per package
< bitcoin-git>
bitcoin/master 51a6e2c Wladimir J. van der Laan: Merge #15681: [mempool] Allow one extra single-ancestor transaction per pa...
< BlueMatt>
woooo
< bitcoin-git>
[bitcoin] laanwj merged pull request #15681: [mempool] Allow one extra single-ancestor transaction per package (master...2019-03-lightning-policy) https://github.com/bitcoin/bitcoin/pull/15681
< sipa>
elichai2: what kind of hardware?
< sipa>
ah, just saw the later message
< sipa>
elichai2: 16 * 6 hours = 4 days
< sipa>
i don't think i'm far off
< elichai2>
Most laptops these days have at least 4 threads. And I'm not even entirely sure what was the bottleneck because nothing maxed out
< elichai2>
Probably the peers internet connection
< sipa>
with 4 threads it would be a day...
< sipa>
6 hours is already pretty terrible on itself
< sipa>
though admittedly there is little we can do about the time to download the chain (except assumeutxo like security model changes)
< sipa>
most laptop cpus will also downclock when many cores are used at once, so it may not be fair to assume more threads linearly oncreases speed
< elichai2>
one thing I saw is that there's no logic that chooses prefers peers by their bandwidth
< elichai2>
so when I manually when to https://bitnodes.earn.com and looked for closer peers that share the same ISP and added then via `addnode` I got faster download
< elichai2>
and if everyone had good cpus we could try and use compression algorithms before sending the blocks around to move some of the weight from the internet to the cpu
< sipa>
elichai2: not explicitly, though there is the "stall detection" logic which tends to kick out the slowest peers from time to time in many cases
< sipa>
elichai2: yeah, we have someone working on transaction compression (though primarily aimed at satellite links, where a large amount of CPU to gain some bandwidth is much more acceptable than on the general P2P network), still some of the results are likely useful for a P2P compression mechanism too
< elichai2>
cool. I was actually impressed by the 6 hours. really thought it will take me at least a day with the `assumevalid=0`
< wumpus>
so no-one else has had issues with c-lightning and recent bitcoind yet?
< wumpus>
trying to bisect it
< dongcarl>
wumpus: Thinking about implementation of addrv2... I'm thinking: rework CAddress to be more enum-y and less weird-IPv6-y, and do the weird-IPV6-y tricks in the serialization code only if `s.GetVersion() < GOSSIP_ADDRV2_VERSION`
< dongcarl>
lmk if that makes sense
< bitcoin-git>
[bitcoin] MarcoFalke opened pull request #16424: build: Treat -Wswitch as error when --enable-werror (master...1907-buildSwitchError) https://github.com/bitcoin/bitcoin/pull/16424
< sipa>
you can leverage that to just use old serializtion for things compatible with it
< sipa>
and new serialization otherwise
< sipa>
(which you can define i guess identical to how the addrv2 bip encodes addressrs)
< sipa>
that would be automatically forward compatible
< sipa>
and backward compatible as long as no new style addresses are stored
< dongcarl>
sipa: Ah... So if I understand you correctly, this way disk serialization of ipv4, v6, and torv2 will be in the older format... Which means if I generate a peers.dat in a newer client and feed it to an older one, the older client will pick up everything except torv3, i2p, and CJDNS?
< sipa>
it will just fail to deserialize entirely if it contains torv3/i2p/cjdns
< sipa>
as it has no idea about the length of the field it's trying to read
< sipa>
so i think there are two ways:
< * dongcarl>
listening
< sipa>
1) have a global peers.dat-wide field (it has an nVersion for this) that says "this file uses new encoding", and then all addresses in it use new encoding; you'd have read support for the old format, but always write in the new format
< sipa>
2) you use the per-address nVersion field to distinguish between old and new (right now the nVersion field there is just the client version number which is a terrible practice anyway; define some cutoff equivalent to say 0.99.0 which means "new format" and stick to 0.18.x for old format), but use old encoding when writing tor/ipv4/ipv6
< sipa>
the first idea is perhaps simpler and much more compact
< sipa>
as ipv4 is hugely wasteful in the old serialization
< dongcarl>
sipa: Thanks for helping think this thru!
< sipa>
to switch between old and new encoding, i suggest not actually using nVersion directly (it's messy...), but instead ORing some field flag into like, like SERIALIZE_TRANSACTION_NO_WITNESS for transactions
< sipa>
then you can also use OverrideStream in the P2P code to select between old and new encoding based on what the peer supports
< sipa>
dongcarl: i mean, don't use the p2p protocol nVersion directly to select between features
< sipa>
that feels like an enormous layer violation, and makes things messy in addrman (where nVersion is the _client_ version, while on the wire it's the protocol version...)
< sipa>
even if at connection time nVersion is used to signal readiness for the new protocol, set a flag in CNode or so to remember what addr protocol is used, and then explicitly choose a flag to set when serializing addr messages based on that
< * dongcarl>
reading and trying to understand
< sipa>
dongcarl: basically my opinion is that it's a mistake that there is a single "stream version" number that influences serialization
< dongcarl>
sipa: You mean like our existing logic for CADDR_TIME_VERSION?
< dongcarl>
That's the mistake?
< sipa>
yes
< sipa>
well, that's historical, and dates from a time when client versions and protocol versions were the same thing
< sipa>
but right now those two are distinct things, and it's not the CAddress that should know exactly which protocols require which serializations
< sipa>
it should be the protocol implementations that decide that, and just tell CAddress whether to use v1 or v2 (which is something CAddress should know about)
< dongcarl>
Okay, I think I understand that. What's a good way of passing the v1 vs v2 info to CAddress so it knows how to serialize?
< sipa>
by essentially using the high bits of nVersion as a bit field, and using one bit in it to indicate which encoding to use
< sipa>
like SERIALIZE_TRANSACTION_NO_WITNESS (which is equal to 0x40000000)
< dongcarl>
Oh I see... Seems hacky but there's no way we're serializing a transaction while we're also serializing an address so that's okay, right?
< sipa>
right, they don't need to live in the same namespace
< sipa>
for segwit, based on whether segwit support was negotiated with the peer (and/or whether the witness flag in a CInv request is set), a wrapper makes the stream temporarily report that flag in its nVersion
< dongcarl>
Okay that makes sense!
< dongcarl>
sipa: Glad you pointed this out because I was about to copy the CADDR_TIME_VERSION logic
< sipa>
for example, see the line in net_processing:
< dongcarl>
Ah! Very good examples... I should probably define a SERIALIZE_ADDR_AS_V2 or something
< sipa>
right
< sipa>
in the v2 stuff you can probably drop the nVersion/SER_DISK/... logic too
< sipa>
no need to encode a per-addr version number if that version is being communicated externally
< dongcarl>
sipa: True!
< * dongcarl>
is glad he understood
< sipa>
much of this is my personal opinion btw, i don't want to give a false impression that this approach is some commonly agreed upon way of doing things
< dongcarl>
sipa: It's certainly given me a solid direction for implementation... Much better than me designing in the dark, and I'm happy to iterate once I have an MVP working. :-)