< GitHub127>
[bitcoin] sipa closed pull request #9053: IBD using chainwork instead of height and not using header timestamps (master...no_checkpoint_for_ibd) https://github.com/bitcoin/bitcoin/pull/9053
< GitHub156>
[bitcoin] laanwj closed pull request #9061: Ignore getheaders prior to passing all checkpoints. (master...FixGetheadersResponseWhenSyncing) https://github.com/bitcoin/bitcoin/pull/9061
< GitHub52>
bitcoin/master aff6584 Cory Fields: net: constify a few CNode vars to indicate that they're threadsafe
< GitHub52>
bitcoin/master 59ac5c5 Cory Fields: net: Use deterministic randomness for CNode's nonce, and make it const
< GitHub52>
bitcoin/master fcf61b8 Wladimir J. van der Laan: Merge #9050: net: make a few values immutable, and use deterministic randomness for the localnonce...
< GitHub54>
[bitcoin] laanwj closed pull request #9050: net: make a few values immutable, and use deterministic randomness for the localnonce (master...connman-const) https://github.com/bitcoin/bitcoin/pull/9050
< GitHub2>
[bitcoin] instagibbs opened pull request #9073: Trivial: Add common failure cases for rpc server connection failure (master...rpcnoconnectstring) https://github.com/bitcoin/bitcoin/pull/9073
< BlueMatt>
wumpus: any update on #8969? Should I solicit more review?
< gribble>
https://github.com/bitcoin/bitcoin/issues/8969 | Decouple peer-processing-logic from block-connection-logic (#2) by TheBlueMatt · Pull Request #8969 · bitcoin/bitcoin · GitHub
< BlueMatt>
thanks gribble!
<@wumpus>
seems ok to me
< BlueMatt>
lol, go to test some latency issues i was observing on fibre network....realize i cant because all the chinese peering in tokyo is currently entirely fucked
< BlueMatt>
oh well, good thing i have other routes....
< instagibbs>
where is the rpc test cache directory by default?
< sdaftuar>
qa/
< sdaftuar>
qa/cache, i guess
<@wumpus>
yes
< BlueMatt>
/tmp/${shit}
< BlueMatt>
oh, cache, yea, qa/
< BlueMatt>
test runs are /tmp/thing
< instagibbs>
ok, looking thanks
< instagibbs>
actually it appears to be top level directory
< instagibbs>
bitcoin/cache
< sdaftuar>
actually i think it depends on where you run the tests from, there are relative paths used in test_framework.py
< instagibbs>
Ah, I'm running individually from top-level
< instagibbs>
My cache somehow got corrupted and tests were failing due to this
< instagibbs>
and if it finds all the nodeX's directories, it never tries to rebuilt it
<@wumpus>
sdaftuar: I'm fairly sure that was made consistent recently
< GitHub136>
[bitcoin] TheBlueMatt opened pull request #9075: Decouple peer-processing-logic from block-connection-logic (#3) (master...net_processing_4) https://github.com/bitcoin/bitcoin/pull/9075
< BlueMatt>
second to last one ^
<@wumpus>
BlueMatt: you just keep reopening it after we merge it, isn't it :)
<@wumpus>
proposed topics?
< BlueMatt>
wumpus: E_NO_PARSE, but, yes, all this stuff is pretty much queued up, once one gets merged another gets pr'd
<@wumpus>
no topics at all for this meeting?
<@wumpus>
did the pre-final alert go out gmaxwell?
< achow101>
it did
<@wumpus>
ok, great
< sipa>
did anyone see it?
< btcdrak>
hi
< sdaftuar>
i did
< achow101>
I caught it on a 0.12.0 node I fired up just for it
< sipa>
gmaxwell and i were talking recently about some improvements to the block/header fetch logic
<@wumpus>
#topic block header/fetch logic
< sipa>
there are a bunch of related points there
< sipa>
one is that we don't have a timeout for headers requests
< jtimon>
BlueMatt: is there a long branch where I can see the code movements you're planning?
< sipa>
and that we don't respond to headers requests while in IBD, which can cause stalls if nodes mistakenly believe they are in IBD
< sipa>
bit it goes even further... the block fetch logic only disconnects peers who slow down the process
< sipa>
we may just have a peer who has no blocks we can fetch at all from, and we never try, and we never disconnect them
< arubi>
I have it on onlynet=onion
< BlueMatt>
jtimon: working on an updated version now
< sdaftuar>
sipa: eg non-NODE_WITNESS nodes?
< jtimon>
BlueMatt: cool
< sipa>
sdaftuar: or nodes who are legitimately behind
< sdaftuar>
ah, right
< sipa>
so it seems there is a simple solution: disconnect outgoing connections you're not downloading from while in IBD
< sipa>
but remove the non-response to getheaders in IBD
< jonasschnelli>
sipa: ack
< sipa>
if the peer actually is behind, we won't fetch from them, and we'll disconnect them instead of stalling yhem
< sipa>
gmaxwell suggested something even stronger: ever minute, disconnect the peer that is slowest to give you blocks overall (during IBD)
< sdaftuar>
if we remove the non-response to getheaders in IBD, mightn't we disconnect people who are downloading from us?
< jonasschnelli>
During IBD?
< sipa>
sdaftuar: note that this is only for outgoing connections
< sdaftuar>
ah
< sipa>
i think serving blocks to someone who is even more behind than us, whike we are in IBD, is perfectly fine
< gmaxwell>
I believe my suggestion went further, if you have MAX outbound, and it's been a minute since you disconnected anyone, and you're downloading blocks, disconnect the outbound peer you recieved the fewest blocks from in the last minute. (or, least recently recieved a block from). Presumably excempting -connect peers.
< gmaxwell>
(ah pieter just said some of that)
< jonasschnelli>
What about prioritize peers that can server "faster" (don't know if it's really measurable)
< sdaftuar>
sipa: one complication with serving headers during IBD is that we might be on a bogus chain
< sipa>
jonasschnelli: that's what we already do... the slowest ones get disconnected if they slow your overall sync speed down
< jonasschnelli>
sipa: ah
< sipa>
sdaftuar: that's something better IBD/chainpoint replacement is for, i guess?
< sipa>
*checkpoint
< jonasschnelli>
during my SPV work I encountered some stalling because of peers serving blocks each in a ~5min rhythm.
< sdaftuar>
i thought gmaxwell's PR to replcae the checkpointed-height with a checkpointed work as a way to determine if you're in IBD makes sense
< sdaftuar>
so if we eliminate the IBD restriction on serving headers, we'd still want to keep some version of that checkpointed-work requirement i think
< sdaftuar>
which i guess would be fine?
< sdaftuar>
and an improvement over the current situation
< sdaftuar>
?
< kanzure>
hi.
< sipa>
i think so
< sipa>
it's hard to reason about this. if you're truly sybilled during IBD, none of this will have an effect
< sipa>
if not, you'll quickly learn about the real chain anyway
< Chris_St1>
sipa: So that pull request does not help if you are fully sybilled? Won't you at least be able to determine if there was a lot of work expended in the sybil attack? (not sure how reassuring that is)
< gmaxwell>
this is going offtopic. :)
< gmaxwell>
sipa: sdaftuar is pointing out that if we're on a checkpoint invalid chain, and serve headers for it, our peers will ban us. So thats a complication with serving headers while below the top checkpoint.
< sipa>
ah.
< sdaftuar>
right, so assuming we are keeping the checkpoint-work-requirement (or some version of it) as a gate on responding to a getheaders, then which of the IBD checks are we trying to eliminate?
< sdaftuar>
from past conversations i think the concern is that we might have some long headers chain that we can't access/download blocks towards, like on testnet or something
< sdaftuar>
is that basicalyl right?
< gmaxwell>
We'd like to eliminate all cases where we simply ignore a getheaders request (potentially replace it with hanging up on the peer)-- because it DOS attacks peers unlucky enough to select us for their initial headers fetch.
< sipa>
we only serve headers for blocks in our main chain, no?
< sdaftuar>
sipa: yes
< sipa>
which indeed may contains dummy low difficulty blocks
< gmaxwell>
In any case, the download part of this can be done first before any change to how we respond to getheaders.
< sipa>
right
< morcos>
gmaxwell: thanks, i think it was important to clearly delineate the problem. i didn't know what we were trying to accomplish. it doesn't seem that having a bunch of IBD nodes able to serve each other as much as they have is _that_ beneficial
< morcos>
however freeing them up to ask someone else for headers withotu waiting for a long timeout seems valuable
< gmaxwell>
morcos: yes, thats mostly irrelevant, the concern is primarily that we cause harm to peers by not responding.
< gmaxwell>
morcos: My recollection is that currently we don't even have a timeout for the initial headers fetch! the 'timeout' is a new block being offered by some other peer.
< sdaftuar>
so step 1: #9068?
< sipa>
morcos: right... i think by using a "kick peers that aren't useful for sync" generic approach, we won't need the "don't serve headers while in IBD" anyway... less comllexity
< sipa>
*complexity
< luke-jr>
(suggested topic: when to halt changes to BIPs; 0.13.1 is no longer BIP 152-compatible I think)
< sipa>
so the issue is only when potential other bip152 implementations are oresent on the network
< sdaftuar>
right
< BlueMatt>
sipa: yes, but no such implementations exist, and if they do it now they must wait for the protocol version
< sipa>
so i believe 0.13.1 is compliant with the updated bip152
< BlueMatt>
yes
< sdaftuar>
yes, i think so as well.
< gmaxwell>
sipa: unfortunately because of the checkpoint stupidity we may still. :(
< gmaxwell>
but lets think about that outside of the meeting.
< gmaxwell>
0.13.0 did too.
< gmaxwell>
But I think what luke was referring to is that BIP152 didn't originally document version 2 compact blocks that use the wtxid instead.
< luke-jr>
wumpus: people are changing BIP 152 still
< gmaxwell>
Luke's question was really about when should someone be told 'write a new BIP' rather than changing an existing one.
< sipa>
yes, this is a good question
< luke-jr>
I may be wrong about the current status of v0.13.1 and BIP 152, but yes, the general principle is what I think needs to be discussed
< gmaxwell>
Not really much of a question for this meeting though, perhaps solicit input on the mailing list?
< luke-jr>
I didn't realise v0.13.1 bumped the protocol version-number
< sdaftuar>
it didn't
< luke-jr>
hmm, ok
< morcos>
i don't think its realistic to think we're going to not want to make small tweaks to complicated BIP's like this after releasing implementations of it. and it seems much clearer in the future to just edit the original bip to reflect the fully thought out final design
< luke-jr>
sdaftuar: then how is it BIP 152 compatible iwth your change?
< sdaftuar>
luke-jr: it imposes a restriction on code to not do something (which no one is currently doing) unless the recipient is at-or-above the bumped version number
< sdaftuar>
in this case, relay before full validation
< gmaxwell>
luke-jr: for the specifics here, 0.13.1 is compatible with BIP152 because it implements a new version number that the original bip152 was just silent on.
< luke-jr>
it says "nodes SHOULD NOT ban a peer for announcing a new block with a CMPCTBLOCK message that is invalid, but has a valid header" unconditionally, and says nodes should bump the version number
< gmaxwell>
and BIP152 already explained how versions were to be handled in a compatible way.
< gmaxwell>
luke-jr: re banning it's just a bug that all prior versions have as well.
< BlueMatt>
oh, well that is a language mistake
< BlueMatt>
by that language, indeed, 0.13.1 violates a SHOULD NOT
< BlueMatt>
however, this wont effect functionality, as we're a) fixing this as if it were a bug, b) we say you SHOULD NOT announce without validation if the number is below
< luke-jr>
ok
< luke-jr>
BlueMatt: with this change, as the author are you comfortable with a freeze to the BIP so we can move it forward to Final status?
<@wumpus>
I thought it mentioned a BIP change, but doesn't seem to mention that anymore
< luke-jr>
BlueMatt: how will existing nodes react if they get a full block message there?
< morcos>
i don't see why we should Finalize it at all until we've stopped changing it for 6-12 mos
<@wumpus>
morcos: makes sense to not finalize too soon, it's unrealistic to expect a bip to come into being completely perfect
< sipa>
i think it depends
< gmaxwell>
Agreed with Morcos. Though for things like consensus code, really being widely active on the network defines final.
< sipa>
there shouldn't be material changes to ideas
< luke-jr>
no particular reason to rush, I guess, just feels like a moving goal for anyone who wanted to be compatible with it
< gmaxwell>
luke-jr: at least they are minor alterations.
<@wumpus>
that's always the case. Others could also report problems encountered during implementing it
< sipa>
but clarifications and elaborating on edge cases is something else
< morcos>
gmaxwell: heh, even there, its a matter of whether we are confident that we've really understood what the consensus is. but yeah i agree it shoudl depend on the changes we want to make.
< luke-jr>
sipa: sure, we still make clarifications to Final BIPs even now I think
< BlueMatt>
luke-jr: agreed, it sucks that its still moving, but currently there are no other implementors (except XT, which I believe is still moving as well)
<@wumpus>
it could also move because of problems other implementors find
< luke-jr>
since it's being used live on the network, changes also should probably address backwards compatibility, which they aren't in this case
<@wumpus>
it's just unrealsitic to expect not even small issues in wording/clarity/definitions, although I guess if it is in a release there should not be substantial incompatible changes anymore
< gmaxwell>
morcos: so interesting point, say we discovered that BIP30 was implemented differently from the BIP tomorrow. What should we do? IETF way would be to attach an erratum to the document right away. But I find that this often confuses people who manage to read the document without an erratum. Then later a new document is published that reflects reality. Though this has a problem that people reme
< gmaxwell>
mber the original number.
< gmaxwell>
If no one of consequence actually implemented BIP30 as specified in the doc, what use does keeping the old doc around (except in the git history) serve?
<@wumpus>
yes, those at least will need to address backwards compatibilty
< luke-jr>
gmaxwell: I *think* we've fixed such issues in Final BIPs already
< BlueMatt>
gmaxwell: we're not plaintext, lets highlight it in red! :p
< luke-jr>
>_<
< morcos>
yes, BIP 34 for example
< morcos>
ehh, i guess that was just wrong explanation
<@wumpus>
I don't think there's time enough to discuss all potential backports to 0.13.2, but that one would do
< gmaxwell>
I think it would be harmless to backport, and helpful for testnet nodes. But I don't have a strong opinion.
< gmaxwell>
oh I see sipa mentioned testnet.
<@wumpus>
so I guess in practice it fixes testnet issues only on 0.13.2, so the question would be is that worth it to potential regressions?
< sdaftuar>
it's not that much of a fix for testnet right, it just allows you to reorg out the non-segwit chain?
< gmaxwell>
it does actually fix a misbehavior that we see on testnet. <famous last words>I can't see it causing a regression.</famous last words>
< gmaxwell>
sdaftuar: because of the 20 minute rule in general it's very easy to get testnet nodes into a state where they just stop mining. Trivial vulnerablity, the active issue is that the non-segwit chain there unintentionally triggers it from time to time.
< gmaxwell>
('stop mining' is ambigious, they won't mine after they're restarted)
< sdaftuar>
fwiw i have a few bridges of my own back up now that i hope will keep that from happening again
< sdaftuar>
can you elaborate on the 20 minute rule though?
< sdaftuar>
oh
<@wumpus>
I think personally I'd prefer to keep it for 0.14, so the new rule/logic can prove itself a while
< gmaxwell>
sdaftuar: the issue is that if anyone produces a lot of headers beyond your current tip which you accept (made computationally easy by the diff1 blocks) then you'll not leave IBD.
< sdaftuar>
got it
< gmaxwell>
sdaftuar: the non-segwit chain does this by accident through a confluence of other behaviors (the not fetching blocks from non-witness peers). But the real bug is just using forward header count to cause you to not leave ibd.
< gmaxwell>
which that PR fixes.
< sipa>
wumpus: ok, we can of course later decide to backport to whatever 0.13.x at that time
<@wumpus>
sipa: yes
< gmaxwell>
sounds fine to me.
<@wumpus>
that doesn't prohibit backporting it later
< gmaxwell>
because of the latching in IBD this code is pretty robust against mistbehavior to begin with.
< sipa>
$-+(#(_+$+ PC LOAD LETTER
<@wumpus>
#endmeeting
< lightningbot>
Meeting ended Thu Nov 3 20:00:22 2016 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
< BakSAj>
sipa: ok thanks, thats also good, i guess schnorr is longrun task
< BlueMatt>
BakSAj: 0.14 has been a lot of fixes/refactoring/cleanups/optimizations/etc/etc...due to a focus on segwit some of these things got put off and/or big features werent something that was a major focus for lots of people
< luke-jr>
BakSAj: does that one on the right really exist? :O
< sipa>
luke-jr: no, it's a math joke
< BakSAj>
luke-jr: only after you forget your bear in the box for some time
< luke-jr>
someone made a fake bear for a math joke? XD
< jtimon>
luke-jr: photoshop or something, yeah
< luke-jr>
o.o
< BlueMatt>
oh, sipa said that
< BlueMatt>
BakSAj: bumpfee might make it in though, so that would be cool
< jtimon>
luke-jr: I'm actually surprised that you are surprised
< luke-jr>
jtimon: I would have tried to get bear skin and paste it on a set of boxes :p
< luke-jr>
didn't even occur to me to try to make such an image with a computer
< jtimon>
not an expert, but I would bet that one is actually not that hard
< BakSAj>
yeah like segwit :-)
< BakSAj>
btw i really wonder how ecosystem will change once sw activates and LN will come to existence
< BakSAj>
lol rereading the log: this is really a good one :-)) BlueMatt making 0.14 great again!
< BakSAj>
i vote it for comic relief
< luke-jr>
hum, after taking forever to start in valgrind, I got a segfault. fun
< luke-jr>
not sure how to interpret the stack trace however
< nanotube>
heh well, so let's go over this... you want PR#\d+\W to go to github pr, I#\d+\W to go to github issues, and just #\d+\W to just assume issue?
< luke-jr>
(^|\s)(PR|I)#\d+\b I think
< luke-jr>
explicitly *not* #\d+
< BlueMatt>
lol
< BlueMatt>
jtimon: the main.cpp split is my net_processing_file branch, which includes #9026, #9075, #8930 in rebased form, 4 more commits to form a future pr, and then -2990 +3085 LOC in a single code-move commit
< jouke>
What does it mean when a wallet says: "Transaction commit failed" I blieve it's a .12 wallet
< jouke>
Hmm, corrupt /win 109
< jouke>
whoops
< jouke>
Hmm, but the specific transaction seems to be broadcasted though?
< sipa>
i believe it means it wasn't accepted by your own mempool, but still broadcast
< sipa>
that shouldn't happen
< sipa>
if it confirms, it will show up in the wallet fine, though
< jouke>
Hmm, rpc doesn't give a tx-id back, so I am not sure if it's absolutely the same
< jouke>
but the outputs match, so I guess so
< jouke>
Other transaction from the same wallet: "Error: The transaction was rejected! This might happen if some of the coins in your wallet were already spent, such as if you used a copy of wallet.dat and coins were spent in the copy but not marked as spent here."
< jouke>
sipa: that wallet node is connected to an other node on the same host via 127.0.0.1. Could that be a reason why it's broadcasted even though the wallet itself says it's not valid?
< sipa>
i don't think so
< jtimon>
rebased #8994 and jtimon/0.13-blocksign branch too for the curious (the latter still needs more work)