< GitHub12>
[bitcoin] gmaxwell closed pull request #7119: Add option to opt into full-RBF when sending funds (master...2015-11-opt-into-full-rbf-option) https://github.com/bitcoin/bitcoin/pull/7119
< phantomcircuit>
gmaxwell, what do you think about removing RelayTransaction entirely and simply sending the top n MB of the mempool every m seconds?
< gmaxwell>
phantomcircuit: I want to try a protocol which does an efficient set reconcillation of the top N mb of mempool. Which is like the pro version of what you're thinking.
< gmaxwell>
Not the actual data, but set reconcile the TXids and then getdata the bodies you need.
< phantomcircuit>
gmaxwell, yes long term goal for sure
< phantomcircuit>
but i can do this version like
< phantomcircuit>
today
< phantomcircuit>
gmaxwell, can you rename setInventoryKnown to filterInventoryKnown
< phantomcircuit>
there's a bunch of those now that are a bit confusing because of not changing the name
< phantomcircuit>
gmaxwell, im pretty sure there's a bug in 7100
< phantomcircuit>
it looks like you can get a false positive inv for blocks
< gmaxwell>
phantomcircuit: where?
< phantomcircuit>
gmaxwell, "getblocks" calls pfrom->PushInventory which uses setInventoryKnown to decide whether to actually do that
< phantomcircuit>
so a false positive could prevent a block inv as well as tx inv
< phantomcircuit>
i dont much care about tx inv's getting droped but it would be bad for block inv's to be also
< gmaxwell>
agreed.
< phantomcircuit>
so im thinking there needs to be a setInventoryKnownBlocks and filterInventoryKnownTransactions
< sipa>
with 6494 we can just drop the known blocks
< sipa>
as we keep track much more efficiently what peers know
< phantomcircuit>
sipa, yes i agree that's a better solution
< phantomcircuit>
except it's optional so we would still need the setInventoryKnownBlocks
< gmaxwell>
woot, managed to apply that without loading the webpage.
< sipa>
you have now passed the entry exam for the school for git wizardry and magic
< gmaxwell>
well the git part was never a problem, just the github part. :P
< phantomcircuit>
just gotta know the special pulls refspec
< phantomcircuit>
sipa, in 7113 you calculate the optimal number of hash functions and then restrict it to 1-50
< phantomcircuit>
why?
< phantomcircuit>
an fp rate of 1/10^15 would get you nHashFuncs ~= 50
< phantomcircuit>
i dont see this being an issue
< sipa>
eh, i guess i wrote that part before realizing that the number of hash finctions only dependee on the fprate :)
< sipa>
functions, depended
< phantomcircuit>
anybody have an opinion of changing gbt to default not mine a transaction until it's been in the mempool for more than a few seconds
< sipa>
phantomcircuit: do that on top of morcos' rewrite then :)
< phantomcircuit>
sipa, it's a trivial 1 line change so i was mostly just wondering about the concept
< sipa>
ok
< phantomcircuit>
it seems like a good idea to me but i just thought of it soo
< sipa>
among miners who are not in a position to exploit high hashrate or fast relay to a majority, that should improve their relay
< gmaxwell>
phantomcircuit: I think it's a good idea. I thought of it before but figured it would screw up the recent improvements. :(
< sipa>
should be trivial now that we remember time-in-mempool
< gmaxwell>
hm. actually yea, you'd just skip them, so it doesn't change the sort.
< sipa>
indeed
< phantomcircuit>
sipa, it's been a trivial 1 loc patch for about a year now
< sipa>
orly?
< phantomcircuit>
actually more than that
< phantomcircuit>
Date: Mon Nov 11 17:35:14 2013 +1000
< sipa>
ha, wow
< phantomcircuit>
on a related note
< phantomcircuit>
we want to maximally prime the sigcache for receiving a block
< phantomcircuit>
while also limiting the mempool
< phantomcircuit>
these are kind of at odds at the moment
< gmaxwell>
I tried to have sipa verify rejects and it polluted the crap out of his cache.
< phantomcircuit>
i've got a few ideas of how to deal with this but they are admittedly mostly insane
< gmaxwell>
it actually makes sense to verify rejects for several reasons.
< gmaxwell>
but I think we have to improve cache management first.
< phantomcircuit>
gmaxwell, well for a miner you can just run with a huge sigcache
< phantomcircuit>
i've got mine set to 4GB...
< gmaxwell>
Sipa found it was polluting it even with a stupidly big one. (but not that stupidly big)
< phantomcircuit>
yeah the definition of stupidly big varies here :)
< gmaxwell>
phantomcircuit: we'd like to uh.. not centeralize mining "you can mine but if you want a low orphan rate you'll have to dedicate 16GB ram to it.. not great. :P
< sipa>
hey let's use a rolling bloom filter for tue sigcache!!!
< sipa>
*ducks*
< gmaxwell>
but I think ultimately thats the best thing to do, we could try to estimate "reject but still likely to get mined soon based on recent history" but I'd rather spend the complexity on making the cache smarter.
< phantomcircuit>
the cache doesn't evict when a block is found does it?
< phantomcircuit>
that would be an easy win
< sipa>
it does
< gmaxwell>
e.g. attach feerate and "tip change counter" to entries in the cache, and evict using them.
< phantomcircuit>
oh
< phantomcircuit>
nvm
< phantomcircuit>
:|
< sipa>
phantomcircuit: well, it evicts after use in a block
< gmaxwell>
e.g. so on full it evits the lowest feerate that went into the cache the most blocks ago.
< sipa>
but not after use in mempool
< phantomcircuit>
so there is also the issue that processing transactions into the mempool acquires cs_main
< phantomcircuit>
which is bad for latency of gbt calls
< gmaxwell>
(by lowest I don't mean a sort, I mean a N random draw...)
< phantomcircuit>
soooo rpc "addsigcachentries"
< * phantomcircuit>
runs away
< gmaxwell>
die
< sipa>
rpc "pollutecache"
< gmaxwell>
should just hash random items in memory and add them, some might be signatures.
< phantomcircuit>
:)
< phantomcircuit>
<phantomcircuit> i've got a few ideas of how to deal with this but they are admittedly mostly insane
< phantomcircuit>
i wasn't lying
< gmaxwell>
phantomcircuit: the obvious thing to do is to just have a EWMA minimum feerate for blocks, and any transaction over that, you verify even if its rejected.
< sipa>
phantomcircuit: when you say insane, i should probably start worrying
< gmaxwell>
(or over 0.95 * that)
< gmaxwell>
sipa: dude, not like he's suggesting turning the G/2 nonce into a cryptosystem.
< tulip>
would having two isolated sigcache (rejects, accepts) do roughly the same job?
< gmaxwell>
tulip: probably no, because the rejects would get polluted and then not be useful (why have it)
< tulip>
people can perform an eviction attack against the rejects cache of course, but that doesn't completely destroy your validation time.
< sipa>
we don't have a negative cache now
< tulip>
gmaxwell: suppose.
< sipa>
gmaxwell: European Momputer Manufacturers Organization minimum feerate (sorry, google was slow in telling me what ECMA stood for)
< sipa>
Womputer, of course
< gmaxwell>
exponentially weighed moving average.
< sipa>
ah, of course
< phantomcircuit>
gmaxwell, EWMA ?
< sipa>
phantomcircuit: European Momputer Manufacturers Organization.
< phantomcircuit>
oh
< phantomcircuit>
lol derp
< phantomcircuit>
i should have continued reading before asking :)
< GitHub47>
[bitcoin] sipa opened pull request #7129: Direct headers announcement (rebase of #6494) (master...direct-headers-announcement) https://github.com/bitcoin/bitcoin/pull/7129
< gmaxwell>
"that isn't merged yet"
< gmaxwell>
"?"
< sipa>
gmaxwell: i'll merge upon happy travis (though maybe someone should proofread my docs)
< gmaxwell>
hm. so I wonder if the sigcache rejects stuff will have less pollution problems once limited mempool is more common?
< sipa>
perhaps yes
< tulip>
sipa: the docs read fine.
< phantomcircuit>
gmaxwell, probably
< phantomcircuit>
i actually like tulip's suggestion of having two sigcaches
< phantomcircuit>
it handles the "i have lots of memory and dont care" case pretty well
< gmaxwell>
why not have a seperate sigcache for each band of feerate? :P and if the highest feerate cache is full it evits to a lower feerate cache.
< gmaxwell>
oh you could also put a neural network in it, and it could do unsupervised classification to decide which transactions will get confirmed... and ... :P
< sipa>
and then skynet
< sipa>
and a genisys block
< phantomcircuit>
gmaxwell, hr
< phantomcircuit>
har har
< phantomcircuit>
but in all seriousness it's potentially a large win for miners and just annoying for everybody else
< gmaxwell>
oh, and if your cache gets too big you can sign chunks of it and ship it off to peers, and they can send it back if you need it later...
< gmaxwell>
phantomcircuit: not the kind of wins we should be hunting, because it presumes a more centeralized world of mining where running a mining node takes a lot of resources.
< gmaxwell>
Better to spend effort on optimizations that don't need that.
< phantomcircuit>
gmaxwell, which reminds me, what kind of work would you want to see before revising the advise on mining with a pruned node?
< gmaxwell>
I think it's fine already.
< gmaxwell>
"You mean you don't mine exclusively on pruned nodes?" ... really the worse problem is that you're hosed in index corruption...
< phantomcircuit>
gmaxwell, miners are hosed if their index is corrupt even without pruning
< phantomcircuit>
hmm i seem to remember asking someone and getting a "wat dont do dat" response recently
< gmaxwell>
it just means a three hour outage vs ... more.
< gmaxwell>
I don't think you got that from me.
< sipa>
you can always make a backup of the block chain data
< gmaxwell>
like... half the reason I care about pruning is to try to rescue p2pool.
< phantomcircuit>
gmaxwell, yes but it might mean running 5 nodes instead of 1
< sipa>
and use that to recover a mining node
< gmaxwell>
the main risk from pruning + mining is that you can't reorg past some depth, which means consensus fault; but we won't let you prune shallower than 288 blocks back to make that not much of a pratical concern.
< gmaxwell>
(or at least if there is a 288 block reorg, manual intervention is.. least of the worries)
< gmaxwell>
though we should make sure that that it cleanly fails (And importantly stops mining) if it tries a reorg beyond pruning.
< gmaxwell>
phantomcircuit: if you're looking for mining related improvments; bringing back the old lukejr patch to forward unverified blocks would be an obvious candidate.
< gmaxwell>
E.g. a new proto messages activated like sendheaders where you can say "here is a block, I haven't verified it"; you're allowed to relay it to others (as non-validated block) without validatiting long as the hash and headers checkout, and so long as it extends the current tip.
< gmaxwell>
when luke did it before, it didn't speed anything up, but that was presumably because of all the dumb sleeps in networking.
< gmaxwell>
I think this would reduce your mining complex by one node, since those relays wouldn't need to lock the chainstate, and so they wouldn't compete with create new block.
< sipa>
checking whether it extends the current tip would need a lock
< phantomcircuit>
gmaxwell, what happens generally if you try a reorg past the pruning depth?
< phantomcircuit>
it should probably pull those blocks from peers
< gmaxwell>
phantomcircuit: it can't. :( :(
< gmaxwell>
undo data is gone too.
< sipa>
though i guess you could have a pindexTipCopy which has an R/W lock, and is updated by the normal sync, but can be read (and used for verification) by other things
< gmaxwell>
someday we could make undo data normative and commit to it, perhaps and then you could pull it.
< sipa>
gmaxwell: i think the biggest bottleneck is the fact that message processing is single threaded
< tulip>
phantomcircuit: 288 blocks is the minimum, if you reorg that far all of your peers have pinged out as well, it gets super messy and everything is likely on fire anyway.
< gmaxwell>
sipa: doesn't happen that all our messaging hashes the data...
< sipa>
gmaxwell: parse error
< gmaxwell>
I'm just saying that message handling is computationally expensive.
< sipa>
gmaxwell: so?
< phantomcircuit>
gmaxwell, iirc the undo data is much smaller than the blocks
< sipa>
phantomcircuit: a factor of 9
< phantomcircuit>
ok so probably we should keep it going back 9x deeper than we do blocks
< gmaxwell>
phantomcircuit: yes, and we could prune it to different depths, but keeping it all removes a lot of the pruning gains.
< sipa>
gmaxwell: the fact that we're processing a new incoming block for half a second is no reason why we couldn't respond to a ping from another peer
< gmaxwell>
On earlier; though I guess if we introduced a new p2p message for block relay, it should do RNC like compression.
< phantomcircuit>
gmaxwell, for the average pool i suspect relaying before validating anything but the header would be a win even without the new p2p message to prevent getting banned
< gmaxwell>
... and bring MRU sets back to track what transactions we've sent a peer? :P
< gmaxwell>
phantomcircuit: it's really important to not hand unvalidated blocks to spv clients.
< tulip>
how do you prevent unvalidated blocks from becoming a DoS vector? 25BTC is expensive, but if I can make a block which takes you 2 minutes to validate it when you see it that could be a problem.
< tulip>
(two minutes of grinding SHA256, then you find you have to reject it)
< phantomcircuit>
gmaxwell, iirc the message hashing is done in the networking thread so it's at least partially threaded
< sipa>
phantomcircuit: no
< sipa>
only the checksum
< sipa>
i think?
< sipa>
yes
< phantomcircuit>
i thought that's what he was talking about
< sipa>
not the txid's, for example
< sipa>
or sighashes which are even more work
< tulip>
me? I was talking about sighash hashing.
< phantomcircuit>
tulip, no i was talking about what gmaxwell said
< tulip>
right.
< sipa>
message handling is of course expensive; it's where block validation (and a part of signature validation even) happens
< sipa>
doesn't mean it needs to be done single threadedly