< GitHub42>
[bitcoin] gmaxwell closed pull request #6851: Optimisation: Store transaction list order in memory rather than compute it every need (master...opti_txorder) https://github.com/bitcoin/bitcoin/pull/6851
< gmaxwell>
midnightmagic: interesting, so that had been running without interruption and just crashed while syncing?
< midnightmagic>
gmaxwell: I did a -reindex and -connect to an internal node.
< * midnightmagic>
shrugs and reindexes the main node.
< gmaxwell>
in any case, my 2xG5 host seems to be catching up fine with git master.. taking like 10 seconds for some blocks. :)
< midnightmagic>
what magic incantation do you do to make it go fast
< jtimon>
morcos the linked commit doesn't touch TrimToSize, that's the next one
< jtimon>
ACK on policy estimator called directly instead of through the mempool (but I've been asked not to worked not to work on policy refactors for now) and that one is certainly disruptive
< jtimon>
setting the attribute by calling TrimToSize seems weird, what's wrong with doing it in the constructor (or in a setter if necessary)?
< jtimon>
morcos_: the estimator is currently not asking questions to the mempool and it doesn't need to, please don't couple that again
< jtimon>
you want the mempool decoupled from policy/fees and I agree, and have done it already in a "private" outdated branch
< jtimon>
but to do so, it is not necessary to couple policy/fees to the mempool instead, they can be completely independent
< jtimon>
whenever we think is the moment, I can completely decouple them
< morcos>
jtimon: the question is where shoould the logic live in smartly estimate fees. in the policyestimator. that logic requires asking question of the mempool. otherwise you're going to have to put that logic repeated in several different places in wallet and gui.
< morcos>
s/live in/live to/
< morcos>
jtimon: the reason I like setting the mempool size in TrimToSize is you can easily imagine logic where the size is non-constant. So after you have trimmed it the mempool knows what size it currently is (the logic as to what size to trim it lives outside mempool)
< morcos>
for instance we discussed using less size for the mempool during IBD so you could use more of your allocated memory for the dbcache
< morcos>
or you can imagine other scenarios
< gmaxwell>
1h 11m to rescan a wallet with 11.7m transactions now.
< sipa>
how long for a wallet with 0?
< gmaxwell>
sipa: less than 10 minutes.
< gmaxwell>
listunspent on that wallet is taking 2m10s right now (to return 15870 coins)
< gmaxwell>
Wallet.dat is 11G which isn't bad.
< gmaxwell>
bleh, and getinfo takes 1m8s.
< gmaxwell>
Luke-Jr: It might be useful to add an index of which transactions have unspent coins to make listunspent fast. But I think with the way balance calculations work getinfo is going to remain slow. :(
< Luke-Jr>
I have never once had a reason to use listunspent..
< gmaxwell>
Luke-Jr: it also means selectcoins will be superslow.
< gmaxwell>
4m 15s for a getbalance "*" 0 true, I dunno why getinfo is faster.
< gmaxwell>
::sigh:: we really need a remove feature for the wallet, but the remove needs to keep track of what was removed so rescan doesn't read... and we can't remove things with spendable txouts because they're not seperate.
< Luke-Jr>
jonas is still rewriting it, right?
< gmaxwell>
We've fucked over the project for years with that kind of thinking; we can't stop improving the wallet because of a grand rewrite.
< Luke-Jr>
sure, it's just one of those things that if I were to personally try to do it, I would end up probably rewriting the wallet myself :p
< sipa>
gmaxwell: what do you mean by remove?
< sipa>
remove keys?
< gmaxwell>
sipa: no, no reason to remove keys. Remove transactions.
< gmaxwell>
sipa: right now large parties using bitcoin core have to periodically rotate out wallets to keep things managable. Things are much better now because of varrious fat trimming. (Including the addtowallet fix we just merged from luke)
< sipa>
so more like mark transactions as archived
< sipa>
?
< sipa>
so they are no longer considered for balance computations etc
< gmaxwell>
Yea, in particular, get them out of the linear scans used to getbalance/listunspent/etc.
< gmaxwell>
But not do so in a way that causes coins to fall out of balance and listunspent, because that will cause funds loss when you think a wallet is empty and it's not.
< gmaxwell>
so that could be refusing to archive when there are unspent outputs, but thats kind of obnoxious, since it would make visible wallet internal behavior that the caller shouldn't really care about.
< gmaxwell>
(Though, otoh, it might encourage sweeping the utxo set. :) )
< gmaxwell>
Also, 'the must be spent completely' rule wouldn't be reorg robust.
< gmaxwell>
And, of course, an archived transaction shouldn't be added back by rescanning.
< gmaxwell>
Some parties (e.g. bitstamp about a year ago) also complained about the size of the wallet files when they had long histories because it complicated backups; but I think the key exports patch that somewhat.
< sipa>
seems easy enough to build a second map inside that does not contain transactions whose outputs have been spent for ages
< sipa>
andnuse that for balance calculations etc
< gmaxwell>
I think accounts mess this up somewhat; or at least make it not a transparent implementation detail.
< sipa>
uh, right
< gmaxwell>
thats why I was talking about remove/archive. :-/
< gmaxwell>
phantomcircuit: were you going to fix the trickle logic? it really is broken.
< sipa>
yes, please!
< gmaxwell>
I'm testing the blocks only mode with relaynetwork client as a whitelisted peer (and I cut out the forced broadcasting for whitelisted peers)... and I'm watching it instantly inv any tx that comes in to all it's neighbors.
< gmaxwell>
of course, there is some node connected to me which is sending me every transaction without an INV.